Are AI Coding Agents More Efficient When They Produce More Code? 2026 Research Review

Who should read this: Engineering leaders deciding whether to expand an agent pilot, reviewers dealing with larger or more frequent pull requests, and engineering-effectiveness teams choosing measures for a trial.

Last updated October 13, 2026. The research findings and version date were checked against the authors’ working paper; recent coverage was checked against the October reporting.

1What the study measured—and what it did not

The phrase “AI coding efficiency” can refer to several different things: how much code a tool helps produce, how quickly a person completes a task, whether a change reaches production, or whether the change works without costly follow-up. These are related questions, but they are not interchangeable measures.

The Harvard AI coding study discussed in recent coverage is a working paper by Fiona Chen and James Stratton. Its authors analyze engineering data from a particular platform, covering the organizations and observation period available in that dataset. That makes the results useful for generating questions about workplace coding activity. It does not make the sample a census of software teams, nor does it establish a universal causal effect.

The paper distinguishes agent-style tools from assistant-style tools and examines engineering activity and work-tracking outcomes. That distinction matters. A tool that independently takes on more of a coding task may affect activity differently from a tool that suggests completions while a developer directs each step. The label “AI tool” alone does not tell you which workflow the data represents.

The reported association between agent use and greater code activity is not the same as evidence that teams delivered more finished software. In the paper’s sample, the authors did not find a statistically significant increase in completed Jira work. That is a result about the measures, sample, and analysis in this paper. It is not proof that agents never help teams complete more work, or that every team will see the same pattern.

Keep the boundary visible: a working paper can inform a trial design before it can settle a broad question. Treat its results as evidence about the studied dataset, not as a forecast for your backlog.

What did the study count, and what should you count? The paper reports code activity and tracked work outcomes. Your team should also inspect the path between a proposed change and an accepted, usable result. For a pull request, that path may involve review, requested changes, follow-up commits, merge, deployment, and later repair. The pull request process documentation describes the collaborative review steps that sit between code being proposed and a change being incorporated.

2For engineering leaders: activity is not delivery

Code lines, commits, and pull requests measure activity or workflow events. A completed feature, resolved customer problem, or safely released change is an outcome. More activity can accompany more delivery, but it can also reflect different task sizes, more generated scaffolding, additional review rounds, or work that is later discarded.

That gap is why “the agent produced more code” is not enough to justify a rollout. Ask whether the change appeared in the measures your organization already uses for delivery. Did the time from work starting to work completing move? Did the number of finished, accepted items change? Did reopened issues or follow-up fixes increase? If activity rose while outcomes stayed flat, the next task is diagnosis—not expansion based on code volume.

DORA’s guidance sets out five delivery measures: change lead time, deployment frequency, failed deployment recovery time, change failure rate, and deployment rework rate (DORA’s metric definitions and guidance). These measures do not automatically isolate the effect of a coding agent. They can, however, help you check whether a change in coding activity coincides with a change in delivery flow or operational stability.

Developer productivity is broader still. The SPACE framework groups measurement into five dimensions: satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow (the SPACE framework paper). The implication for a pilot is practical: do not replace an outcome measure with an activity counter, and do not let one metric stand in for the whole developer experience.

Why can more agent-written code fail to produce more delivered software? Because code is only one input to a release. Work may wait for product decisions, integration, security checks, tests, review, or a developer to resolve an incomplete change. An agent can increase the volume entering those queues without increasing the rate at which the team clears them.

Before extending a pilot, compare the same work definitions and time windows before and during the trial. If the team changes its definition of “done,” includes different project types, or changes staffing at the same time, you will not know whether the agent or the changed conditions explain the movement.

3For code reviewers: measure the queue, not just the PR count

A coding agent can shift work from writing code to checking and correcting it. That does not mean it always does so; the effect depends on change size, codebase conventions, test coverage, risk, and how much context the agent receives. But reviewers should check for a possible queue effect instead of assuming faster code production shortens the path to merge.

Can an AI coding agent slow code review? It can contribute to slower review when it creates changes that take longer to understand or require more revision, but this study does not prove that agents alone cause review delays in every organization. Look for evidence in your own workflow: time waiting for first review, time from review to merge, number of requested changes, discussion volume, and follow-up commits. Interpret those signals alongside the risk and size of each change.

A higher comment count is not automatically a sign of poorer work. A team that reviews more carefully may leave more comments while preventing defects. Likewise, a low comment count does not show that changes are safe if reviewers are overloaded or approvals are cursory. Compare like with like: similar repositories, change types, risk levels, and review policies.

PR size can distort comparisons. A small documentation update and a broad architectural change do not impose the same review burden. If an agent changes the distribution of PR sizes, total review time may rise even if the median review time for comparable changes does not. Track both the shape of the work entering review and what happens after it arrives.

For a review-focused check, follow one representative change through the process. Record when it becomes reviewable, when a reviewer begins, what is requested, how many revision cycles occur, and whether the final change meets the team’s acceptance criteria. This reveals where time accumulates without assuming that the coding tool is the only cause.

4For engineering-effectiveness teams: set a fair pilot baseline

A trial is only useful if the comparison remains interpretable. If teams change their PR policy, task sizing, staffing, or release cadence during the same period, a before-and-after comparison may blend those changes with the effect of the agent. Set the scope and definitions before the first trial change enters review.

Use a shared measurement plan that covers three layers:

• Delivery outcomes: completed and accepted work, time from start to completion, and whether the change reached the intended release or user.
• Review flow: wait time, time spent in review, requested changes, revision cycles, and queue size.
• Quality and stability: defects, rollbacks, rework, or other signals your team already uses to detect an unsafe or incomplete change.

Choose measures that suit your release model. A service team that deploys frequently may inspect delivery and failure signals differently from a team shipping a desktop application on a scheduled cadence. Do not force a single industry metric into a workflow where it has no meaningful interpretation. Keep the measure, definition, team scope, and observation window consistent across the comparison.

First, define what counts as “done”

Write down the event that makes a work item complete. Is it accepted by a product owner, merged, released, or verified in production? Choose the definition that matches the claim you want to test. If your aim is to know whether users receive useful changes sooner, a merged PR by itself is an incomplete endpoint.

Next, match the comparison

Select teams or work types with similar risk, review requirements, and release patterns. If a direct comparison is not possible, document the differences and avoid presenting the result as a clean causal estimate. Capture concurrent changes—such as a staffing adjustment or a release freeze—because they can affect both activity and delivery.

Then, inspect where work accumulates

Review the handoffs from task selection through coding, testing, review, and release. Separate time spent actively working from time waiting in a queue where your records allow it. An increase in coding activity paired with longer review waits points to a different intervention from a rise in defects or a fall in completed work.

Finally, agree on a stop-or-expand rule

Decide in advance what result would support expansion, what would trigger a change to the workflow, and what would pause the trial. A rule should reflect trade-offs: a modest delivery gain may not justify substantially higher rework or unacceptable review risk. Do not make the rule depend on code volume alone.

Team self-check before expanding an agent pilot

• [ ] We have defined completed work in a way that matches the outcome we want to improve.
• [ ] We can distinguish proposed code activity from accepted or released work.
• [ ] Our before-and-during comparison uses the same team scope and metric definitions.
• [ ] We are recording review waits and requested changes, not only the number of PRs.
• [ ] We have selected quality signals and know who will inspect them.
• [ ] We have a named owner and a decision rule for pausing, adjusting, or expanding the pilot.

Use this as a team self-check, not as a universal prescription validated by the paper. A trial with inconsistent definitions can produce confident-looking charts that answer no useful question.

Watch the denominator: if you compare review time per PR, note whether the mix of small and large changes has shifted. If it has, report the change in mix rather than attributing the entire movement to the tool.

5For management: make this a pilot question, not a shutdown order

The study does not show that every coding agent is ineffective. It also does not show that more agent-generated code must produce more value. Both claims go beyond the evidence described in the paper. The practical decision is narrower: is this workflow improving a defined outcome for your team without creating an unacceptable review or quality burden?

Start with one bounded use case and an agreed baseline. For example, you might test an agent on a specific class of changes while leaving higher-risk work under the existing process. The point is not to cherry-pick a favorable result. It is to make the comparison specific enough that a positive or negative result leads to a decision you can explain.

How can you tell whether an AI coding tool actually improves delivery? Check whether the team completes the work it intended to complete, whether the wait and rework around review remain manageable, and whether quality signals stay within the limits you set. Compare the same definitions over a consistent observation window. If those conditions are not met, you may have learned something about the workflow, but you have not established a general productivity gain.

The research is therefore most useful as a prompt to inspect the gap between code activity and delivery. Before expanding a pilot, ask whether your team has more finished work, whether review is absorbing the additional changes, and whether the quality cost is acceptable. If you cannot answer those questions from your current tracking, improve the measurement before increasing adoption.

If a pilot also depends on a temporary remote development environment, include its setup and operating cost in the comparison rather than treating it as invisible overhead. For background on the team and its editorial focus, see about vpstix. To turn the research’s review and delivery concerns into acceptance criteria for your own trial, browse the vpstix engineering and AI articles.