Skip to main content

How to Measure AI Coding Productivity

Updated 6 min readMarkdown

Measure the engineering work produced, its quality and its cost.

A coding session marked “complete” is useful operational data. It does not establish that code merged, reached production or solved the intended problem.

Weave’s Q3 research makes that distinction measurable. The Python calculation below uses the report’s published spend shares. The rest of the audit specifies the records needed to investigate the same question on your own data.

Check what “complete” means in the data

In the retained Q3 2026 sample, 23% of traced session spend had both a completion signal and a merge link. Another 31% had completion without a merge link, while 17% had a merge link without completion.

The inputs below are rounded percentages of spend. They are not session counts.

spend_share_pct = {
    "completed_and_merged": 23,
    "merged_without_completion": 17,
    "completed_without_merge": 31,
    "unclassified": 17,
    "abandoned_or_blocked": 12,
}
assert sum(spend_share_pct.values()) == 100

completed = (
    spend_share_pct["completed_and_merged"]
    + spend_share_pct["completed_without_merge"]
)
merge_linked = (
    spend_share_pct["completed_and_merged"]
    + spend_share_pct["merged_without_completion"]
)

print(
    "Merge-linked share of completion-labelled spend: "
    f'{spend_share_pct["completed_and_merged"] / completed:.1%}'
)
print(
    "No-completion share of merge-linked spend: "
    f'{spend_share_pct["merged_without_completion"] / merge_linked:.1%}'
)

The results are 42.6% and 42.5%. Completion labels miss some merge-linked spending and include spending with no recorded merge. Neither ratio is a task-success rate.

The source report excludes one organization responsible for 77% of traced spend and roughly half the sessions. These figures describe the remaining sample. A missing merge link also does not prove that work failed: some tasks need no PR, and some links are missing.

Keep the events separate

Store session outcome, PR merge time, deployment time and task acceptance as separate fields. A task can require several sessions and PRs; a single PR can contain work from several sessions. Keep the link IDs and the attribution rule so joining those tables does not duplicate cost or output.

Use an explicit observation cutoff. If a PR merges after the reporting period, retain that fact instead of silently rewriting the original result. Report unmatched sessions and tasks alongside matched records.

Add an output measure and quality checks

Merged PRs vary in difficulty. Weave’s Code Output estimates the expert effort represented by the final merged diff and explains the assessment. It measures work represented by the code; it does not measure an engineer’s actual working hours.

For a defined team and period, calculate output per attributed dollar. Keep subscriptions, separately billed usage and estimates identifiable, and count each expense once. Compare similar work and preserve the team’s membership in the comparison.

Then inspect review effort, reverts, defects and follow-up changes over the same observation window. Interpret a rise in output per dollar alongside the quality and type of work produced.

Compare coding tools on output, cost and quality

Weave’s Tools view compares connected coding tools such as Claude Code, Cursor, GitHub Copilot and Codex on engineering output, cost and bug ratio. Cost per output by tool puts spending on a common output basis. The Models view adds average cost per session, first-shot success, average AI output per session and code turnover. These views help a team choose which tools to expand and which workflows need attention.

Use the same engineers, repositories and period for the comparison, then inspect representative changes. Check the source of AI authorship evidence and whether collection coverage differs across tools. A session associated with a tool and an attributable code change provide different evidence. If the decision depends on faster delivery, compare cycle time and review delays as well as output.

Weave’s Silk 1 evaluation reports 3.3× lower average prediction error than the previous Weave model, with 88% of PRs within ±1 expert-effort hour of the reference estimate. Engineers can open a PR’s assessment and check its reasoning. Those estimates make different kinds of code changes easier to compare; a controlled rollout is still needed to isolate a tool’s effect.

Compare Cursor and Copilot on the same work

Illustrative example, not customer data or a benchmark: a backend team wants to compare Cursor and Copilot on API bug fixes. Use the same engineers and repository for an October 1–14 baseline and October 15–28 follow-up. Check each PR’s review and rework at the same elapsed time after merge, with records available through November 11.

Check Cursor records Copilot records
Work and users API bug fixes from the selected engineers The same work and engineers; some use both tools
Attribution Line-level git-ai attestations available across both periods Activity matching begins October 22; earlier tool activity is missing
Shared PRs Several changes also involve Copilot The same shared changes appear here; whole-PR output cannot be credited again
Cost Paid seats and separately billed overage identified; list-price consumption kept separately Paid seats and usage charges identified; consumption estimates kept separately
Delivery and quality PR timestamps, review waits and follow-up changes available Those PR records exist, but the incomplete tool attribution prevents a comparable before-and-after tool breakdown
Decision Inspect the work and attributed output Collect a comparable period before declaring a tool winner

In this example, the team cannot distinguish which assistant made developers faster. Copilot’s collection start falls inside the follow-up period, so the apparent change in its AI-attributed output could reflect newly captured activity. Weave’s attribution documentation explains how source availability affects attribution and why activity matching can over- or under-attribute a PR.

For shared changes, retain each PR once and use the supported contribution shares. An engineer using both assistants does not create two independent comparison groups. Keep unresolved attribution visible instead of splitting it evenly. Weave also distinguishes whole-PR output from the human or AI slice; adding whole-PR totals across overlapping lists overstates the work.

Use Weave’s Tools view and cost-per-output comparison to inspect each tool’s attributed output, cost and bug ratio. For speed, inspect cycle time and review delays for the relevant work. Estimated expert-effort output measures the work represented by a change, while those timings describe its delivery. Keep paid subscriptions and overage separate from list-price consumption when evaluating the bill.

The next action is to collect a comparable period and inspect the shared PRs before expanding or retiring either tool. Matching tasks makes the comparison more useful; it does not establish causality. A lower cost-per-output ratio alone cannot settle the decision when quality, delivery or missing records could change the conclusion.

Validate the joins before comparing tools

  • Confirm that multiple session links do not multiply one PR’s output.
  • Keep missing cost separate from zero cost.
  • Handle zero denominators explicitly.
  • Include tasks that legitimately produce no PR.
  • Give both comparison periods equal time to accumulate later merges and defects.

Weave’s AI report brings session, cost and engineering-output views together. Use the spend-share calculation to check how completion and merge signals overlap. Validate the joins and acceptance records in your own data before comparing tools or changing a budget.

About the author

  • Weave builds engineering intelligence, token intelligence, and routing for AI-assisted development. We publish practical guides on software delivery, model costs, and measurement. Learn more at https://workweave.ai.

No comments yet. Be the first to comment.

More from this blog