<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Weave Engineering]]></title><description><![CDATA[Practical guides from Weave on engineering productivity, token costs, and AI-assisted software delivery. We explain measurement methods, share technical researc]]></description><link>https://workweave.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Mon, 16 Nov 2026 21:50:13 GMT</lastBuildDate><atom:link href="https://workweave.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[How to Measure a Shared Coding Skill Without Double-Counting Sessions]]></title><description><![CDATA[Measure shared coding skills with distinct session counts, explicit cost definitions and a PostgreSQL example that avoids double-counting.]]></description><link>https://workweave.hashnode.dev/how-to-measure-a-shared-coding-skill-without-double-counting-sessions</link><guid isPermaLink="true">https://workweave.hashnode.dev/how-to-measure-a-shared-coding-skill-without-double-counting-sessions</guid><category><![CDATA[PostgreSQL]]></category><category><![CDATA[Software Engineering]]></category><dc:creator><![CDATA[Weave]]></dc:creator><pubDate>Sun, 15 Nov 2026 00:48:09 GMT</pubDate><content:encoded><![CDATA[<p>A skill called <code>add-endpoint</code> can be invoked three times in one coding session. That session can also use <code>write-tests</code> and <code>update-docs</code>. Counting each invocation as a completed task, or assigning the session’s full cost to each skill and summing the rows, produces misleading totals.</p>
<p>Weave’s skill reporting separates definitions, invocations and session results. The same separation is useful when building a measurement pipeline: record the instructions, their use, the sessions they appeared in and the resulting work as distinct entities.</p>
<p>The schema, query and worked example below illustrate that design. They are not a Weave API contract or a customer dataset.</p>
<h2>Give each record a precise identity</h2>
<p>Use four logical tables in a measurement pipeline:</p>
<table>
<thead>
<tr>
<th>Entity</th>
<th>Suggested fields</th>
<th>Purpose</th>
</tr>
</thead>
<tbody><tr>
<td>Skill definition</td>
<td>Repository, path, content hash, observed timestamp</td>
<td>Identify the instructions you can inspect</td>
</tr>
<tr>
<td>Skill invocation</td>
<td>Event ID, session ID, repository, skill name, invocation timestamp</td>
<td>Record observed use and deduplicate ingestion</td>
</tr>
<tr>
<td>Session</td>
<td>Session ID, status, cost amount, cost basis, cost availability, start and end</td>
<td>Count work sessions and retain cost semantics</td>
</tr>
<tr>
<td>Engineering result</td>
<td>Result ID, linked session IDs, result type, validation status</td>
<td>Keep the delivered work separate from invocation events</td>
</tr>
</tbody></table>
<p>Include the repository in the identity. Two repositories can contain different skills called <code>review</code>. Also distinguish the file you can inspect today from the instructions a past session loaded. If runtime version capture is unavailable, leave that version unknown. Assigning the latest default-branch content hash would invent a historical record.</p>
<p><a href="https://docs.weaveos.com/reports/skill-library">Weave’s Skill Library</a> combines definitions from connected repositories with skill-invocation telemetry. It retains definitions with no observed use and invocations with no matched definition. Preserve both states in your pipeline so a missing file or missing telemetry remains visible.</p>
<h2>Count invocations and sessions separately</h2>
<p>An invocation count records how often a skill was called. A distinct-session count records how many sessions included it. A task-completion count needs its own success criteria; repeated invocations and sessions are insufficient.</p>
<p>This PostgreSQL query reports sessions associated with each skill. It assumes <code>skill_invocations</code> has been deduplicated by event ID and <code>sessions</code> contains one row per session. All included costs use the same API-equivalent basis. The date filter selects sessions containing a skill invocation in the window. Each selected session contributes its full recorded cost, including activity outside that window.</p>
<p>Check that every invocation's session ID has a matching session before using the inner join. Enforce a foreign key or report orphan invocation rows separately. Otherwise, missing sessions disappear from the denominator without an error.</p>
<pre><code class="language-sql">WITH skill_sessions AS (
  SELECT DISTINCT repository_id, skill_name, session_id
  FROM skill_invocations
  WHERE invoked_at &gt;= :start_at
    AND invoked_at &lt; :end_at
)
SELECT
  ss.repository_id,
  ss.skill_name,
  COUNT(*) AS sessions_using_skill,
  COUNT(*) FILTER (WHERE s.status = 'completed')
    AS completed_sessions,
  COUNT(*) FILTER (
    WHERE s.cost_available
      AND s.api_equivalent_cost_usd IS NOT NULL
  )
    AS sessions_with_cost,
  AVG(s.api_equivalent_cost_usd)
    FILTER (
      WHERE s.cost_available
        AND s.api_equivalent_cost_usd IS NOT NULL
    )
    AS average_session_cost_usd
FROM skill_sessions ss
JOIN sessions s USING (session_id)
GROUP BY ss.repository_id, ss.skill_name;
</code></pre>
<p>Each result row describes sessions that included the skill. Several skills can share the same sessions, so the row does not assign the session’s outcome exclusively to that skill. If you add completion rate, define which statuses enter the denominator and how you handle ongoing sessions.</p>
<p>Weave’s <a href="https://docs.weaveos.com/reports/ai">AI reporting</a> uses this session-level view in Skill session stats. It helps a team investigate the work associated with a skill. Establishing whether the instructions caused an improvement requires a separate evaluation.</p>
<h2>Do not sum overlapping skill rows into a budget total</h2>
<p>Two sessions show how the totals can diverge:</p>
<table>
<thead>
<tr>
<th>Session</th>
<th>Skills observed</th>
<th align="right">API-equivalent session cost</th>
</tr>
</thead>
<tbody><tr>
<td>S1</td>
<td><code>add-endpoint</code>, <code>write-tests</code></td>
<td align="right">$4</td>
</tr>
<tr>
<td>S2</td>
<td><code>write-tests</code></td>
<td align="right">$2</td>
</tr>
</tbody></table>
<p>The two sessions cost $6 on the stated basis. The sessions associated with <code>add-endpoint</code> total $4; those associated with <code>write-tests</code> total $6. Summing those skill rows would report $10 because S1 appears in both groups.</p>
<p>Deduplicate by session ID before calculating an organization total. A per-skill budget needs an explicit allocation policy. Dividing a session’s cost equally among its skills is one possible policy, but it cannot establish how much spending each skill caused. Turn-level attribution measures a narrower part of the session; label that boundary when reporting it.</p>
<p>An API-equivalent estimate prices model usage at API rates. A subscription payment measures what the organization paid. Keep those amounts separate, and show how many sessions have cost data. Missing cost stays missing: replacing it with zero would lower the average without any change in efficiency.</p>
<h2>Connect results without multiplying them</h2>
<p>A pull request may have several contributing sessions. One session may contribute to more than one change. Keep those relationships in a link table, and choose an attribution method before aggregating output.</p>
<p>Review a sample of the linked work before building a complicated allocation model. Record whether the intended task was completed, the required checks passed and the resulting change was accepted. Keep generated lines, session completion and accepted work as separate fields.</p>
<p>When a skill changes, compare similar tasks and record the model, tool and repository context. A new version may coincide with easier tickets, a model upgrade or more experienced users. To estimate the skill’s causal effect, use a controlled assignment or rollout with success criteria defined in advance.</p>
<h2>Use the report to review the instructions</h2>
<p>Review a shared skill’s definition coverage, observed use, session outcomes and cost coverage together. Then inspect examples behind the totals. An incomplete session can reveal instructions that need work; an accepted change can help the team understand where the procedure was useful.</p>
<p>Weave connects those records so engineering teams can evaluate a reusable procedure from its repository definition through its use in coding sessions. Use that evidence to decide whether to expand its use, revise it or retire it.</p>
]]></content:encoded></item><item><title><![CDATA[How to Measure AI Coding Productivity]]></title><description><![CDATA[Use Weave to compare AI coding output, cost and quality, with a Python audit of completion and merge signals.]]></description><link>https://workweave.hashnode.dev/how-to-measure-ai-coding-productivity</link><guid isPermaLink="true">https://workweave.hashnode.dev/how-to-measure-ai-coding-productivity</guid><category><![CDATA[Productivity]]></category><category><![CDATA[Software Engineering]]></category><dc:creator><![CDATA[Weave]]></dc:creator><pubDate>Sun, 15 Nov 2026 00:45:18 GMT</pubDate><content:encoded><![CDATA[<p>Measure the engineering work produced, its quality and its cost.</p>
<p>A coding session marked “complete” is useful operational data. It does not establish that code merged, reached production or solved the intended problem.</p>
<p>Weave’s Q3 research makes that distinction measurable. The Python calculation below uses the report’s published spend shares. The rest of the audit specifies the records needed to investigate the same question on your own data.</p>
<h2>Check what “complete” means in the data</h2>
<p>In the retained <a href="https://weaveos.com/research/q3-2026">Q3 2026 sample</a>, 23% of traced session spend had both a completion signal and a merge link. Another 31% had completion without a merge link, while 17% had a merge link without completion.</p>
<p>The inputs below are rounded percentages of spend. They are not session counts.</p>
<pre><code class="language-python">spend_share_pct = {
    "completed_and_merged": 23,
    "merged_without_completion": 17,
    "completed_without_merge": 31,
    "unclassified": 17,
    "abandoned_or_blocked": 12,
}
assert sum(spend_share_pct.values()) == 100

completed = (
    spend_share_pct["completed_and_merged"]
    + spend_share_pct["completed_without_merge"]
)
merge_linked = (
    spend_share_pct["completed_and_merged"]
    + spend_share_pct["merged_without_completion"]
)

print(
    "Merge-linked share of completion-labelled spend: "
    f'{spend_share_pct["completed_and_merged"] / completed:.1%}'
)
print(
    "No-completion share of merge-linked spend: "
    f'{spend_share_pct["merged_without_completion"] / merge_linked:.1%}'
)
</code></pre>
<p>The results are <strong>42.6%</strong> and <strong>42.5%</strong>. Completion labels miss some merge-linked spending and include spending with no recorded merge. Neither ratio is a task-success rate.</p>
<p>The source report excludes one organization responsible for 77% of traced spend and roughly half the sessions. These figures describe the remaining sample. A missing merge link also does not prove that work failed: some tasks need no PR, and some links are missing.</p>
<h2>Keep the events separate</h2>
<p>Store session outcome, PR merge time, deployment time and task acceptance as separate fields. A task can require several sessions and PRs; a single PR can contain work from several sessions. Keep the link IDs and the attribution rule so joining those tables does not duplicate cost or output.</p>
<p>Use an explicit observation cutoff. If a PR merges after the reporting period, retain that fact instead of silently rewriting the original result. Report unmatched sessions and tasks alongside matched records.</p>
<h2>Add an output measure and quality checks</h2>
<p>Merged PRs vary in difficulty. Weave’s <a href="https://docs.weaveos.com/metrics/code-output">Code Output</a> estimates the expert effort represented by the final merged diff and explains the assessment. It measures work represented by the code; it does not measure an engineer’s actual working hours.</p>
<p>For a defined team and period, calculate output per attributed dollar. Keep subscriptions, separately billed usage and estimates identifiable, and count each expense once. Compare similar work and preserve the team’s membership in the comparison.</p>
<p>Then inspect review effort, reverts, defects and follow-up changes over the same observation window. Interpret a rise in output per dollar alongside the quality and type of work produced.</p>
<h2>Compare coding tools on output, cost and quality</h2>
<p>Weave’s <a href="https://docs.weaveos.com/reports/ai">Tools view</a> compares connected coding tools such as Claude Code, Cursor, GitHub Copilot and Codex on engineering output, cost and bug ratio. Cost per output by tool puts spending on a common output basis. The Models view adds average cost per session, first-shot success, average AI output per session and code turnover. These views help a team choose which tools to expand and which workflows need attention.</p>
<p>Use the same engineers, repositories and period for the comparison, then inspect representative changes. Check <a href="https://weaveos.com/features/ai-code-authorship">the source of AI authorship evidence</a> and whether collection coverage differs across tools. A session associated with a tool and an attributable code change provide different evidence. If the decision depends on faster delivery, compare cycle time and review delays as well as output.</p>
<p>Weave’s <a href="https://weaveos.com/products/code-intelligence">Silk 1 evaluation</a> reports 3.3× lower average prediction error than the previous Weave model, with 88% of PRs within ±1 expert-effort hour of the reference estimate. Engineers can open a PR’s assessment and check its reasoning. Those estimates make different kinds of code changes easier to compare; a controlled rollout is still needed to isolate a tool’s effect.</p>
<h3>Compare Cursor and Copilot on the same work</h3>
<p><strong>Illustrative example, not customer data or a benchmark:</strong> a backend team wants to compare Cursor and Copilot on API bug fixes. Use the same engineers and repository for an October 1–14 baseline and October 15–28 follow-up. Check each PR’s review and rework at the same elapsed time after merge, with records available through November 11.</p>
<table>
<thead>
<tr>
<th>Check</th>
<th>Cursor records</th>
<th>Copilot records</th>
</tr>
</thead>
<tbody><tr>
<td>Work and users</td>
<td>API bug fixes from the selected engineers</td>
<td>The same work and engineers; some use both tools</td>
</tr>
<tr>
<td>Attribution</td>
<td>Line-level git-ai attestations available across both periods</td>
<td>Activity matching begins October 22; earlier tool activity is missing</td>
</tr>
<tr>
<td>Shared PRs</td>
<td>Several changes also involve Copilot</td>
<td>The same shared changes appear here; whole-PR output cannot be credited again</td>
</tr>
<tr>
<td>Cost</td>
<td>Paid seats and separately billed overage identified; list-price consumption kept separately</td>
<td>Paid seats and usage charges identified; consumption estimates kept separately</td>
</tr>
<tr>
<td>Delivery and quality</td>
<td>PR timestamps, review waits and follow-up changes available</td>
<td>Those PR records exist, but the incomplete tool attribution prevents a comparable before-and-after tool breakdown</td>
</tr>
<tr>
<td>Decision</td>
<td>Inspect the work and attributed output</td>
<td>Collect a comparable period before declaring a tool winner</td>
</tr>
</tbody></table>
<p>In this example, the team cannot distinguish which assistant made developers faster. Copilot’s collection start falls inside the follow-up period, so the apparent change in its AI-attributed output could reflect newly captured activity. <a href="https://docs.weaveos.com/analysis/ai-output-attribution">Weave’s attribution documentation</a> explains how source availability affects attribution and why activity matching can over- or under-attribute a PR.</p>
<p>For shared changes, retain each PR once and use the supported contribution shares. An engineer using both assistants does not create two independent comparison groups. Keep unresolved attribution visible instead of splitting it evenly. Weave also distinguishes <a href="https://docs.weaveos.com/metrics/ai-output-percentage">whole-PR output from the human or AI slice</a>; adding whole-PR totals across overlapping lists overstates the work.</p>
<p>Use Weave’s <a href="https://docs.weaveos.com/reports/ai">Tools view and cost-per-output comparison</a> to inspect each tool’s attributed output, cost and bug ratio. For speed, inspect cycle time and review delays for the relevant work. Estimated expert-effort output measures the work represented by a change, while those timings describe its delivery. Keep <a href="https://docs.weaveos.com/metrics/ai-tool-costs">paid subscriptions and overage separate from list-price consumption</a> when evaluating the bill.</p>
<p>The next action is to collect a comparable period and inspect the shared PRs before expanding or retiring either tool. Matching tasks makes the comparison more useful; it does not establish causality. A lower cost-per-output ratio alone cannot settle the decision when quality, delivery or missing records could change the conclusion.</p>
<h2>Validate the joins before comparing tools</h2>
<ul>
<li>Confirm that multiple session links do not multiply one PR’s output.</li>
<li>Keep missing cost separate from zero cost.</li>
<li>Handle zero denominators explicitly.</li>
<li>Include tasks that legitimately produce no PR.</li>
<li>Give both comparison periods equal time to accumulate later merges and defects.</li>
</ul>
<p>Weave’s <a href="https://docs.weaveos.com/reports/ai">AI report</a> brings session, cost and engineering-output views together. Use the spend-share calculation to check how completion and merge signals overlap. Validate the joins and acceptance records in your own data before comparing tools or changing a budget.</p>
]]></content:encoded></item></channel></rss>