A skill called add-endpoint can be invoked three times in one coding session. That session can also use write-tests and update-docs. Counting each invocation as a completed task, or assigning the session’s full cost to each skill and summing the rows, produces misleading totals.
Weave’s skill reporting separates definitions, invocations and session results. The same separation is useful when building a measurement pipeline: record the instructions, their use, the sessions they appeared in and the resulting work as distinct entities.
The schema, query and worked example below illustrate that design. They are not a Weave API contract or a customer dataset.
Give each record a precise identity
Use four logical tables in a measurement pipeline:
| Entity | Suggested fields | Purpose |
|---|---|---|
| Skill definition | Repository, path, content hash, observed timestamp | Identify the instructions you can inspect |
| Skill invocation | Event ID, session ID, repository, skill name, invocation timestamp | Record observed use and deduplicate ingestion |
| Session | Session ID, status, cost amount, cost basis, cost availability, start and end | Count work sessions and retain cost semantics |
| Engineering result | Result ID, linked session IDs, result type, validation status | Keep the delivered work separate from invocation events |
Include the repository in the identity. Two repositories can contain different skills called review. Also distinguish the file you can inspect today from the instructions a past session loaded. If runtime version capture is unavailable, leave that version unknown. Assigning the latest default-branch content hash would invent a historical record.
Weave’s Skill Library combines definitions from connected repositories with skill-invocation telemetry. It retains definitions with no observed use and invocations with no matched definition. Preserve both states in your pipeline so a missing file or missing telemetry remains visible.
Count invocations and sessions separately
An invocation count records how often a skill was called. A distinct-session count records how many sessions included it. A task-completion count needs its own success criteria; repeated invocations and sessions are insufficient.
This PostgreSQL query reports sessions associated with each skill. It assumes skill_invocations has been deduplicated by event ID and sessions contains one row per session. All included costs use the same API-equivalent basis. The date filter selects sessions containing a skill invocation in the window. Each selected session contributes its full recorded cost, including activity outside that window.
Check that every invocation's session ID has a matching session before using the inner join. Enforce a foreign key or report orphan invocation rows separately. Otherwise, missing sessions disappear from the denominator without an error.
WITH skill_sessions AS (
SELECT DISTINCT repository_id, skill_name, session_id
FROM skill_invocations
WHERE invoked_at >= :start_at
AND invoked_at < :end_at
)
SELECT
ss.repository_id,
ss.skill_name,
COUNT(*) AS sessions_using_skill,
COUNT(*) FILTER (WHERE s.status = 'completed')
AS completed_sessions,
COUNT(*) FILTER (
WHERE s.cost_available
AND s.api_equivalent_cost_usd IS NOT NULL
)
AS sessions_with_cost,
AVG(s.api_equivalent_cost_usd)
FILTER (
WHERE s.cost_available
AND s.api_equivalent_cost_usd IS NOT NULL
)
AS average_session_cost_usd
FROM skill_sessions ss
JOIN sessions s USING (session_id)
GROUP BY ss.repository_id, ss.skill_name;
Each result row describes sessions that included the skill. Several skills can share the same sessions, so the row does not assign the session’s outcome exclusively to that skill. If you add completion rate, define which statuses enter the denominator and how you handle ongoing sessions.
Weave’s AI reporting uses this session-level view in Skill session stats. It helps a team investigate the work associated with a skill. Establishing whether the instructions caused an improvement requires a separate evaluation.
Do not sum overlapping skill rows into a budget total
Two sessions show how the totals can diverge:
| Session | Skills observed | API-equivalent session cost |
|---|---|---|
| S1 | add-endpoint, write-tests |
$4 |
| S2 | write-tests |
$2 |
The two sessions cost $6 on the stated basis. The sessions associated with add-endpoint total $4; those associated with write-tests total $6. Summing those skill rows would report $10 because S1 appears in both groups.
Deduplicate by session ID before calculating an organization total. A per-skill budget needs an explicit allocation policy. Dividing a session’s cost equally among its skills is one possible policy, but it cannot establish how much spending each skill caused. Turn-level attribution measures a narrower part of the session; label that boundary when reporting it.
An API-equivalent estimate prices model usage at API rates. A subscription payment measures what the organization paid. Keep those amounts separate, and show how many sessions have cost data. Missing cost stays missing: replacing it with zero would lower the average without any change in efficiency.
Connect results without multiplying them
A pull request may have several contributing sessions. One session may contribute to more than one change. Keep those relationships in a link table, and choose an attribution method before aggregating output.
Review a sample of the linked work before building a complicated allocation model. Record whether the intended task was completed, the required checks passed and the resulting change was accepted. Keep generated lines, session completion and accepted work as separate fields.
When a skill changes, compare similar tasks and record the model, tool and repository context. A new version may coincide with easier tickets, a model upgrade or more experienced users. To estimate the skill’s causal effect, use a controlled assignment or rollout with success criteria defined in advance.
Use the report to review the instructions
Review a shared skill’s definition coverage, observed use, session outcomes and cost coverage together. Then inspect examples behind the totals. An incomplete session can reveal instructions that need work; an accepted change can help the team understand where the procedure was useful.
Weave connects those records so engineering teams can evaluate a reusable procedure from its repository definition through its use in coding sessions. Use that evidence to decide whether to expand its use, revise it or retire it.
About the author
- Weave builds engineering intelligence, token intelligence, and routing for AI-assisted development. We publish practical guides on software delivery, model costs, and measurement. Learn more at https://workweave.ai.
Comments
Join the discussionNo comments yet. Be the first to comment.