Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude/skills/check-dbt-conventions/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ Every silver model and every dbt-owned staging model with append/event semantics
For each `.sql` file under `src/ingestion/silver/` and `src/ingestion/connectors/*/dbt/`:

- If `materialized='table'` AND model name is in the allow-list above → PASS
- If `materialized='table'` AND model name is NOT in the allow-list → FAIL with suggestion: "Convert to `materialized='incremental'`. For **silver** use `incremental_strategy='delete+insert'` + `unique_key='unique_key'`; for **staging** use `incremental_strategy='append'`. Add `WHERE _version > (SELECT max(_version) FROM {{ this }})`. If upstream lacks `_version`, amend the SELECT to project `toUnixTimestamp64Milli(_airbyte_extracted_at) AS _version`."
- If `materialized='table'` AND model name is NOT in the allow-list → FAIL with suggestion: "Convert to `materialized='incremental'`. For **silver** use `incremental_strategy='delete+insert'` + `unique_key='unique_key'`; for **staging** use `incremental_strategy='append'`. Scope the incremental boundary to one source instance with `{{ silver_incremental_watermark([...]) }}` — never a table-wide `max(_version)`, which lets one producer's boundary permanently exclude a slower producer's rows. If upstream lacks `_version`, amend the SELECT to project `toUnixTimestamp64Milli(_airbyte_extracted_at) AS _version`."
- If `materialized='view'` for a silver `class_*` / `fct_*` / `mtr_*` → FAIL (views forbidden in silver per check 1)
- If `materialized='ephemeral'` → cross-checked by Check 6

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,18 @@ against its eight. The real gaps are elsewhere:
3. **Two facts stall inside our own pipeline**: `last_active` never leaves bronze, and rows
with a non-active `status` are dropped without anyone knowing what is lost.

**Deferred out of this issue.** `prs_with_cc_count` and `prs_total_count` also reach silver
unread, but they answer whether a pull request involved Claude Code — the subject of `#1660`,
not of cost. No requirement here calls for them, and FR-9 forbids the per-PR cost figure they
would invite. Recorded as a candidate for `#1660`; the vendor populates them only where
Anthropic's GitHub app is connected.
**Resolved since.** The `status` half of gap 3 is closed: the value is carried into
class-contract `seat_status` and reaches gold as a dimension, because filtering on it was
retroactive — the vendor restates the status for every day it re-reads, so dropping non-active
rows deleted the whole history of whoever left. See `audit-claude-team.md` D1. `last_active`
still stops at bronze (D4).

**Deferred out of this issue, then revised.** `prs_with_cc_count` and `prs_total_count` answer
whether a pull request involved Claude Code — the subject of `#1660`, not of cost — and FR-9
forbids the per-PR cost figure they would invite. That still holds for anything per-PR. The two
counts themselves no longer stop in silver: they are served as `ai.prs_with_assistant` and
`ai.prs_total`, emitted only where the vendor supplies a value, since the alternative was data
nobody could reach. The vendor populates them only where Anthropic's GitHub app is connected.

**Decomposition strategy**:

Expand Down Expand Up @@ -381,8 +388,8 @@ branch is picked up.
paid seat nobody used is visible as money. Room left under the extra-usage ceiling is
**not** underuse and is never counted as such: it was never purchased.
- The seat-state filter lives inside the overage branch rather than being inherited from
the activity stream. A deactivated person keeps an overage row but loses activity rows,
and would otherwise read as an idle seat. The gate is `credit_limit_cents IS NOT NULL`;
the activity stream: a seat with no usage has no activity row to inherit a state from.
The gate is `credit_limit_cents IS NOT NULL`;
`is_enabled` is carried as a dimension and never filtered on (2.1).
- e2e for both.

Expand Down
5 changes: 3 additions & 2 deletions docs/components/backend/analytics/specs/ai-cost/PRD.md
Original file line number Diff line number Diff line change
Expand Up @@ -559,8 +559,9 @@ rest classified but unattributed. *Confirming the real value distribution on pro
would make the rule exhaustive rather than defensive.*

**OD-5 — Seat state filter.** *(resolved — recorded for traceability)* A deactivated person
keeps an overage row but loses activity rows, so they would read as an unused seat. The
filter must live inside the overage branch. `is_enabled` was the candidate; its semantics
keeps an overage row and, since the seat status became a carried column rather than a filter,
keeps their activity rows too. The filter must still live inside the overage branch, because a
seat with no usage has no activity row to inherit a state from. `is_enabled` was the candidate; its semantics
remain undocumented and one observation cannot separate "extra usage is disabled" from "the
seat is not assigned", so it is carried as a dimension and never used as a filter. Seat state
gates on `credit_limit_cents IS NOT NULL` instead.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,19 @@ FROM bronze_claude_team.claude_team_code_metrics FINAL
GROUP BY status
```

**Decision: keep the filter.** The vendor documents no value set for this field, so a value
other than `active` may appear at any time and would be dropped without a trace. Add a
build-time check that counts non-active rows rather than removing the filter.
**Decision: carry the field, do not filter on it.** The first reading — keep the filter, add
a check that counts non-active rows — missed that the filter is retroactive. The endpoint
restates the status for every day it re-reads, so the value describes the seat as of the read
and not as of the metric date: the day a person's seat is deactivated, the filter deletes
their entire history, and a full refresh removes it from silver altogether. Nothing about a
seat's present state licenses deleting the work it recorded.

So staging carries the value into the class-contract column `seat_status`, gold exposes it as
the `seat_status` dimension, and emission is gated on the activity counters instead — the same
gate every other contributor to the class already uses, and the one the class contract asks
for. Rows written before the column existed carry no value and read as `unknown`, which is
what they are: re-materialising them from bronze would mean a full refresh, and that is the
operation that costs months of overage history.

## 3. `is_enabled` — resolves PRD OD-5

Expand Down Expand Up @@ -124,6 +134,15 @@ Anthropic populates both counters only for organisations that have connected its
whether a pull request involved Claude Code, which is `#1660`'s subject, not cost — the
entry was removed from the decomposition and recorded as a candidate for `#1660`.

**Revised on the counters.** Leaving them in silver unread meant nobody could see them at all,
which is a worse position than serving them with the vendor's own caveat attached. Both now
reach gold as `ai.prs_with_assistant` and `ai.prs_total`, emitted only where the vendor
supplies a value, so an organisation without the app returns no value rather than a zero —
consequence 1 above, honoured by construction. Consequence 2 stands: a live demonstration
still needs a tenant with the app connected. `ai.prs_total` is served as context for the other
measure and not as a goal of its own, because the vendor's window may not be the day the row
is dated.

## 7. Seat economics — what to show for 2.3

```sql
Expand Down Expand Up @@ -155,11 +174,11 @@ The properties the demonstration rests on, each visible in that query:

| # | Decision | Affects |
|---|---|---|
| D1 | Keep the `status = 'active'` filter; add a build-time count of non-active rows | 2.1 → build |
| D1 | Carry `status` into class-contract `seat_status` and gate emission on the activity counters instead; the filter was retroactive and deleted a deactivated person's whole history | 2.1 → build |
| D2 | Seat-state gate is `credit_limit_cents IS NOT NULL` alone; `is_enabled` is carried as a dimension, never as a filter | 2.6; closes PRD OD-5 |
| D3 | Do not carry `api_key_name`, `avg_cost_per_day`, `avg_lines_accepted_per_day`, `prs_with_cc_percentage` | 2.1 → closed |
| D4 | Do not carry `last_active` yet; record as a candidate for an "idle seat" signal | 2.6 |
| D5 | PR attribution leaves this decomposition; recorded as a candidate for `#1660` | — |
| D5 | PR attribution leaves this decomposition. Superseded on the counters themselves: the two counts already in silver now reach gold as `ai.prs_with_assistant` / `ai.prs_total`, honest-NULL where the vendor does not report. The per-PR cost figure FR-9 forbids stays out, and the "did this PR involve the assistant" question remains `#1660`'s | — |
| D6 | Guard `used_credits_basis` — the used-vs-limit comparison is only valid while every row shares one basis | 2.3 |
| D7 | `members` and `invites` stop at bronze in both implementations; no parity gap, no action | — |
| D8 | Invoices are the only genuine extraction gap against the reference | 2.5 |
Expand Down
28 changes: 17 additions & 11 deletions docs/components/backend/analytics/specs/ai-cost/research-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -348,16 +348,19 @@ by `cost_type`, not by which connector produced it.** A `billing_model` column o
| Stream (`connector.yaml`) | `claude_team_overage_spend` (line 168) | `claude_team_code_metrics` (line 288) |
| Bronze table | `bronze_claude_team.claude_team_overage_spend` | `bronze_claude_team.claude_team_code_metrics` |
| Grain | one row per seat, monthly snapshot | one row per (`metric_date`, `email`) |
| Used in | `claude_team__ai_overage.sql` → column `is_enabled` | `claude_team__ai_dev_usage.sql:93` → `WHERE status = 'active'` |
| Used in | `claude_team__ai_overage.sql` → column `is_enabled` | `claude_team__ai_dev_usage.sql` → class-contract column `seat_status` (was a `WHERE status = 'active'` filter; audit D1) |
| Meaning | undocumented | undocumented beyond `'active'` |

Consequence for seat-utilisation metrics: the denominator comes from `class_ai_overage`
while activity comes from `class_ai_dev_usage` filtered to `status='active'`. A deactivated
person keeps their overage row but loses their activity rows, and would register as an
under-utilised seat. The state filter has to live inside the overage branch. `is_enabled`
looked like the candidate when this section was written; §16 and audit decision D2 settle it
the other way — the gate is `credit_limit_cents IS NOT NULL` alone, and `is_enabled` is
carried as a dimension. This paragraph is kept for the reasoning, not for its conclusion.
while activity comes from `class_ai_dev_usage`, which at the time was filtered to
`status='active'`. A deactivated person kept their overage row but lost their activity rows,
and would register as an under-utilised seat. That half no longer applies — audit D1 removed
the filter, so the activity rows stay. The conclusion is unchanged for the other reason: a
person with no usage at all has no activity row to inherit a state from, so the state filter
still has to live inside the overage branch. `is_enabled` looked like the candidate when this
section was written; §16 and audit decision D2 settle it the other way — the gate is
`credit_limit_cents IS NOT NULL` alone, and `is_enabled` is carried as a dimension. This
paragraph is kept for the reasoning, not for its conclusion.

### Other `overage_spend_limits` fields

Expand Down Expand Up @@ -415,10 +418,13 @@ gap independent of #1607 and should be tracked separately.
5. **`dbt source freshness` is scheduled nowhere** — see §15. Every connector declares
thresholds; no workflow runs the command. Platform-wide, not AI-specific; should become
its own issue linked to `#1607`.
6. **Vendor pull-request attribution deferred** — `prs_with_cc_count` and `prs_total_count`
reach silver unread. Removed from this decomposition as `#1660` territory; no requirement
here needs them, and the vendor populates them only where Anthropic's GitHub app is
connected — everywhere else both counters read zero without meaning it.
6. **Vendor pull-request attribution deferred, then served** — `prs_with_cc_count` and
`prs_total_count` reached silver unread. Removed from this decomposition as `#1660`
territory; no requirement here needed them, and the vendor populates them only where
Anthropic's GitHub app is connected — everywhere else both counters read zero without
meaning it. Reversed on the counters alone: they are served as `ai.prs_with_assistant` and
`ai.prs_total`, emitted only where the vendor supplies a value so that the zeros never
become assertions. Anything per-PR remains `#1660`'s and FR-9's.
7. **#1986 "Verified state" corrections** — the Cursor hard-NULL claim is stale, and
`ai_cost_person_period` is described as the grain the catalog could read, though the
unified registry reads `ai_metric_observations`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ mod tests {
#[test]
fn registry_declares_the_expected_counts() {
assert_eq!(builtin_sources().len(), 6, "builtin source count");
assert_eq!(builtin_metrics().len(), 62, "builtin metric count");
assert_eq!(builtin_metrics().len(), 64, "builtin metric count");
}

#[test]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ this file and the registry disagree.
- Reads: dev_conversations
- Formula: sum(dev_conversations)
- Shape: integer, higher_is_better, unit conversations
- Notes: Person-attributed coding conversations from dev tools that report them.
- Notes: Person-attributed conversations with coding AI tools, counted as the vendor counts them. For the agent tools this is the session or thread count — a session is the unit of conversation there, and no vendor publishes a separate conversation counter — so the number reads as "times the person started working with the assistant", not as messages exchanged. Tools that report no such unit, inline-completion tools among them, return no value rather than a zero.

## ai.chat_assistant_conversations — AI chat conversations

Expand All @@ -100,6 +100,22 @@ this file and the registry disagree.
- Shape: integer, higher_is_better, unit conversations
- Notes: Person-attributed chat assistant conversations from supported AI chat tools.

## ai.prs_with_assistant — PRs with AI assistance

- Source: ai_usage (ai_metric_observations)
- Reads: prs_with_assistant
- Formula: sum(prs_with_assistant)
- Shape: integer, higher_is_better, unit PRs
- Notes: Pull requests where the coding assistant was active at least once, as the vendor attributes them. Reported only by sources that connect to the code host themselves, so a person working without that connection returns no value rather than a zero. Counts pull requests, not commits or lines, and says nothing about how much of the change the assistant wrote.

## ai.prs_total — PRs seen by the AI vendor

- Source: ai_usage (ai_metric_observations)
- Reads: prs_total
- Formula: sum(prs_total)
- Shape: integer, neutral, unit PRs
- Notes: Pull requests the AI vendor observed for the person, served as the context for PRs with AI assistance rather than as a goal of its own. It is the vendor's count over the vendor's own window, which need not be the day it is reported against and need not agree with the git sources — read it next to that measure, not next to git pull-request metrics.

## git.commits — Commits

- Source: git (git_metric_observations)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,14 @@ sources:
evidence_granularity: source_summary
- key: chat_assistant_conversations
evidence_granularity: source_summary
- key: prs_with_assistant
evidence_granularity: source_summary
- key: prs_total
evidence_granularity: source_summary
dimensions:
- tool
- surface
- seat_status
- source:
key: git
kind: managed_observation
Expand Down Expand Up @@ -382,7 +387,7 @@ metrics:
label: AI dev conversations
short_label: AI dev chats
description: Coding tool conversations where the source reports them
explanation: Person-attributed coding conversations from dev tools that report them.
explanation: Person-attributed conversations with coding AI tools, counted as the vendor counts them. For the agent tools this is the session or thread count — a session is the unit of conversation there, and no vendor publishes a separate conversation counter — so the number reads as "times the person started working with the assistant", not as messages exchanged. Tools that report no such unit, inline-completion tools among them, return no value rather than a zero.
unit: conversations
format: integer
direction: higher_is_better
Expand Down Expand Up @@ -413,6 +418,44 @@ metrics:
dimensions:
- tool
- surface
- metric_key: ai.prs_with_assistant
source_key: ai_usage
subject: pull_requests
label: PRs with AI assistance
short_label: PRs with AI
description: Pull requests the coding assistant touched
explanation: Pull requests where the coding assistant was active at least once, as the vendor attributes them. Reported only by sources that connect to the code host themselves, so a person working without that connection returns no value rather than a zero. Counts pull requests, not commits or lines, and says nothing about how much of the change the assistant wrote.
unit: PRs
format: integer
direction: higher_is_better
entity_type: person
computation: sum
peer_cohort_key: org_unit
inputs:
- input_role: value
measure_key: prs_with_assistant
dimensions:
- tool
- seat_status
- metric_key: ai.prs_total
source_key: ai_usage
subject: pull_requests
label: PRs seen by the AI vendor
short_label: PRs seen
description: Pull requests the AI vendor counted in the same window
explanation: Pull requests the AI vendor observed for the person, served as the context for PRs with AI assistance rather than as a goal of its own. It is the vendor's count over the vendor's own window, which need not be the day it is reported against and need not agree with the git sources — read it next to that measure, not next to git pull-request metrics.
unit: PRs
format: integer
direction: neutral
entity_type: person
computation: sum
peer_cohort_key: org_unit
inputs:
- input_role: value
measure_key: prs_total
dimensions:
- tool
- seat_status
- metric_key: git.commits
source_key: git
subject: commits
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@
-- Mapping notes:
-- tool='codex' — dev-tool discriminator (cf. 'claude_code', 'cursor').
-- session_count ← n_threads — a Codex thread is the closest analogue to a coding session.
-- conversation_count ← n_threads — the same thread count: a thread IS the
-- unit of conversation, and no separate
-- conversation counter exists upstream.
-- lines_added ← lines_added — AI-accepted lines (from code_attribution.lines_of_code.added).
-- cost_cents ← NULL — `credits` are Codex usage credits, not a currency amount.
-- Codex-only counters (credits, n_turns, text_tokens, current_streak) are
Expand Down Expand Up @@ -71,7 +74,9 @@ SELECT
'chatgpt_team' AS source,
data_source,
CAST(_airbyte_extracted_at AS Nullable(DateTime64(3))) AS collected_at,
toUnixTimestamp64Milli(_airbyte_extracted_at) AS _version
toUnixTimestamp64Milli(_airbyte_extracted_at) AS _version,
-- The usage endpoint carries no seat lifecycle state.
CAST(NULL AS Nullable(String)) AS seat_status
FROM (
-- Bronze dedup: keep the latest extract per (email, date). Defensive depth —
-- becomes a no-op once promote_bronze_to_rmt merges (ADR-0002), but guards
Expand Down
Loading
Loading