diff --git a/docs/components/connectors/ai/README.md b/docs/components/connectors/ai/README.md index 35d04892b..050cb2bfe 100644 --- a/docs/components/connectors/ai/README.md +++ b/docs/components/connectors/ai/README.md @@ -173,6 +173,9 @@ This feeds the **planned** `class_ai_org_usage` (see below). | `class_ai_api_usage` (existing) | ❌ N/A | ✅ feeds | ✅ feeds | ❌ tagged as `openai`, not silver-class | ❌ N/A (flat-seat) | ❌ no API | ❌ N/A | ❌ N/A | ❌ N/A | | `class_ai_assistant_usage` (proposed, NEW) | ❌ N/A | ❌ N/A | ✅ feeds (chat/cowork/office/web_search) | ❌ N/A | ✅ feeds (planned — Compliance Logs `conversations` events) | ❌ no API | ❌ N/A | 🟡 chat surface | ❌ N/A | | `class_ai_cost` (proposed, NEW) | ❌ | ✅ feeds | ❌ | ✅ feeds | ❌ (flat-seat, not exposed) | ❌ | ❌ | ✅ `promptCreditsUsed` | ❌ | +| `class_ai_overage` (NEW — Claude Team BUILT †) | ❌ | ❌ | ❌ | 🟡 `credits_used` vs quota | ❌ | 🟡 `credits_used` vs quota | ❌ | 🟡 `promptCreditsUsed` vs plan | ❌ | + +† **Claude Team** is the first live source for `class_ai_overage` but is **not a column above** (this matrix predates the claude-team connector). Source `claude_team__ai_overage` reads `/overage_spend_limits` via the customer proxy: per-seat `used_credits` vs `monthly_credit_limit` (already cents, USD) → `overage_cents = max(0, used − limit)`. Verified live (149 seats, Owner role required). See the dedicated section below. ## Known gaps (todo backlog) @@ -193,15 +196,16 @@ This feeds the **planned** `class_ai_org_usage` (see below). Goal: each class is one **product domain** with a homogeneous schema. No NULL-padded columns from sources that "don't expose this kind of data". When a provider grows (Anthropic adds Compliance API, OpenAI splits ChatGPT/Codex, etc.) — rows get added to existing classes or a new class appears, but **existing class schemas don't get smeared**. -## Proposed structure (5 core + 2 optional classes) +## Proposed structure (6 core + 2 optional classes) -``` +```text silver/ ├── ai/ │ ├── class_ai_dev_usage (per-person-day code activity) │ ├── class_ai_assistant_usage (per-person-day chat/cowork/office/web) │ ├── class_ai_api_usage (per-key-or-project-day tokens + multi-modal API) -│ ├── class_ai_cost (per-line-item-day financial) +│ ├── class_ai_cost (per-line-item-day financial — org/project total) +│ ├── class_ai_overage (per-person-month spend-over-limit — BUILT: Claude Team) │ └── class_ai_audit_log (event-stream — admin actions, compliance) └── _shared/ ├── class_people (existing) — identity unification @@ -319,6 +323,31 @@ provider, source, data_source, collected_at | Claude Enterprise | ❌ (no cost data) | | OpenAI | `openai__ai_cost` (existing `to_ai_cost.sql` — re-tag `silver:class_ai_cost`) | +### 4a. `class_ai_overage` (NEW — sibling of `class_ai_cost`; **BUILT for Claude Team**) + +**Grain:** `(insight_tenant_id, source, account/email, billing month)` — one row per person × month. + +**Why separate from `class_ai_cost`:** `class_ai_cost` answers *"how much did the org/project spend"* (financial line items, per-project/day, not attributable to a person). `class_ai_overage` answers *"how much did **each person** spend **above their own monthly limit**"* — a per-seat budget-vs-actual comparison. Overage needs a per-seat **limit** that cost line items don't carry, and is the metric product asked for ("сколько денег человек потратил свыше лимита"). Keeping it separate avoids smearing a per-person limit column across the org-grain cost class. + +**Schema (19 positional cols — first contributor `claude_team__ai_overage` defines the contract; vendor extras → `overage_metrics_json`, never new columns):** +```text +insight_tenant_id, source_id, unique_key, email, account_id +period_month, tool, seat_tier, currency +credit_limit_cents, used_amount_cents, overage_cents -- minor units (cents); overage = max(0, used − limit), honest-NULL if no limit +is_over_limit, is_enabled, overage_metrics_json +source, data_source, collected_at, _version +``` + +**Provider feeding:** + +| Provider | Source | Status | +|---|---|---| +| **Claude Team** | `claude_team__ai_overage` — `/overage_spend_limits` per-seat `used_credits` vs `monthly_credit_limit` (already cents, USD) | ✅ **BUILT** (needs proxy sessionKey with `billing:view`/Owner — verified live, 149 seats) | +| OpenAI / ChatGPT | `openai__ai_overage` (FUTURE) — `chat_activity.credits_used` + codex `credits` per user | 🟡 needs per-seat included quota + credit→$ rate (product input); `overage_cents` stays NULL until quota known | +| Windsurf | `windsurf__ai_overage` (FUTURE) — `promptCreditsUsed` per user vs plan | 🟡 possible if plan limit is known | + +**Units:** `used_credits` / `monthly_credit_limit` from the Claude web API are **already cents** (USD, `decimal_places=2`) — mapped straight to `*_cents` with **no ×100** (unlike `claude_team__ai_dev_usage.cost_cents`, which ×100 a dollar-string). **Month grain:** the endpoint is a current-period snapshot with no period field → `period_month = toStartOfMonth(_airbyte_extracted_at)`, latest snapshot per (seat, month) kept (month in `unique_key`) so history accrues and the in-flight month stays idempotent. + ### 5. `class_ai_audit_log` (NEW) **Grain:** event-stream, `(event_id, event_at)` diff --git a/docs/components/connectors/ai/claude-team/specs/DESIGN.md b/docs/components/connectors/ai/claude-team/specs/DESIGN.md index 6cdb95d7a..f317c2c46 100644 --- a/docs/components/connectors/ai/claude-team/specs/DESIGN.md +++ b/docs/components/connectors/ai/claude-team/specs/DESIGN.md @@ -196,9 +196,21 @@ allowed via `additionalProperties: true`. ### 4.4 Silver layer -Out of scope for MVP. `descriptor.dbt_select: ''`. When Silver lands -(Phase 6+), models will be tagged `claude-team` and selected with -`tag:claude-team+`. +Out of scope for the **Bronze MVP** — but Silver has since landed +(`descriptor.dbt_select: 'tag:claude-team+'`, not `''`). Models are +tagged `claude-team` and contribute to shared Silver classes: + +- `claude_team__ai_dev_usage` → `class_ai_dev_usage` (per-user-per-day + Claude Code usage from `claude_team_code_metrics`; INSIGHT-458). +- `claude_team__ai_overage` → `class_ai_overage` (per-seat-per-month + spend over the monthly credit limit from `claude_team_overage_spend`; + descriptor 1.3.0). `overage_cents = max(0, used_credits − + monthly_credit_limit)`, units already cents (no ×100). Gold surfaces + it as the `cc_overage` AI bullet. See the cross-connector contract in + `docs/components/connectors/ai/README.md` and `src/ingestion/silver/ai/schema.yml`. + +`bronze_promoted` (ADR-0002) promotes all populated Bronze streams, +including `claude_team_overage_spend`, to ReplacingMergeTree. ## 5. Operational limitations (MVP) diff --git a/docs/components/connectors/ai/claude-team/specs/FEATURE.md b/docs/components/connectors/ai/claude-team/specs/FEATURE.md index 3769efe8a..5e3062e27 100644 --- a/docs/components/connectors/ai/claude-team/specs/FEATURE.md +++ b/docs/components/connectors/ai/claude-team/specs/FEATURE.md @@ -10,7 +10,9 @@ Bring up the `claude-team` declarative Airbyte source against the customer-deployed proxy (see PRD §1.2 and DESIGN §1). Land four -streams in `bronze_claude_team.*`. No Silver, no Gold. +streams in `bronze_claude_team.*`. (Bronze MVP scope — Silver/Gold +landed later: `class_ai_dev_usage` per INSIGHT-458 and `class_ai_overage` +/ Gold `cc_overage` per descriptor 1.3.0; see DESIGN §4.4.) ## 2. Deliverables diff --git a/docs/components/connectors/ai/claude-team/specs/PRD.md b/docs/components/connectors/ai/claude-team/specs/PRD.md index fb9c34d31..29a2ac212 100644 --- a/docs/components/connectors/ai/claude-team/specs/PRD.md +++ b/docs/components/connectors/ai/claude-team/specs/PRD.md @@ -87,7 +87,11 @@ Proxy source code, Dockerfile, and deployment instructions: - Proxy implementation. Lives in `secure-enclave`. The Insight side treats it as an opaque HTTP endpoint. -- Silver / Gold transformations. `dbt_select: ''` in descriptor. +- Silver / Gold transformations were out of scope for the Bronze MVP + but have since landed (`dbt_select: 'tag:claude-team+'`): Silver + `claude_team__ai_dev_usage` → `class_ai_dev_usage` (INSIGHT-458) and + `claude_team__ai_overage` → `class_ai_overage` (descriptor 1.3.0, + Gold bullet `cc_overage`). See DESIGN §4.4. - Real-time / streaming sync. Daily cron only. - Multi-org. One connector instance per claude.ai org. To serve multiple orgs, deploy multiple proxy containers (one per org) and diff --git a/src/backend/services/analytics-api/src/migration/m20260610_000001_fix_ai_label_drift.rs b/src/backend/services/analytics-api/src/migration/m20260610_000001_fix_ai_label_drift.rs index fa807f432..f8b4b5457 100644 --- a/src/backend/services/analytics-api/src/migration/m20260610_000001_fix_ai_label_drift.rs +++ b/src/backend/services/analytics-api/src/migration/m20260610_000001_fix_ai_label_drift.rs @@ -107,8 +107,10 @@ mod tests { // asserts every metric_key the gold view emits is classified into exactly // one bucket — so adding a connector key without classifying it fails CI. // ===================================================================== - /// `ai_person_period` sum-branch (counters). Mirrors the multiIf in - /// 20260610000000_ai-person-period-rollup-fix.sql — keep in sync. + /// `ai_person_period` sum-branch (counters). Mirrors the multiIf in the + /// latest period-rollup migration (20260618000000_ai-claude-team-overage-gold.sql, + /// which re-set the view to add `cc_overage`; previously 20260610000000) — + /// keep in sync. const SUM_KEYS: &[&str] = &[ "chatgpt", "cc_lines", @@ -123,6 +125,7 @@ mod tests { "cc_offered", "cc_tool_accept", "cc_cost", + "cc_overage", "prs_total", "prs_with_cc", "cursor_offered", @@ -138,13 +141,17 @@ mod tests { ]; /// The `metric_keys` actually EMITTED into `insight.ai_bullet_rows` by the gold - /// view 20260609000000 (its ARRAY JOIN branches) — these are the only keys - /// that reach `ai_person_period` and must therefore be classified. NB this is - /// the GOLD key set, NOT the `query_ref` ARRAY JOIN: the latter also lists - /// query_ref-computed ratios (`cursor_acceptance`, `cc_tool_acceptance`, - /// `ai_loc_share2`) and the `claude_web` stub, which are never emitted to - /// `ai_bullet_rows` and so never hit the period rollup. prs_* were removed - /// (honest-NULL) so they are absent here too. + /// view (its ARRAY JOIN branches; latest = 20260618000000, branches 1–6) — + /// these are the only keys that reach `ai_person_period` and must therefore be + /// classified. NB this is the GOLD key set, NOT the `query_ref` ARRAY JOIN: the + /// latter also lists query_ref-computed ratios (`cursor_acceptance`, + /// `cc_tool_acceptance`, `ai_loc_share2`) and the `claude_web` stub, which are + /// never emitted to `ai_bullet_rows` and so never hit the period rollup. prs_* + /// were removed (honest-NULL) so they are absent here too. + /// + /// ⚠️ This list is hand-maintained (mirrors the gold view's branches). A new + /// gold branch key MUST be added here AND classified in `SUM_KEYS/MAX_KEYS`, or + /// the guard gives false-green (it only checks keys present in this list). const BULLET_ROWS_KEYS: &[&str] = &[ // branch 1 (all dev tools) "active_ai_members", @@ -170,6 +177,8 @@ mod tests { // branch 5 (chatgpt chat) "chatgpt_active", "chatgpt", + // branch 6 (claude overage) + "cc_overage", ]; /// Every key the gold view emits must be classified into EXACTLY one of @@ -199,4 +208,19 @@ mod tests { assert!(!SUM_KEYS.contains(&"chatgpt_active")); assert!(!MAX_KEYS.contains(&"codex_lines")); } + + /// `cc_overage` is a per-period spend counter (twin of `cc_cost`) → sum, never + /// max/avg. Avg would divide a monthly snapshot by active-day count (#1286). + #[test] + fn cc_overage_sums_like_cc_cost() { + assert!(SUM_KEYS.contains(&"cc_overage"), "cc_overage must sum"); + assert!( + !MAX_KEYS.contains(&"cc_overage"), + "cc_overage is not an active flag" + ); + assert!( + BULLET_ROWS_KEYS.contains(&"cc_overage"), + "cc_overage must be listed as an emitted gold key" + ); + } } diff --git a/src/backend/services/analytics-api/src/migration/m20260618_000001_ai_claude_team_overage_metric.rs b/src/backend/services/analytics-api/src/migration/m20260618_000001_ai_claude_team_overage_metric.rs new file mode 100644 index 000000000..46b83bc9c --- /dev/null +++ b/src/backend/services/analytics-api/src/migration/m20260618_000001_ai_claude_team_overage_metric.rs @@ -0,0 +1,316 @@ +//! Add Claude Team `cc_overage` to Team / IC Bullet AI `query_ref`s +//! (per-seat spend over the monthly credit limit). +//! +//! Pairs with ingestion migration +//! `20260618000000_ai-claude-team-overage-gold.sql`, which adds Branch 6 +//! (`metric_key = 'cc_overage'`) to `insight.ai_bullet_rows`, emitting +//! `overage_cents` from `silver.class_ai_overage` (source='`claude_team`'). +//! +//! Changes to each `query_ref` (Team + IC), extending the m20260609 head: +//! 1. New `cc_overage_v` in the wide-aggregate — honest-NULL guarded +//! (`if(countIf(key) > 0, sumIf(value), NULL)`), like the prs metrics: +//! a seat with no overage reading renders `ComingSoon`, while a seat +//! within its limit emits a real `0` (the gold view emits 0 only when +//! a limit is known). +//! 2. New `('cc_overage', cc_overage_v)` entry in the ARRAY JOIN unpivot. +//! +//! `cc_overage` is a plain counter (cents over limit), NOT an active-marker — +//! it is NOT added to `ACTIVE_LIST`, so the outer dispatch aggregates it with +//! `avg(v_period)` (average per-person overage over the period), identical to +//! `cc_cost`. +//! +//! Backend-emitted `metric_key`s: 22 → 23. +//! Catalog metadata (label / unit '¢' / lower-is-better / thresholds) is +//! seeded by the paired migration +//! `m20260618_000002_seed_claude_team_overage_catalog`. FE renders it once a +//! `bullet-layout-groups` entry is added in `cyber-insight-front`. + +use sea_orm_migration::prelude::*; + +#[derive(DeriveMigrationName)] +pub struct Migration; + +const TEAM_BULLET_AI_ID: &str = "00000000000000000001000000000006"; +const IC_BULLET_AI_ID: &str = "00000000000000000001000000000013"; + +/// Active-marker `metric_key`s — unchanged from m20260609. `cc_overage` is a +/// counter, NOT an active-marker, so it is deliberately absent here. +const ACTIVE_LIST: &str = + "'active_ai_members', 'cursor_active', 'cc_active', 'codex_active', 'chatgpt_active'"; + +/// Inner wide-aggregate: one row per `person_id`, every FE-visible +/// `metric_key` in its own column. Extends m20260609 with `cc_overage_v`. +fn wide_aggregate_pp() -> &'static str { + "SELECT person_id, any(org_unit_id) AS org_unit_id, \ + if(countIf(metric_key = 'active_ai_members') > 0, toFloat64(1), CAST(NULL AS Nullable(Float64))) AS active_ai_members_v, \ + if(countIf(metric_key = 'cursor_active') > 0, toFloat64(1), CAST(NULL AS Nullable(Float64))) AS cursor_active_v, \ + if(countIf(metric_key = 'cc_active') > 0, toFloat64(1), CAST(NULL AS Nullable(Float64))) AS cc_active_v, \ + sumIf(metric_value, metric_key = 'cursor_completions') AS cursor_completions_v, \ + sumIf(metric_value, metric_key = 'cursor_agents') AS cursor_agents_v, \ + sumIf(metric_value, metric_key = 'cursor_lines') AS cursor_lines_v, \ + sumIf(metric_value, metric_key = 'cc_sessions') AS cc_sessions_v, \ + sumIf(metric_value, metric_key = 'cc_lines') AS cc_lines_v, \ + sumIf(metric_value, metric_key = 'cc_tool_accept') AS cc_tool_accept_v, \ + sumIf(metric_value, metric_key = 'team_ai_loc') AS team_ai_loc_v, \ + sumIf(metric_value, metric_key = 'cc_cost') AS cc_cost_v, \ + if(countIf(metric_key = 'prs_with_cc') > 0, sumIf(metric_value, metric_key = 'prs_with_cc'), CAST(NULL AS Nullable(Float64))) AS prs_with_cc_v, \ + if(countIf(metric_key = 'prs_total') > 0, sumIf(metric_value, metric_key = 'prs_total'), CAST(NULL AS Nullable(Float64))) AS prs_total_v, \ + if(sumIf(metric_value, metric_key = 'cursor_offered') > 0, \ + round(toFloat64(100) \ + * sumIf(metric_value, metric_key = 'cursor_completions') \ + / sumIf(metric_value, metric_key = 'cursor_offered'), 1), \ + CAST(NULL AS Nullable(Float64))) AS cursor_acceptance_v, \ + if(sumIf(metric_value, metric_key = 'cc_offered') > 0, \ + round(toFloat64(100) \ + * sumIf(metric_value, metric_key = 'cc_tool_accept') \ + / sumIf(metric_value, metric_key = 'cc_offered'), 1), \ + CAST(NULL AS Nullable(Float64))) AS cc_tool_acceptance_v, \ + if(sumIf(metric_value, metric_key = 'cursor_total_lines') > 0, \ + round(toFloat64(100) \ + * sumIf(metric_value, metric_key = 'cursor_lines') \ + / sumIf(metric_value, metric_key = 'cursor_total_lines'), 1), \ + CAST(NULL AS Nullable(Float64))) AS ai_loc_share2_v, \ + if(countIf(metric_key = 'codex_active') > 0, toFloat64(1), CAST(NULL AS Nullable(Float64))) AS codex_active_v, \ + sumIf(metric_value, metric_key = 'chatgpt') AS chatgpt_v, \ + CAST(NULL AS Nullable(Float64)) AS claude_web_v, \ + sumIf(metric_value, metric_key = 'codex_lines') AS codex_lines_v, \ + sumIf(metric_value, metric_key = 'codex_sessions') AS codex_sessions_v, \ + if(countIf(metric_key = 'chatgpt_active') > 0, toFloat64(1), CAST(NULL AS Nullable(Float64))) AS chatgpt_active_v, \ + if(countIf(metric_key = 'cc_overage') > 0, sumIf(metric_value, metric_key = 'cc_overage'), CAST(NULL AS Nullable(Float64))) AS cc_overage_v \ + FROM insight.ai_bullet_rows \ + GROUP BY person_id" +} + +/// `ARRAY JOIN` unpivot: wide columns → long rows per person. +/// 22 keys from m20260609 + `cc_overage` = 23 total. +fn array_join_kv() -> &'static str { + "ARRAY JOIN [ \ + ('active_ai_members', active_ai_members_v), \ + ('cursor_active', cursor_active_v), \ + ('cc_active', cc_active_v), \ + ('cursor_completions', cursor_completions_v), \ + ('cursor_agents', cursor_agents_v), \ + ('cursor_lines', cursor_lines_v), \ + ('cc_sessions', cc_sessions_v), \ + ('cc_lines', cc_lines_v), \ + ('cc_tool_accept', cc_tool_accept_v), \ + ('team_ai_loc', team_ai_loc_v), \ + ('cc_cost', cc_cost_v), \ + ('prs_with_cc', prs_with_cc_v), \ + ('prs_total', prs_total_v), \ + ('cursor_acceptance', cursor_acceptance_v), \ + ('cc_tool_acceptance', cc_tool_acceptance_v), \ + ('ai_loc_share2', ai_loc_share2_v), \ + ('codex_active', codex_active_v), \ + ('chatgpt', chatgpt_v), \ + ('claude_web', claude_web_v), \ + ('codex_lines', codex_lines_v), \ + ('codex_sessions', codex_sessions_v), \ + ('chatgpt_active', chatgpt_active_v), \ + ('cc_overage', cc_overage_v) \ + ] AS kv" +} + +fn team_query() -> String { + let pp = wide_aggregate_pp(); + let kv = array_join_kv(); + format!( + "SELECT p.metric_key AS metric_key, \ + multiIf(p.metric_key IN ({ACTIVE_LIST}), sum(p.v_period), avg(p.v_period)) AS value, \ + any(c.company_median) AS median, \ + any(c.company_min) AS range_min, \ + any(c.company_max) AS range_max \ + FROM ( \ + SELECT person_id, org_unit_id, \ + kv.1 AS metric_key, kv.2 AS v_period \ + FROM ({pp}) pp \ + {kv} \ + ) p \ + LEFT JOIN ( \ + SELECT metric_key, \ + multiIf(metric_key IN ({ACTIVE_LIST}), \ + if(count(v_period) = 0, CAST(NULL AS Nullable(Float64)), toFloat64(0)), \ + quantileExact(0.5)(v_period)) AS company_median, \ + multiIf(metric_key IN ({ACTIVE_LIST}), \ + if(count(v_period) = 0, CAST(NULL AS Nullable(Float64)), toFloat64(0)), \ + min(v_period)) AS company_min, \ + multiIf(metric_key IN ({ACTIVE_LIST}), \ + if(count(v_period) = 0, CAST(NULL AS Nullable(Float64)), toFloat64(count())), \ + max(v_period)) AS company_max \ + FROM ( \ + SELECT kv.1 AS metric_key, kv.2 AS v_period \ + FROM ({pp}) ppc \ + {kv} \ + ) inner_c \ + GROUP BY metric_key \ + ) c ON c.metric_key = p.metric_key \ + GROUP BY p.metric_key" + ) +} + +fn ic_query() -> String { + let pp = wide_aggregate_pp(); + let kv = array_join_kv(); + format!( + "SELECT p.metric_key AS metric_key, \ + multiIf(p.metric_key IN ({ACTIVE_LIST}), sum(p.v_period), avg(p.v_period)) AS value, \ + any(c.team_median) AS median, \ + any(c.team_min) AS range_min, \ + any(c.team_max) AS range_max \ + FROM ( \ + SELECT person_id, org_unit_id, \ + kv.1 AS metric_key, kv.2 AS v_period \ + FROM ({pp}) pp \ + {kv} \ + ) p \ + LEFT JOIN ( \ + SELECT metric_key, org_unit_id, \ + multiIf(metric_key IN ({ACTIVE_LIST}), \ + if(count(v_period) = 0, CAST(NULL AS Nullable(Float64)), toFloat64(0)), \ + quantileExact(0.5)(v_period)) AS team_median, \ + multiIf(metric_key IN ({ACTIVE_LIST}), \ + if(count(v_period) = 0, CAST(NULL AS Nullable(Float64)), toFloat64(0)), \ + min(v_period)) AS team_min, \ + multiIf(metric_key IN ({ACTIVE_LIST}), \ + if(count(v_period) = 0, CAST(NULL AS Nullable(Float64)), toFloat64(count())), \ + max(v_period)) AS team_max \ + FROM ( \ + SELECT person_id, org_unit_id, \ + kv.1 AS metric_key, kv.2 AS v_period \ + FROM ({pp}) ppc \ + {kv} \ + ) inner_c \ + GROUP BY metric_key, org_unit_id \ + ) c ON c.metric_key = p.metric_key AND c.org_unit_id = p.org_unit_id \ + GROUP BY p.metric_key" + ) +} + +#[async_trait::async_trait] +impl MigrationTrait for Migration { + async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> { + let db = manager.get_connection(); + for (hex_id, query) in [ + (TEAM_BULLET_AI_ID, team_query()), + (IC_BULLET_AI_ID, ic_query()), + ] { + db.execute_unprepared(&format!( + "UPDATE metrics SET query_ref = '{qr}' WHERE id = UNHEX('{hex_id}')", + qr = query.replace('\'', "''"), + )) + .await?; + } + Ok(()) + } + + /// Irreversible — roll back the paired CH migration + /// `20260618000000_ai-claude-team-overage-gold.sql` first (which removes + /// the `cc_overage` branch from the view), then restore the previous + /// `query_ref` from `m20260609_000001` manually. + async fn down(&self, _manager: &SchemaManager) -> Result<(), DbErr> { + Err(DbErr::Custom( + "m20260618_000001_ai_claude_team_overage_metric is irreversible: \ + roll back the paired CH migration \ + 20260618000000_ai-claude-team-overage-gold.sql first." + .to_string(), + )) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// All 23 FE-visible `metric_keys` must appear in the ARRAY JOIN unpivot + /// (22 from m20260609 + `cc_overage`). + const EXPECTED_METRIC_KEYS: &[&str] = &[ + "active_ai_members", + "cursor_active", + "cc_active", + "cursor_completions", + "cursor_agents", + "cursor_lines", + "cc_sessions", + "cc_lines", + "cc_tool_accept", + "team_ai_loc", + "cc_cost", + "prs_with_cc", + "prs_total", + "cursor_acceptance", + "cc_tool_acceptance", + "ai_loc_share2", + "codex_active", + "chatgpt", + "claude_web", + "codex_lines", + "codex_sessions", + "chatgpt_active", + "cc_overage", + ]; + + #[test] + fn array_join_emits_all_23_keys() { + let kv = array_join_kv(); + for key in EXPECTED_METRIC_KEYS { + assert!( + kv.contains(&format!("('{key}',")), + "ARRAY JOIN missing key {key}" + ); + } + assert_eq!( + kv.matches("('").count(), + 23, + "ARRAY JOIN must emit exactly 23 keys" + ); + assert_eq!(EXPECTED_METRIC_KEYS.len(), 23); + } + + /// `cc_overage` is a cost counter → must NOT be in `ACTIVE_LIST` + /// (it is averaged per person, not counted as a DAU marker). + #[test] + fn cc_overage_is_not_active_marker() { + assert!( + !ACTIVE_LIST.contains("'cc_overage'"), + "cc_overage is a counter, not an active marker" + ); + } + + /// `cc_overage` must be honest-NULL guarded (countIf > 0 … else NULL), + /// like the prs metrics: no reading → `ComingSoon`, never a fake 0. + #[test] + fn cc_overage_is_honest_null_guarded() { + let pp = wide_aggregate_pp(); + // countIf guard present, followed by the sumIf and the NULL-as alias. + assert!( + pp.contains("if(countIf(metric_key = 'cc_overage') > 0"), + "cc_overage_v must be countIf-guarded (honest-NULL), not a bare sumIf" + ); + assert!( + pp.contains("sumIf(metric_value, metric_key = 'cc_overage')"), + "cc_overage_v must read its value via sumIf" + ); + assert!( + pp.contains("AS cc_overage_v"), + "cc_overage_v alias must be present in the wide-aggregate" + ); + } + + /// Both `query_refs` must embed the new key end-to-end. + #[test] + fn queries_reference_cc_overage() { + for q in [team_query(), ic_query()] { + assert!(q.contains("cc_overage"), "query missing cc_overage"); + assert!(q.contains("ARRAY JOIN"), "query missing ARRAY JOIN unpivot"); + } + } + + /// Sanity: the m20260609 keys are still present (no regression). + #[test] + fn prior_keys_preserved() { + let kv = array_join_kv(); + for key in ["cc_cost", "codex_lines", "chatgpt_active", "team_ai_loc"] { + assert!(kv.contains(&format!("('{key}',")), "regression: lost {key}"); + } + } +} diff --git a/src/backend/services/analytics-api/src/migration/m20260618_000002_seed_claude_team_overage_catalog.rs b/src/backend/services/analytics-api/src/migration/m20260618_000002_seed_claude_team_overage_catalog.rs new file mode 100644 index 000000000..015e15297 --- /dev/null +++ b/src/backend/services/analytics-api/src/migration/m20260618_000002_seed_claude_team_overage_catalog.rs @@ -0,0 +1,231 @@ +//! Seed `metric_catalog` + `product-default` `metric_threshold` for the +//! Claude Team `cc_overage` `metric_key` introduced in +//! `m20260618_000001_ai_claude_team_overage_metric`. +//! +//! `cc_overage` — per-seat spend in cents ABOVE the monthly credit limit +//! (max(0, used − limit)) from the Claude Team plan. +//! `higher_is_better = false` (overage is a cost/risk signal: +//! any spend over the limit is undesirable). +//! +//! Threshold placeholders (cents, "lower is better"): +//! `good ≤ 0¢` — the seat is within its monthly limit (no overage). +//! `warn ≤ 5000¢` — up to $50 over the limit. Above that → bad. +//! Tune per-tenant via the admin CRUD API (#525). +//! +//! ⚠️ FE visibility: this catalog row registers label / unit / threshold +//! metadata. The metric renders once a `bullet-layout-groups` entry for +//! `cc_overage` is added in `cyber-insight-front` (paired FE change). + +use sea_orm::{ConnectionTrait, Statement, Value}; +use sea_orm_migration::prelude::*; +use uuid::Uuid; + +#[derive(DeriveMigrationName)] +pub struct Migration; + +struct SeedRow { + metric_key: &'static str, + label: &'static str, + sublabel: Option<&'static str>, + description: Option<&'static str>, + unit: Option<&'static str>, + format: Option<&'static str>, + higher_is_better: bool, + is_member_scale: bool, + source_tags: &'static [&'static str], + good: f64, + warn: f64, +} + +fn source_tags_json(tags: &[&'static str]) -> String { + let mut out = String::from("["); + for (i, tag) in tags.iter().enumerate() { + if i > 0 { + out.push(','); + } + out.push('"'); + for c in tag.chars() { + match c { + '"' => out.push_str("\\\""), + '\\' => out.push_str("\\\\"), + _ => out.push(c), + } + } + out.push('"'); + } + out.push(']'); + out +} + +const SEEDS: &[SeedRow] = &[SeedRow { + metric_key: "ai_bullet_rows.cc_overage", + label: "Claude Overage", + sublabel: Some("Claude Team \u{b7} spend over monthly limit \u{b7} cents \u{b7} period total"), + description: Some( + "Per-seat spend in cents ABOVE the monthly Claude Team credit limit \ + (max(0, used_credits \u{2212} monthly_credit_limit)). Sourced from \ + class_ai_overage (/overage_spend_limits). 0 means the seat is within \ + its limit; ComingSoon means no computable overage (unknown limit).", + ), + unit: Some("\u{a2}"), + format: None, + higher_is_better: false, + is_member_scale: false, + source_tags: &["claude_team"], + // good ≤ 0¢ (within limit), warn ≤ 5000¢ ($50 over). Lower-is-better. + good: 0.0, + warn: 5000.0, +}]; + +const INSERT_CATALOG_SQL: &str = "\ + INSERT INTO metric_catalog \ + (id, tenant_id, metric_key, label, sublabel, description, unit, format, \ + higher_is_better, is_member_scale, source_tags, is_enabled) \ + VALUES (?, NULL, ?, ?, ?, ?, ?, ?, ?, ?, ?, TRUE) \ + ON DUPLICATE KEY UPDATE \ + label = VALUES(label), \ + sublabel = VALUES(sublabel), \ + description = VALUES(description), \ + unit = VALUES(unit), \ + format = VALUES(format), \ + higher_is_better = VALUES(higher_is_better), \ + is_member_scale = VALUES(is_member_scale), \ + source_tags = VALUES(source_tags), \ + is_enabled = VALUES(is_enabled)"; + +const INSERT_THRESHOLD_SQL: &str = "\ + INSERT INTO metric_threshold \ + (id, tenant_id, metric_key, scope, role_slug, team_id, good, warn, is_locked) \ + VALUES (?, NULL, ?, 'product-default', '', '', ?, ?, FALSE) \ + ON DUPLICATE KEY UPDATE \ + good = VALUES(good), \ + warn = VALUES(warn)"; + +fn nullable_str_value(v: Option<&str>) -> Value { + match v { + Some(s) => Value::from(s), + None => Value::String(None), + } +} + +#[async_trait::async_trait] +impl MigrationTrait for Migration { + async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> { + let conn = manager.get_connection(); + let backend = manager.get_database_backend(); + + for row in SEEDS { + let catalog_id = Uuid::now_v7(); + let threshold_id = Uuid::now_v7(); + let source_tags_json_str = source_tags_json(row.source_tags); + + conn.execute(Statement::from_sql_and_values( + backend, + INSERT_CATALOG_SQL, + [ + Value::Bytes(Some(Box::new(catalog_id.as_bytes().to_vec()))), + Value::from(row.metric_key), + Value::from(row.label), + nullable_str_value(row.sublabel), + nullable_str_value(row.description), + nullable_str_value(row.unit), + nullable_str_value(row.format), + Value::from(row.higher_is_better), + Value::from(row.is_member_scale), + Value::from(source_tags_json_str.as_str()), + ], + )) + .await?; + + conn.execute(Statement::from_sql_and_values( + backend, + INSERT_THRESHOLD_SQL, + [ + Value::Bytes(Some(Box::new(threshold_id.as_bytes().to_vec()))), + Value::from(row.metric_key), + Value::from(row.good), + Value::from(row.warn), + ], + )) + .await?; + } + + tracing::info!( + seeded = SEEDS.len(), + "claude_team overage metric_catalog seed migration applied" + ); + + Ok(()) + } + + async fn down(&self, _manager: &SchemaManager) -> Result<(), DbErr> { + Err(DbErr::Custom( + "m20260618_000002_seed_claude_team_overage_catalog is irreversible: \ + delete the cc_overage catalog row manually if needed." + .to_string(), + )) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn seed_count_is_one() { + assert_eq!( + SEEDS.len(), + 1, + "expected exactly 1 new catalog row (cc_overage)" + ); + } + + #[test] + fn key_routes_to_ai_bullet_rows() { + assert!( + SEEDS[0].metric_key.starts_with("ai_bullet_rows."), + "metric_key {:?} does not route to ai_bullet_rows", + SEEDS[0].metric_key + ); + assert_eq!(SEEDS[0].metric_key, "ai_bullet_rows.cc_overage"); + } + + /// Overage is a cost/risk signal — `higher_is_better` must be false. + #[test] + fn overage_is_lower_is_better() { + assert!( + !SEEDS[0].higher_is_better, + "cc_overage is overage spend — higher_is_better must be false" + ); + } + + /// Unit must be cents (matches `cc_cost` and the gold view's cents values). + #[test] + fn unit_is_cents() { + assert_eq!(SEEDS[0].unit, Some("\u{a2}"), "cc_overage unit must be ¢"); + } + + #[test] + fn tagged_claude_team() { + assert_eq!(SEEDS[0].source_tags, &["claude_team"]); + } + + /// good (within limit) must be ≤ warn (some overage tolerated) for a + /// lower-is-better metric. + #[test] + fn thresholds_ordered_for_lower_is_better() { + assert!( + SEEDS[0].good <= SEEDS[0].warn, + "lower-is-better: good ({}) must be ≤ warn ({})", + SEEDS[0].good, + SEEDS[0].warn + ); + } + + #[test] + fn source_tags_json_is_well_formed() { + let json = source_tags_json(SEEDS[0].source_tags); + assert!(json.starts_with('[') && json.ends_with(']')); + assert!(json.len() > 2); + } +} diff --git a/src/backend/services/analytics-api/src/migration/mod.rs b/src/backend/services/analytics-api/src/migration/mod.rs index 6129ec20d..f07e7fa2c 100644 --- a/src/backend/services/analytics-api/src/migration/mod.rs +++ b/src/backend/services/analytics-api/src/migration/mod.rs @@ -27,6 +27,8 @@ mod m20260610_000001_fix_ai_label_drift; mod m20260612_000001_support_metrics; mod m20260612_000002_seed_support_catalog; mod m20260612_000003_link_support_query_catalog; +mod m20260618_000001_ai_claude_team_overage_metric; +mod m20260618_000002_seed_claude_team_overage_catalog; #[cfg(test)] mod live_tests; @@ -66,6 +68,8 @@ impl MigratorTrait for Migrator { Box::new(m20260612_000001_support_metrics::Migration), Box::new(m20260612_000002_seed_support_catalog::Migration), Box::new(m20260612_000003_link_support_query_catalog::Migration), + Box::new(m20260618_000001_ai_claude_team_overage_metric::Migration), + Box::new(m20260618_000002_seed_claude_team_overage_catalog::Migration), ] } } diff --git a/src/ingestion/connectors/ai/claude-team/connector.yaml b/src/ingestion/connectors/ai/claude-team/connector.yaml index d37a89163..cee85b7e0 100644 --- a/src/ingestion/connectors/ai/claude-team/connector.yaml +++ b/src/ingestion/connectors/ai/claude-team/connector.yaml @@ -187,6 +187,21 @@ streams: is_enabled: { type: boolean } monthly_credit_limit: { type: [number, "null"] } used_credits: { type: number } + # Monetary unit of monthly_credit_limit / used_credits. Confirmed + # 'USD' on live data; the numbers are minor units (cents, + # decimal_places=2 per the sibling /usage extra_usage payload), + # e.g. monthly_credit_limit=10000 ⇒ $100.00. + currency: { type: [string, "null"] } + # True once a seat has exhausted its monthly credit limit (hard + # block). Distinct from used_credits > monthly_credit_limit, which + # only flags soft overage. Often null. + out_of_credits: { type: [boolean, "null"] } + # Whether used_credits is measured before or after discount + # (e.g. 'post_discount'). + used_credits_basis: { type: [string, "null"] } + # 'seat_tier' = limit inherited from the plan tier default; + # 'member' = per-member override. + limit_type: { type: [string, "null"] } retriever: type: SimpleRetriever requester: diff --git a/src/ingestion/connectors/ai/claude-team/dbt/claude_team__ai_overage.sql b/src/ingestion/connectors/ai/claude-team/dbt/claude_team__ai_overage.sql new file mode 100644 index 000000000..75b840e21 --- /dev/null +++ b/src/ingestion/connectors/ai/claude-team/dbt/claude_team__ai_overage.sql @@ -0,0 +1,124 @@ +-- depends_on: {{ ref('claude_team__bronze_promoted') }} +-- Bronze → Silver: Claude Team per-seat credit spend vs limit → class_ai_overage +-- +-- Source: bronze_claude_team.claude_team_overage_spend — the per-seat +-- spend-state snapshot pulled via the customer-deployed claude-team-proxy +-- from the claude.ai web API (/api/organizations/{org}/overage_spend_limits). +-- One row per seat (account_uuid). Requires the proxy sessionKey to hold +-- `billing:view` / Owner role — otherwise the Bronze stream is empty +-- (HTTP 403 IGNOREd upstream) and this model yields zero rows (sync GREEN). +-- +-- This is the FIRST contributor to the class_ai_overage Silver class and +-- therefore DEFINES its 19-column positional contract (consumed by +-- `union_by_tag('silver:class_ai_overage')`). Any future source (OpenAI, +-- etc.) MUST emit these columns in this exact order — vendor-specific +-- fields go into overage_metrics_json, never new columns. +-- +-- UNITS — CRITICAL: unlike claude_team__ai_dev_usage.cost_cents (which casts +-- a decimal-as-string dollar amount × 100), `used_credits` and +-- `monthly_credit_limit` here are ALREADY in minor units (cents, USD, +-- decimal_places=2): monthly_credit_limit=10000 ⇒ $100.00, used_credits=699 +-- ⇒ $6.99. So credit_limit_cents / used_amount_cents map straight through +-- with NO ×100. Verified live (149 seats, currency='USD'). +-- +-- GRAIN: the endpoint reports current-billing-period-to-date spend with no +-- explicit period field. We stamp period_month = start-of-month of the +-- snapshot's extraction time and keep the LATEST snapshot per (seat, month). +-- As months roll over this accrues a monthly history; within the current +-- month it always reflects the freshest snapshot. unique_key carries the +-- month so a new month never overwrites a prior month's closing value. +{{ config( + materialized='incremental', + incremental_strategy='append', + unique_key='unique_key', + engine='ReplacingMergeTree(_version)', + order_by=['unique_key'], + on_schema_change='append_new_columns', + settings={'allow_nullable_key': 1}, + schema='staging', + tags=['claude-team', 'silver:class_ai_overage'] +) }} + +WITH latest_per_seat_month AS ( + -- Bronze is full-refresh+append: every sync re-emits all seats with the + -- same unique_key (no date). Collapse to the latest snapshot per seat per + -- calendar month so each (seat, month) keeps its freshest spend reading. + SELECT * + FROM {{ source('bronze_claude_team', 'claude_team_overage_spend') }} + WHERE account_uuid IS NOT NULL + AND trim(account_uuid) != '' + AND account_email IS NOT NULL + AND trim(account_email) != '' + ORDER BY _airbyte_extracted_at DESC + -- Dedup on the FULL grain (tenant + source + seat + month), not just + -- account_uuid: a multi-tenant / multi-instance bronze_claude_team can hold + -- the same account_uuid under different tenant_id/source_id in one month, + -- and keying on account_uuid alone would drop those as false duplicates. + LIMIT 1 BY tenant_id, source_id, account_uuid, toStartOfMonth(_airbyte_extracted_at) +) + +SELECT + tenant_id AS insight_tenant_id, + source_id, + -- Silver dedup key: tenant-source-seat-month. The month component (absent + -- from the Bronze unique_key) preserves monthly history and gives intra- + -- month idempotency (latest snapshot wins via _version, same key). + CAST(concat( + coalesce(tenant_id, ''), '-', + coalesce(source_id, ''), '-', + coalesce(account_uuid, ''), '-', + formatDateTime(toStartOfMonth(_airbyte_extracted_at), '%Y-%m') + ) AS String) AS unique_key, + -- Per-seat identity. Claude Team seats always carry an email; account_uuid + -- is the stable vendor id (identity proxy / join anchor). + lower(trim(account_email)) AS email, + account_uuid AS account_id, + toStartOfMonth(_airbyte_extracted_at) AS period_month, + 'claude' AS tool, + seat_tier, + coalesce(currency, 'USD') AS currency, + -- Already cents (USD minor units) — NO ×100. NULL when no limit applies + -- (e.g. unassigned seats with limit_type NULL). round() guards against a + -- float repr ('10000.0') that toUInt32OrNull would otherwise reject. + toUInt32OrNull(toString(round(monthly_credit_limit))) AS credit_limit_cents, + toUInt32(round(coalesce(used_credits, 0))) AS used_amount_cents, + -- Overage = spend beyond the limit. NULL (not 0) when the limit is unknown + -- — honest-NULL: we cannot compute overage without a limit. + multiIf( + monthly_credit_limit IS NULL, CAST(NULL AS Nullable(UInt32)), + CAST(greatest(0, toInt64(round(coalesce(used_credits, 0))) + - toInt64(round(monthly_credit_limit))) AS Nullable(UInt32)) + ) AS overage_cents, + -- Soft over-limit flag (used > limit). NULL when limit unknown. Distinct + -- from out_of_credits (hard exhaustion), which lives in the JSON blob. + multiIf( + monthly_credit_limit IS NULL, CAST(NULL AS Nullable(UInt8)), + toUInt8(coalesce(used_credits, 0) > monthly_credit_limit) + ) AS is_over_limit, + -- Bronze may store the JSON boolean as Bool, UInt8, or the strings + -- 'true'/'false' depending on destination typing — normalise all forms. + multiIf( + lower(toString(is_enabled)) IN ('true', '1'), toUInt8(1), + lower(toString(is_enabled)) IN ('false', '0'), toUInt8(0), + CAST(NULL AS Nullable(UInt8)) + ) AS is_enabled, + -- Vendor-specific extras kept out of the positional contract. + toJSONString(map( + 'limit_type', ifNull(toString(limit_type), ''), + 'used_credits_basis', ifNull(toString(used_credits_basis), ''), + 'out_of_credits', ifNull(toString(out_of_credits), ''), + 'seat_tier', ifNull(toString(seat_tier), '') + )) AS overage_metrics_json, + 'claude_team' AS source, + data_source, + CAST(_airbyte_extracted_at AS Nullable(DateTime64(3))) AS collected_at, + toUnixTimestamp64Milli(_airbyte_extracted_at) AS _version +FROM latest_per_seat_month +{% if is_incremental() %} + -- Re-evaluate the current and previous month so an in-flight month's + -- closing value keeps updating; older months are immutable. + WHERE toStartOfMonth(_airbyte_extracted_at) >= ( + SELECT coalesce(max(period_month), toDate('1970-01-01')) - INTERVAL 1 MONTH + FROM {{ this }} + ) +{% endif %} diff --git a/src/ingestion/connectors/ai/claude-team/dbt/claude_team__bronze_promoted.sql b/src/ingestion/connectors/ai/claude-team/dbt/claude_team__bronze_promoted.sql index 47fc21652..47b67020f 100644 --- a/src/ingestion/connectors/ai/claude-team/dbt/claude_team__bronze_promoted.sql +++ b/src/ingestion/connectors/ai/claude-team/dbt/claude_team__bronze_promoted.sql @@ -13,7 +13,8 @@ tags=['claude-team'] ) }} -{% do promote_bronze_to_rmt(table='bronze_claude_team.claude_team_members', order_by='unique_key') %} -{% do promote_bronze_to_rmt(table='bronze_claude_team.claude_team_code_metrics', order_by='unique_key') %} +{% do promote_bronze_to_rmt(table='bronze_claude_team.claude_team_members', order_by='unique_key') %} +{% do promote_bronze_to_rmt(table='bronze_claude_team.claude_team_code_metrics', order_by='unique_key') %} +{% do promote_bronze_to_rmt(table='bronze_claude_team.claude_team_overage_spend', order_by='unique_key') %} SELECT 1 AS promoted diff --git a/src/ingestion/connectors/ai/claude-team/dbt/schema.yml b/src/ingestion/connectors/ai/claude-team/dbt/schema.yml index 552957288..eda7d107f 100644 --- a/src/ingestion/connectors/ai/claude-team/dbt/schema.yml +++ b/src/ingestion/connectors/ai/claude-team/dbt/schema.yml @@ -20,6 +20,7 @@ sources: freshness: warn_after: {count: 72, period: hour} error_after: {count: 120, period: hour} + - name: claude_team_overage_spend # per-seat spend snapshot — sync-liveness default models: - name: claude_team__ai_dev_usage @@ -127,3 +128,90 @@ models: description: "ReplacingMergeTree version — toUnixTimestamp64Milli(_airbyte_extracted_at) per ADR-0001." tests: - not_null + + - name: claude_team__ai_overage + description: > + Bronze → Silver: Claude Team per-seat credit spend vs limit → + class_ai_overage. Sourced from claude_team_overage_spend + (/api/organizations/{org}/overage_spend_limits via the claude-team-proxy; + requires billing:view / Owner on the proxy sessionKey, else empty). + First contributor to class_ai_overage — DEFINES its 19-column positional + contract. One row per (tenant, seat, billing month): latest snapshot per + seat per month. UNITS: used_credits / monthly_credit_limit are already + cents (USD) — mapped straight to *_cents with NO ×100 (unlike + claude_team__ai_dev_usage.cost_cents). + columns: + - name: insight_tenant_id + tests: + - not_null + - name: source_id + tests: + - not_null + - name: unique_key + description: "Composite dedup key: tenant-source-account_uuid-YYYY-MM" + tests: + - not_null + - unique + - name: email + description: "Seat work email (lowercased) — identity key" + tests: + - not_null + - name: account_id + description: "claude.ai account_uuid — stable vendor seat id / join anchor" + tests: + - not_null + - name: period_month + description: "First day of the billing month the snapshot was observed in (UTC)" + tests: + - not_null + - name: tool + description: "Always 'claude'" + tests: + - not_null + - accepted_values: + arguments: + values: ['claude'] + - name: seat_tier + description: "Vendor plan tier (team_standard / team_tier_1 / unassigned / …)" + - name: currency + description: "ISO currency of the *_cents amounts — 'USD' for Claude Team (Anthropic bills Team plans in USD)" + tests: + - not_null + - accepted_values: + arguments: + values: ['USD'] + - name: credit_limit_cents + description: > + Monthly credit limit in cents (monthly_credit_limit, already minor + units). NULL when no limit applies (unassigned seats). + - name: used_amount_cents + description: "Credits spent this period in cents (used_credits, already minor units)" + tests: + - not_null + - name: overage_cents + description: "max(0, used_amount_cents - credit_limit_cents). NULL when limit unknown (honest-NULL)" + - name: is_over_limit + description: "1 when used > limit (soft overage). NULL when limit unknown" + - name: is_enabled + description: "Whether overage/extra-usage is enabled for the seat" + - name: overage_metrics_json + description: "Raw vendor extras: limit_type, used_credits_basis, out_of_credits, seat_tier" + - name: source + description: "Always 'claude_team'" + tests: + - not_null + - accepted_values: + arguments: + values: ['claude_team'] + - name: data_source + description: "Passed through from bronze (always 'insight_claude_team')" + tests: + - not_null + - name: collected_at + description: "_airbyte_extracted_at" + tests: + - not_null + - name: _version + description: "ReplacingMergeTree version — toUnixTimestamp64Milli(_airbyte_extracted_at)" + tests: + - not_null diff --git a/src/ingestion/connectors/ai/claude-team/descriptor.yaml b/src/ingestion/connectors/ai/claude-team/descriptor.yaml index a1fa89ca2..ff1dcf85f 100644 --- a/src/ingestion/connectors/ai/claude-team/descriptor.yaml +++ b/src/ingestion/connectors/ai/claude-team/descriptor.yaml @@ -5,7 +5,13 @@ name: claude-team # (bearer token Insight uses to authenticate to the customer's proxy). # MINOR bump per ADR-0015 — Spec gains required fields; existing # deployments must add the new Secret keys before next reconcile. -version: "1.2.0" +# 1.3.0: claude_team_overage_spend now feeds Silver class_ai_overage +# (per-seat spend-vs-limit overage). Additive — Bronze stream gains +# scalar fields (currency / out_of_credits / used_credits_basis / +# limit_type) and new dbt models (claude_team__ai_overage → class_ai_overage). +# Populates only once the proxy sessionKey holds billing:view / Owner; +# otherwise the stream stays empty and the Silver rows are zero (no failure). +version: "1.3.0" # Daily at 04:00 UTC — off-peak vs other connectors. The customer's # proxy runs a single browser instance, so we avoid scheduling # claude-team next to anything else that may also hit it. diff --git a/src/ingestion/scripts/create-bronze-placeholders.sh b/src/ingestion/scripts/create-bronze-placeholders.sh index 95b2fcd55..b760c4db3 100644 --- a/src/ingestion/scripts/create-bronze-placeholders.sh +++ b/src/ingestion/scripts/create-bronze-placeholders.sh @@ -229,6 +229,38 @@ CREATE TABLE IF NOT EXISTS silver.class_ai_dev_usage ( SQL fi +# silver.class_ai_overage — per-seat AI spend-vs-limit (Claude Team). Referenced +# by the cc_overage branch of the ai_bullet_rows gold view +# (20260618000000_ai-claude-team-overage-gold.sql); without this placeholder +# CREATE VIEW fails at migration time (CH 24.x validates view source tables) in +# any env where dbt hasn't built the silver model yet. +if ! ch_table_exists silver class_ai_overage; then + echo " Creating placeholder: silver.class_ai_overage" + run_ch <<'SQL' +CREATE TABLE IF NOT EXISTS silver.class_ai_overage ( + insight_tenant_id String, + source_id String, + unique_key String, + email String, + account_id String, + period_month Date, + tool String, + seat_tier Nullable(String), + currency String, + credit_limit_cents Nullable(UInt32), + used_amount_cents UInt32, + overage_cents Nullable(UInt32), + is_over_limit Nullable(UInt8), + is_enabled Nullable(UInt8), + overage_metrics_json String, + source String, + data_source String, + collected_at DateTime, + _version UInt64 +) ENGINE = ReplacingMergeTree(_version) ORDER BY (email, period_month) COMMENT 'INSIGHT_PLACEHOLDER_v1'; +SQL +fi + # silver.class_support_activity — support (Zendesk) dbt model. Referenced by the # support_bullet_rows gold view (20260611000000_support-bullet-rows.sql); without # this placeholder CREATE VIEW fails at migration time (CH 24.x validates view diff --git a/src/ingestion/scripts/migrations/20260618000000_ai-claude-team-overage-gold.sql b/src/ingestion/scripts/migrations/20260618000000_ai-claude-team-overage-gold.sql new file mode 100644 index 000000000..6ae5470e6 --- /dev/null +++ b/src/ingestion/scripts/migrations/20260618000000_ai-claude-team-overage-gold.sql @@ -0,0 +1,189 @@ +-- ===================================================================== +-- ai_bullet_rows — Claude Team overage (spend-over-limit) extension +-- ===================================================================== +-- +-- Surfaces per-seat Claude Team overage (silver.class_ai_overage) in the +-- AI bullet Gold view. Adds Branch 6; Branches 1–5 are unchanged from +-- 20260609000000_ai-chatgpt-team-gold.sql. +-- +-- Branch 6 — Claude overage (metric_key = 'cc_overage'): +-- cc_overage ← class_ai_overage.overage_cents — cents a seat spent +-- ABOVE its monthly credit limit (max(0, used − limit)). +-- Source-scoped to source='claude_team' (the only overage source today; +-- future OpenAI overage gets its own metric_key, not this one). +-- +-- Grain note: class_ai_overage is a per-seat MONTHLY snapshot (one row +-- per seat per billing month). We date each row at toDate(collected_at) +-- — the day we last read the snapshot — NOT period_month. The snapshot +-- for the current month is therefore always stamped with a recent date, +-- so it is captured by the dashboard's rolling date window (which filters +-- on metric_date); period_month (the 1st) would fall outside short +-- windows. sumIf over a multi-month window adds each month's closing +-- overage — total overage incurred in the window. +-- +-- honest-NULL: rows are emitted ONLY where overage_cents IS NOT NULL +-- (a computable overage — the seat has a known limit). Seats with an +-- unknown limit emit no row, so the backend's countIf-guarded aggregate +-- renders them ComingSoon, never a fake $0. A seat within its limit +-- emits overage_cents = 0 (a real "no overage" reading → renders 0). +-- +-- Paired backend migration: +-- m20260618_000001_ai_claude_team_overage_metric.rs — adds cc_overage to +-- the Team / IC Bullet AI query_ref (counter, avg per person, honest-NULL +-- guarded). Catalog metadata seeded by +-- m20260618_000002_seed_claude_team_overage_catalog.rs. +-- +-- Idempotent: DROP VIEW IF EXISTS + CREATE VIEW. +-- ===================================================================== + +DROP VIEW IF EXISTS insight.ai_bullet_rows; + +CREATE VIEW insight.ai_bullet_rows AS + +-- ─── Branch 1: tool-agnostic (any dev AI tool counts) ──────────────── +SELECT + lower(c.email) AS person_id, + p.org_unit_id AS org_unit_id, + c.day AS metric_date, + kv.1 AS metric_key, + kv.2 AS metric_value +FROM silver.class_ai_dev_usage AS c +LEFT JOIN insight.people AS p ON lower(c.email) = p.person_id +ARRAY JOIN [ + ('active_ai_members', toFloat64(1)), + ('team_ai_loc', toFloat64(coalesce(c.lines_added, 0))) +] AS kv +WHERE c.email IS NOT NULL AND c.email != '' + +UNION ALL + +-- ─── Branch 2: Cursor (tool = 'cursor') ────────────────────────────── +SELECT + lower(c.email) AS person_id, + p.org_unit_id AS org_unit_id, + c.day AS metric_date, + kv.1 AS metric_key, + kv.2 AS metric_value +FROM silver.class_ai_dev_usage AS c +LEFT JOIN insight.people AS p ON lower(c.email) = p.person_id +ARRAY JOIN [ + ('cursor_active', toFloat64(1)), + ('cursor_completions', toFloat64(coalesce(c.tool_use_accepted, 0))), + ('cursor_agents', toFloat64(coalesce(c.agent_sessions, 0))), + ('cursor_lines', toFloat64(coalesce(c.lines_added, 0))), + ('cursor_offered', toFloat64(coalesce(c.tool_use_offered, 0))), + ('cursor_total_lines', toFloat64(coalesce(c.total_lines_added, 0))) +] AS kv +WHERE c.tool = 'cursor' + AND c.email IS NOT NULL AND c.email != '' + +UNION ALL + +-- ─── Branch 3: Claude Code (tool = 'claude_code') ──────────────────── +SELECT + lower(c.email) AS person_id, + p.org_unit_id AS org_unit_id, + c.day AS metric_date, + kv.1 AS metric_key, + kv.2 AS metric_value +FROM silver.class_ai_dev_usage AS c +LEFT JOIN insight.people AS p ON lower(c.email) = p.person_id +ARRAY JOIN [ + ('cc_active', toFloat64(1)), + ('cc_sessions', toFloat64(coalesce(c.session_count, 0))), + ('cc_lines', toFloat64(coalesce(c.lines_added, 0))), + ('cc_tool_accept', toFloat64(coalesce(c.tool_use_accepted, 0))), + ('cc_offered', toFloat64(coalesce(c.tool_use_offered, 0))), + ('cc_cost', toFloat64(coalesce(c.cost_cents, 0))) +] AS kv +WHERE c.tool = 'claude_code' + AND c.email IS NOT NULL AND c.email != '' + +UNION ALL + +-- ─── Branch 4: Codex (tool = 'codex', ChatGPT Team) ────────────────── +SELECT + lower(c.email) AS person_id, + p.org_unit_id AS org_unit_id, + c.day AS metric_date, + kv.1 AS metric_key, + kv.2 AS metric_value +FROM silver.class_ai_dev_usage AS c +LEFT JOIN insight.people AS p ON lower(c.email) = p.person_id +ARRAY JOIN [ + ('codex_active', toFloat64(1)), + ('codex_lines', toFloat64(coalesce(c.lines_added, 0))), + ('codex_sessions', toFloat64(coalesce(c.session_count, 0))) +] AS kv +WHERE c.tool = 'codex' + AND c.email IS NOT NULL AND c.email != '' + +UNION ALL + +-- ─── Branch 5: ChatGPT chat (tool = 'chatgpt') ─────────────────────── +-- Conversational surface — sourced from class_ai_assistant_usage. +SELECT + lower(a.email) AS person_id, + p.org_unit_id AS org_unit_id, + a.day AS metric_date, + kv.1 AS metric_key, + kv.2 AS metric_value +FROM silver.class_ai_assistant_usage AS a +LEFT JOIN insight.people AS p ON lower(a.email) = p.person_id +ARRAY JOIN [ + ('chatgpt_active', toFloat64(1)), + ('chatgpt', toFloat64(coalesce(a.message_count, 0))) +] AS kv +WHERE a.tool = 'chatgpt' + AND a.email IS NOT NULL AND a.email != '' + +UNION ALL + +-- ─── Branch 6: Claude overage (metric_key = 'cc_overage') ──────────── +-- Per-seat spend over the monthly credit limit — from class_ai_overage. +-- Dated at the snapshot's collection day (see header). Only computable +-- overages (overage_cents IS NOT NULL) are emitted. +SELECT + lower(o.email) AS person_id, + p.org_unit_id AS org_unit_id, + toDate(o.collected_at) AS metric_date, + 'cc_overage' AS metric_key, + toFloat64(o.overage_cents) AS metric_value +FROM silver.class_ai_overage AS o +LEFT JOIN insight.people AS p ON lower(o.email) = p.person_id +WHERE o.source = 'claude_team' + AND o.email IS NOT NULL AND o.email != '' + AND o.overage_cents IS NOT NULL; + +-- ===================================================================== +-- ai_person_period — classify the new cc_overage key (issue #1286 guard) +-- ===================================================================== +-- The period-rollup view (last set in 20260610000000) reclassifies each +-- bullet metric_key: counters → sum, active flags → max, ratios → avg. +-- A key absent from the multiIf silently defaults to avg() — the #1286 +-- defect. cc_overage is a per-period spend counter (twin of cc_cost, a +-- "period total" sibling), so it MUST sum, not average — otherwise a +-- monthly overage snapshot would be divided by each person's active-day +-- count. Recreated here with cc_overage added to the sum branch; the +-- Rust guard test (m20260610_000001) mirrors this list. +-- ===================================================================== +DROP VIEW IF EXISTS insight.ai_person_period; +CREATE VIEW insight.ai_person_period AS +SELECT + metric_key, + person_id, + any(org_unit_id) AS org_unit_id, + max(metric_date) AS metric_date, + multiIf( + metric_key IN ('chatgpt','cc_lines','cc_sessions','cursor_agents', + 'cursor_lines','claude_web','cursor_completions','team_ai_loc', + 'codex_lines','codex_sessions','cc_offered','cc_tool_accept', + 'cc_cost','cc_overage','prs_total','prs_with_cc', + 'cursor_offered','cursor_total_lines'), + sum(metric_value), + metric_key IN ('active_ai_members','cursor_active','cc_active','codex_active', + 'chatgpt_active'), + max(metric_value), + avg(metric_value)) AS v +FROM insight.ai_bullet_rows +GROUP BY metric_key, person_id; diff --git a/src/ingestion/silver/ai/class_ai_overage.sql b/src/ingestion/silver/ai/class_ai_overage.sql new file mode 100644 index 000000000..337529bf0 --- /dev/null +++ b/src/ingestion/silver/ai/class_ai_overage.sql @@ -0,0 +1,25 @@ +{{ config( + materialized='incremental', + incremental_strategy='delete+insert', + unique_key='unique_key', + schema='silver', + engine='ReplacingMergeTree(_version)', + order_by=['unique_key'], + settings={'allow_nullable_key': 1}, + tags=['silver'] +) }} + +-- Unified per-person per-month AI spend-vs-limit (overage) across vendors. +-- Grain: one row per (tenant, source, seat, billing month). Homogeneous +-- monetary contract in minor units (cents) + ISO currency so Claude Team and +-- (future) OpenAI / ChatGPT seats are directly comparable. overage_cents = +-- max(0, used_amount_cents - credit_limit_cents), NULL when no limit is known. +-- +-- depends_on: {{ ref('claude_team__ai_overage') }} + +SELECT * FROM ( + {{ union_by_tag('silver:class_ai_overage') }} +) +{% if is_incremental() %} +WHERE _version > (SELECT max(_version) FROM {{ this }}) +{% endif %} diff --git a/src/ingestion/silver/ai/schema.yml b/src/ingestion/silver/ai/schema.yml index 41fed8533..43505258f 100644 --- a/src/ingestion/silver/ai/schema.yml +++ b/src/ingestion/silver/ai/schema.yml @@ -374,3 +374,97 @@ models: description: "When the record was collected" tests: - not_null + + - name: class_ai_overage + description: > + Unified per-person per-month AI spend-vs-limit (overage) across vendors. + Grain: one row per (tenant, source, seat, billing month). Answers "how + much did each person spend above their monthly limit". Homogeneous + monetary contract in minor units (cents) + ISO currency so seats from + Claude Team and future sources (OpenAI / ChatGPT) are directly + comparable. overage_cents = max(0, used_amount_cents - credit_limit_cents), + NULL when no limit is known (honest-NULL). First/only source today: + Claude Team (claude_team__ai_overage, from /overage_spend_limits via the + claude-team-proxy, billing:view/Owner-gated). OpenAI joins later by + emitting the same 19 columns under tag silver:class_ai_overage. + columns: + - name: insight_tenant_id + description: "Tenant isolation field" + tests: + - not_null + - name: source_id + description: "Connector instance identifier" + tests: + - not_null + - name: unique_key + description: "Composite dedup key: tenant-source-seat-YYYY-MM (per source/month)" + tests: + - not_null + - unique + - name: email + description: > + Seat work email (lowercased). Present for Claude Team (every seat + carries an email). For sources where overage is only attributable + via an API/account id, email MAY be NULL and account_id carries the + identity proxy (resolved in Silver Step 2). + - name: account_id + description: "Stable vendor seat/account id (Claude: account_uuid). Identity proxy / join anchor." + - name: period_month + description: "First day of the billing month (UTC). Monthly billing grain." + tests: + - not_null + - name: tool + description: "AI vendor discriminator." + tests: + - not_null + - accepted_values: + arguments: + values: ['claude', 'chatgpt', 'openai'] + - name: seat_tier + description: "Vendor plan/seat tier (free text per vendor). NULL when not exposed." + - name: currency + description: > + ISO currency of the *_cents monetary columns. 'USD' today (both + Claude Team and OpenAI bill in USD); extend accepted_values when a + non-USD billing tenant first appears. + tests: + - not_null + - accepted_values: + arguments: + values: ['USD'] + - name: credit_limit_cents + description: > + Monthly spend/credit limit in minor units (cents). NULL when no limit + applies or the limit is unknown for the source. + - name: used_amount_cents + description: "Spend so far in the period, in minor units (cents)." + tests: + - not_null + - name: overage_cents + description: > + Spend beyond the limit: max(0, used_amount_cents - credit_limit_cents). + NULL (not 0) when credit_limit_cents is unknown — overage is + uncomputable without a limit. + - name: is_over_limit + description: "1 when used_amount_cents > credit_limit_cents (soft overage). NULL when limit unknown." + - name: is_enabled + description: "Whether overage / extra-usage is enabled for the seat. NULL when not exposed." + - name: overage_metrics_json + description: > + Raw vendor-specific extras kept out of the positional contract. + Claude Team: limit_type / used_credits_basis / out_of_credits / seat_tier. + - name: source + description: "Staging-model discriminator." + tests: + - not_null + - accepted_values: + arguments: + values: ['claude_team', 'openai', 'chatgpt_team'] + - name: data_source + description: "Framework source discriminator." + tests: + - not_null + - name: collected_at + description: "When the record was collected (_airbyte_extracted_at)." + tests: + - not_null