Repository navigation
fix(claude-admin): send day-aligned exclusive ending_at for usage/cost streams #1926
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
5f83d3c
fix(claude-admin): send day-aligned exclusive ending_at for usage/cos…
cyberdima b6a6f08
fix(claude-admin): address review comments and bump descriptor version
cyberdima f2b18ab
docs(claude-admin): add docstrings to test helpers and functions
cyberdima d43e43a
Merge branch 'main' of github.com:constructorfabric/insight into fix/…
cyberdima b7fac1c
Merge branch 'main' into fix/1901-claude-admin-cost-report-date-range
cyberdima ec785d0
Merge branch 'main' into fix/1901-claude-admin-cost-report-date-range
mitasovr 049a356
Merge branch 'main' into fix/1901-claude-admin-cost-report-date-range
cyberdima 450d4eb
Merge branch 'main' into fix/1901-claude-admin-cost-report-date-range
cyberantonz File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
148 changes: 148 additions & 0 deletions
148
...nectors/ai/claude-admin/specs/ADR/0002-cost-report-exclusive-ending-boundary.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,148 @@ | ||
| --- | ||
| status: accepted | ||
| date: 2026-07-25 | ||
| supersedes: 0001-cursor-granularity-boundary-fix.md | ||
| --- | ||
|
|
||
| # ADR-0002: `cursor_granularity: P0D` for exclusive, day-aligned `ending_at` on the usage/cost streams | ||
|
|
||
| **ID**: `cpt-insightspec-adr-claude-admin-cost-report-boundary` | ||
|
|
||
| <!-- toc --> | ||
|
|
||
| - [Context and Problem Statement](#context-and-problem-statement) | ||
| - [Decision Drivers](#decision-drivers) | ||
| - [Considered Options](#considered-options) | ||
| - [Decision Outcome](#decision-outcome) | ||
| - [Consequences](#consequences) | ||
| - [Confirmation](#confirmation) | ||
| - [Pros and Cons of the Options](#pros-and-cons-of-the-options) | ||
| - [More Information](#more-information) | ||
| - [Traceability](#traceability) | ||
|
|
||
| <!-- /toc --> | ||
|
|
||
| ## Context and Problem Statement | ||
|
|
||
| [ADR-0001](0001-cursor-granularity-boundary-fix.md) set `cursor_granularity: PT1S` on the three incremental streams (`claude_admin_messages_usage`, `claude_admin_cost_report`, `claude_admin_code_usage`) to avoid an empty `starting_at == ending_at` window. That fix was incomplete for the two streams that inject `ending_at` (`messages_usage`, `cost_report`): a live staging sync (`docs/components/connectors/testing/claude-admin-sync-log.md`) showed `claude_admin_cost_report` returning **HTTP 400 `Invalid date range: ending date must be after starting date`** on **every** daily slice — 0 rows ingested. | ||
|
|
||
| The Anthropic Admin usage/cost API contract (confirmed from | ||
| `https://platform.claude.com/docs/en/api/admin-api/usage-cost/get-cost-report` | ||
| and `.../get-messages-usage-report`): | ||
|
|
||
| - `starting_at` is **required** and is snapped to the **start of the UTC day** for `bucket_width=1d`. | ||
| - `ending_at` is **optional and EXCLUSIVE** — "time buckets that **end before** this timestamp". | ||
| - Each returned bucket carries `starting_at` (inclusive) and `ending_at` (exclusive); the documented example is a full midnight→midnight day: `starting_at: 2025-08-01T00:00:00Z`, `ending_at: 2025-08-02T00:00:00Z`. | ||
|
|
||
| Airbyte CDK's `DatetimeBasedCursor` computes each slice as | ||
| `ending_at = (partition_start + step) − cursor_granularity` (verified in | ||
| `airbyte_cdk 6.60.16`, `datetime_based_cursor.py:270`, `_partition_daterange`). | ||
| With `step: P1D`, `cursor_granularity: PT1S` this yields `00:00:00Z → 23:59:59Z`. | ||
| The only 1d bucket `[00:00, next-00:00)` ends at the **next midnight**, which is | ||
| **not before** `23:59:59Z`, so **zero buckets qualify** — the API rejects it. The | ||
| sibling `messages_usage` endpoint tolerates the sub-day `ending_at` (returns | ||
| nothing useful) while `cost_report` hard-400s; the underlying defect is identical. | ||
|
|
||
| ## Decision Drivers | ||
|
|
||
| - `ending_at` is exclusive and `starting_at` snaps to day start → a | ||
| `midnight → next-midnight` window selects exactly one daily bucket. | ||
| - CDK couples `step` and `cursor_granularity`: defining one without the other is | ||
| a validation error (`datetime_based_cursor.py:80-85`). `cursor_granularity` | ||
| cannot simply be dropped while `step` remains. | ||
| - The cursor advances by `step` each iteration (`start = next_start`, | ||
| `datetime_based_cursor.py:280`) **independent of** `cursor_granularity`, so | ||
| `step: P1D` keeps one-day-per-slice advancement with no skipped days. | ||
| - Per-row `date`/`unique` are derived from `stream_interval['start_time'][:10]` | ||
| (one day per slice) and the dbt Silver models | ||
| (`dbt/claude_admin__ai_api_usage.sql`, `dbt/claude_admin__ai_dev_usage.sql`) | ||
| aggregate one row per day — the fix must keep exactly one bucket per slice. | ||
|
|
||
| ## Considered Options | ||
|
|
||
| 1. **`cursor_granularity: P0D`** (`PT0S` is equivalent) — keep `step: P1D`, | ||
| subtract zero → `ending_at = starting_at + P1D` = next midnight. | ||
| 2. **Widen `step` to `P7D`/`P30D`** — fewer requests, but each returns multiple | ||
| daily buckets all stamped with the slice-start `date` → corrupts Silver. | ||
| 3. **Drop `end_time_option`** — send only `starting_at`. The usage/cost | ||
| endpoints default to a multi-day window when `ending_at` is omitted | ||
| (`messages_usage` `1d` default = 7 days), pulling multiple buckets per slice — | ||
| same corruption as option 2, plus dependence on an undocumented default. | ||
|
|
||
| ## Decision Outcome | ||
|
|
||
| **Chosen option: Option 1 — `cursor_granularity: P0D` on `messages_usage` and | ||
| `cost_report`.** `_parse_timedelta("P0D")` → `timedelta(0)` | ||
| (`datetime_based_cursor.py:319`), so: | ||
|
|
||
| ```text | ||
| ending_at = (partition_start + P1D) − P0D = next UTC midnight | ||
| ``` | ||
|
|
||
| which is exclusive-correct: it selects exactly the one daily bucket | ||
| `[start_midnight, next_midnight)`. | ||
|
|
||
| `claude_admin_code_usage` is left on `cursor_granularity: PT1S` because it does | ||
| **not** inject `end_time_option` (sends only `starting_at`) — the boundary defect | ||
| never applied there, and its cursor uses the bare `%Y-%m-%d` format. No change. | ||
|
|
||
| ### Consequences | ||
|
|
||
| - `cost_report` and `messages_usage` reach COMPLETE with rows > 0 and zero 400s. | ||
| - Exactly one bucket per daily slice — no gap, no double-count (verified against | ||
| the exclusive contract), so the one-row-per-day dbt Silver models are correct. | ||
| - Historical and current-day partitions both produce a valid non-empty window | ||
| (`start → start + P1D`); the terminal boundary slice where `start == end` is | ||
| benign (the PT1S version produced the same tail slice). | ||
|
|
||
| ### Confirmation | ||
|
|
||
| - `connector.yaml` carries `cursor_granularity: P0D` on `claude_admin_messages_usage` | ||
| and `claude_admin_cost_report`; `claude_admin_code_usage` remains `PT1S`. | ||
| - L1 mock tests (`tests/test_claude_admin_cost_report.py`, | ||
| `tests/test_claude_admin_messages_usage.py`) freeze the clock and assert the | ||
| emitted `ending_at` query param equals `starting_at + P1D` (next midnight), | ||
| never `23:59:59Z`. | ||
| - L2 live staging smoke confirms rows > 0, zero 400s, and one row per day (no | ||
| duplicate days) in `bronze_claude_admin`. | ||
|
|
||
| ## Pros and Cons of the Options | ||
|
|
||
| ### Option 1: cursor_granularity: P0D | ||
|
|
||
| - Good — minimal, one value changes per stream; keeps explicit windowing. | ||
| - Good — matches the exclusive-`ending_at` contract exactly; one bucket per slice. | ||
| - Good — no change to `step`, so per-day `date`/`unique` and Silver stay correct. | ||
| - Neutral — relies on CDK's documented `(start + step) − granularity` arithmetic, | ||
| pinned and covered by the L1 test. | ||
|
|
||
| ### Option 2: Widen step (P7D/P30D) | ||
|
|
||
| - Good — cuts request volume ~7–30×. | ||
| - Bad — returns multiple daily buckets per request stamped with the slice-start | ||
| date → corrupts `claude_admin__ai_api_usage` / `claude_admin__ai_dev_usage` | ||
| unless the `date`/`unique` derivation is reworked to read each bucket's own | ||
| `starting_at`. Out of scope for a boundary fix. | ||
|
|
||
| ### Option 3: Drop end_time_option | ||
|
|
||
| - Good — sidesteps the boundary calculation. | ||
| - Bad — the API defaults to a multi-day window when `ending_at` is omitted → | ||
| same multi-bucket corruption as option 2, plus reliance on an undocumented | ||
| default. | ||
|
|
||
| ## More Information | ||
|
|
||
| - Airbyte CDK `DatetimeBasedCursor` (6.60.x): slice end | ||
| `= (partition_start + step) − cursor_granularity`; cursor advances by `step`; | ||
| `step`/`cursor_granularity` must be defined together. | ||
| - Request-volume reduction (BUG-2 / #1902) is handled separately via | ||
| `concurrency_level: 1` + a `Retry-After`-aware error handler, not by widening | ||
| the step. | ||
|
|
||
| ## Traceability | ||
|
|
||
| | Artifact | Requirement ID | Relationship | | ||
| |----------|---------------|--------------| | ||
| | [ADR-0001](0001-cursor-granularity-boundary-fix.md) | `cpt-insightspec-adr-claude-admin-cursor-granularity` | Supersedes — corrects the cost/usage-stream boundary | | ||
| | [DESIGN.md](../DESIGN.md) | `cpt-insightspec-constraint-claude-admin-date-range` | Satisfies — prevents the `Invalid date range` API rejection | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| """Claude Admin connector test config builder.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| from connector_tests import ConfigBuilder | ||
|
|
||
| API_BASE = "https://api.anthropic.com" | ||
|
|
||
|
|
||
| class ClaudeAdminConfigBuilder(ConfigBuilder): | ||
| def __init__(self) -> None: | ||
| """Seed the base config with a test admin key and a fixed, time-full start_date.""" | ||
| super().__init__() | ||
| self._config.update( | ||
| { | ||
| "admin_api_key": "test-admin-key", | ||
| # Full ISO 8601 with time — the messages_usage and cost_report | ||
| # cursors parse strictly and reject bare YYYY-MM-DD. With the clock | ||
| # frozen at 2026-04-27 and step P1D this yields a small, fixed set | ||
| # of one-day slices from 2026-04-24. | ||
| "start_date": "2026-04-24T00:00:00Z", | ||
| } | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| import sys | ||
| from pathlib import Path | ||
|
|
||
| # Local builder modules (config.py) are importable under --import-mode=importlib. | ||
|
cyberantonz marked this conversation as resolved.
Resolved
|
||
| sys.path.insert(0, str(Path(__file__).parent)) | ||
|
|
||
| from connector_tests.plugin import * # noqa: E402,F401,F403 — http_mocker fixture + hooks | ||
12 changes: 12 additions & 0 deletions
12
src/ingestion/connectors/ai/claude-admin/tests/fixtures/cost_result.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| { | ||
| "amount": "123.45", | ||
| "currency": "USD", | ||
| "workspace_id": "wrkspc_01TESTAAAAAAAAAAAAAAAAAA", | ||
| "description": "Claude Sonnet 4 Usage - Input Tokens", | ||
| "cost_type": "tokens", | ||
| "model": "claude-sonnet-4-6", | ||
| "service_tier": "standard", | ||
| "context_window": "0-200k", | ||
| "token_type": "uncached_input_tokens", | ||
| "inference_geo": "global" | ||
| } |
17 changes: 17 additions & 0 deletions
17
src/ingestion/connectors/ai/claude-admin/tests/fixtures/messages_result.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| { | ||
| "model": "claude-sonnet-4-6", | ||
| "api_key_id": "apikey_01TESTAAAAAAAAAAAAAAAAAA", | ||
| "workspace_id": "wrkspc_01TESTAAAAAAAAAAAAAAAAAA", | ||
| "service_tier": "standard", | ||
| "context_window": "0-200k", | ||
| "uncached_input_tokens": 1500, | ||
| "cache_read_input_tokens": 200, | ||
| "cache_creation": { | ||
| "ephemeral_5m_input_tokens": 500, | ||
| "ephemeral_1h_input_tokens": 1000 | ||
| }, | ||
| "output_tokens": 500, | ||
| "server_tool_use": { | ||
| "web_search_requests": 10 | ||
| } | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.