Skip to content
Merged
Original file line number Diff line number Diff line change
@@ -1,12 +1,20 @@
---
status: accepted
status: superseded
date: 2026-04-08
superseded-by: 0002-cost-report-exclusive-ending-boundary.md
---

# ADR-0001: Cursor granularity `PT1S` to avoid empty date-boundary windows

**ID**: `cpt-insightspec-adr-claude-admin-cursor-granularity`

> **Superseded by [ADR-0002](0002-cost-report-exclusive-ending-boundary.md).**
> `PT1S` still 400s the `cost_report` stream: the Admin API treats `ending_at`
> as exclusive, so `23:59:59Z` selects zero daily buckets. The
> `messages_usage` and `cost_report` streams now use `cursor_granularity: P0D`
> (`ending_at = starting_at + P1D` = next midnight). `code_usage` keeps `PT1S`
> because it does not send `ending_at`. See ADR-0002.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

<!-- toc -->

- [Context and Problem Statement](#context-and-problem-statement)
Expand Down Expand Up @@ -40,7 +48,7 @@ ending_at = today_00:00 + P1D - P1D = today_00:00 = starting_at

The Anthropic Admin API rejects requests where `ending_at == starting_at` with HTTP 400 (`"ending_at must be after starting_at"`). This caused incremental sync to fail every time the cursor reached the current day.

The affected streams in `claude-admin` (`claude_admin_messages_usage`, `claude_admin_cost_report`, `claude_admin_code_usage`) include `end_time_option` in their configuration, meaning they send `ending_at` as a query parameter to the API. All three streams were vulnerable to the boundary condition.
At the time of this decision, the affected streams in `claude-admin` (`claude_admin_messages_usage`, `claude_admin_cost_report`, `claude_admin_code_usage`) included `end_time_option` in their configuration, meaning they sent `ending_at` as a query parameter to the API. All three streams were vulnerable to the boundary condition. (Superseded: `claude_admin_code_usage` no longer sends `ending_at` — see ADR-0002.)

## Decision Drivers

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
---
status: accepted
date: 2026-07-25
supersedes: 0001-cursor-granularity-boundary-fix.md
---

# ADR-0002: `cursor_granularity: P0D` for exclusive, day-aligned `ending_at` on the usage/cost streams

**ID**: `cpt-insightspec-adr-claude-admin-cost-report-boundary`

<!-- toc -->

- [Context and Problem Statement](#context-and-problem-statement)
- [Decision Drivers](#decision-drivers)
- [Considered Options](#considered-options)
- [Decision Outcome](#decision-outcome)
- [Consequences](#consequences)
- [Confirmation](#confirmation)
- [Pros and Cons of the Options](#pros-and-cons-of-the-options)
- [More Information](#more-information)
- [Traceability](#traceability)

<!-- /toc -->

## Context and Problem Statement

[ADR-0001](0001-cursor-granularity-boundary-fix.md) set `cursor_granularity: PT1S` on the three incremental streams (`claude_admin_messages_usage`, `claude_admin_cost_report`, `claude_admin_code_usage`) to avoid an empty `starting_at == ending_at` window. That fix was incomplete for the two streams that inject `ending_at` (`messages_usage`, `cost_report`): a live staging sync (`docs/components/connectors/testing/claude-admin-sync-log.md`) showed `claude_admin_cost_report` returning **HTTP 400 `Invalid date range: ending date must be after starting date`** on **every** daily slice — 0 rows ingested.

The Anthropic Admin usage/cost API contract (confirmed from
`https://platform.claude.com/docs/en/api/admin-api/usage-cost/get-cost-report`
and `.../get-messages-usage-report`):

- `starting_at` is **required** and is snapped to the **start of the UTC day** for `bucket_width=1d`.
- `ending_at` is **optional and EXCLUSIVE** — "time buckets that **end before** this timestamp".
- Each returned bucket carries `starting_at` (inclusive) and `ending_at` (exclusive); the documented example is a full midnight→midnight day: `starting_at: 2025-08-01T00:00:00Z`, `ending_at: 2025-08-02T00:00:00Z`.

Airbyte CDK's `DatetimeBasedCursor` computes each slice as
`ending_at = (partition_start + step) − cursor_granularity` (verified in
`airbyte_cdk 6.60.16`, `datetime_based_cursor.py:270`, `_partition_daterange`).
With `step: P1D`, `cursor_granularity: PT1S` this yields `00:00:00Z → 23:59:59Z`.
The only 1d bucket `[00:00, next-00:00)` ends at the **next midnight**, which is
**not before** `23:59:59Z`, so **zero buckets qualify** — the API rejects it. The
sibling `messages_usage` endpoint tolerates the sub-day `ending_at` (returns
nothing useful) while `cost_report` hard-400s; the underlying defect is identical.

## Decision Drivers

- `ending_at` is exclusive and `starting_at` snaps to day start → a
`midnight → next-midnight` window selects exactly one daily bucket.
- CDK couples `step` and `cursor_granularity`: defining one without the other is
a validation error (`datetime_based_cursor.py:80-85`). `cursor_granularity`
cannot simply be dropped while `step` remains.
- The cursor advances by `step` each iteration (`start = next_start`,
`datetime_based_cursor.py:280`) **independent of** `cursor_granularity`, so
`step: P1D` keeps one-day-per-slice advancement with no skipped days.
- Per-row `date`/`unique` are derived from `stream_interval['start_time'][:10]`
(one day per slice) and the dbt Silver models
(`dbt/claude_admin__ai_api_usage.sql`, `dbt/claude_admin__ai_dev_usage.sql`)
aggregate one row per day — the fix must keep exactly one bucket per slice.

## Considered Options

1. **`cursor_granularity: P0D`** (`PT0S` is equivalent) — keep `step: P1D`,
subtract zero → `ending_at = starting_at + P1D` = next midnight.
2. **Widen `step` to `P7D`/`P30D`** — fewer requests, but each returns multiple
daily buckets all stamped with the slice-start `date` → corrupts Silver.
3. **Drop `end_time_option`** — send only `starting_at`. The usage/cost
endpoints default to a multi-day window when `ending_at` is omitted
(`messages_usage` `1d` default = 7 days), pulling multiple buckets per slice —
same corruption as option 2, plus dependence on an undocumented default.

## Decision Outcome

**Chosen option: Option 1 — `cursor_granularity: P0D` on `messages_usage` and
`cost_report`.** `_parse_timedelta("P0D")` → `timedelta(0)`
(`datetime_based_cursor.py:319`), so:

```text
ending_at = (partition_start + P1D) − P0D = next UTC midnight
```

which is exclusive-correct: it selects exactly the one daily bucket
`[start_midnight, next_midnight)`.

`claude_admin_code_usage` is left on `cursor_granularity: PT1S` because it does
**not** inject `end_time_option` (sends only `starting_at`) — the boundary defect
never applied there, and its cursor uses the bare `%Y-%m-%d` format. No change.

### Consequences

- `cost_report` and `messages_usage` reach COMPLETE with rows > 0 and zero 400s.
- Exactly one bucket per daily slice — no gap, no double-count (verified against
the exclusive contract), so the one-row-per-day dbt Silver models are correct.
- Historical and current-day partitions both produce a valid non-empty window
(`start → start + P1D`); the terminal boundary slice where `start == end` is
benign (the PT1S version produced the same tail slice).

### Confirmation

- `connector.yaml` carries `cursor_granularity: P0D` on `claude_admin_messages_usage`
and `claude_admin_cost_report`; `claude_admin_code_usage` remains `PT1S`.
- L1 mock tests (`tests/test_claude_admin_cost_report.py`,
`tests/test_claude_admin_messages_usage.py`) freeze the clock and assert the
emitted `ending_at` query param equals `starting_at + P1D` (next midnight),
never `23:59:59Z`.
- L2 live staging smoke confirms rows > 0, zero 400s, and one row per day (no
duplicate days) in `bronze_claude_admin`.

## Pros and Cons of the Options

### Option 1: cursor_granularity: P0D

- Good — minimal, one value changes per stream; keeps explicit windowing.
- Good — matches the exclusive-`ending_at` contract exactly; one bucket per slice.
- Good — no change to `step`, so per-day `date`/`unique` and Silver stay correct.
- Neutral — relies on CDK's documented `(start + step) − granularity` arithmetic,
pinned and covered by the L1 test.

### Option 2: Widen step (P7D/P30D)

- Good — cuts request volume ~7–30×.
- Bad — returns multiple daily buckets per request stamped with the slice-start
date → corrupts `claude_admin__ai_api_usage` / `claude_admin__ai_dev_usage`
unless the `date`/`unique` derivation is reworked to read each bucket's own
`starting_at`. Out of scope for a boundary fix.

### Option 3: Drop end_time_option

- Good — sidesteps the boundary calculation.
- Bad — the API defaults to a multi-day window when `ending_at` is omitted →
same multi-bucket corruption as option 2, plus reliance on an undocumented
default.

## More Information

- Airbyte CDK `DatetimeBasedCursor` (6.60.x): slice end
`= (partition_start + step) − cursor_granularity`; cursor advances by `step`;
`step`/`cursor_granularity` must be defined together.
- Request-volume reduction (BUG-2 / #1902) is handled separately via
`concurrency_level: 1` + a `Retry-After`-aware error handler, not by widening
the step.

## Traceability

| Artifact | Requirement ID | Relationship |
|----------|---------------|--------------|
| [ADR-0001](0001-cursor-granularity-boundary-fix.md) | `cpt-insightspec-adr-claude-admin-cursor-granularity` | Supersedes — corrects the cost/usage-stream boundary |
| [DESIGN.md](../DESIGN.md) | `cpt-insightspec-constraint-claude-admin-date-range` | Satisfies — prevents the `Invalid date range` API rejection |
5 changes: 4 additions & 1 deletion scripts/ci/components.py
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,10 @@
"pytest_args": "--suites-only",
"cover": False,
"triggered_by": ["connector-tests-harness"],
"paths": ["src/ingestion/connectors/task-tracking/jira"],
"paths": [
"src/ingestion/connectors/task-tracking/jira",
"src/ingestion/connectors/ai/claude-admin",
],
},
]

Expand Down
2 changes: 1 addition & 1 deletion src/ingestion/connectors/ai/claude-admin/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ Silver-level `silver:class_*` tags will be added in a separate PR; this connecto

- **Rate limits**: organization-level, enforced by the Anthropic Admin API. The connector follows `Retry-After` on HTTP 429 and retries transient 5xx with exponential backoff.
- **31-day window**: usage/cost endpoints cap date ranges at 31 days per request. The connector steps at `P1D` (one day per request) to avoid boundary-day loss caused by Airbyte's inclusive-inclusive cursor arithmetic.
- **`cursor_granularity: PT1S`**: applied on incremental streams to prevent empty date-boundary windows (`starting_at == ending_at`) that the API rejects with HTTP 400. See the historical `claude-api` ADRs for background.
- **Date-window boundary**: the usage/cost API treats `ending_at` as **exclusive** and snaps `starting_at` to the UTC day start, so a `midnight → next-midnight` window selects exactly one daily bucket. `claude_admin_messages_usage` and `claude_admin_cost_report` use `cursor_granularity: P0D` (`ending_at = starting_at + P1D`, the next midnight); the earlier `PT1S` sent `23:59:59Z`, which is not before the bucket's midnight end, so `cost_report` 400'd with `Invalid date range` and ingested 0 rows. `claude_admin_code_usage` keeps `PT1S` because it sends only `starting_at`. See [ADR-0002](../../../../../docs/components/connectors/ai/claude-admin/specs/ADR/0002-cost-report-exclusive-ending-boundary.md) (supersedes ADR-0001).
- **No 3-day reporting lag**: unlike the Enterprise Analytics API (`claude-enterprise`), the Admin API makes day `D` data queryable the same day it is aggregated. This connector's default `start_date` is 90 days ago (not 14).
- **Substream for workspace members**: the members stream iterates over every workspace ID from the parent stream. Large organizations with many workspaces may see proportional sync time.

Expand Down
20 changes: 17 additions & 3 deletions src/ingestion/connectors/ai/claude-admin/connector.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
version: "7.0.4"
version: "6.60.16"
type: DeclarativeSource

check:
Expand Down Expand Up @@ -248,8 +248,15 @@ streams:
type: MinMaxDatetime
datetime: "{{ config.get('start_date', day_delta(-90, format='%Y-%m-%dT00:00:00Z')) }}"
datetime_format: "%Y-%m-%dT%H:%M:%SZ"
# cursor_granularity: P0D so each slice sends ending_at = starting_at + step
# = the next UTC midnight. The Admin usage/cost API treats ending_at as
# EXCLUSIVE ("buckets that end before this timestamp") and snaps starting_at
# to the UTC day start, so a midnight→midnight window selects exactly one
# daily bucket — no gap, no double-count. A positive granularity (the old
# PT1S) pulls ending_at back to 23:59:59Z, which is not before the bucket's
# midnight end, so zero buckets qualify. See ADR-0002 (supersedes ADR-0001).
step: P1D
cursor_granularity: PT1S
cursor_granularity: P0D
start_time_option:
type: RequestOption
field_name: starting_at
Expand Down Expand Up @@ -364,8 +371,15 @@ streams:
type: MinMaxDatetime
datetime: "{{ config.get('start_date', day_delta(-90, format='%Y-%m-%dT00:00:00Z')) }}"
datetime_format: "%Y-%m-%dT%H:%M:%SZ"
# cursor_granularity: P0D so each slice sends ending_at = starting_at + step
# = the next UTC midnight. The Admin usage/cost API treats ending_at as
# EXCLUSIVE ("buckets that end before this timestamp") and snaps starting_at
# to the UTC day start, so a midnight→midnight window selects exactly one
# daily bucket — no gap, no double-count. A positive granularity (the old
# PT1S) pulls ending_at back to 23:59:59Z, which is not before the bucket's
# midnight end, so zero buckets qualify. See ADR-0002 (supersedes ADR-0001).
step: P1D
cursor_granularity: PT1S
cursor_granularity: P0D
start_time_option:
type: RequestOption
field_name: starting_at
Expand Down
2 changes: 1 addition & 1 deletion src/ingestion/connectors/ai/claude-admin/descriptor.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: claude-admin
version: "2026.05.04"
version: "2026.07.25"
schedule: '0 2 * * *'
dbt_select: 'tag:claude-admin+'
workflow: sync
Expand Down
23 changes: 23 additions & 0 deletions src/ingestion/connectors/ai/claude-admin/tests/config.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
"""Claude Admin connector test config builder."""

from __future__ import annotations

from connector_tests import ConfigBuilder

API_BASE = "https://api.anthropic.com"


class ClaudeAdminConfigBuilder(ConfigBuilder):
def __init__(self) -> None:
"""Seed the base config with a test admin key and a fixed, time-full start_date."""
super().__init__()
self._config.update(
{
"admin_api_key": "test-admin-key",
# Full ISO 8601 with time — the messages_usage and cost_report
# cursors parse strictly and reject bare YYYY-MM-DD. With the clock
# frozen at 2026-04-27 and step P1D this yields a small, fixed set
# of one-day slices from 2026-04-24.
"start_date": "2026-04-24T00:00:00Z",
}
)
7 changes: 7 additions & 0 deletions src/ingestion/connectors/ai/claude-admin/tests/conftest.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
import sys
from pathlib import Path

# Local builder modules (config.py) are importable under --import-mode=importlib.
Comment thread
cyberantonz marked this conversation as resolved.
Resolved
sys.path.insert(0, str(Path(__file__).parent))

from connector_tests.plugin import * # noqa: E402,F401,F403 — http_mocker fixture + hooks
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"amount": "123.45",
"currency": "USD",
"workspace_id": "wrkspc_01TESTAAAAAAAAAAAAAAAAAA",
"description": "Claude Sonnet 4 Usage - Input Tokens",
"cost_type": "tokens",
"model": "claude-sonnet-4-6",
"service_tier": "standard",
"context_window": "0-200k",
"token_type": "uncached_input_tokens",
"inference_geo": "global"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"model": "claude-sonnet-4-6",
"api_key_id": "apikey_01TESTAAAAAAAAAAAAAAAAAA",
"workspace_id": "wrkspc_01TESTAAAAAAAAAAAAAAAAAA",
"service_tier": "standard",
"context_window": "0-200k",
"uncached_input_tokens": 1500,
"cache_read_input_tokens": 200,
"cache_creation": {
"ephemeral_5m_input_tokens": 500,
"ephemeral_1h_input_tokens": 1000
},
"output_tokens": 500,
"server_tool_use": {
"web_search_requests": 10
}
}
Loading
Loading