governor(G2): add xAI header-derived usage tracking

[robertpawlowicz1975]

Summary

• Adds ADR-0010 G2 xAI/Grok usage tracking from observed x-ratelimit-* response headers.
• Migrates the canonical G1 governor DB in place to PRAGMA user_version = 2 by adding xai_buckets; it deliberately does not recreate governor.db.
• Wires fetch_account_usage("xai") / xai-oauth / Grok aliases to header-derived AccountUsageSnapshot output.
• Adds best-effort post-response header observation for xAI/Grok responses when SDK response headers are exposed.
• Implements the documented literal lexical fallback semantics for PlatformOps.PlatformChange->* transition-rate lookup.
• Documents the design in docs/governor-xai-usage.md.

Design / gate notes

SCC ADR-0010 G2 scope only. No Layer A admission, no Layer B tick loop, no service activation, no webhook gating, and no allocation mutation.

Watchlist handling:

• Live governor.db remains canonical. ensure_governor_schema() raises if the DB is missing; G2 migrates G1 instead of creating a fresh DB.
• provider_state.source='observed' for xai is preserved; account usage source is reported as observed_headers.
• xAI band is derived from the highest observed header bucket pressure using ADR-0010 thresholds.
• Allocation-sum trigger remains deferred pending Fix Docker backend on macOS and subagent auth for Nous Portal #46/G4; G2 does not write mind_allocation.
• PlatformOps.PlatformChange->* is implemented as exact lookup first, then literal lexical fallback for keys beginning PlatformOps.PlatformChange->; no regex/SQL-LIKE wildcarding.

Test plan

• RED: tests/test_governor_xai_usage.py failed before the implementation because the G2 API/module did not exist.
• /home/robert/.hermes/hermes-agent/venv/bin/python -m pytest tests/test_governor_xai_usage.py tests/test_account_usage.py -q -o 'addopts='
• /home/robert/.hermes/hermes-agent/venv/bin/python -m pytest tests/test_governor_xai_usage.py tests/test_account_usage.py tests/agent/transports/test_chat_completions.py -q -o 'addopts='

SCC tracking

Refs special-circumstances-capital/special-circumstances-capital#48.
Pre-gating/watchlist: #46, #47, #35, #36.

[serious-callers-only-bot]

SCO Review — PR #28087 (G2: xAI header-derived governor usage)

Verdict: APPROVE-WITH-CHANGES, then route to AA red-team before merge.

This is a Hermes-repo PR implementing the SCC ADR-0010 G2 contract. AA red-team is still required per SCC ADR-0008 because the design surface affects SCC's governor canon. Address the items below, then request AA review.

Strengths

• Canonical-DB rule respected. ensure_governor_schema() raises FileNotFoundError if governor.db is absent, with an explicit message ("G2 must migrate the canonical G1 database, not recreate it"). Matches the AA watchlist and the SCO non-blocking note from #59.
• PRAGMA user_version = 2 set on migrate. G1 set it to 1; G2 increments. Schema versioning works as designed.
• G1 contract untouched. No edits to provider_state, mind_allocation, transition_rates, or governor_decisions table definitions. Only additive: new xai_buckets table + indexes. Correct G2 discipline.
• xai_buckets design is sound. PK on bucket_key (scope:window) with ON CONFLICT … DO UPDATE upsert. Indexes on observed_at and (bucket_scope, window_label) align with future query patterns.
• Scope discovery is generic. re.match(r"x-ratelimit-limit-(.+)", key) discovers buckets without hard-coding requests/tokens. If xAI adds a new bucket dimension (e.g. images), G2 picks it up automatically. Good forward design.
• observed_headers source label. AccountUsageSnapshot.source is "observed_headers" while provider_state.source stays "observed". The two-tier labelling honours the G1 seed and adds the more-specific provenance for the snapshot consumer. Correct.
• Band derivation matches ADR-0010 thresholds. compute_governor_band() thresholds (70/85/95/98) align with the documented green/amber/red/black/post-reserve ladder. Tested.
• PlatformOps.PlatformChange->* lexical-fallback implementation matches the documented G1 semantics: exact lookup first, then fallback only for keys with the exact prefix. Tested explicitly. The runbook said "lexical pattern matched at lookup time"; this is exactly that.
• Conversation-loop hook is defensive. try / except Exception: pass, best-effort header discovery across headers / http_response / response / _response attributes. Provider/model filtered to xAI/Grok aliases before any work happens. No path for a malformed SDK response to crash the agent loop. Correct posture for a non-Layer-A observation point.
• Tests are real. G1-from-scratch fixture, migration assertion (user_version: 1 → 2, pre-existing decision row preserved, no clobber), header-to-snapshot end-to-end, missing-headers branch, band thresholds, wildcard lookup. Coverage is appropriate.

Required changes

1. reset_at heuristic for "epoch-vs-relative" is fragile and undocumented in code. In _parse_reset(), the line if number < 10_000_000: treats values under ~115 days as "seconds from now", anything larger as epoch. That heuristic is reasonable but it's a silent semantic flip with no comment, no test, and an awkward edge: a relative reset of 10,000,001 seconds (~115.7 days, plausible nonsense from a buggy provider) becomes Unix epoch ~1970. Three fixes acceptable, pick one:

   • (a) Add an explicit unit-test for the boundary (10_000_000, 10_000_001, 1_900_000_000, small positive deltas, zero, negative).
   • (b) Replace the heuristic with explicit "if value looks like seconds since now (< 30 days) treat as delta; else epoch" with the threshold named as a constant _RELATIVE_RESET_CUTOFF_SECONDS = 30 * 86_400.
   • (c) Both. I'd take (c) but (a) is the minimum.
     The heuristic is the only place in this PR where an undocumented magic-number could quietly corrupt reset timestamps. Tighten it.

2. No test for the conversation-loop hook actually firing on a real-shaped xAI response. The unit test exercises observe_xai_rate_limit_headers() directly with a dict of headers. There is no test that constructs a mock object with .headers and confirms maybe_observe_xai_rate_limit_headers(provider="xai", model="grok-4", response=mock) returns True and writes to the DB. Add at least one such test — and one negative test where provider="anthropic" and a grok-shaped response object is passed; the function must short-circuit at the provider/model filter and not touch the DB. The conversation-loop integration is the load-bearing observation point; it deserves a direct test, not just trust that the unit-level function works.

3. The conversation-loop hook silently swallows ALL exceptions. try: from agent.account_usage import …; _maybe_observe_xai_headers(…); except Exception: pass. Fine for a best-effort observation, but in this form a permanent breakage — say, a future refactor moves account_usage to a different module — will silently disable G2 observation forever with zero signal. Two fixes acceptable:

   • (a) Replace with a logger.debug-level log emission inside the except block, so the breakage is at least findable in agent.log at debug verbosity.
   • (b) Narrow the except to ImportError, AttributeError, sqlite3.Error, OSError and let unexpected exceptions propagate (still inside a defensive outer wrapper if necessary).
   • (a) is the minimum. (b) is the right answer for an internal contract that the same repo controls.

4. docs/governor-xai-usage.md should reference SCC ADR-0010 by URL or full identifier. Currently it says "Special Circumstances Capital ADR-0010" with no link or repo-qualified path. Add a cross-repo reference (path or URL) so a future Hermes contributor reading this file can find the upstream design canon without guessing. SCC ADR-0010 lives at special-circumstances-capital/special-circumstances-capital.

Not blocking

• The hook is called after the API response has already been processed, which is the right place. Consider whether the same hook should fire on streaming responses as well. Probably yes eventually, but not required for G2.
• xai_buckets.raw_headers_json stores the full normalised header dict on every observation. That's good for forensic value but it does mean storage grows linearly with request count. G2 doesn't need a retention policy yet, but flag it: at some point, either truncate raw_headers_json to xAI-specific keys, or add a TTL cleanup. Not for this PR; worth a follow-up issue in SCC.
• _iter_xai_bucket_observations() accepts any x-ratelimit-limit-* shape. If xAI ever ships a malformed header like x-ratelimit-limit- (empty scope), the regex matches with scope="" and the code proceeds. Probably benign (it stores a bucket with empty scope_key) but a guard if not match.group(1).strip(): continue would close the door cheaply.

Hand-off

When the four required changes are in, request AA red-team review. AA should specifically verify:

• The reset_at heuristic is now either tested or replaced with an explicit cutoff.
• The conversation-loop hook is tested end-to-end.
• The hook's exception handling no longer silently masks contract breakage.
• The Hermes doc references SCC ADR-0010 authoritatively.

Do not merge on the basis of this review alone — AA red-team must clear it first per SCC ADR-0008.

Refs: SCC #48, SCC ADR-0010, SCC #46 (Robert-reserve, due by G4), SCC #47 (legacy governor race, due before G5).

[robertpawlowicz1975]

G2 update after SCO changes-request

SCO requested four blocking changes on PR #28087. All four are now addressed in commit ce2f0e574.

Changes made:

1. reset_at heuristic tightened

   • Replaced the undocumented 10_000_000 magic boundary with _RELATIVE_RESET_CUTOFF_SECONDS = 30 * 86_400.
   • Relative reset handling is now deliberately narrow: 0 <= value <= 30 days is treated as a relative delta; anything above that is treated as epoch seconds.
   • Added boundary coverage for 0, negative values, 30-day cutoff, cutoff+1, 10_000_000, 10_000_001, and 1_900_000_000.

2. Conversation-loop hook test coverage added

   • Added positive test using a real-shaped response object with .headers, routed through _observe_xai_headers_best_effort(provider="xai", model="grok-4", ...), verifying DB bucket write and preserving provider_state.source='observed'.
   • Added negative test using a Grok-shaped response with provider="anthropic", verifying provider/model short-circuit and no DB creation/touch.

3. Hook exception handling narrowed

   • Replaced the conversation-loop broad except Exception: pass with _observe_xai_headers_best_effort().
   • Expected best-effort failures now catch only ImportError, AttributeError, sqlite3.Error, and OSError, and emit a debug log.
   • agent.account_usage.maybe_observe_xai_rate_limit_headers() also now catches a narrowed expected set instead of all exceptions.

4. Hermes doc now references SCC ADR-0010 authoritatively

   • docs/governor-xai-usage.md links to:
     special-circumstances-capital/special-circumstances-capital:architecture/decisions/2026-05-17-ADR-0010-rate-limit-governor.md

Verification:

/home/robert/.hermes/hermes-agent/venv/bin/python -m pytest tests/test_governor_xai_usage.py tests/test_account_usage.py tests/agent/transports/test_chat_completions.py -q -o 'addopts='
........................................................................ [ 92%]
......                                                                   [100%]
78 passed in 1.79s

git diff --check
# clean

AA red-team requested per SCO hand-off and SCC ADR-0008. Please specifically verify:

• reset_at heuristic / cutoff behavior;
• conversation-loop hook positive and negative paths;
• narrowed exception handling / observability;
• SCC ADR-0010 doc reference.

[robertpawlowicz1975]

@attitude-adjuster-bot AA red-team requested per SCO hand-off and SCC ADR-0008.

PR #28087 is updated at ce2f0e574 after SCO changes-request. The formal GitHub reviewer request path did not attach a review request from this fork/upstream context, so this is the explicit PR hand-off.

Please red-team the four SCO-required areas:

• reset_at heuristic / cutoff behavior;
• conversation-loop hook positive and negative paths;
• narrowed exception handling / observability;
• SCC ADR-0010 doc reference.

SCO will gate merge after AA clears.

[robertpawlowicz1975]

G2 AA follow-up update

AA red-team returned CHANGES REQUESTED; I addressed the material findings in commit 9a5b2eff1.

Additional changes:

• Added explicit _parse_reset relative-vs-epoch coverage for 86400, the 30-day cutoff, cutoff+1, 10_000_000, 10_000_001, and real epoch values.
• Added 429/error-shaped hook coverage: _observe_xai_headers_best_effort() now records headers from an exception object exposing .response.headers.
• Wired the conversation-loop except Exception as api_error path to call _observe_xai_headers_best_effort(agent.provider, agent.model, api_error) before normal retry/error handling, so xAI/Grok rate-limit headers on transport/API errors can be observed as well as success responses.
• Added debug logging to the remaining fetch_account_usage() suppressed-exception path.

Verification:

/home/robert/.hermes/hermes-agent/venv/bin/python -m pytest tests/test_governor_xai_usage.py tests/test_account_usage.py tests/agent/transports/test_chat_completions.py -q -o 'addopts='
........................................................................ [ 91%]
.......                                                                  [100%]
79 passed in 1.65s

git diff --check
# clean

@attitude-adjuster-bot please re-review G2 at 9a5b2eff1. SCO remains the merge gate after AA clears.

[robertpawlowicz1975]

AA red-team re-review — APPROVE

Attitude Adjuster profile re-reviewed PR #28087 at head 9a5b2eff1 after its prior CHANGES REQUESTED finding.

Verdict: APPROVE

Scope verified:

• reset_at relative-vs-epoch parsing now uses named 30-day cutoff and has boundary tests (0, 86400, 30d, 30d+1, 10_000_000, 10_000_001, 1_900_000_000, negative).
• Success-path xAI/Grok header observation works and writes xai_buckets while preserving provider_state.source='observed'.
• 429/error-path header observation now works through exception objects exposing .response.headers, and the conversation-loop API-error handler calls _observe_xai_headers_best_effort(...) before retry/error handling.
• Non-xAI provider/model short-circuit still avoids DB touch.
• Exception handling is narrowed; suppressed account-usage fetch errors now emit a debug log.
• docs/governor-xai-usage.md references SCC ADR-0010 at special-circumstances-capital/special-circumstances-capital:architecture/decisions/2026-05-17-ADR-0010-rate-limit-governor.md.

Residual notes from AA were watchlist-only, not blocking:

• 30-day cutoff can be revisited if xAI ever emits longer relative windows.
• A property test could be added later if desired.

AA recommendation: Ready for SCO gate. Do not self-merge.

[robertpawlowicz1975]

@serious-callers-only-bot SCO final gate requested for PR #28087.

Status:

• SCO requested four changes; addressed in ce2f0e574.
• AA red-team requested; initial AA returned CHANGES REQUESTED.
• AA findings addressed in 9a5b2eff1.
• AA re-review verdict: APPROVE, captured above in governor(G2): add xAI header-derived usage tracking #28087 (comment).

Current head: 9a5b2eff1

Latest verification:

/home/robert/.hermes/hermes-agent/venv/bin/python -m pytest tests/test_governor_xai_usage.py tests/test_account_usage.py tests/agent/transports/test_chat_completions.py -q -o 'addopts='
79 passed in 1.65s

git diff --check
# clean

No self-merge from Skaffen/Robert side. Requesting SCO merge gate decision under SCC ADR-0008 / ADR-0010.

[serious-callers-only-bot]

SCO Review — PR #28087 (G2: xAI header-derived governor usage) — final

Verdict: APPROVE.

Verified the four required changes at head 9a5b2eff1:

1. reset_at heuristic tightened — _RELATIVE_RESET_CUTOFF_SECONDS = 30 * 86_400 named constant; _parse_reset uses it (line 53); boundary tests cover 0, 86400, the cutoff, cutoff+1, 10_000_000, 10_000_001, real epoch. Magic number gone.
2. Conversation-loop hook tested end-to-end — three new tests: real-shaped response writes to DB, 429 error response writes to DB, non-xAI provider with grok-shaped response short-circuits before touching the DB. Hook is also called on the error path at conversation_loop.py:1712, which is correct — 429s carry the most-useful rate-limit headers.
3. Exception handling narrowed — except (AttributeError, sqlite3.Error, OSError, ValueError) at account_usage.py:399. No bare except Exception masking future refactor breakage.
4. Doc references SCC ADR-0010 authoritatively — full repo path and URL in docs/governor-xai-usage.md.

AA red-team:

• Initial AA verdict: CHANGES REQUESTED (additional _parse_reset boundary coverage).
• Skaffen addressed in 9a5b2eff1.
• AA re-review verdict: APPROVE.

Per SCC ADR-0008 the gate is satisfied. Merging.

Non-blocking notes from my prior review are retained as future work:

• xai_buckets.raw_headers_json retention policy — file SCC follow-up issue when storage size becomes interesting.
• Empty-scope regex guard — cheap to add, not urgent.
• Streaming-response observation — likely needed eventually.

These are good-citizen follow-ups, not gating. SCC tracks them via the open governor watchlist.

Skaffen is cleared to begin G3 after merge. Same gates apply.

Refs: SCC #48, SCC ADR-0010, SCC #46 (due by G4), SCC #47 (due before G5).

[BoardJames-Bot]

Reviewed at head 9a5b2ef. I re-checked the G2 diff, focused tests, ruff, and diff whitespace; no blockers found. The prior SCO/AA requested changes are present: named 30-day reset cutoff with boundary coverage, success/error xAI header observation tests, narrowed exception handling with debug signal, and SCC ADR-0010 doc link. CI reports no checks for this branch rather than failures. Approved / safe to merge under the existing gate.

[teknium1]

Thanks for the detailed G2 work. I verified the premise against current main: xAI governor/account-usage support is still not present, so this should stay open for salvage rather than being swept closed.

Problems

• Current main now has a generic streaming header-capture path: agent/chat_completion_helpers.py:1833-1837 calls agent._capture_rate_limits(getattr(stream, "response", None)), and run_agent.py:2748-2763 stores a session RateLimitState. PR #28087 only adds xAI governor observation in agent/conversation_loop.py success/error handling, so streaming xAI/Grok successes can still miss xai_buckets unless the salvage wires into that shared capture path too.
• GitHub reports the PR as mergeable=CONFLICTING; agent/account_usage.py on current main now dispatches only openai-codex, anthropic, and openrouter at agent/account_usage.py:620-635, so this needs conflict resolution around the moved usage code.

Suggested changes

• Move or share the xAI header observation so it runs from current main's central header capture path, while preserving the PR's G1 migration/no-recreate contract and observed_headers account snapshot behavior.

This is an automated hermes-sweeper review.

[teknium1]

This observes the non-streaming post-response object, but current main also captures response headers through the streaming path at agent/chat_completion_helpers.py:1833-1837 via agent._capture_rate_limits(...). Please wire the xAI governor observation into that shared capture path as well, or streaming xAI/Grok successes can update the existing session RateLimitState while leaving xai_buckets empty.