fix(telemetry): persist first accounted fallback route - #62302
Conversation
teknium1
left a comment
There was a problem hiding this comment.
Thanks for the focused reproduction and narrow scope. I found one persistence issue that needs resolution.
Problems
hermes_state.py:2524overwrites onlysessions.model. TUI resume chooses that value beforemodel_config.model(tui_gateway/server.py:2245), but choosesmodel_config.providerbeforebilling_provider(tui_gateway/server.py:2251-2257). A session whose primary config remains inmodel_configcan therefore resume with the fallback model paired with the primary provider.- Automatic fallback is explicitly turn-scoped:
restore_primary_runtime()restores the primary model/provider on the next turn (agent/agent_runtime_helpers.py:1138-1169). Persisting the transient fallback as the resume model conflicts with that lifecycle.
Suggested changes
- Separate first-call accounting attribution from the durable runtime selection, or reconcile
model,model_config, and resume precedence as one atomic contract. - Add a regression covering a primary-configured TUI session, first-call fallback persistence, and
_stored_session_runtime_overrides().
Automated hermes-sweeper review.
| if first_accounted_route: | ||
| conn.execute( | ||
| """UPDATE sessions | ||
| SET model = ?, billing_provider = ?, |
There was a problem hiding this comment.
This changes the value TUI resume treats as authoritative for the model (tui_gateway/server.py:2245) without updating model_config, whose provider takes precedence over billing_provider (tui_gateway/server.py:2251-2257). A primary-configured session can resume with this fallback model paired with the primary provider; please preserve a coherent resume contract or separate accounting attribution from runtime selection.
|
Merged via PR #62610 after incorporating your first-accounted fallback-route fix into the per-model usage work, with your authorship preserved in git history. The landing tightens the trigger to genuinely accounted usage, clears stale route fields atomically, and covers zero-usage Codex turns. Thanks for identifying the legacy summary-row gap and supplying the focused regression shape. |
Problem
A session row is created with the requested primary model before the first API call. If that primary fails before producing any accounted usage and automatic fallback succeeds,
update_token_counts()currently keeps the requested model viaCOALESCE(model, ?), while other route fields may describe the fallback.That can persist an incoherent pair such as a GPT model with
custom:zai, even though every accounted API call used the fallback route.Fix
Treat the first accounted usage event as authoritative only while the existing session has
api_call_count == 0:modelandbilling_providertogether;billing_base_urlandbilling_modewhen supplied;This keeps the legacy aggregate row internally coherent without pretending it can represent mixed-provider usage.
Scope and related work
This is intentionally narrower than the other open fallback/usage PRs:
This PR only fixes the zero-accounted-call case. It does not attempt to attribute mid-session mixed usage in the single legacy row.
Tests
Added regression coverage proving that:
Validation on current
main:git diff --checkpasses for both changed files.