Skip to content

fix(pricing): track cost for direct Gemini providers + add Gemini 3.x entries (#32400) - #32404

Closed
hbentel wants to merge 2 commits into
NousResearch:mainfrom
hbentel:fix/gemini-cost-tracking-32400
Closed

fix(pricing): track cost for direct Gemini providers + add Gemini 3.x entries (#32400)#32404
hbentel wants to merge 2 commits into
NousResearch:mainfrom
hbentel:fix/gemini-cost-tracking-32400

Conversation

@hbentel

@hbentel hbentel commented May 26, 2026

Copy link
Copy Markdown
Contributor

Closes #32400.

Problem

estimated_cost_usd is always $0.00 for every session using provider: gemini or provider: google-gemini-cli, on every Gemini model (including ones with existing pricing entries like gemini-2.5-flash). Token counts are recorded correctly (after #15253 / commit ba8337464), but the cost-calculation chain never fires.

Root cause (two layers)

  1. resolve_billing_route() has no Gemini branch. Native Gemini falls through to billing_mode="unknown", which skips _lookup_official_docs_pricing entirely. Affects every Gemini model, not just 3.5-flash.
  2. No pricing entry for gemini-3.5-flash. Even after fixing the route, the static table only has Gemini 2.5 / 2.0 entries.

Changes

agent/usage_pricing.py

  • Added a branch in resolve_billing_route() that maps provider="gemini" and provider="google-gemini-cli" to provider="google" (matching the existing _OFFICIAL_DOCS_PRICING key prefix), with billing_mode="official_docs_snapshot".
  • Added pricing entries for:
    • gemini-3.5-flash — $1.50 / $9.00 / M; cache_read $0.15/M
    • gemini-3-flash-preview — $0.50 / $3.00 / M; cache_read $0.05/M
    • gemini-3.1-pro-preview — $2.00 / $12.00 / M; cache_read $0.20/M
    • gemini-3.1-flash-lite — $0.25 / $1.50 / M; cache_read $0.025/M

Source: https://ai.google.dev/pricing (cross-referenced with OpenRouter's GET /api/v1/models for the same model IDs). Marked pricing_version="google-pricing-2026-05".

tests/agent/test_usage_pricing.py

5 new regression tests:

  • resolve_billing_route maps gemini → google with official_docs_snapshot mode
  • Same for google-gemini-cli
  • gemini-3.5-flash has a pricing entry
  • gemini-2.5-flash (existing entry) now reachable from provider="gemini"
  • End-to-end estimate_usage_cost for a typical 3.5-flash turn produces a real $ amount

Live end-to-end verification

Same gemini-3.5-flash session before / after on Hermes CLI:

$ sqlite3 ~/.hermes/profiles/gemini35/state.db \
    "SELECT input_tokens, output_tokens, estimated_cost_usd, cost_status \
     FROM sessions ORDER BY started_at DESC LIMIT 1" -header

# Before this PR:
input_tokens=22106  output_tokens=239  estimated_cost_usd=0.0       cost_status=unknown

# With this PR:
input_tokens=19914  output_tokens=9    estimated_cost_usd=0.029952  cost_status=estimated

Math check: 19914 × $1.50/M + 9 × $9.00/M = $0.029952 ✅ matches stored value exactly.

Relationship to #15253

#15253 ("token counts always 0") is already fixed in commit ba8337464; the remaining symptom (cost = 0) has a different root cause documented here. Recommend closing #15253 as stale and tracking the cost issue under #32400 instead.

Test plan

  • scripts/run_tests.sh tests/agent/test_usage_pricing.py — 16/16 pass (5 new + 11 existing)
  • scripts/run_tests.sh tests/agent/ — only pre-existing failures (test_anthropic_adapter.py OAuth-env issues on macOS, unrelated)
  • Live verification on gemini-3.5-flash — see above

🤖 Generated with Claude Code

@hbentel @claude
fix(pricing): track cost for direct Gemini providers + add Gemini 3.x…
60d9ff0 
… entries (NousResearch#32400)

resolve_billing_route had no branch for provider="gemini" or
"google-gemini-cli", so every native Gemini session fell through to
billing_mode="unknown" and skipped pricing lookup. estimated_cost_usd
stayed $0 for every direct-Gemini user on every Gemini model — even
gemini-2.5-flash where the pricing entry already existed in the table.

Changes:
- agent/usage_pricing.py: add a `gemini` / `google-gemini-cli` branch
  to resolve_billing_route() that normalizes to provider="google" so
  the existing ("google", "gemini-X.Y-...") pricing entries match.
- agent/usage_pricing.py: add pricing entries for gemini-3.5-flash,
  gemini-3-flash-preview, gemini-3.1-pro-preview, and
  gemini-3.1-flash-lite. Values sourced from
  https://ai.google.dev/pricing.
- tests: 5 regression tests covering both layers + the end-to-end
  estimate for a typical 3.5-flash turn.

Live verified on gemini-3.5-flash: a session that previously stored
estimated_cost_usd=0.0, cost_status="unknown" now stores
estimated_cost_usd=0.029952, cost_status="estimated".

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@hbentel hbentel mentioned this pull request May 26, 2026
Closed
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint provider/gemini Google Gemini (AI Studio, Cloud Code) labels May 26, 2026
@alt-glitch

alt-glitch commented May 26, 2026

Copy link
Copy Markdown
Collaborator

Competing with #31382 — both fix #32400 by adding a gemini/google-gemini-cli branch to resolve_billing_route() and adding Gemini 3.x pricing entries. Same files modified (agent/usage_pricing.py, tests/agent/test_usage_pricing.py). Recommend consolidating into a single PR.

@hbentel hbentel mentioned this pull request May 26, 2026
Closed
5 tasks
@hbentel @claude
fix(pricing): cover all gemini provider aliases + add gemini-3.1-flas…
b02de3d 
…h-lite-preview entry

Extends resolve_billing_route to include the full set of gemini profile
aliases (google, google-gemini, google-ai-studio) alongside the already-
covered gemini and google-gemini-cli names, so all entry points map to
the google pricing table.  Adds the gemini-3.1-flash-lite-preview pricing
entry that was present in the competing fix (NousResearch#31382).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@hbentel hbentel mentioned this pull request May 27, 2026
Closed
14 tasks
@hbentel

hbentel commented Jun 14, 2026

Copy link
Copy Markdown
Contributor Author

Closing in favor of #31382 which covers the same fix. Happy to help if that PR needs any changes.

@hbentel hbentel closed this Jun 14, 2026
hbentel added a commit to hbentel/hermes-agent that referenced this pull request Jun 14, 2026
@hbentel @claude
test: characterization tests for prompt-cache pipeline (groundwork for …
81e17fc 
…NousResearch#29818)

Pure additive test groundwork that locks down current prompt-cache behavior
at four layers. Lands ahead of a planned refactor that splits the
Anthropic-only caching surface into a pluggable strategy on ProviderProfile
(removes the 8-branch anthropic_prompt_cache_policy() and the
_use_prompt_caching / _use_native_cache_layout AIAgent flags) and unblocks
Gemini context caching for NousResearch#29818. None of those changes are in this PR —
this is only the regression net.

Adds:

- tests/agent/test_usage_pricing.py: 5 new tests covering coverage gaps in
  normalize_usage — Codex Responses shape (subtract path), Gemini native
  SimpleNamespace via the OpenAI branch, Anthropic no-cache zeros,
  reasoning_tokens extraction from output_tokens_details.

- tests/agent/transports/test_extract_cache_stats_realistic.py: 10 new
  tests + 1 xfail. Uses production-shaped fixtures (rather than minimal
  mocks) for Anthropic and chat-completions transports. The Codex
  Responses transport has no extract_cache_stats override today; the
  xfail documents that gap and flips to xpass once the impl lands.

- tests/agent/test_conversation_loop_caching.py: 14 new tests covering
  the policy + apply integration that conversation_loop.py:898 performs.
  Parameterized matrix exercises every branch of the current 8-branch
  anthropic_prompt_cache_policy() function via the full join.

- tests/agent/test_cache_wire_payload_snapshot.py: 6 snapshot tests
  asserting the literal output dict of apply_anthropic_cache_control()
  for fixed inputs across (native/envelope × 5m/1h × text/tool-message).
  Any byte change in the marker wire format breaks these.

- tests/integration/test_prompt_cache_baseline.py +
  scripts/baseline_prompt_cache.sh: opt-in live regression check
  (HERMES_LIVE_BASELINE=1). Runs a real gemini-3.5-flash turn against a
  test profile and asserts shape invariants on the resulting state.db
  row. Two assertions (cost_status=='estimated', estimated_cost_usd > 0)
  currently fail on plain main — they document the bug fixed by NousResearch#32404
  and flip to green automatically once that PR merges. Must be invoked
  via direct `python -m pytest`, not scripts/run_tests.sh (the latter
  uses env -i by design and strips non-allowlisted env vars).

Test plan:
  - scripts/run_tests.sh tests/agent/ tests/integration/test_prompt_cache_baseline.py
    → 3534 pass, 14 fail (the 14 are pre-existing test_anthropic_adapter.py
      OAuth-credential leaks on developer machines, unrelated to this PR)
  - All 45 PR-added tests pass under the standard runner
  - 6 baseline tests correctly skip without HERMES_LIVE_BASELINE=1
  - 1 expected xfail in extract_cache_stats_realistic (Codex impl gap)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
hbentel added a commit to hbentel/hermes-agent that referenced this pull request Jun 19, 2026
@hbentel @claude
test: characterization tests for prompt-cache pipeline (groundwork for …
ded458e 
…NousResearch#29818)

Pure additive test groundwork that locks down current prompt-cache behavior
at four layers. Lands ahead of a planned refactor that splits the
Anthropic-only caching surface into a pluggable strategy on ProviderProfile
(removes the 8-branch anthropic_prompt_cache_policy() and the
_use_prompt_caching / _use_native_cache_layout AIAgent flags) and unblocks
Gemini context caching for NousResearch#29818. None of those changes are in this PR —
this is only the regression net.

Adds:

- tests/agent/test_usage_pricing.py: 5 new tests covering coverage gaps in
  normalize_usage — Codex Responses shape (subtract path), Gemini native
  SimpleNamespace via the OpenAI branch, Anthropic no-cache zeros,
  reasoning_tokens extraction from output_tokens_details.

- tests/agent/transports/test_extract_cache_stats_realistic.py: 10 new
  tests + 1 xfail. Uses production-shaped fixtures (rather than minimal
  mocks) for Anthropic and chat-completions transports. The Codex
  Responses transport has no extract_cache_stats override today; the
  xfail documents that gap and flips to xpass once the impl lands.

- tests/agent/test_conversation_loop_caching.py: 14 new tests covering
  the policy + apply integration that conversation_loop.py:898 performs.
  Parameterized matrix exercises every branch of the current 8-branch
  anthropic_prompt_cache_policy() function via the full join.

- tests/agent/test_cache_wire_payload_snapshot.py: 6 snapshot tests
  asserting the literal output dict of apply_anthropic_cache_control()
  for fixed inputs across (native/envelope × 5m/1h × text/tool-message).
  Any byte change in the marker wire format breaks these.

- tests/integration/test_prompt_cache_baseline.py +
  scripts/baseline_prompt_cache.sh: opt-in live regression check
  (HERMES_LIVE_BASELINE=1). Runs a real gemini-3.5-flash turn against a
  test profile and asserts shape invariants on the resulting state.db
  row. Two assertions (cost_status=='estimated', estimated_cost_usd > 0)
  currently fail on plain main — they document the bug fixed by NousResearch#32404
  and flip to green automatically once that PR merges. Must be invoked
  via direct `python -m pytest`, not scripts/run_tests.sh (the latter
  uses env -i by design and strips non-allowlisted env vars).

Test plan:
  - scripts/run_tests.sh tests/agent/ tests/integration/test_prompt_cache_baseline.py
    → 3534 pass, 14 fail (the 14 are pre-existing test_anthropic_adapter.py
      OAuth-credential leaks on developer machines, unrelated to this PR)
  - All 45 PR-added tests pass under the standard runner
  - 6 baseline tests correctly skip without HERMES_LIVE_BASELINE=1
  - 1 expected xfail in extract_cache_stats_realistic (Codex impl gap)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This was referenced Jul 28, 2026
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists provider/gemini Google Gemini (AI Studio, Cloud Code) type/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Direct Gemini cost tracking is always $0.00 — resolve_billing_route() lacks gemini/google-gemini-cli branches + missing gemini-3.5-flash pricing entry

2 participants

@hbentel @alt-glitch