fix(gemini): recover v0.39 stream-json usage - #518
Merged
junhoyeo merged 1 commit intoMay 18, 2026
Conversation
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Recover Gemini CLI stream-json stats that expose per-model token counts directly under stats.models rather than nested stats.models.*.tokens. Treat stream-json input as already net when only input is present while preserving cache-inclusive normalization for legacy tokens wrappers. Validation: - cargo fmt --check: PASS - cargo test -p tokscale-core gemini: PASS, 32 passed; codebuff 0 passed, 10 filtered out; hermes 0 passed, 3 filtered out - cargo test -p tokscale-core: PASS, 662 passed, 1 ignored; codebuff 10 passed; hermes 3 passed; doc-tests 0 passed - cargo clippy -p tokscale-core --lib -- -D warnings: PASS - git diff --check: PASS - git diff --cached --check: PASS - rg -n "[А-Яа-яЁё]" crates/tokscale-core/src/sessions/gemini.rs: PASS, no matches Rollback: - git revert HEAD
IvGolovach
force-pushed
the
codex/gemini-0391-usage-diagnostics
branch
from
May 12, 2026 17:12
96edd27 to
616f66b
Compare
makoMakoGo
added a commit
to makoMakoGo/tokscale
that referenced
this pull request
May 19, 2026
* fix(hermes): include Hermes usage in agent totals (junhoyeo#552) fix(hermes): include agent metadata in parsed usage Validation * Validation tier: Tier 2 — Narrow runtime change, Hermes parser metadata only with targeted parser and TUI aggregation coverage. * git diff --check: PASS * git diff --cached --check: PASS * cargo test -p tokscale-core --test hermes: PASS (3 passed) * cargo test -p tokscale-cli tui::data::tests::test_aggregate_messages_builds_agent_usage: PASS (1 passed) * cargo fmt --check: PASS * Ledger: not applicable — not required for selected validation tier/change family. * Version: not applicable — not required for selected validation tier/change family. * Not run: full workspace tests — not required for selected validation tier; remote CI can provide broad proof. Rollback * git revert HEAD * fix(gemini): recover v0.39 stream-json usage (junhoyeo#518) Recover Gemini CLI stream-json stats that expose per-model token counts directly under stats.models rather than nested stats.models.*.tokens. Treat stream-json input as already net when only input is present while preserving cache-inclusive normalization for legacy tokens wrappers. Validation: - cargo fmt --check: PASS - cargo test -p tokscale-core gemini: PASS, 32 passed; codebuff 0 passed, 10 filtered out; hermes 0 passed, 3 filtered out - cargo test -p tokscale-core: PASS, 662 passed, 1 ignored; codebuff 10 passed; hermes 3 passed; doc-tests 0 passed - cargo clippy -p tokscale-core --lib -- -D warnings: PASS - git diff --check: PASS - git diff --cached --check: PASS - rg -n "[А-Яа-яЁё]" crates/tokscale-core/src/sessions/gemini.rs: PASS, no matches Rollback: - git revert HEAD * fix(antigravity): decode chunked identity probe responses (junhoyeo#562) * fix(antigravity): decode chunked identity probe responses * fix(antigravity): prefer Transfer-Encoding chunked over Content-Length When a response advertises both Transfer-Encoding: chunked and Content-Length, RFC 7230 §3.3.3 requires the receiver to ignore Content-Length and decode the body as chunked. Both identity_probe_request and rpc_request previously checked Content-Length first, so a server that sent both headers would be decoded as a fixed-length read on top of chunk framing and the body would silently desync. Reorder both code paths to check chunked first, then fall back to Content-Length, then to close-delimited reads. Add a regression test that serves chunked Antigravity JSON alongside a bogus Content-Length: 1 header and asserts the marker is still detected. --------- Co-authored-by: Junho Yeo <i@junho.io> * fix(tui): keep tab sort defaults in sync (junhoyeo#553) Validation * Validation tier: Tier 2 — narrow runtime change, localized TUI state/footer behavior. * git diff --check: PASS * git diff --cached --check: PASS * cargo test -p tokscale-cli test_initial_hourly_tab_uses_hourly_sort_default: PASS * cargo test -p tokscale-cli test_switch_tab_preserves_daily_sort_after_hourly_roundtrip: PASS * cargo test -p tokscale-cli test_current_count_label_matches_active_tab: PASS * cargo test -p tokscale-cli --bin tokscale: PASS, 482 passed, 1 ignored * cargo fmt --all --check: PASS * rustup run stable cargo clippy -p tokscale-cli --bin tokscale --all-features -- -D warnings: PASS * Ledger: not applicable — not required for selected validation tier/change family. * Version: not applicable — not required for selected validation tier/change family. * Not run: no frontend tests — not required for CLI TUI-only change. * Additional note: cargo test -p tokscale-cli was attempted and failed only in pricing integration tests due external LiteLLM/OpenRouter fetch failures, outside this TUI diff. Rollback * git revert HEAD * chore: bump version to 2.1.2 * feat(tui): open daily detail rows for selected date (junhoyeo#564) * feat(tui): open daily detail rows for selected date * fix(tui): re-anchor daily detail close + sanitize on data refresh Two daily-detail state-machine fixes: - close_daily_detail() re-anchors the Daily summary selection by date rather than by the cached list index. The user can change the Daily sort while in detail mode, which makes daily_list_selected_index stale; finding the date in the freshly-sorted list keeps the same day highlighted on close. The viewport is preserved when the restored row is still in it, otherwise centered around the row. - update_data() exits detail mode when the refreshed UsageData no longer contains the date the user was viewing. Without this guard selected_daily_detail_date stays Some(...) while get_sorted_daily_detail_rows() returns empty, leaving the UI on a ghost detail view that cannot be navigated out of with arrow keys. Tests: - close after sort change: open detail on a target date, switch sort from Date to Cost, close, assert the restored selection still points at the original date. - update_data drops detail when the selected date disappears, with daily_detail_date returning None and the detail rows empty. - update_data preserves detail mode when the selected date still exists after the refresh. --------- Co-authored-by: Junho Yeo <i@junho.io> * feat(tui): add Minutely tab for per-minute token breakdown (junhoyeo#542) * feat(tui): add Minutely tab for per-minute token breakdown * feat(settings): gate Minutely tab behind opt-in flag, default off Add `minutelyTabEnabled` to settings.json (camelCase, default false) and plumb it through three surfaces so users who do not need per-minute aggregation do not pay for it: - DataLoader skips the minute-bucket map in aggregate_messages when the flag is off. The flag is set via `with_minutely_enabled(bool)` builder on construction in App::new_with_cached_data. - The header tab strip and click-area registration filter Tab::Minutely out when the flag is off, so the tab is invisible and not focusable via mouse. - Tab / BackTab / Left / Right navigation skip Tab::Minutely when the flag is off via new App::next_visible_tab / prev_visible_tab helpers that walk Tab::next/prev until they land on a visible variant. - App::new_with_cached_data clamps initial_tab to Tab::Overview when the requested tab is invisible under the current flag, so a stale TuiConfig pointing at Minutely cannot strand the user on a hidden view. Tests: - settings round-trip and default-false coverage in tui::settings. - DataLoader::aggregate_messages aggregation gate: skip when flag off, run when flag on, group same-minute messages, split adjacent-minute messages, clamp negative tokens and cost to zero. - App tab navigation: default cycle excludes Minutely; with the flag enabled the full 7-tab cycle is restored. Stale Tab::Minutely initial request clamps to Tab::Overview when the flag is off. - Footer current_count_label keeps an explicit "(0 minutes)" arm, verified by a dedicated test that enables the flag. --------- Co-authored-by: Patrick Kühn <patrick.kuehn@von-poll.com> Co-authored-by: Junho Yeo <i@junho.io> * docs(readme): document minutelyTabEnabled and the optional Minutely tab (junhoyeo#568) Add the new `minutelyTabEnabled` setting to the configuration tables in README.md, README.ja.md, README.ko.md, and README.zh-cn.md, plus a short "Enabling the Minutely tab" subsection (localized in each language) explaining when to flip it on, the cost/benefit, the settings.json snippet, and where the tab appears in the strip. The English README also gets a small parenthetical in the features list and the TUI features section noting that Minutely is opt-in. The language READMEs already lag behind the English view list (some still say "4 views"); a full re-sync of those is out of scope for this change \u2014 this PR only adds the new flag's documentation. Cross-references the existing `autoRefreshEnabled` setting for the near-real-time monitoring use case. * docs(readme): correct TUI keyboard map and sync language readmes (junhoyeo#569) The `1-6` tab-switch hint appears in every README but the TUI key handler has no number-key branches \u2014 tab switching is only `\u2190/\u2192/Tab/BackTab`. The same handler also exposes nine keys that the docs never listed: `Enter`/`Esc`/`Backspace` for the daily-detail and graph-cell flow added by junhoyeo#564, `Home`/`End` for list navigation, `j` to jump to today, `y` to copy the selected row, `h` to toggle the Overview chart granularity, `v` to toggle the Hourly Table/Profile view, `Shift+R` to toggle auto-refresh, and `+`/`-` to adjust the refresh interval. `Ctrl+C` was also missing. Replace the keyboard navigation block in README.md, README.ja.md, README.ko.md, and README.zh-cn.md with an accurate list covering these keys. Each language version preserves the surrounding bullet ordering so the diff is local. The three language READMEs also still claimed "4 interactive views" in the Features list at line 140 \u2014 missing Hourly and Agents (added long before this round) and the new opt-in Minutely. Their TUI Features subsection already said 6 views, so the inconsistency was only at the top. Align the line 140 wording with the English version (six views + opt-in Minutely) in all three translations. No code changes; touches only README*.md. * chore: bump version to 2.1.3 * fix(tui): populate minutely data after cache hits --------- Co-authored-by: Ivan Golovach <20299097+IvGolovach@users.noreply.github.com> Co-authored-by: Dohyun Jung <ddark.kr@gmail.com> Co-authored-by: Junho Yeo <i@junho.io> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: xczllgit <52604975+xczllgit@users.noreply.github.com> Co-authored-by: Patrick Kühn <patrick@kuehntechlabs.de> Co-authored-by: Patrick Kühn <patrick.kuehn@von-poll.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
stream-jsonusage rows whose per-model stats are emitted directly understats.models.<model>.tokens-wrapped stats behavior, including cache-inclusive input normalization.input-only stream stats as already net input so cache hits are not subtracted twice.Why
Gemini CLI v0.39.1 emits stream-json usage as
input_tokens,output_tokens,cached, andinputwithout a nestedtokensobject. Tokscale only read model stats fromstats.models.<model>.tokens, so recoverable usage could be missed. This change counts token-bearing v0.39.x stats while still avoiding fabricated usage when logs contain no token metadata.Upstream shapes reviewed:
Diff scope
crates/tokscale-core/src/sessions/gemini.rsBranch integrity
mainorigin/mainat3a9045bc1b9aa245ca44ffcdca68bdaaa09520eb96edd2777df67126c42056fe2a448f026ba27e501 / 03a9045bc1b9aa245ca44ffcdca68bdaaa09520eborigin/mainis an ancestor of this branch.Commit integrity
96edd2777df67126c42056fe2a448f026ba27e50 fix(gemini): recover v0.39 stream-json usageLedger/version proof
Not applicable —
scripts/ledger/does not exist in this repository.Not applicable — no version metadata changed.
Diff hygiene
git diff --name-status origin/main...HEAD:M crates/tokscale-core/src/sessions/gemini.rsgit diff --check origin/main...HEAD: PASS, no outputValidation mode and proof
Mode 2 — narrow runtime parser change. This is sufficient because the diff only changes Gemini usage extraction/normalization and is covered by targeted Gemini parser tests plus the full
tokscale-coresuite.cargo fmt --check: PASScargo test -p tokscale-core gemini: PASS, 32 passedcargo test -p tokscale-core: PASS, 662 passed, 1 ignored; codebuff 10 passed; hermes 3 passed; doc-tests 0 passedcargo clippy -p tokscale-core --lib -- -D warnings: PASSgit diff --check: PASSgit diff --cached --check: PASSNot used as required proof —
cargo clippy -p tokscale-core --all-targets -- -D warningscurrently reports an unrelated pre-existing test-helper lint incrates/tokscale-core/src/lib.rs:2238.Migration notes
Not applicable — no DB migration changed.
CI context confirmation
Pending — PR has not been opened yet, so remote CI has not run.
CI context names unchanged. Not applicable — no CI/workflow context changes.
Runtime safety
Parser-only change: no new blocking locks, queues, network calls, persistence behavior, or invariant removal.
No invariant regression introduced.
Documentation integrity
Not applicable — no docs/runbooks/commands changed.
Rollback plan
Use
git revert <post_merge_commit_sha>after merge.No DB downgrade or data repair required.
Known residual risks
Gemini chat recordings that contain no usage metadata or token stats still cannot be counted honestly. This PR only recovers token-bearing v0.39.x
stream-jsonstats.