Skip to content

feat(sessions): add Command Code as a tracked source - #717

Merged
junhoyeo merged 4 commits into
junhoyeo:mainfrom
baanish:main
Jun 17, 2026
Merged

feat(sessions): add Command Code as a tracked source#717
junhoyeo merged 4 commits into
junhoyeo:mainfrom
baanish:main

Conversation

@baanish

@baanish baanish commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

What

Adds Command Code (commandcode.ai) as a tracked session source, scanning ~/.commandcode/projects/<slug>/*.jsonl.

Why it's estimated

Command Code does not persist token usage on disk — the CLI computes usage in memory and ships it to its backend (confirmed by decompiling the npm bundle; the local transcript carries only message text). So tokens are estimated from the transcript, the same way tokscale already handles kiro:

  • input = cumulative conversation context preceding each assistant turn (Command Code re-sends the full context per request)
  • output = that turn's own content
  • ~4 chars/token, counted from each message's canonical JSON serialization so tool args/results are included

The model id comes from ~/.commandcode/config.json and is canonicalized (e.g. MiniMaxAI/MiniMax-M3-FreeMiniMax-M3) so pricing resolves to the real paid model instead of the free-promo entry or a fuzzy mismatch. The source bypasses the parse cache because that model dependency lives outside the fingerprinted transcript file.

Known limitation

The estimate approximates tokens processed and will not match Command Code's server dashboard: it counts full tool outputs that the tool truncates before billing, and cannot see auxiliary (tool-desc / taste-1) runs. On a real ~880-turn session it estimates 270M input / ~$81 at the MiniMax-M3 rate, versus the dashboard's 181M / $0.00 (a promo). This is consistent with how tokscale treats its other estimated sources.

Testing

  • 9 parser unit tests (estimation, model canonicalization, checkpoint/malformed/empty edge cases)
  • registry + CLI mirror-point tests updated (ClientId::COUNT, ClientFilter, TUI client list)
  • real-session smoke via tokscale clients and tokscale models -c commandcode
  • full suites green: core 927, CLI 778

Reviewed adversarially with Codex (2 rounds); fixes for model-pricing resolution and config-aware (cache-bypass) parsing folded in.

🤖 Generated with Claude Code

Related to #668


Summary by cubic

Add commandcode as a tracked session source with estimated token usage and pricing so Command Code sessions show up with costs and analytics. Scans local transcripts and applies model-aware pricing without relying on server usage; registered as client id 29.

  • New Features
    • Scan ~/.commandcode/projects/<slug>/*.jsonl; skip *.checkpoints.jsonl.
    • Estimate tokens from canonical JSON (~4 chars/token): input = cumulative context before each assistant turn; output = that turn’s content; cache_read = 0.
    • Read model from ~/.commandcode/config.json, canonicalize for pricing (e.g., MiniMaxAI/MiniMax-M3-FreeMiniMax-M3), and bypass the parse cache so config changes take effect immediately.
    • Register across the app: CLI filter flag --commandcode and name mapping, TUI entry “Command Code” with hotkey d, client id 29 (ClientId::COUNT = 30), and frontend display name/logo/color.

Written for commit 9c1bcd1. Summary will update on new commits.

Review in cubic

@vercel

vercel Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
tokscale Ready Ready Preview, Comment Jun 17, 2026 6:42pm

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 9 files

Re-trigger cubic

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aab8e1edfe

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


let chars = entry.content.as_ref().map(content_chars).unwrap_or(0);

match entry.role.as_deref() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Parse Command Code records by type

When scanning current Command Code JSONL sessions, the records use top-level type values (user/assistant) rather than role, so entry.role is None and every line falls into the catch-all arm. That means no assistant UnifiedMessages are ever emitted and tokscale models -c commandcode reports zero usage despite discovering session files; read type (and the corresponding message content shape) instead of matching role here.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: this reply is posted by Claude Code on behalf of the author, not written by them.

This finding doesn't apply to Command Code's format. Command Code transcripts use a top-level role field (user / assistant / tool), not a top-level type. The type field only appears inside content parts (e.g. {"type":"text", ...}), which the match entry.role here never touches.

Verified against a real session (1,766 lines): every line carries role (880 assistant, 862 tool, 24 user), and zero lines have a top-level type. Running it end to end, tokscale models -c commandcode reports ~270M input across 880 messages — not zero usage.

The suggestion looks like it assumed the OpenAI/Codex transcript shape, which does use a top-level type.

Command Code (commandcode.ai) stores session transcripts under
~/.commandcode/projects but does not persist token usage on disk — the
CLI computes usage in memory and sends it to its backend. Estimate usage
from the transcript: input from the cumulative conversation context
preceding each assistant turn and output from the assistant's own
content, at ~4 characters per token (matching tokscale's other estimated
sources), counted from each message's canonical JSON serialization so
structured tool args/results are included.

Canonicalize the configured model id from ~/.commandcode/config.json
(e.g. "MiniMaxAI/MiniMax-M3-Free" -> "MiniMax-M3") so pricing resolves to
the real paid model rather than the free-promo entry or a fuzzy
mismatch.

The estimate approximates tokens processed and intentionally does not
match Command Code's server-reported usage (which reflects tool-output
truncation and auxiliary model runs absent from the transcript).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Directive: input estimation is intentionally an upper bound; changing re-sent context to cache_read is a maintainer decision needing real billing data
Confidence: high
Scope-risk: narrow

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="crates/tokscale-core/src/sessions/commandcode.rs">

<violation number="1" location="crates/tokscale-core/src/sessions/commandcode.rs:411">
P2: Test claiming to pin exact token values derives expectations from the same helpers as production, defeating its purpose</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

let user2_content = json!([{"type": "text", "text": "cccc"}]);
let asst2_content = json!([{"type": "text", "text": "dddd"}]);

let user1_chars = content_chars(&user1_content);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Test claiming to pin exact token values derives expectations from the same helpers as production, defeating its purpose

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At crates/tokscale-core/src/sessions/commandcode.rs, line 411:

<comment>Test claiming to pin exact token values derives expectations from the same helpers as production, defeating its purpose</comment>

<file context>
@@ -355,6 +368,108 @@ mod tests {
+        let user2_content = json!([{"type": "text", "text": "cccc"}]);
+        let asst2_content = json!([{"type": "text", "text": "dddd"}]);
+
+        let user1_chars = content_chars(&user1_content);
+        let asst1_chars = content_chars(&asst1_content);
+        let user2_chars = content_chars(&user2_content);
</file context>

@junhoyeo
junhoyeo merged commit 18c7e87 into junhoyeo:main Jun 17, 2026
2 checks passed
junhoyeo added a commit that referenced this pull request Jun 18, 2026
…t cmd, Warp/Cline (#720)

* docs(readme): sync EN/ja/ko/zh — add Command Code, Antigravity CLI, report cmd, Warp/Cline

EN is the canonical lead. Brings the localized READMEs into sync:
- Add Command Code (#717) and Antigravity CLI (#713) to the client support
  table, multi-platform feature list, source-filtering list, and the
  `--client` possible-values list (all four locales).
- Document the `tokscale report` command (summarize-on-by-default, apple-fm
  backend, --no-summarize opt-out, wiki DB, task grouping) in ja/ko/zh.
- Back-fill pre-existing translation drift: Warp/Oz + Cline client rows,
  feature-list entries, and the Warp/Oz Commands section in ja/ko/zh; fix a
  malformed (logo-less) Grok Build row in ko/zh.

All four files now share an identical 33-client roster and --client list.

Constraint: README.md is the lead; localized files mirror it
Confidence: high
Scope-risk: narrow
Not-tested: native-speaker review of ja/ko/zh translation nuance

* docs(readme): address Codex/cubic review on the report summarizer docs

Apply review feedback from PR #720 (Codex + cubic), verified against the code,
across all four locales (EN lead):

- apple-fm prerequisite (Codex/cubic P2): scripts/wiki-summarizer.py is NOT
  bundled in the published npm/bunx packages (files lists ship only bin/dist),
  so the documented default backend fails with "wiki-summarizer.py not found".
  Add a note that npm/bunx users must copy the script into the config dir or
  use a CLI backend (--summarizer claude/codex/gemini/kiro).
- Apple FM fallback (cubic P3): when the Apple FM SDK is unavailable the script
  falls back to a heuristic classifier (fallback_classify), not "empty
  summaries" — corrected the wording in all locales.
- wiki.db path (cubic P3): WikiDb::default_path() uses dirs::config_dir(), so
  it is platform-specific; qualified the path instead of hardcoding ~/.config.

Confidence: high
Scope-risk: narrow
junhoyeo added a commit that referenced this pull request Jun 22, 2026
…turn input (#750)

Three confirmed bugs in the Command Code session parser (PR #717):

(a) PANIC: canonicalize_model byte-sliced `base[base.len()-N..]` guarded only
    by a length check. A non-ASCII model id from the untrusted
    ~/.commandcode/config.json could place that byte index mid-codepoint and
    panic. Switch to a boundary-safe `str::get(..)` slice that returns None
    (instead of panicking) when the tail is not on a char boundary, preserving
    the case-insensitive `-free` strip.

(b) PRICING: dropping the org prefix turned `MiniMaxAI/MiniMax-M3-Free` into
    bare `MiniMax-M3`, but the client's own `command-code` provider is not a
    pricing provider, so the resolver never reached a `minimax/...` key and
    pricing was lost. Recover the real provider from the gateway id via
    `provider_identity::inferred_provider_from_model` and use it as the
    message provider_id (matching the claudecode convention), falling back to
    `command-code` when nothing is inferred.

(c) O(N^2) INPUT: per-assistant-turn input was estimated from `context_chars`,
    a running cumulative sum of ALL prior messages, inflating input across the
    session versus other clients. Estimate per-turn input from only the new
    context introduced since the previous response (user prompt + tool
    results). Summed over the session this charges each message's content once.

Regression tests added: non-ASCII / emoji model ids do not panic; a MiniMax
model resolves non-zero pricing; per-turn input does not accumulate. Updated
the module doc-comment and renamed the pinning test to match the new model.

Confidence: high
Scope-risk: moderate
Directive: provider_id for MiniMax sessions is now `minimax`, not
`command-code`; this is intentional so pricing resolves. Do not revert without
restoring a pricing path for the gateway provider.
Not-tested: real-world non-MiniMax gateway ids that infer no provider (fall
back to `command-code`, unchanged behavior)
junhoyeo added a commit that referenced this pull request Jun 22, 2026
…746)

* docs: sync EN/ja/ko/zh-cn for unreleased clients + breaking flag removal

Pre-v3.2.0 documentation sync for the unreleased range (v3.1.3..main):

- #465: replace the now-false legacy per-client flag notices in ja/ko/zh-cn
  (they claimed the removed flags still work) with a v3.2.0 breaking-change
  migration note; add the same note to README.md (which had none).
- #728: document the MiniMax Token Plan subscription source (distinct from the
  MINIMAX_API_KEY row) in all locales; port the entire Subscription Usage
  section into ja/ko/zh-cn (was English-only).
- #718: add the Jcode table row + detail section to the locales missing them.
- #726: document the TOKSCALE_FM_DEBUG env var in all locales.
- #633: add the missing task-attributed report bullet to README.ja Key Features.
- drift: add Junie to the frontend Source-filtering list (all locales).
- #710: fix the MiMo Code repo link (XiaomiMiMo/MiMo -> XiaomiMiMo/MiMo-Code).
- #717: disclose Command Code token usage is estimated (~4 chars/token).

Confidence: medium
Scope-risk: narrow
Directive: ja/ko/zh-cn translations of the ported Subscription Usage section are machine-generated and should get a native-speaker review pass
Not-tested: #713 Antigravity CLI detail section was not added — no English source section exists to port from

* fix(report): feed real session content to the summarizer (#633)

extract_content_for_session unconditionally returned metadata_only_content()
(first_user_message hardcoded None), so the report summarizer never saw any
conversation content and the four real per-client extractors were dead code.

Add content_extractor::extract_session_content, which dispatches to the correct
per-client extractor (opencode/claude/codex/gemini) and falls back to
metadata-only — never erroring or panicking — for unknown clients, missing
candidates, or unreadable/unparseable files. report.rs builds a SessionPathIndex
once (session_id -> transcript file, plus opencode DBs) and threads it through
run_summarizer so each payload carries the real first user message.

Confidence: high
Scope-risk: moderate
Rejected: thread file paths through core's scanner/WikiEntry | too invasive; indexed at the report layer instead
Not-tested: end-to-end opencode/codex/gemini extraction in report.rs (core dispatcher covers claude + all fallback paths; per-client extractors are pre-existing)

* docs: name the breaking release v4.0.0 (was v3.2.0)

The per-client flag removal (#465) is a breaking change, so the next release is v4.0.0, not v3.2.0. Update the migration notes in all four README locales and the main.rs doc comment accordingly.

* fix(report): real Codex/Gemini extraction + (client,session_id) index keying

Addresses automated review feedback on the #633 report-summarizer-content
fix (PR #746). The summarizer still surfaced (none) for normal Codex/Gemini
sessions and could mis-route cross-client session_id collisions.

- content_extractor: parse the current on-disk Codex format (event_msg with
  payload.type == "user_message", text in payload.message) and skip
  harness-injected context blocks (<environment_context>/<system-reminder>/
  <user_instructions>), mirroring sessions::codex.
- content_extractor: Gemini extractor now handles chat-recording JSON
  (messages[].type == "user" / content) and falls back to scanning
  line-delimited JSONL; empty/whitespace user text is treated as not-found.
- extract_session_content: an empty/whitespace first_user_message no longer
  counts as success, so scanning continues to a later candidate with real text.
- report: SessionPathIndex is keyed by (client, session_id) to prevent
  cross-client collisions, and Gemini files are keyed by their in-file
  sessionId (via gemini_session_id_for_file) rather than the filename stem,
  since the wiki entry's session_id is derived from inside the file.
- Added fixture-based regression tests for all of the above.

Constraint: wiki session_id for Gemini comes from the in-file sessionId, not the path stem
Rejected: match any leading '<' for Codex injected blocks | drops legit prompts starting with markup
Confidence: high
Scope-risk: narrow
pinion05 added a commit to pinion05/tokscale that referenced this pull request Jun 23, 2026
* feat(sessions): add Command Code as a tracked source

Command Code (commandcode.ai) stores session transcripts under
~/.commandcode/projects but does not persist token usage on disk — the
CLI computes usage in memory and sends it to its backend. Estimate usage
from the transcript: input from the cumulative conversation context
preceding each assistant turn and output from the assistant's own
content, at ~4 characters per token (matching tokscale's other estimated
sources), counted from each message's canonical JSON serialization so
structured tool args/results are included.

Canonicalize the configured model id from ~/.commandcode/config.json
(e.g. "MiniMaxAI/MiniMax-M3-Free" -> "MiniMax-M3") so pricing resolves to
the real paid model rather than the free-promo entry or a fuzzy
mismatch.

The estimate approximates tokens processed and intentionally does not
match Command Code's server-reported usage (which reflects tool-output
truncation and auxiliary model runs absent from the transcript).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(commandcode): document and pin cumulative-input estimation behavior

Directive: input estimation is intentionally an upper bound; changing re-sent context to cache_read is a maintainer decision needing real billing data
Confidence: high
Scope-risk: narrow

* style: apply rustfmt

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Junho Yeo <i@junho.io>
pinion05 added a commit to pinion05/tokscale that referenced this pull request Jun 23, 2026
…t cmd, Warp/Cline (junhoyeo#720)

* docs(readme): sync EN/ja/ko/zh — add Command Code, Antigravity CLI, report cmd, Warp/Cline

EN is the canonical lead. Brings the localized READMEs into sync:
- Add Command Code (junhoyeo#717) and Antigravity CLI (junhoyeo#713) to the client support
  table, multi-platform feature list, source-filtering list, and the
  `--client` possible-values list (all four locales).
- Document the `tokscale report` command (summarize-on-by-default, apple-fm
  backend, --no-summarize opt-out, wiki DB, task grouping) in ja/ko/zh.
- Back-fill pre-existing translation drift: Warp/Oz + Cline client rows,
  feature-list entries, and the Warp/Oz Commands section in ja/ko/zh; fix a
  malformed (logo-less) Grok Build row in ko/zh.

All four files now share an identical 33-client roster and --client list.

Constraint: README.md is the lead; localized files mirror it
Confidence: high
Scope-risk: narrow
Not-tested: native-speaker review of ja/ko/zh translation nuance

* docs(readme): address Codex/cubic review on the report summarizer docs

Apply review feedback from PR junhoyeo#720 (Codex + cubic), verified against the code,
across all four locales (EN lead):

- apple-fm prerequisite (Codex/cubic P2): scripts/wiki-summarizer.py is NOT
  bundled in the published npm/bunx packages (files lists ship only bin/dist),
  so the documented default backend fails with "wiki-summarizer.py not found".
  Add a note that npm/bunx users must copy the script into the config dir or
  use a CLI backend (--summarizer claude/codex/gemini/kiro).
- Apple FM fallback (cubic P3): when the Apple FM SDK is unavailable the script
  falls back to a heuristic classifier (fallback_classify), not "empty
  summaries" — corrected the wording in all locales.
- wiki.db path (cubic P3): WikiDb::default_path() uses dirs::config_dir(), so
  it is platform-specific; qualified the path instead of hardcoding ~/.config.

Confidence: high
Scope-risk: narrow
t1000040 pushed a commit to tmobi-internal/tokscale that referenced this pull request Jun 30, 2026
* feat(sessions): add Command Code as a tracked source

Command Code (commandcode.ai) stores session transcripts under
~/.commandcode/projects but does not persist token usage on disk — the
CLI computes usage in memory and sends it to its backend. Estimate usage
from the transcript: input from the cumulative conversation context
preceding each assistant turn and output from the assistant's own
content, at ~4 characters per token (matching tokscale's other estimated
sources), counted from each message's canonical JSON serialization so
structured tool args/results are included.

Canonicalize the configured model id from ~/.commandcode/config.json
(e.g. "MiniMaxAI/MiniMax-M3-Free" -> "MiniMax-M3") so pricing resolves to
the real paid model rather than the free-promo entry or a fuzzy
mismatch.

The estimate approximates tokens processed and intentionally does not
match Command Code's server-reported usage (which reflects tool-output
truncation and auxiliary model runs absent from the transcript).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(commandcode): document and pin cumulative-input estimation behavior

Directive: input estimation is intentionally an upper bound; changing re-sent context to cache_read is a maintainer decision needing real billing data
Confidence: high
Scope-risk: narrow

* style: apply rustfmt

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Junho Yeo <i@junho.io>
t1000040 pushed a commit to tmobi-internal/tokscale that referenced this pull request Jun 30, 2026
…t cmd, Warp/Cline (junhoyeo#720)

* docs(readme): sync EN/ja/ko/zh — add Command Code, Antigravity CLI, report cmd, Warp/Cline

EN is the canonical lead. Brings the localized READMEs into sync:
- Add Command Code (junhoyeo#717) and Antigravity CLI (junhoyeo#713) to the client support
  table, multi-platform feature list, source-filtering list, and the
  `--client` possible-values list (all four locales).
- Document the `tokscale report` command (summarize-on-by-default, apple-fm
  backend, --no-summarize opt-out, wiki DB, task grouping) in ja/ko/zh.
- Back-fill pre-existing translation drift: Warp/Oz + Cline client rows,
  feature-list entries, and the Warp/Oz Commands section in ja/ko/zh; fix a
  malformed (logo-less) Grok Build row in ko/zh.

All four files now share an identical 33-client roster and --client list.

Constraint: README.md is the lead; localized files mirror it
Confidence: high
Scope-risk: narrow
Not-tested: native-speaker review of ja/ko/zh translation nuance

* docs(readme): address Codex/cubic review on the report summarizer docs

Apply review feedback from PR junhoyeo#720 (Codex + cubic), verified against the code,
across all four locales (EN lead):

- apple-fm prerequisite (Codex/cubic P2): scripts/wiki-summarizer.py is NOT
  bundled in the published npm/bunx packages (files lists ship only bin/dist),
  so the documented default backend fails with "wiki-summarizer.py not found".
  Add a note that npm/bunx users must copy the script into the config dir or
  use a CLI backend (--summarizer claude/codex/gemini/kiro).
- Apple FM fallback (cubic P3): when the Apple FM SDK is unavailable the script
  falls back to a heuristic classifier (fallback_classify), not "empty
  summaries" — corrected the wording in all locales.
- wiki.db path (cubic P3): WikiDb::default_path() uses dirs::config_dir(), so
  it is platform-specific; qualified the path instead of hardcoding ~/.config.

Confidence: high
Scope-risk: narrow
t1000040 pushed a commit to tmobi-internal/tokscale that referenced this pull request Jun 30, 2026
…turn input (junhoyeo#750)

Three confirmed bugs in the Command Code session parser (PR junhoyeo#717):

(a) PANIC: canonicalize_model byte-sliced `base[base.len()-N..]` guarded only
    by a length check. A non-ASCII model id from the untrusted
    ~/.commandcode/config.json could place that byte index mid-codepoint and
    panic. Switch to a boundary-safe `str::get(..)` slice that returns None
    (instead of panicking) when the tail is not on a char boundary, preserving
    the case-insensitive `-free` strip.

(b) PRICING: dropping the org prefix turned `MiniMaxAI/MiniMax-M3-Free` into
    bare `MiniMax-M3`, but the client's own `command-code` provider is not a
    pricing provider, so the resolver never reached a `minimax/...` key and
    pricing was lost. Recover the real provider from the gateway id via
    `provider_identity::inferred_provider_from_model` and use it as the
    message provider_id (matching the claudecode convention), falling back to
    `command-code` when nothing is inferred.

(c) O(N^2) INPUT: per-assistant-turn input was estimated from `context_chars`,
    a running cumulative sum of ALL prior messages, inflating input across the
    session versus other clients. Estimate per-turn input from only the new
    context introduced since the previous response (user prompt + tool
    results). Summed over the session this charges each message's content once.

Regression tests added: non-ASCII / emoji model ids do not panic; a MiniMax
model resolves non-zero pricing; per-turn input does not accumulate. Updated
the module doc-comment and renamed the pinning test to match the new model.

Confidence: high
Scope-risk: moderate
Directive: provider_id for MiniMax sessions is now `minimax`, not
`command-code`; this is intentional so pricing resolves. Do not revert without
restoring a pricing path for the gateway provider.
Not-tested: real-world non-MiniMax gateway ids that infer no provider (fall
back to `command-code`, unchanged behavior)
t1000040 pushed a commit to tmobi-internal/tokscale that referenced this pull request Jun 30, 2026
…unhoyeo#746)

* docs: sync EN/ja/ko/zh-cn for unreleased clients + breaking flag removal

Pre-v3.2.0 documentation sync for the unreleased range (v3.1.3..main):

- junhoyeo#465: replace the now-false legacy per-client flag notices in ja/ko/zh-cn
  (they claimed the removed flags still work) with a v3.2.0 breaking-change
  migration note; add the same note to README.md (which had none).
- junhoyeo#728: document the MiniMax Token Plan subscription source (distinct from the
  MINIMAX_API_KEY row) in all locales; port the entire Subscription Usage
  section into ja/ko/zh-cn (was English-only).
- junhoyeo#718: add the Jcode table row + detail section to the locales missing them.
- junhoyeo#726: document the TOKSCALE_FM_DEBUG env var in all locales.
- junhoyeo#633: add the missing task-attributed report bullet to README.ja Key Features.
- drift: add Junie to the frontend Source-filtering list (all locales).
- junhoyeo#710: fix the MiMo Code repo link (XiaomiMiMo/MiMo -> XiaomiMiMo/MiMo-Code).
- junhoyeo#717: disclose Command Code token usage is estimated (~4 chars/token).

Confidence: medium
Scope-risk: narrow
Directive: ja/ko/zh-cn translations of the ported Subscription Usage section are machine-generated and should get a native-speaker review pass
Not-tested: junhoyeo#713 Antigravity CLI detail section was not added — no English source section exists to port from

* fix(report): feed real session content to the summarizer (junhoyeo#633)

extract_content_for_session unconditionally returned metadata_only_content()
(first_user_message hardcoded None), so the report summarizer never saw any
conversation content and the four real per-client extractors were dead code.

Add content_extractor::extract_session_content, which dispatches to the correct
per-client extractor (opencode/claude/codex/gemini) and falls back to
metadata-only — never erroring or panicking — for unknown clients, missing
candidates, or unreadable/unparseable files. report.rs builds a SessionPathIndex
once (session_id -> transcript file, plus opencode DBs) and threads it through
run_summarizer so each payload carries the real first user message.

Confidence: high
Scope-risk: moderate
Rejected: thread file paths through core's scanner/WikiEntry | too invasive; indexed at the report layer instead
Not-tested: end-to-end opencode/codex/gemini extraction in report.rs (core dispatcher covers claude + all fallback paths; per-client extractors are pre-existing)

* docs: name the breaking release v4.0.0 (was v3.2.0)

The per-client flag removal (junhoyeo#465) is a breaking change, so the next release is v4.0.0, not v3.2.0. Update the migration notes in all four README locales and the main.rs doc comment accordingly.

* fix(report): real Codex/Gemini extraction + (client,session_id) index keying

Addresses automated review feedback on the junhoyeo#633 report-summarizer-content
fix (PR junhoyeo#746). The summarizer still surfaced (none) for normal Codex/Gemini
sessions and could mis-route cross-client session_id collisions.

- content_extractor: parse the current on-disk Codex format (event_msg with
  payload.type == "user_message", text in payload.message) and skip
  harness-injected context blocks (<environment_context>/<system-reminder>/
  <user_instructions>), mirroring sessions::codex.
- content_extractor: Gemini extractor now handles chat-recording JSON
  (messages[].type == "user" / content) and falls back to scanning
  line-delimited JSONL; empty/whitespace user text is treated as not-found.
- extract_session_content: an empty/whitespace first_user_message no longer
  counts as success, so scanning continues to a later candidate with real text.
- report: SessionPathIndex is keyed by (client, session_id) to prevent
  cross-client collisions, and Gemini files are keyed by their in-file
  sessionId (via gemini_session_id_for_file) rather than the filename stem,
  since the wiki entry's session_id is derived from inside the file.
- Added fixture-based regression tests for all of the above.

Constraint: wiki session_id for Gemini comes from the in-file sessionId, not the path stem
Rejected: match any leading '<' for Codex injected blocks | drops legit prompts starting with markup
Confidence: high
Scope-risk: narrow
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants