chore(release): pre-v4.0.0 doc sync + fix report summarizer content - #746
Conversation
Pre-v3.2.0 documentation sync for the unreleased range (v3.1.3..main): - #465: replace the now-false legacy per-client flag notices in ja/ko/zh-cn (they claimed the removed flags still work) with a v3.2.0 breaking-change migration note; add the same note to README.md (which had none). - #728: document the MiniMax Token Plan subscription source (distinct from the MINIMAX_API_KEY row) in all locales; port the entire Subscription Usage section into ja/ko/zh-cn (was English-only). - #718: add the Jcode table row + detail section to the locales missing them. - #726: document the TOKSCALE_FM_DEBUG env var in all locales. - #633: add the missing task-attributed report bullet to README.ja Key Features. - drift: add Junie to the frontend Source-filtering list (all locales). - #710: fix the MiMo Code repo link (XiaomiMiMo/MiMo -> XiaomiMiMo/MiMo-Code). - #717: disclose Command Code token usage is estimated (~4 chars/token). Confidence: medium Scope-risk: narrow Directive: ja/ko/zh-cn translations of the ported Subscription Usage section are machine-generated and should get a native-speaker review pass Not-tested: #713 Antigravity CLI detail section was not added — no English source section exists to port from
extract_content_for_session unconditionally returned metadata_only_content() (first_user_message hardcoded None), so the report summarizer never saw any conversation content and the four real per-client extractors were dead code. Add content_extractor::extract_session_content, which dispatches to the correct per-client extractor (opencode/claude/codex/gemini) and falls back to metadata-only — never erroring or panicking — for unknown clients, missing candidates, or unreadable/unparseable files. report.rs builds a SessionPathIndex once (session_id -> transcript file, plus opencode DBs) and threads it through run_summarizer so each payload carries the real first user message. Confidence: high Scope-risk: moderate Rejected: thread file paths through core's scanner/WikiEntry | too invasive; indexed at the report layer instead Not-tested: end-to-end opencode/codex/gemini extraction in report.rs (core dispatcher covers claude + all fallback paths; per-client extractors are pre-existing)
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d8d33473f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for (_client, path) in scan.all_files() { | ||
| if let Some(stem) = path.file_stem().and_then(|s| s.to_str()) { | ||
| by_session_id | ||
| .entry(stem.to_string()) |
There was a problem hiding this comment.
For Gemini JSONL sessions (and any transcript whose parser overrides the filename with an in-file session id), the wiki entry's session_id is not necessarily path.file_stem(); parse_gemini_headless_jsonl replaces it from session_id/sessionId in the init line. Indexing only by the stem makes candidates_for return empty for those entries, so the new summarizer path silently falls back to metadata-only and never sends the first prompt. Build this index from the parsed session ids, or carry the source path while populating the wiki, instead of relying on the filename.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in c81dbbe. The index is now built from each Gemini transcript's real session id (parsed via the new gemini_session_id_for_file, which reads the in-file sessionId/session_id from the chat recording or the init line of headless JSONL), with the filename stem kept only as a fallback alias. Lookups by the wiki's session_id now resolve instead of silently falling back to metadata-only. Regression test: build_session_path_index_keys_gemini_by_inner_session_id.
| "codex" => extract_codex_content, | ||
| "gemini" => extract_gemini_content, |
There was a problem hiding this comment.
For sessions where a candidate path is found, routing codex/gemini here still falls back to metadata for the current formats: Codex human input is event_msg with payload.type == "user_message" and text in payload.message, and Gemini chat recordings are line-delimited JSONL (init/result lines), but the extractors only look for top-level user_message/input or a single JSON document with messages. This means the new report summarizer still sees (none) for normal Codex/Gemini sessions; extend the extractors before marking these clients as content-backed.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in c81dbbe. extract_codex_content now parses the current format (event_msg with payload.type == "user_message", text in payload.message) and skips harness-injected context blocks (<environment_context>/<system-reminder>/<user_instructions>), mirroring sessions::codex. extract_gemini_content parses the chat-recording JSON (messages[].type == "user"/content) and also scans line-delimited JSONL as a fallback. Fixture tests added: extract_codex_content_parses_current_event_msg_format, extract_codex_content_skips_only_injected_returns_none, extract_gemini_content_parses_chat_recording_format.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
4 issues found across 6 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
The per-client flag removal (#465) is a breaking change, so the next release is v4.0.0, not v3.2.0. Update the migration notes in all four README locales and the main.rs doc comment accordingly.
…-sync-and-report-fix
junhoyeo
changed the title
…-sync-and-report-fix # Conflicts: # README.md
junhoyeo
changed the title
… keying Addresses automated review feedback on the #633 report-summarizer-content fix (PR #746). The summarizer still surfaced (none) for normal Codex/Gemini sessions and could mis-route cross-client session_id collisions. - content_extractor: parse the current on-disk Codex format (event_msg with payload.type == "user_message", text in payload.message) and skip harness-injected context blocks (<environment_context>/<system-reminder>/ <user_instructions>), mirroring sessions::codex. - content_extractor: Gemini extractor now handles chat-recording JSON (messages[].type == "user" / content) and falls back to scanning line-delimited JSONL; empty/whitespace user text is treated as not-found. - extract_session_content: an empty/whitespace first_user_message no longer counts as success, so scanning continues to a later candidate with real text. - report: SessionPathIndex is keyed by (client, session_id) to prevent cross-client collisions, and Gemini files are keyed by their in-file sessionId (via gemini_session_id_for_file) rather than the filename stem, since the wiki entry's session_id is derived from inside the file. - Added fixture-based regression tests for all of the above. Constraint: wiki session_id for Gemini comes from the in-file sessionId, not the path stem Rejected: match any leading '<' for Codex injected blocks | drops legit prompts starting with markup Confidence: high Scope-risk: narrow
There was a problem hiding this comment.
2 issues found across 3 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="crates/tokscale-core/src/sessions/gemini.rs">
<violation number="1" location="crates/tokscale-core/src/sessions/gemini.rs:119">
P2: Full-file `read_to_string` makes session-id lookup O(file size) memory per file. This can significantly slow report indexing and spike memory on large Gemini JSONL transcripts.</violation>
</file>
<file name="crates/tokscale-core/src/content_extractor.rs">
<violation number="1" location="crates/tokscale-core/src/content_extractor.rs:132">
P3: New Codex injected-message filter duplicates existing Codex session logic instead of reusing one shared helper. This creates divergence risk when tags/rules change.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| .map(|s| s.to_string()) | ||
| }; | ||
|
|
||
| let content = std::fs::read_to_string(path).ok()?; |
There was a problem hiding this comment.
P2: Full-file read_to_string makes session-id lookup O(file size) memory per file. This can significantly slow report indexing and spike memory on large Gemini JSONL transcripts.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At crates/tokscale-core/src/sessions/gemini.rs, line 119:
<comment>Full-file `read_to_string` makes session-id lookup O(file size) memory per file. This can significantly slow report indexing and spike memory on large Gemini JSONL transcripts.</comment>
<file context>
@@ -101,6 +101,52 @@ pub fn parse_gemini_file(path: &Path) -> Vec<UnifiedMessage> {
+ .map(|s| s.to_string())
+ };
+
+ let content = std::fs::read_to_string(path).ok()?;
+
+ if path.extension().and_then(|s| s.to_str()) == Some("jsonl") {
</file context>
| /// reported as the user's first prompt. Mirrors the detection in | ||
| /// `sessions::codex` — matching specific tags (not any leading `<`) avoids | ||
| /// dropping legitimate prompts that happen to start with markup. | ||
| const CODEX_SYSTEM_INJECTED_PREFIXES: [&str; 3] = [ |
There was a problem hiding this comment.
P3: New Codex injected-message filter duplicates existing Codex session logic instead of reusing one shared helper. This creates divergence risk when tags/rules change.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At crates/tokscale-core/src/content_extractor.rs, line 132:
<comment>New Codex injected-message filter duplicates existing Codex session logic instead of reusing one shared helper. This creates divergence risk when tags/rules change.</comment>
<file context>
@@ -124,6 +124,24 @@ pub fn extract_claudecode_content(jsonl_path: &Path, session_id: &str) -> Option
+/// reported as the user's first prompt. Mirrors the detection in
+/// `sessions::codex` — matching specific tags (not any leading `<`) avoids
+/// dropping legitimate prompts that happen to start with markup.
+const CODEX_SYSTEM_INJECTED_PREFIXES: [&str; 3] = [
+ "<environment_context>",
+ "<system-reminder>",
</file context>
…unhoyeo#746) * docs: sync EN/ja/ko/zh-cn for unreleased clients + breaking flag removal Pre-v3.2.0 documentation sync for the unreleased range (v3.1.3..main): - junhoyeo#465: replace the now-false legacy per-client flag notices in ja/ko/zh-cn (they claimed the removed flags still work) with a v3.2.0 breaking-change migration note; add the same note to README.md (which had none). - junhoyeo#728: document the MiniMax Token Plan subscription source (distinct from the MINIMAX_API_KEY row) in all locales; port the entire Subscription Usage section into ja/ko/zh-cn (was English-only). - junhoyeo#718: add the Jcode table row + detail section to the locales missing them. - junhoyeo#726: document the TOKSCALE_FM_DEBUG env var in all locales. - junhoyeo#633: add the missing task-attributed report bullet to README.ja Key Features. - drift: add Junie to the frontend Source-filtering list (all locales). - junhoyeo#710: fix the MiMo Code repo link (XiaomiMiMo/MiMo -> XiaomiMiMo/MiMo-Code). - junhoyeo#717: disclose Command Code token usage is estimated (~4 chars/token). Confidence: medium Scope-risk: narrow Directive: ja/ko/zh-cn translations of the ported Subscription Usage section are machine-generated and should get a native-speaker review pass Not-tested: junhoyeo#713 Antigravity CLI detail section was not added — no English source section exists to port from * fix(report): feed real session content to the summarizer (junhoyeo#633) extract_content_for_session unconditionally returned metadata_only_content() (first_user_message hardcoded None), so the report summarizer never saw any conversation content and the four real per-client extractors were dead code. Add content_extractor::extract_session_content, which dispatches to the correct per-client extractor (opencode/claude/codex/gemini) and falls back to metadata-only — never erroring or panicking — for unknown clients, missing candidates, or unreadable/unparseable files. report.rs builds a SessionPathIndex once (session_id -> transcript file, plus opencode DBs) and threads it through run_summarizer so each payload carries the real first user message. Confidence: high Scope-risk: moderate Rejected: thread file paths through core's scanner/WikiEntry | too invasive; indexed at the report layer instead Not-tested: end-to-end opencode/codex/gemini extraction in report.rs (core dispatcher covers claude + all fallback paths; per-client extractors are pre-existing) * docs: name the breaking release v4.0.0 (was v3.2.0) The per-client flag removal (junhoyeo#465) is a breaking change, so the next release is v4.0.0, not v3.2.0. Update the migration notes in all four README locales and the main.rs doc comment accordingly. * fix(report): real Codex/Gemini extraction + (client,session_id) index keying Addresses automated review feedback on the junhoyeo#633 report-summarizer-content fix (PR junhoyeo#746). The summarizer still surfaced (none) for normal Codex/Gemini sessions and could mis-route cross-client session_id collisions. - content_extractor: parse the current on-disk Codex format (event_msg with payload.type == "user_message", text in payload.message) and skip harness-injected context blocks (<environment_context>/<system-reminder>/ <user_instructions>), mirroring sessions::codex. - content_extractor: Gemini extractor now handles chat-recording JSON (messages[].type == "user" / content) and falls back to scanning line-delimited JSONL; empty/whitespace user text is treated as not-found. - extract_session_content: an empty/whitespace first_user_message no longer counts as success, so scanning continues to a later candidate with real text. - report: SessionPathIndex is keyed by (client, session_id) to prevent cross-client collisions, and Gemini files are keyed by their in-file sessionId (via gemini_session_id_for_file) rather than the filename stem, since the wiki entry's session_id is derived from inside the file. - Added fixture-based regression tests for all of the above. Constraint: wiki session_id for Gemini comes from the in-file sessionId, not the path stem Rejected: match any leading '<' for Codex injected blocks | drops legit prompts starting with markup Confidence: high Scope-risk: narrow
1 participant
Pre-v4.0.0 cleanup: documentation sync + report summarizer fix
Follow-up to the release-readiness review of the unreleased range (
v3.1.3..main). Two independent tracks, separate commits.🔧
fix(report): feed real session content to the summarizer (#633)extract_content_for_sessionalways returnedmetadata_only_content()(first_user_messagehardcodedNone), so thereportcommand's LLM summarizer never saw any conversation content — the four real per-client extractors were dead code.content_extractor::extract_session_contentdispatches to the right per-client extractor (opencode/claude/codex/gemini) and falls back to metadata-only — never erroring or panicking — for unknown clients, missing candidates, or unreadable/unparseable files.report.rsbuilds aSessionPathIndexonce (session_id→ transcript file, plus opencode DBs) and threads it throughrun_summarizer.report.rs, 4 incontent_extractor.rs);cargo test+clippyclean for both crates.📚
docs: sync EN / ja / ko / zh-cnTOKSCALE_FM_DEBUGenv var in all localesXiaomiMiMo/MiMo-Code### Antigravitycache entry, which all locales already have). If a dedicated Antigravity-CLI detail section is wanted, it should be authored in README.md first.canonicalize_modelnon-ASCII panic, feat: add MiMo Code client support #710 MiMotime.createdms assumption, Add JetBrains Junie client support #727 Junie dedup-key collision, feat(report): add multi-backend summarizer and LLM task grouping #633 report UTC-vs-Local date filtering, fix(apple-fm): real on-device generation (programmatic schema) + dlopen packaging so npm arm64 never crashes on macOS <26 #726 title-clustering over-merge, feat(report): native Rust apple-fm summarizer via FoundationModels FFI (drop Python) #721 apple-fm zero FFI test coverage, feat(kiro): add globalStorage support #715 Kiro snapshot date bucketing, feat(jcode): add support for Jcode #718 Jcode first-wins dedup.🤖 Generated with Claude Code
Summary by cubic
Fixes the report summarizer to include real session content and syncs EN/ja/ko/zh-cn docs for v4.0.0, including the
--clientmigration and subscription usage updates. Adds robust Codex/Gemini extraction and safer session indexing to prevent empty or misrouted summaries.Bug Fixes
extract_session_contentwith safe fallback; empty/whitespace prompts no longer count as found.SessionPathIndexkeyed by(client, session_id); Gemini files indexed by in-filesessionId.event_msg.user_messageformat and skip injected<environment_context>/<system-reminder>/<user_instructions>; support Gemini chat-recording JSON and JSONL.Docs
--client/-c.TOKSCALE_FM_DEBUG.Written for commit c81dbbe. Summary will update on new commits.