Skip to content
This repository was archived by the owner on Jul 30, 2026. It is now read-only.

refactor(core): remove parsed message pipeline - #103

Merged
makoMakoGo merged 1 commit into
codex/issue-92-cache-wrapped-clientsfrom
codex/issue-93-remove-parsed-message
Jul 3, 2026
Merged

refactor(core): remove parsed message pipeline#103
makoMakoGo merged 1 commit into
codex/issue-92-cache-wrapped-clientsfrom
codex/issue-93-remove-parsed-message

Conversation

@makoMakoGo

@makoMakoGo makoMakoGo commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Summary

  • Remove the legacy ParsedMessage / ParsedMessages structs and conversion helpers.
  • Delete the public parse_local_clients path now that local consumers use unified-message folds from P1: wrapped 与 clients 走一次解析和缓存路径 #92.
  • Move the remaining local-loader tests onto UnifiedMessage plus count_local_client_messages.
  • Keep parse_all_messages_with_pricing test-only while runtime paths use the cached unified fold.

Closes #93

Depends on #101.

Validation

  • rg -n "ParsedMessage|ParsedMessages|parse_local_clients|parsed_to_unified|unified_to_parsed|filter_parsed_messages" crates --glob '*.rs' returns no matches
  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets
  • cargo test -p tokscale-core
  • cargo test --workspace
  • cargo build --release -p tokscale-cli

Local memory check

Measured on WSL2 Ubuntu 24.04 with live local client data. Base is the previously installed local tokscale binary before replacement; PR is target/release/tokscale from this branch. Each command had one warmup run, then 3 measured runs with /usr/bin/time -f "%M,%e"; RSS values are %M / 1024.

Command Base median RSS PR median RSS RSS change Base median time PR median time
tokscale --no-spinner clients --json 42.0 MiB 41.7 MiB -0.7% 8.69s 8.13s
tokscale wrapped --year 2026 --agents --no-spinner --output ... 145.0 MiB 143.5 MiB -1.0% 9.13s 8.61s
tokscale wrapped --year 2026 --clients --no-spinner --output ... 139.2 MiB 140.7 MiB +1.1% 8.95s 8.34s

The wrapped commands exited successfully and produced non-empty PNG output. This PR removes the legacy parsed-message API path; local runtime RSS is effectively unchanged against the #101 cached-fold baseline.


Summary by cubic

Removed the legacy parsed-message pipeline in tokscale-core. Local loading now uses the cached unified message fold and consolidates on UnifiedMessage.

  • Refactors

    • Deleted ParsedMessage, ParsedMessages, and helpers (unified_to_parsed, parsed_to_unified, filter_parsed_messages).
    • Removed public parse_local_clients; tests now load via the unified fold plus count_local_client_messages.
    • Marked parse_all_messages_with_pricing as test-only (#[cfg(test)]).
    • Updated tests to use UnifiedMessage (including tokens.* fields) and renamed them to “local message loader” cases.
  • Migration

    • Replace parse_local_clients and ParsedMessage with:
      • UnifiedMessage for data.
      • count_local_client_messages for per-client counts.
      • Runtime consumers should call load_usage_data or load_aggregated_views_with_pricing (both use the cached unified fold).

Written for commit fd5be33. Summary will update on new commits.

Review in cubic

@coderabbitai

coderabbitai Bot commented Jul 2, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 6fb80984-67b9-4806-abd2-3ee428df746e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/issue-93-remove-parsed-message

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@makoMakoGo makoMakoGo left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我看了 #93 的决策记录和这次 diff。删除 ParsedMessage / ParsedMessagesparse_local_clientsparsed_to_unified / unified_to_parsed / filter_parsed_messages 的方向是对的,也符合 #93 的范围:统一到 UnifiedMessage,不再保留第二套消息形态。

但 ready 前有一个需要处理的问题:#103 依赖 #101,而 #101 当前 head 后续新增了 client_count_bucket,把 cc-mirror/* 归到 ClientId::Claude,并加了 test_client_count_sink_attributes_cc_mirror_variants_to_claude#103 当前分支里的 ClientCountSink 仍然直接走 ClientId::from_str(&message.client),这会让 cc-mirror/* 的本地 client count 被丢掉,相当于回退 #101 的修复。

建议先 rebase/merge #101 当前 head,然后保留 client_count_bucket 和对应测试。除此之外,我没有看到删除 ParsedMessage 管线本身的问题。

@makoMakoGo
makoMakoGo force-pushed the codex/issue-93-remove-parsed-message branch from 2181f2d to fd5be33 Compare July 3, 2026 08:34

@makoMakoGo makoMakoGo left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

重新看过当前 head fd5be33e,上次指出的问题已处理。

确认点:

  • #103 现在基于 #101 最新 head 7011c325,只 ahead 1 commit。
  • client_count_bucket 已保留,cc-mirror/* 会归到 ClientId::Claude
  • ClientCountSink 已恢复通过 client_count_bucket 计数。
  • headless Codex count 也已按 folded message_count.max(0) 计数。
  • test_client_count_sink_attributes_cc_mirror_variants_to_claude 和 folded headless Codex count 测试都在。
  • ParsedMessage 管线删除方向保持干净,没有看到新的回归点。

LGTM from code review. 当前 workflow 还在 in_progress,合并前等 CI 结果即可。

@makoMakoGo
makoMakoGo marked this pull request as ready for review July 3, 2026 09:02

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Re-trigger cubic

@makoMakoGo

Copy link
Copy Markdown
Owner Author

/juya review

@juya-review-bot juya-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpenCodeReview found 1 issue(s).

Comment thread crates/tokscale-core/src/lib.rs
@makoMakoGo
makoMakoGo merged commit e4a1ed3 into codex/issue-92-cache-wrapped-clients Jul 3, 2026
4 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant