Skip to content
This repository was archived by the owner on Jul 30, 2026. It is now read-only.

fix(cost): derive local costs from token pricing - #72

Merged
makoMakoGo merged 6 commits into
personal/local-clientsfrom
fix/token-derived-local-cost
Jun 19, 2026
Merged

fix(cost): derive local costs from token pricing#72
makoMakoGo merged 6 commits into
personal/local-clientsfrom
fix/token-derived-local-cost

Conversation

@makoMakoGo

@makoMakoGo makoMakoGo commented Jun 19, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add ADR 0011 for token-derived local report cost semantics.
  • Ignore app/vendor cost fields in local parsers and always derive UnifiedMessage.cost through Tokscale pricing.
  • Drop cost-only rows and disable both Crush and Warp usage rows until token-level sources exist.
  • Bump source-message cache schema to rebuild cached app-reported costs.

Verification

  • cargo test -p tokscale-core
  • cargo test -p tokscale-cli
  • cargo run -p tokscale-cli -- --no-spinner --json --client crush
  • cargo run -p tokscale-cli -- --no-spinner --json --client opencode

Summary by cubic

Local report costs now derive only from token pricing. Parsers ignore app/vendor spend, rows without positive tokens are dropped, and cost-only clients (Crush, Warp) are fully disabled and removed from scanning.

  • Refactors

    • Token-only pricing: compute UnifiedMessage.cost from tokens via crate::finalize_token_priced_messages; strip embedded costs in parsers; normalize token buckets; skip zero-token rows; clean totals (including -0.0) in reports/TUI; offline totals are 0.0 without pricing.
    • Remove cost-only clients: delete crush/warp adapters and session parsers, drop their local scan paths, mark them as not locally parsed, update the scanner to ignore them, remove Warp cache checks and leftover scan/submit paths, and trim related CLI/tests.
  • Migration

    • Bump source-message cache schema to 26; cache rebuilds on first run.

Written for commit 9ea9f30. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

发布说明

  • Bug Fixes

    • 统一成本口径:本地报告的费用仅从 token 使用量派生计算;当缺少正 token 时会丢弃/归零仅费用类信息,且聚合总成本会进行净化。
    • 相关会话/适配流程不再依赖供应商上报的费用字段;改进集成结果一致性。
    • 命令行离线模式下的成本输出断言更严格(缺少定价缓存场景 totalCost 现在精确为 0.0)。
    • 报表中对无 token 的 Warp/Crush 数据不再纳入,并更新本地告警文案。
  • Chores

    • 更新缓存架构版本,触发缓存重建以适配新的定价语义。

@coderabbitai

coderabbitai Bot commented Jun 19, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

将所有本地 session 解析器中对应用/vendor 上报的 cost/credits 字段的读取全部移除,改为在解析后通过新的 apply_token_pricing 函数统一从 token buckets 派生 UnifiedMessage.cost(无定价时为 0.0);聚合层新增 clean_total_cost 归一化;缓存版本升至 26;新增 ADR 0011 文档记录该架构决策。

Changes

Token 派生本地成本重构

Layer / File(s) Summary
核心定价入口与缓存版本升级
crates/tokscale-core/src/lib.rs, crates/tokscale-core/src/message_cache.rs, docs/adr/0011-token-derived-local-cost.md
新增 apply_token_pricingfinalize_token_priced_messages:进入时强制清零 message.cost,无定价时直接返回,有定价时按 token 计算并仅在结果 >0 时写回;finalize 先做 token 归一化与派生字段刷新,过滤无正 token 记录,再调用 pricing。CACHE_SCHEMA_VERSION 升至 26。ADR 0011 文档化规则。
删除 PricingPolicy 并简化 file 适配器
crates/tokscale-core/src/adapters/file.rs
NonCachedFileAdapter 移除 pricing_policy 字段与构造参数,parse 改为调用新的 finalize_token_priced_messagesWARP_ADAPTER 静态初始化被删除。
cache 与 codex 适配器迁移定价入口
crates/tokscale-core/src/adapters/cache.rs, crates/tokscale-core/src/adapters/codex.rs
cache 的三处定价位点(fingerprint 缺失、缓存命中、缓存插入前)与 codex 的消息定价都改为调用 finalize_token_priced_messages,删除私有的 apply_pricing_to_messages 函数。
其他适配器从逐消息定价改为批量处理
crates/tokscale-core/src/adapters/crush.rs, crates/tokscale-core/src/adapters/gjc.rs, crates/tokscale-core/src/adapters/goose.rs, crates/tokscale-core/src/adapters/hermes.rs, crates/tokscale-core/src/adapters/kilo.rs, crates/tokscale-core/src/adapters/kiro.rs, crates/tokscale-core/src/adapters/trae.rs
删除 PricingPolicy 导入,parse 内对消息逐条应用定价的逻辑改为调用 finalize_token_priced_messages;测试期望与 cost 数值同步调整。
适配器注册表与模块导出清理
crates/tokscale-core/src/adapters/mod.rs, crates/tokscale-core/src/sessions/mod.rs, crates/tokscale-core/src/adapters/opencode.rs
LOCAL_SOURCE_ADAPTERS 移除 WARP_ADAPTERCRUSH_ADAPTER;从 sessions/mod.rs 移除 pub mod crush/warp 导出;新增单测验证 Warp/Crush 不注册为本地适配器;opencode 适配器移除 Crush 分支。
Session 解析器移除成本字段:AMP、Codebuff、Cursor、GJC
crates/tokscale-core/src/sessions/amp.rs, crates/tokscale-core/src/sessions/codebuff.rs, crates/tokscale-core/src/sessions/cursor.rs, crates/tokscale-core/src/sessions/gjc.rs
删除结构体中的 cost/credits 字段及对应读取逻辑,统一 token 归一化(各项 max(0))与正性过滤(positive_token_total==0 时跳过),消息初始成本改为 0.0,更新单测期望。
Session 解析器成本清理与 token 过滤:Hermes、Kilo、Mux、OpenClaw
crates/tokscale-core/src/sessions/hermes.rs, crates/tokscale-core/src/sessions/kilo.rs, crates/tokscale-core/src/sessions/mux.rs, crates/tokscale-core/src/sessions/openclaw.rs
SQL 查询删除 estimated/actual cost 列与相关条件,JSON 反序列化删除 cost 字段,UnifiedMessage 初始成本改为 0.0,统一 token 的 max(0) 与正性过滤,更新单测。
Session 解析器指纹清理与 token 过滤:OpenCode、RooCode、Trae
crates/tokscale-core/src/sessions/opencode.rs, crates/tokscale-core/src/sessions/roocode.rs, crates/tokscale-core/src/sessions/trae.rs
JSON 与 SQLite 反序列化删除 cost 字段,SQLite 指纹逻辑移除成本维度,token 统一 clamp 与 positive_token_total 过滤,消息初始成本改为 0.0,更新单测。
Crush 解析器置空实现
crates/tokscale-core/src/sessions/crush.rs
parse_crush_sqlite 改为直接返回空向量,删除所有 SQLite 查询/递归会话树逻辑;测试精简为仅两种空返回场景(缺失数据库、cost-only 记录)。
聚合层成本零值归一化
crates/tokscale-core/src/aggregator.rs, crates/tokscale-core/src/aggregate/engine.rs, crates/tokscale-core/src/aggregate/tui.rs
新增 clean_total_cost 函数确保 cost==0.0 时写入精确 0.0;在 MonthlyReportHourlyReportModelReportcalculate_summary 中应用于 total_cost/max_costsane_cost 条件从 >=0.0 改为 >0.0
lib.rs 定价测试切换与重基线
crates/tokscale-core/src/lib.rs
大量定价测试导入与函数名从 apply_pricing_if_available 改为 apply_token_pricing;无定价场景断言改为 0.0;新增 finalize 过滤、token 正性汇总与饱和行为测试;所有定价相关测试保持 cost 期望不变。
Session 与集成测试重基线
crates/tokscale-core/src/sessions/cline.rs, crates/tokscale-core/src/sessions/kilocode.rs, crates/tokscale-core/tests/codebuff.rs, crates/tokscale-core/tests/gjc.rs, crates/tokscale-core/tests/hermes.rs
cline、kilocode 单元测试 cost 期望从 0.05/0.05 改为 0.0;codebuff 集成测试从 1.25 改为 0.0;gjc/hermes 从"嵌入/回退"语义改为"统一 token 定价",期望值基于 token·price 计算。
CLI 离线与 Warp 告警更新
crates/tokscale-cli/tests/cli_tests.rs, crates/tokscale-cli/src/main.rs, crates/tokscale-cli/src/warp.rs
CLI 离线无定价缓存场景的 totalCost 断言统一改为严格等于 0.0(四个命令);warp_setup_warnings_for_report 简化为仅在显式请求时返回固定告警;删除 has_usage_cache_in_home 辅助函数。

估计代码审查工作量

🎯 4 (Complex) | ⏱️ ~60 minutes

可能相关的 PR

  • makoMakoGo/tokscale#19:同样修改了 amp.rs 的 AMP 会话解析/成本语义,将解析得到的 credits 字段通过定价处理,与本 PR 将 credits 完全移除属同一模块的强相关变更。

🐇 小兔子算账忙,
旧成本字段统统藏。
Token 桶里量一量,
apply_token_pricing 新路畅!
零点零归零点零,
缓存版本 26 上场。✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed PR标题清晰准确地概括了本次变更的核心目标:从本地应用成本转变为基于Token定价推导成本。
Docstring Coverage ✅ Passed Docstring coverage is 80.77% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/token-derived-local-cost

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@makoMakoGo
makoMakoGo marked this pull request as ready for review June 19, 2026 12:44
@makoMakoGo

Copy link
Copy Markdown
Owner Author

/juya review

@codspeed-hq

codspeed-hq Bot commented Jun 19, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 18 untouched benchmarks


Comparing fix/token-derived-local-cost (9ea9f30) with personal/local-clients (51ca010)1

Open in CodSpeed

Footnotes

  1. No successful run was found on personal/local-clients (eef2342) during the generation of this report, so 51ca010 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 34 files

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Comment thread crates/tokscale-core/src/lib.rs
@makoMakoGo

Copy link
Copy Markdown
Owner Author

/juya review

@juya-review-bot

Copy link
Copy Markdown

OpenCodeReview failed before posting review comments. Check bot logs for this pull request and retry after fixing the service configuration.

@makoMakoGo

Copy link
Copy Markdown
Owner Author

/juya review

@juya-review-bot

Copy link
Copy Markdown

OpenCodeReview failed before posting review comments. Check bot logs for this pull request and retry after fixing the service configuration.

@makoMakoGo

Copy link
Copy Markdown
Owner Author

/juya review

@juya-review-bot juya-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpenCodeReview inline comment.

Comment thread crates/tokscale-core/src/adapters/cache.rs

@juya-review-bot juya-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpenCodeReview inline comment.

Comment thread crates/tokscale-core/src/aggregate/engine.rs

@juya-review-bot juya-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpenCodeReview inline comment.

Comment thread crates/tokscale-core/src/aggregator.rs
Repository owner deleted a comment from juya-review-bot Bot Jun 19, 2026
@makoMakoGo
makoMakoGo merged commit a85fb8b into personal/local-clients Jun 19, 2026
9 checks passed
@makoMakoGo
makoMakoGo deleted the fix/token-derived-local-cost branch June 25, 2026 12:11
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant