Skip to content
This repository was archived by the owner on Jul 30, 2026. It is now read-only.

feat(warp): parse local sqlite token usage - #124

Merged
makoMakoGo merged 7 commits into
personal/local-clientsfrom
codex/release-4.0.5-provenance
Jul 5, 2026
Merged

feat(warp): parse local sqlite token usage#124
makoMakoGo merged 7 commits into
personal/local-clientsfrom
codex/release-4.0.5-provenance

Conversation

@makoMakoGo

@makoMakoGo makoMakoGo commented Jul 5, 2026

Copy link
Copy Markdown
Owner

Summary

  • parse local Warp warp.sqlite usage from the local source adapter, including scanner.extraScanPaths.warp
  • discover Warp databases from the official state-directory locations used by Warp itself, while keeping extra scan roots for cross-system data such as WSL Windows paths
  • allocate Warp and Grok total-only usage with the fixed bucket ratio documented in ADR 0017
  • batch-round Grok total-only rows per source file and bump the Grok parser revision so stale source-message cache shards are rebuilt
  • keep Warp model/provider normalization in the global finalization path, treat Warp local rows as aggregates rather than turns, and parse Warp naive SQLite timestamps as UTC
  • fail fast if a total-only imputation batch exceeds i64::MAX, and remove tracked empty TUI export JSON artifacts

Validation

  • cargo fmt
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test -p tokscale-core --lib
  • cargo test -p tokscale-cli
  • cargo test
  • cargo build --release -p tokscale-cli
  • local installed tokscale --no-spinner --client warp --json --group-by client,provider,model reports 24 Windows Warp rows and 37,204,845 total tokens
  • local installed tokscale hourly --no-spinner --client warp --json reports Warp aggregate rows with turnCount: 0

Summary by CodeRabbit

  • 新功能

    • 新增对 Warp 本地会话的扫描与导入支持,客户端列表与 JSON 输出现在会显示相关路径与存在状态。
    • 扩展了对仅提供总 token 的来源的处理方式,报告中的 token 分布更完整。
  • 文档

    • 更新客户端、配置、定价与 ADR 说明,补充 Warp 和总 token 分配规则的使用方式。
  • Bug 修复

    • 调整了若干客户端的识别与路径发现逻辑,减少本地报告中的遗漏。

Allocate total-only Warp and Grok usage with fixed local-history bucket ratios, document the projection, and remove tracked empty TUI exports.
@coderabbitai

coderabbitai Bot commented Jul 5, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 81ab3c6f-775b-43d7-a2a7-3558a7a76c47

📥 Commits

Reviewing files that changed from the base of the PR and between 380f4a9 and 91f84fd.

📒 Files selected for processing (10)
  • crates/tokscale-cli/src/commands/clients.rs
  • crates/tokscale-cli/tests/cli_tests.rs
  • crates/tokscale-core/src/adapters/warp.rs
  • crates/tokscale-core/src/clients.rs
  • crates/tokscale-core/src/lib.rs
  • crates/tokscale-core/src/local_clients.rs
  • crates/tokscale-core/src/scanner.rs
  • crates/tokscale-core/src/sessions/warp.rs
  • crates/tokscale-core/src/token_imputation.rs
  • docs/clients.md
✅ Files skipped from review due to trivial changes (1)
  • docs/clients.md
🚧 Files skipped from review as they are similar to previous changes (5)
  • crates/tokscale-core/src/lib.rs
  • crates/tokscale-core/src/adapters/warp.rs
  • crates/tokscale-core/src/token_imputation.rs
  • crates/tokscale-core/src/scanner.rs
  • crates/tokscale-core/src/sessions/warp.rs

Walkthrough

本次变更新增Warp本地SQLite解析与适配器接入,加入固定比例的total-only token归因模块并用于Warp与Grok,同时更新本地扫描、CLI行为、模型别名规范和相关文档说明,最后统一升级包版本到4.0.5。

Changes

Warp本地解析与Token固定桶归因

Layer / File(s) Summary
固定比例Token归因核心算法
crates/tokscale-core/src/lib.rs, crates/tokscale-core/src/token_imputation.rs
新增token_imputation内部模块,实现单行与批量的total-only token桶分配。
Warp SQLite会话解析器
crates/tokscale-core/src/sessions/mod.rs, crates/tokscale-core/src/sessions/warp.rs
新增parse_warp_sqlite及其辅助结构和函数,从Warp SQLite中提取会话token用量、时间戳与workspace信息,并生成UnifiedMessage
Warp本地Source Adapter注册
crates/tokscale-core/src/adapters/warp.rs, crates/tokscale-core/src/adapters/mod.rs, crates/tokscale-core/src/message_cache.rs
新增Warp本地适配器并注册到本地适配器列表,ParserId新增Warp变体,相关单测同步更新。
本地客户端配置与扫描器启用Warp
crates/tokscale-core/src/local_clients.rs, crates/tokscale-core/src/scanner.rs, crates/tokscale-core/src/clients.rs, crates/tokscale-cli/src/commands/clients.rs, crates/tokscale-cli/tests/cli_tests.rs
Warp本地扫描路径改为warp.sqlite,扫描器显式识别并扫描Warp路径,CLI客户端输出补充Warp路径展示与测试。
CLI Warp警告移除
crates/tokscale-cli/src/commands/shared.rs, crates/tokscale-cli/src/main_tests.rs, crates/tokscale-cli/src/commands/wrapped.rs
移除Warp相关setup警告逻辑,并删除对应测试断言。
Grok解析改为批量Token归因
crates/tokscale-core/src/adapters/file.rs, crates/tokscale-core/src/sessions/grok.rs
Grok解析改为先收集pending消息再批量归因,相关adapter revision常量和测试断言一并更新。
人类展示名模型ID规范化
crates/tokscale-core/src/model_aliases.rs
新增从展示名推导规范化slug的逻辑,并扩展Claude的max tier qualifier支持。
ADR与客户端文档更新
docs/adr/0011-token-derived-local-cost.md, docs/adr/0017-fixed-token-bucket-imputation.md, docs/clients.md, docs/configuration.md, docs/fork.md, docs/pricing.md, README.md
新增ADR 0017并更新相关文档中Crush、Grok与Warp的token归因说明。

版本号发布提升

Layer / File(s) Summary
版本号统一升级
Cargo.toml, packages/cli-darwin-arm64/package.json, packages/cli-linux-x64-gnu/package.json, packages/cli-win32-x64-msvc/package.json, packages/cli/package.json, packages/tokscale/package.json
workspace、CLI平台包与tokscale包版本从4.0.4更新为4.0.5。

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

Poem

我是一只兔子,轻轻跳过树梢,
看见 Warp.sqlite 里 token 在舞蹈。
固定桶分配,咔哒一声分好,
Grok 与 Warp 都乖乖归到。
版本升到 4.0.5,胡萝卜都知道,🥕
今天的发布,真妙!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed 标题准确概括了本次 PR 的核心变化:为 Warp 增加本地 SQLite token usage 解析。
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/release-4.0.5-provenance

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@makoMakoGo
makoMakoGo marked this pull request as ready for review July 5, 2026 09:19
@codspeed-hq

codspeed-hq Bot commented Jul 5, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 27 untouched benchmarks


Comparing codex/release-4.0.5-provenance (91f84fd) with personal/local-clients (f2e15a6)1

Open in CodSpeed

Footnotes

  1. No successful run was found on personal/local-clients (ddcf95f) during the generation of this report, so f2e15a6 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 32 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread crates/tokscale-core/src/sessions/warp.rs Outdated
Comment thread crates/tokscale-core/src/token_imputation.rs Outdated
Bump the Grok parser revision for the imputed bucket semantics, batch-round Grok total-only rows per source file, and stop counting Warp aggregate rows as turns.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
crates/tokscale-core/src/model_aliases.rs (1)

476-479: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

is_claude_tier_qualifier 目前仅识别 "max"

当前实现足够满足本 PR 范围(仅 "max"),但如果未来出现其他 tier qualifier(如 "pro"、"plus"),需要记得在此处补充。可考虑用一个常量数组承载,便于扩展。

♻️ 可选的扩展写法
+const CLAUDE_TIER_QUALIFIERS: &[&str] = &["max"];
+
 fn is_claude_tier_qualifier(value: &str) -> bool {
-    matches!(value, "max")
+    CLAUDE_TIER_QUALIFIERS.contains(&value)
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/tokscale-core/src/model_aliases.rs` around lines 476 - 479,
`is_claude_tier_qualifier` 目前只匹配单个字符串,扩展性较差;请在该函数中改为基于一个集中维护的常量集合/数组来判断 tier
qualifier,并通过这个符号统一管理 `"max"` 以及未来可能加入的 `"pro"`、`"plus"` 等值,便于后续在
`model_aliases` 中扩展时只改一处。
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/tokscale-core/src/adapters/warp.rs`:
- Around line 18-37: The Warp adapter discovery currently only scans the default
root from ClientId::Warp.local_def(), so it can miss the actual warp.sqlite
locations on macOS/Linux. Update discover() in warp.rs to add platform-specific
candidate roots before calling adapter_discover::scan_roots, alongside the
existing default_root and extra_roots_for_client handling. Keep the discovery
flow intact, but make sure the added roots cover Warp’s real data locations on
those platforms.

In `@crates/tokscale-core/src/sessions/warp.rs`:
- Around line 260-279: The parse_warp_timestamp helper currently interprets bare
timestamps through chrono::Local, which can shift last_modified_at/start_ts
based on host timezone and DST. Update parse_warp_timestamp in warp.rs so the
non-RFC3339 formats are treated as UTC by converting the parsed NaiveDateTime
with UTC semantics instead of Local, and add a test covering the bare timestamp
branch to verify the UTC millisecond result.

---

Nitpick comments:
In `@crates/tokscale-core/src/model_aliases.rs`:
- Around line 476-479: `is_claude_tier_qualifier`
目前只匹配单个字符串,扩展性较差;请在该函数中改为基于一个集中维护的常量集合/数组来判断 tier qualifier,并通过这个符号统一管理 `"max"`
以及未来可能加入的 `"pro"`、`"plus"` 等值,便于后续在 `model_aliases` 中扩展时只改一处。
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9ad267c0-95b8-405f-904d-88ca2d734b40

📥 Commits

Reviewing files that changed from the base of the PR and between ddcf95f and 380f4a9.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (32)
  • Cargo.toml
  • README.md
  • crates/tokscale-cli/src/commands/shared.rs
  • crates/tokscale-cli/src/commands/wrapped.rs
  • crates/tokscale-cli/src/main_tests.rs
  • crates/tokscale-cli/tokscale-export-20260215-203203.json
  • crates/tokscale-cli/tokscale-export-20260215-203814.json
  • crates/tokscale-cli/tokscale-export-20260215-203901.json
  • crates/tokscale-cli/tokscale-export-20260215-203908.json
  • crates/tokscale-core/src/adapters/file.rs
  • crates/tokscale-core/src/adapters/mod.rs
  • crates/tokscale-core/src/adapters/warp.rs
  • crates/tokscale-core/src/lib.rs
  • crates/tokscale-core/src/local_clients.rs
  • crates/tokscale-core/src/message_cache.rs
  • crates/tokscale-core/src/model_aliases.rs
  • crates/tokscale-core/src/scanner.rs
  • crates/tokscale-core/src/sessions/grok.rs
  • crates/tokscale-core/src/sessions/mod.rs
  • crates/tokscale-core/src/sessions/warp.rs
  • crates/tokscale-core/src/token_imputation.rs
  • docs/adr/0011-token-derived-local-cost.md
  • docs/adr/0017-fixed-token-bucket-imputation.md
  • docs/clients.md
  • docs/configuration.md
  • docs/fork.md
  • docs/pricing.md
  • packages/cli-darwin-arm64/package.json
  • packages/cli-linux-x64-gnu/package.json
  • packages/cli-win32-x64-msvc/package.json
  • packages/cli/package.json
  • packages/tokscale/package.json
💤 Files with no reviewable changes (5)
  • crates/tokscale-cli/tokscale-export-20260215-203901.json
  • crates/tokscale-cli/tokscale-export-20260215-203814.json
  • crates/tokscale-cli/tokscale-export-20260215-203203.json
  • crates/tokscale-cli/tokscale-export-20260215-203908.json
  • crates/tokscale-cli/src/main_tests.rs

Comment thread crates/tokscale-core/src/adapters/warp.rs
Comment thread crates/tokscale-core/src/sessions/warp.rs
Interpret Warp naive SQLite timestamps as UTC and fail fast when total-only batch imputation totals exceed i64::MAX.
Discover Warp databases from the official state-directory locations used by Warp itself, and keep extraScanPaths for cross-system roots such as WSL Windows data.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 7 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread crates/tokscale-cli/src/commands/clients.rs Outdated
Keep Warp clients diagnostics sessionsPathExists tied to the displayed sessionsPath while alternate roots remain listed separately.
@makoMakoGo

Copy link
Copy Markdown
Owner Author

/juya review

@juya-review-bot juya-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpenCodeReview found 6 issue(s).

Comment thread crates/tokscale-cli/src/commands/clients.rs
Comment thread crates/tokscale-core/src/local_clients.rs
Comment thread crates/tokscale-core/src/local_clients.rs
Comment thread crates/tokscale-core/src/token_imputation.rs Outdated
Comment thread crates/tokscale-core/src/token_imputation.rs Outdated
Comment thread crates/tokscale-core/src/token_imputation.rs Outdated
Make Warp scan-root platform behavior explicit, cover cross-platform root expectations, and enforce token imputation invariants in release builds.
@makoMakoGo
makoMakoGo merged commit 3e95ee8 into personal/local-clients Jul 5, 2026
10 checks passed
@makoMakoGo
makoMakoGo deleted the codex/release-4.0.5-provenance branch July 5, 2026 11:07
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant