B3: measure and publish the token breakdown (12.3) - #189
Merged
Conversation
Ran the wired instrument: cold build, 2 questions, both arms accepted at 100%. Counted with Microsoft.ML.Tokenizers reporting CountMethod=Exact and Encoding=gpt-4o, so these are token counts rather than the chars/4 estimate this claim could not honestly have been made from. Structured 351 tokens vs 98,409 transcript = 280x Structured 340 tokens vs 100,767 transcript = 296x Hybrid 1,067 tokens vs 98,409 transcript = 92x Hybrid 2,676 tokens vs 100,767 transcript = 38x The structured arm's budget is preferences 179, facts 112, entities 60 -- ten items each. The hybrid arm's is dominated by raw messages, 904 and 2,520 tokens for 15 of them, which is exactly what the comparison is for: one arm answers from extracted memory, the other also carries transcript. The baseline is the full transcript, what a memoryless agent would have to send to answer the same question. A truncated window would flatter the result by measuring against a system that has already given up on remembering. Documented as a verification run and not as a headline. Two questions proves the instrument works end to end and gives an order of magnitude; it says nothing about variance across question types, and the two transcripts here happen to be nearly the same size. Publishing needs 50 questions and a distribution rather than a mean -- a compression ratio is a ratio of two skewed quantities, and its mean is not a typical case. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
There was a problem hiding this comment.
Pull request overview
Adds a performance documentation note that records an end-to-end verification run measuring token costs for “answering from memory” versus sending the full transcript, and links it from the performance docs index.
Changes:
- Added a new doc page describing measured context-token counts for structured vs hybrid memory context assembly, including reproduction guidance.
- Linked the new token breakdown doc from the performance documentation README.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| docs/performance/token-breakdown.md | New documentation page capturing measured token breakdown/compression ratios and how to reproduce the measurement. |
| docs/performance/README.md | Adds a table entry linking to the new token breakdown documentation page. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This was referenced Aug 27, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ran the wired instrument. Cold build, 2 questions, both arms accepted at 100%.
CountMethod=Exact,Encoding=gpt-4o— real token counts, not the chars/4 estimate this claim could not honestly have been made from.Structured budget: preferences 179, facts 112, entities 60 — ten items each. Hybrid's is dominated by raw messages (904 and 2,520 tokens for 15), which is the comparison's whole point: one arm answers from extracted memory, the other also carries transcript.
Baseline is the full transcript — what a memoryless agent must send for the same question. A truncated window would flatter the result by measuring against a system that already gave up on remembering.
Documented as a verification run, not a headline. n=2 proves the instrument works end to end and gives an order of magnitude; it says nothing about variance across question types, and these two transcripts happen to be nearly the same size. Publishing needs 50 questions and a distribution — a compression ratio is a ratio of two skewed quantities and its mean is not a typical case.
🤖 Generated with Claude Code
https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE