fix(core): size compression side-query maxOutputTokens to available window - #7962
fix(core): size compression side-query maxOutputTokens to available window#7962zambalee wants to merge 3 commits into
Conversation
…indow The compaction side-query always requested a fixed maxOutputTokens (COMPACT_MAX_OUTPUT_TOKENS=20_000) regardless of how much of the context window the prompt itself already consumed. On smaller-window deployments (e.g. self-hosted vLLM with --max-model-len 65536) this can push promptTokens + 20_000 past the model's context length, causing the backend to reject the request with a 400 before generation even starts — surfaced identically to a genuinely empty model summary as COMPRESSION_FAILED_EMPTY_SUMMARY. Add computeCompactionOutputBudget() to size maxOutputTokens dynamically from the actual remaining window, and bail out early with the existing COMPRESSION_FAILED_OUTPUT_TRUNCATED status (instead of sending a request guaranteed to fail) when the remaining budget is too small to be useful. Reproduced via direct curl payloads against a self-hosted backend using the real compression prompt and a real failed session's history at increasing token-count tiers; confirmed root cause and fix by observing the 400 only appears once promptTokens approaches the window ceiling.
…low-up) computeCompactionOutputBudget's flat COMPACT_OUTPUT_SAFETY_MARGIN (1_000) was not enough headroom for the caller's estimatedPromptTokens to be off from the backend's true tokenizer count. A real production case (KAT-Coder-V2.5-Dev, 2026-07-28, verified live via direct curl against the deployment) fed a 43,549-token hard-tier-rescue estimate into this function while the true prompt was 50,951 tokens (a 7,402-token gap far past the flat margin), computing a budget generous enough to reproduce the exact 400-then-empty-summary failure this function exists to prevent. Switch to `outputClampMargin(window)` (max(10_000, 5% of window)) — the same window-scaled margin already used by the main-turn clamp (clampOutputTokensToWindow) — so both request-construction sites share one safety-margin policy instead of drifting apart. Verified against the real regression by re-sending the failed session's exact history directly to the backend with the new budget: 200 OK, well-formed non-empty <state_snapshot>, finish_reason=stop.
github-actions
Bot
added
the
review/self-reported
|
Thanks for the PR — the linked issue has one of the more thorough reproductions I've seen on this repo. Template: the body doesn't follow the repo template headings ( Problem: observed bug, well-evidenced. Issue #7960 includes curl-level reproduction against a real vLLM backend with exact token counts from two independently-failed sessions (43,549 estimated vs 50,951 true prompt tokens on a 65,536 window). The 400 is structurally guaranteed when Direction: aligned. The main-turn request path already got this treatment in #6556 ( Size: 128 production lines (chatCompressionService.ts: 123+/5-), 214 test lines (chatCompressionService.test.ts: 168+/46-). Well within bounds. Approach: the scope is right — one new pure function ( Risk: no elevated risk signals — no high-risk paths matched. Moving on to code review. 🔍 中文说明感谢贡献！关联 issue 的复现非常详尽。 模板：PR 正文没有使用仓库模板的标题格式，但内容覆盖了模板要求的所有信息——动机、验证步骤、关联 issue。内容到位，不因标题格式阻塞。 问题：已观测到的 bug，证据充分。Issue #7960 包含对真实 vLLM 后端的 curl 级复现，附带两次独立失败会话的精确 token 数（65,536 窗口下，客户端估计 43,549 vs 后端真实 50,951 prompt tokens）。当 方向：对齐。主请求路径已在 #6556 中通过 规模：128 行生产代码，214 行测试代码。在合理范围内。 方案：范围恰当——一个新的纯函数、一个提前退出、调用处一行改动、以及 风险：无升级风险信号。 进入代码审查 🔍 — Qwen Code · qwen3.8-max-preview Reviewed at |
Code reviewIndependent proposal: given the problem (fixed 20,000-token The PR does exactly this. Two non-blocking observations:
Everything else is clean: the TestingFinal CI results for
One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行（取最新一次运行），省略 skipped，失败项排在最前。 CI has not run yet (fork PRs require a maintainer to approve the workflow run). No check results to quote. The author reports 94 tests passing locally including 4 new ones — this is the author's claim, not verified evidence. The sandboxed verification lanes ( 中文说明代码审查：PR 的方案与我的独立提案完全一致——在 side-query 前动态计算输出预算，复用 两个非阻塞观察：(1) 测试：CI 尚未运行（fork PR 需维护者批准工作流）。作者报告 94 个测试通过（含 4 个新测试），这是作者声明，非验证证据。沙箱验证通道需要作者有写权限，本 PR 作者没有。维护者可在一次性容器中检出分支并针对小窗口端点复现，或手动验证算术。 — Qwen Code · qwen3.8-max-preview Reviewed at |
|
Confidence: 4/5 — clean, minimal fix for a well-reproduced bug; two non-blocking nits (dead export, truncation guard heuristic). This is the kind of PR that's easy to review because it does exactly one thing and does it the way the codebase already does it elsewhere. The main-turn request path got window-aware output sizing in #6556; the compression side-query was the last call site still firing a fixed 20,000-token The implementation is three lines of arithmetic in a pure function, reusing Non-blocking nits for a follow-up: the CI note: the fork PR's CI workflow completed with 中文说明这是一个干净的单点修复：主请求路径在 #6556 中已实现窗口感知的输出大小调整，压缩 side-query 是最后一个仍使用固定 20,000 token 实现是三行算术的纯函数，复用 非阻塞建议： CI 说明：fork PR 的 CI 工作流以 — Qwen Code · qwen3.8-max-preview Reviewed at |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅
|
|
|
@qwen-code /verify |
|
Sandboxed verification not started — the PR author does not have write access to this repository, and 沙箱验证未启动 —— 该 PR 作者不具备本仓库写权限，而 |
Maintainer verification round — built and ran a real backend locallyThe triage bot approved this on static review and explicitly left the behavioural claim unverified ("a maintainer could check out the branch in a disposable container and reproduce against a small-window OpenAI-compatible endpoint"). I did that. Verdict: fix-then-merge. The fix is real and I can prove it end to end. But the bail-out gate it adds fires far more often than intended, and I can reproduce it turning a working compaction into a hard failure on four separate paths — including the reactive overflow recovery and the hard-tier rescue on the default 1M-token window. One line fixes it; I verified that too. HarnessIsolated worktree at Everything on the request-construction path is shipped code — 1. The fix works — confirmed at the wire ✅Scenario A replays #7960's exact numbers (65,536 window, client-side estimate 43,549, true backend count 50,978):
That settles it: the dynamic budget prevents the 400, on the real request path. Small factual note: on base the 400 propagates as a thrown error out of 2. Blocker — the bail-out gate over-fires
|
| window | originalTokenCount |
real side-query prompt | base | PR | |
|---|---|---|---|---|---|
| B auto/manual, client over-counts | 65,536 | 55,000 | 43,958 | 200 → COMPRESSED (→12,142) |
budget 536 → bail |
| C reactive overflow recovery | 65,536 | 70,000 | 43,958 | 200 → COMPRESSED (→27,142) |
budget 0 → bail |
| D 32K deployment, auto tier | 32,768 | 28,000 | 12,031 | 200 → COMPRESSED (→17,137) |
budget 0 → bail |
| E 1M window, hard-tier rescue | 1,000,000 | 977,000 | 940,021 | 200 → COMPRESSED (→38,215) |
budget 0 → bail |
Two of these are structural, not edge cases:
- C — reactive overflow recovery.
geminiChat.ts:2646passesoriginalTokenCountOverride = contextOverflow.actualTokens ?? contextOverflow.limitTokens ?? contextWindowSize. WhenevercontextWindowSizematches the backend's real limit — the normal case, and exactly the small-window deployment this issue is about — every one of those branches is≥ windowby construction (that's why the main turn overflowed). Sowindow − promptTokens − margin ≤ −10,000→ budget0→ the last-ditch recovery bails before sending anything. It is the only thing standing between the user and a dead session. - E — 1,000,000-token window (the shipping default for the qwen3-max class). The largest
originalTokenCountstill compressible is 948,000 — 29,000 tokens below thehardthreshold of 977,000. The hard-tier rescue's whole operating range is inside the dead band.
Same shape on 32,768, the deployment class next door to the one in the issue: the compressible range ends at 20,768, 7,085 tokens below the auto threshold of 27,853 — so auto-compaction bails every time it fires.
3. Test coverage — the behaviour change itself is unpinned
Mutation matrix over the 94-test suite, one mutant at a time:
| mutant | outcome |
|---|---|
M1 window-scaled margin → flat COMPACT_OUTPUT_SAFETY_MARGIN |
killed |
M2 call site un-wired: maxOutputTokens back to fixed COMPACT_MAX_OUTPUT_TOKENS |
SURVIVED — 94/94 still pass |
M3 delete the COMPACT_MIN_OUTPUT_TOKENS bail-out branch |
SURVIVED — 94/94 still pass |
M4 drop the Math.max(0, …) floor |
killed |
M5 drop the COMPACT_MAX_OUTPUT_TOKENS ceiling |
killed |
| M6 margin floor 10,000 → 2,000 | killed |
Undoing the entire fix at the call site leaves the suite green. Running the PR's suite against the base source agrees: only 4 of 94 tests fail, and all four are computeCompactionOutputBudget unit tests — the pure function is well covered, the wiring and the new branch are not.
Also worth a look: the test titled "uses a window-scaled margin (not a flat constant): a larger window gets a larger absolute margin" does not catch M1. Both of its budgets clamp to COMPACT_MAX_OUTPUT_TOKENS, so expect(small).toBeLessThanOrEqual(large) is 20000 <= 20000 — it passes with a flat margin too.
Two tests close both gaps (I wrote and verified them: pass on head, kill M2/M3 respectively):
- T1 —
compress()on a 65,536 window withoriginalTokenCount: 43_549assertsrunSideQueryreceivesmaxOutputTokens === 11_987, not 20,000. - T2 — with a budget below the floor, asserts
runSideQueryis never called and the status isCOMPRESSION_FAILED_OUTPUT_TRUNCATED.
4. Suggested change — one line, verified
Size the budget against the payload that is actually about to be sent. It's already computed two lines above the gate:
// packages/core/src/services/chatCompressionService.ts, replacing
// const estimatedPromptTokens = originalTokenCount + pendingToolResultTokenCount;
const estimatedPromptTokens =
estimateContentTokens(slim.slimmedHistory, slimmingConfig.imageTokenEstimate) +
1_000; // compression system prompt + kick-off turn, per the comment at line ~853slim.slimmedHistory already includes the pending tool result and already has images/documents stripped, so it is the side-query prompt. The outputClampMargin stays exactly as it is — that margin is what absorbs chars/4 tokenizer error, which is the real lesson from the incident.
Re-ran all five wire scenarios with this applied: A 200 ✓ · B 200 ✓ · C 200 ✓ · D 200 ✓ · E 200 ✓ — every one COMPRESSED, none over the window. 93/94 of the PR's own tests still pass; the one failure is a fixture artifact ("does not deep-clone full history while compressing" puts a 1 MiB tool output — ≈262K estimated tokens — inside a declared 100K window, so refusing it is arguably the correct new behaviour, but the fixture would need adjusting).
5. Confirmed, non-blocking
COMPACT_OUTPUT_SAFETY_MARGINis dead. Agreeing with the bot:git show 0c0ca5fconfirms it's introduced by this PR, so there is nothing to be back-compatible with, and nothing reads it — only comments mention it.- The truncation guard still compares against
COMPACT_MAX_OUTPUT_TOKENSrather thancompressionOutputBudget. Pre-existingfinish_reasonTODO; not this PR's to fix.
Housekeeping
chatCompressionService.test.ts— 94/94 pass on the head (confirms the author's number).- Adjacent suites — 500/500 pass (
chatCompressionService+geminiChat+tokenLimits+postCompactAttachments). eslintclean,prettier --checkclean on both changed files.
Thanks for the unusually thorough issue write-up — the reproduction numbers in it are what made this harness cheap to build.
中文说明
维护者本地验证轮次 —— 搭建真实后端实测
triage bot 是静态审查通过的，并明确指出行为层面的结论未经验证（"维护者可在一次性容器中检出分支并针对小窗口端点复现"）。我做了这件事。
结论：修复后再合并。 修复本身是真实有效的，我能端到端证明它。但 PR 新增的提前退出（bail-out）判定触发得远比预期频繁，我在四条不同路径上复现出「原本能成功的压缩变成硬失败」，其中一条在默认的 100 万 token 窗口上是无条件发生的。一行代码即可修复，我也一并验证了。
测试环境
隔离 worktree，PR head 051ec23 vs base 0c0ca5f，对接一个假的 OpenAI 兼容后端，严格执行 issue 中的 vLLM 规则：当 prompt_tokens + max_tokens > max_model_len 时在生成前直接返回 400。
请求构造路径上的每一行都是仓库代码 —— ChatCompressionService.compress() → runSideQuery() → BaseLlmClient.generateText(stream) → OpenAIContentGenerator → openai SDK → 真实 HTTP。仅 stub 了 Config 的接线方法，用来把真实 generator 指向 localhost 并固定 contextWindowSize。
1. 修复有效 —— 已在网络层确认 ✅
场景 A 重放 #7960 的精确数字（65,536 窗口，客户端估算 43,549，后端真实计数 50,978）：
发出的 max_tokens |
prompt + output | 后端 | |
|---|---|---|---|
base 0c0ca5f |
20,000 | 70,978 | 400 —— compress() 抛异常 |
PR 051ec23 |
11,987 | 62,965 | 200 → COMPRESSED ✅ |
结论明确：动态预算确实在真实请求路径上避免了 400。
一点事实补充：在 base 上，400 是以抛出异常的形式从 compress() 传出的，并没有落到 COMPRESSION_FAILED_EMPTY_SUMMARY。issue 中"两者不可区分"的说法可能依赖具体的调用方／provider 路径 —— 如果后续要做状态拆分，这点值得注意。
2. 阻塞项 —— 提前退出判定触发过度 ⚠️
该判定把 estimatedPromptTokens = originalTokenCount + pendingToolResultTokenCount 传给 computeCompactionOutputBudget。但 originalTokenCount 是 GeminiChat.lastPromptTokenCount，即主请求的 prompt 大小，其中包含 core system prompt 和完整的工具声明 —— 而 side-query 两者都不发送。本分支上实测：
getCoreSystemPrompt()≈ 5,872 tok，getCompressionPrompt()≈ 896 tok- 工具声明 ≈ 4,072 tok（仅 8 个内置工具；真实会话注册的远不止，还有 MCP）
也就是说判定起点就偏大约 9K+，然后又额外减去 ≥ 10,000 的 outputClampMargin。复现出四个后果（均为：base 压缩成功，PR 一个请求都不发并返回 COMPRESSION_FAILED_OUTPUT_TRUNCATED）：
| 窗口 | originalTokenCount |
真实 side-query prompt | base | PR | |
|---|---|---|---|---|---|
| B auto/manual，客户端高估 | 65,536 | 55,000 | 43,958 | 200 → COMPRESSED (→12,142) |
预算 536 → 退出 |
| C 反应式溢出恢复 | 65,536 | 70,000 | 43,958 | 200 → COMPRESSED (→27,142) |
预算 0 → 退出 |
| D 32K 部署，auto 档 | 32,768 | 28,000 | 12,031 | 200 → COMPRESSED (→17,137) |
预算 0 → 退出 |
| E 100 万窗口，hard 档兜底 | 1,000,000 | 977,000 | 940,021 | 200 → COMPRESSED (→38,215) |
预算 0 → 退出 |
其中两条是结构性的，不是边缘情况：
- C —— 反应式溢出恢复。
geminiChat.ts:2646传入originalTokenCountOverride = contextOverflow.actualTokens ?? contextOverflow.limitTokens ?? contextWindowSize。只要contextWindowSize与后端真实上限一致（常规情形，也正是本 issue 所述的小窗口部署），这三个分支按定义都≥ window（正是因为溢出才走到这里），于是window − promptTokens − margin ≤ −10,000→ 预算0→ 这个最后的兜底恢复在发送任何请求前就退出了。它是用户与「会话彻底卡死」之间唯一的屏障。 - E —— 100 万 token 窗口（qwen3-max 一类模型的出厂默认值）。 仍可压缩的最大
originalTokenCount是 948,000，比hard阈值 977,000 低 29,000。也就是说 hard 档兜底的整个工作区间都落在死区里。
32,768 窗口（与 issue 所述部署类型相邻的一档）情况相同：可压缩区间止于 20,768，比 auto 阈值 27,853 低 7,085 —— 自动压缩每次触发都会直接退出。
3. 测试覆盖 —— 行为变更本身没有测试锁定
对 94 个测试逐一施加变异（mutation）：
| 变异 | 结果 |
|---|---|
M1 窗口缩放 margin → 扁平 COMPACT_OUTPUT_SAFETY_MARGIN |
被杀死 |
M2 调用点还原：maxOutputTokens 改回固定 COMPACT_MAX_OUTPUT_TOKENS |
存活 —— 94/94 仍通过 |
M3 删除 COMPACT_MIN_OUTPUT_TOKENS 提前退出分支 |
存活 —— 94/94 仍通过 |
M4 去掉 Math.max(0, …) 下限 |
被杀死 |
M5 去掉 COMPACT_MAX_OUTPUT_TOKENS 上限 |
被杀死 |
| M6 margin 下限 10,000 → 2,000 | 被杀死 |
在调用点把整个修复撤销，测试套件依然全绿。用 PR 的测试跑 base 源码也印证了这点：94 个里只有 4 个失败，且全部是 computeCompactionOutputBudget 的纯函数单测 —— 纯函数覆盖良好，接线和新分支没有覆盖。
另外值得一看：标题为 "uses a window-scaled margin (not a flat constant)" 的测试并不能捕获 M1。它的两个预算都被 clamp 到 COMPACT_MAX_OUTPUT_TOKENS，因此 expect(small).toBeLessThanOrEqual(large) 实际是 20000 <= 20000 —— 用扁平 margin 一样能通过。
两个测试可以补上这两个缺口（我已编写并验证：在 head 上通过，分别杀死 M2/M3）：
- T1 —— 65,536 窗口、
originalTokenCount: 43_549时，断言runSideQuery收到的是maxOutputTokens === 11_987而非 20,000。 - T2 —— 预算低于下限时，断言
runSideQuery从未被调用且状态为COMPRESSION_FAILED_OUTPUT_TRUNCATED。
4. 建议修改 —— 一行，已验证
按真正即将发送的载荷来计算预算。它在判定上方两行就已经算好了：
// packages/core/src/services/chatCompressionService.ts，替换
// const estimatedPromptTokens = originalTokenCount + pendingToolResultTokenCount;
const estimatedPromptTokens =
estimateContentTokens(slim.slimmedHistory, slimmingConfig.imageTokenEstimate) +
1_000; // 压缩 system prompt + kick-off 轮次，见第 853 行附近注释slim.slimmedHistory 已包含 pending tool result，且已剥离图片/文档 —— 它就是 side-query 的 prompt。outputClampMargin 完全保持不变：吸收 chars/4 分词误差正是它的职责，也正是本次事故的真正教训。
应用后重跑全部五个网络层场景：A 200 ✓ · B 200 ✓ · C 200 ✓ · D 200 ✓ · E 200 ✓ —— 全部 COMPRESSED，无一超窗。PR 自带测试 93/94 通过；唯一失败是 fixture 造成的（"does not deep-clone full history while compressing" 在声明为 100K 的窗口里放了 1 MiB 工具输出 ≈ 26.2 万估算 token，因此拒绝它可以说才是正确的新行为，但该 fixture 需要相应调整）。
5. 已确认的非阻塞项
COMPACT_OUTPUT_SAFETY_MARGIN是死代码。 同意 bot 的判断：git show 0c0ca5f确认它由本 PR 引入，因此不存在向后兼容对象，且无任何代码读取 —— 只有注释提到它。- 截断守卫仍与
COMPACT_MAX_OUTPUT_TOKENS比较而非compressionOutputBudget。这是既有的finish_reasonTODO，不属于本 PR 范畴。
其他检查
chatCompressionService.test.ts—— head 上 94/94 通过（印证作者的数字）。- 相邻测试套件 —— 500/500 通过（
chatCompressionService+geminiChat+tokenLimits+postCompactAttachments）。 - 两个改动文件
eslint与prettier --check均干净。
感谢这份异常详尽的 issue 说明 —— 里面的复现数字正是让这套测试环境搭建成本很低的原因。
…load Maintainer review on PR QwenLM#7962 found the bail-out gate over-fired: it fed originalTokenCount (the main-turn prompt size, which includes the core system prompt + full tool declarations the side-query never sends) into computeCompactionOutputBudget, understating the real headroom by ~9K+ tokens. This turned working compactions into hard failures on four reproduced paths, two of them structural rather than edge cases — the reactive overflow-recovery path and the 1M-token hard-tier rescue, whose entire operating range fell inside the resulting dead band. Size the budget from slim.slimmedHistory instead — the payload actually about to be sent, already computed two lines above the gate — leaving outputClampMargin as the sole absorber of chars/4 estimation error, which is the real lesson from the original incident. Adds the two tests the mutation-testing pass flagged as missing (M2/M3): one pinning the dynamic budget value end-to-end, one pinning the bail-out path when the real payload leaves no room. Also widens the deep-clone regression test's synthetic tool-output payload down from 1 MiB (which the new history-based estimate would now correctly reject as over-budget on that test's 100K window) to 200 KB, which still exercises the same getHistory-not-called assertion without tripping the budget gate.
|
Qwen precheck requires maintainer approval before automated triage/review. Head SHA: Reason:
A maintainer with write access can inspect the PR and manually request a run with |
|
Thanks for the extremely thorough verification — the wire-level A/B/C/D/E matrix and the mutation-testing pass made the actual defect obvious. Pushed a fix at
|
Maintainer verification — round 2 (
|
| arm | outcome |
|---|---|
051ec23 (R1 head) |
bailed, 0 requests — the R1 blocker |
35e7483 (this head) |
HTTP 200, compressed ✅ |
Same for the other three over-count paths I listed. originalTokenCount is out of the gate, so they're all fixed. Suite is 96/96, prettier --check and eslint --max-warnings 0 clean on both files. No disagreement there.
One bookkeeping note: the PR body's Verification section (the live vLLM repro, "43,549 estimated vs 50,951 true") describes the original commits — that run exercised originalTokenCount, which 35e7483 no longer feeds to the gate. The section is worth re-running or re-wording before merge so it still matches the code.
Finding 1 (blocking) — for CJK histories the fix is a no-op; the original 400 comes back verbatim
estimateContentTokens is chars/4. Measured against the real Qwen3-8B tokenizer (tokenizer.json, 11.4 MB, via hf-mirror):
| script | real chars/token | vs chars/4 |
|---|---|---|
| Chinese | 1.80 | under-counts 2.22× |
| English | 4.50 | over-counts 1.12× |
So on a 50K-token Chinese history the gate is handed a ~22.5K "prompt size", min(20000, 65536 − 23532 − 10000) clamps straight back to 20,000 — the exact fixed constant this PR exists to replace — and vLLM rejects the request before generating anything.
I built a fill-level sweep on a 65,536 window. The rescue band is where base 400s but the window still has room for a valid summary; that band is the entire point of #7960. History is calibrated so the backend's true prompt count hits each target, so the English and Chinese rows are the same size on the wire and differ only in script.
Compactions rescued across the band: base 0/10 → this PR 1/10. For Chinese, 0 of 5 — the PR's 400 is byte-identical to base's: same max_tokens=20000, same total, same error string.
To be fair to the PR: below the band (I also swept 30K / 40K / 45K) all three arms compress successfully, so this isn't a regression against main anywhere I could find — it's that the fix doesn't reach the cases it was written for. And these fixtures are single-script; a real session mixes ASCII code with CJK prose, so the error lands between the two rows above and scales with the CJK fraction.
estimateContentTokens' own doc comment calls this out:
char/4is a conservative lower bound (real tokenizers vary ±30%); using it to TRIGGER compaction earlier is safe (false-positive), using it to SKIP compaction is not.
The gate does exactly the latter — it both skips compaction and sizes a wire field.
Worth flagging: your own #7963 fixes this same chars/4 CJK under-count for the output clamp with a 1.5× conservative inflation. #7962 introduces a new chars/4 consumer with no inflation at all — and per the measurement above, even 1.5× wouldn't cover the 2.22× gap.
Finding 2 (blocking) — the 1.12× English over-count bails out compactions that would have fit
Above ~46K the English over-count pushes the computed budget under COMPACT_MIN_OUTPUT_TOKENS and compaction is skipped entirely. At true=50,939 the correct budget is 65,536 − 50,939 − 10,000 = 4,597 — comfortably over the 2,000 floor, and the summary in my fixture needs 168 tokens. Instead: no request, no summary, COMPRESSION_FAILED_OUTPUT_TRUNCATED.
This is a better diagnosis than base's 400 (which is the PR's stated secondary goal, and real value), but the session is equally stuck.
Finding 3 (non-blocking) — the two new tests can't distinguish the change they're named for
M7 — reverting estimatedPromptTokens to R1's originalTokenCount + pendingToolResultTokenCount, i.e. undoing this round's entire change — fails 0 of 96 tests. The fixtures make the two formulas numerically identical:
- "sizes maxOutputTokens from the real side-query payload, not originalTokenCount" — history estimate is
42,549 + 1,000 = 43,549, and the test passesoriginalTokenCount: 43_549. The comment says "deliberately mismatched", but they're equal, so both formulas yield11,987. - "bails out … when the real payload leaves no budget" — new formula gives
60,000, old gives55,000; both fall underCOMPACT_MIN_OUTPUT_TOKENS, so both bail.
M2/M3 are genuinely killed now, as you said — but those pin the call-site wiring, which was already correct at 051ec23. To pin this round's change, one of the two fixtures needs originalTokenCount far from the history estimate (and ideally a CJK fixture).
Also M9 survives: swapping slim.slimmedHistory for the unslimmed sideQueryHistory fails nothing.
(M7's survival isn't a harness artifact — M8, a different edit to the very same expression, is killed by that same test. The anchor region is covered; it just isn't covered for the thing that changed.)
A tested direction
The estimate needs to be accurate, not merely conservative in one direction — the margin can't absorb a 2.22× error. Combining an API-authoritative anchor with a density-aware fallback rescues 10/10 band cells in both languages (third column in the first screenshot):
const MAIN_TURN_ONLY_OVERHEAD_TOKENS = 8_000; // measured below
const estimatedPromptTokens = Math.min(
// originalTokenCount is the API-reported MAIN-turn count: accurate for any
// script, but includes the system prompt + tool decls the side-query omits.
originalTokenCount > 0
? Math.max(0, originalTokenCount - MAIN_TURN_ONLY_OVERHEAD_TOKENS)
: Number.POSITIVE_INFINITY,
// Density-aware fallback: correct on first send, and caps the synthetic
// >= window value the reactive overflow path passes in.
cjkAwareTokens(slim.slimmedHistory) + 1_000,
);min() matters in both directions: the anchor keeps CJK honest, the history term stops the overflow path's synthetic count from bailing (I verified the anchor alone re-breaks that path — same failure as R1).
I measured the overhead constant with the real tokenizer: core system prompt 4,839 + 8 built-in tool declarations 3,669 − compression prompt 705 = 7,803. Real setups carry more tools, so a slightly-low constant errs conservative.
Two caveats, so this isn't taken as drop-in. It fails the two new tests (19,987 vs the expected 11,987, and the second no longer bails) — but both fixtures are pure-ASCII 'x'.repeat(...), whose real token count is nowhere near chars/4, so those expectations encode the assumption under review. And the 1.5/4.5 divisors are Qwen3-specific. Please treat this as evidence the band is reachable rather than a finished patch — the estimator choice is yours, and it should probably be shared with #7963 rather than invented twice.
Harness / reproduction
Fake OpenAI-compatible server enforcing vLLM's pre-generation check (prompt_tokens + max_tokens > max_model_len → 400), with prompt tokens counted by the real Qwen3-8B tokenizer over HTTP rather than a chars/4 approximation.
Unmocked path under test: ChatCompressionService.compress() → runSideQuery() → BaseLlmClient.generateText(stream) → createOpenAIContentGenerator → openai SDK → real HTTP. Only three Config wiring accessors are patched (getContentGeneratorConfig / getContentGenerator / getBaseLlmClient) so a real Config from makeFakeConfig() points at localhost with a pinned contextWindowSize.
- History is calibrated per cell with a separate tiny-
max_tokensgenerateTextprobe, so calibration is identical in arms wherecompress()bails without sending. originalTokenCountis derived astrueSideQueryPromptTokens + 7,803(the measured main-turn-only overhead) rather than hardcoded, so the anchor tracks history size the way a real session does.- Arms:
wt-base(0c0ca5f) ·wt-r1(051ec23) ·wt-r2(35e7483) · candidate patches, one worktree each, identicalnode_modules. - Every mutant proved it landed via
git diff --statbefore its suite run.
中文版本
维护者验证 —— 第 2 轮（35e7483），本地构建并对真实后端运行
先说清楚一点：35e7483 实现的正是我上一轮建议的方案，而我的建议是错的。 它把我发现的「高估」换成了「低估」，而后者对本仓库最大的用户群体伤害更大。
结论：修复后可合并（fix-then-merge）。
这个提交确实修好了什么
R1 的阻塞问题已解决。在反应式溢出恢复路径上（geminiChat.ts:2646 传入的 actualTokens ?? limitTokens ?? contextWindowSize 按构造必然 >= window）：051ec23 静默跳过、零请求；35e7483 HTTP 200，压缩成功 ✅。我列出的另外三条高估路径同样已修复。96/96 测试通过，prettier 与 eslint --max-warnings 0 均干净。
一点记录性的提醒：PR 描述中的 Verification 段落（真实 vLLM 复现，「43,549 估算 vs 50,951 真实」）描述的是最初那两个提交 —— 那次运行走的是 originalTokenCount，而 35e7483 已不再把它喂给闸门。合并前建议重跑或改写该段，使其与代码一致。
发现 1（阻塞）——对中文历史，该修复完全无效，原始 400 原样复现
estimateContentTokens 是 chars/4。用真实 Qwen3-8B tokenizer 实测：中文 1.80 字符/token（低估 2.22×），英文 4.50（高估 1.12×）。
因此 5 万 token 的中文历史，闸门只看到约 2.25 万的「prompt 大小」，min(20000, 65536 − 23532 − 10000) 直接钳回 20,000 —— 正是本 PR 想要取代的那个固定常量，vLLM 随即在生成前拒绝请求。
在 65,536 窗口上做了填充度扫描。救援区间指 base 会 400、但窗口其实还放得下摘要的区间，也正是 #7960 的核心场景。历史经过校准，使后端真实 prompt 计数命中同一目标值，因此中英文行在链路上体积相同，只有文字种类不同。
该区间内成功压缩数：base 0/10 → 本 PR 1/10；中文 0/5 —— PR 的 400 与 base 逐字节相同：同样的 max_tokens=20000、同样的总数、同样的错误串。
也说句公道话：在该区间以下（我另外扫了 30K / 40K / 45K），三个分支都能成功压缩，因此我没有找到相对 main 的任何回退 —— 问题在于这个修复没能覆盖它本要解决的场景。另外这些夹具是单一字种；真实会话是 ASCII 代码与中文叙述混排，误差会落在上表两行之间，并随 CJK 占比变化。
estimateContentTokens 自己的注释已经写明：用它来提前「触发」压缩是安全的，用它来「跳过」压缩则不安全。而该闸门恰恰是后者。
另外值得一提：你自己的 #7963 正是在修 chars/4 对 CJK 的低估（对输出钳制加了 1.5× 保守膨胀）。#7962 却新增了一个完全没有膨胀的 chars/4 使用点 —— 而按上面的实测，即便 1.5× 也补不上 2.22× 的缺口。
发现 2（阻塞）——英文侧 1.12× 的高估会跳过本可成功的压缩
约 46K 以上，英文高估把预算压到 COMPACT_MIN_OUTPUT_TOKENS 之下，压缩被整个跳过。在 true=50,939 时，正确预算是 65,536 − 50,939 − 10,000 = 4,597，远高于 2,000 下限（我的样例摘要只需 168 token）。结果却是：不发请求、无摘要。这比 base 的 400 诊断性更好（也正是本 PR 的次要目标，确有价值），但会话同样卡住。
发现 3（非阻塞）——两个新测试无法区分它们所命名的改动
M7 —— 把 estimatedPromptTokens 还原为 R1 的 originalTokenCount + pendingToolResultTokenCount，即撤销本轮全部改动 —— 96 个测试零失败。 因为夹具让新旧公式数值完全相同：
- "sizes maxOutputTokens from the real side-query payload…"：历史估算
42,549 + 1,000 = 43,549，而测试传入originalTokenCount: 43_549。注释写「deliberately mismatched」，实际两者相等，新旧公式都得11,987。 - "bails out … when the real payload leaves no budget"：新公式
60,000、旧公式55,000，双双低于COMPACT_MIN_OUTPUT_TOKENS，都会跳过。
M2/M3 现在确实被杀死了 —— 但它们钉住的是调用点接线，而那在 051ec23 时就已正确。要钉住本轮改动，需要让某个夹具的 originalTokenCount 明显偏离历史估算（最好再加一个 CJK 夹具）。另外 M9 存活：把 slim.slimmedHistory 换成未精简的 sideQueryHistory，无一测试失败。
（M7 的存活不是测试框架的假象 —— 对同一个表达式做另一处改动的 M8 会被同一个测试杀死。该处是有覆盖的，只是没覆盖到本轮真正改动的那一点。）
一个经过验证的方向
估算需要的是准确，而不只是单向保守 —— 安全边际吸收不了 2.22× 的误差。把 API 权威锚点与字种感知回退取 min() 组合后，该区间中英文 10/10 全部救回（见第一张图第三列）：
const MAIN_TURN_ONLY_OVERHEAD_TOKENS = 8_000;
const estimatedPromptTokens = Math.min(
originalTokenCount > 0
? Math.max(0, originalTokenCount - MAIN_TURN_ONLY_OVERHEAD_TOKENS)
: Number.POSITIVE_INFINITY,
cjkAwareTokens(slim.slimmedHistory) + 1_000,
);min() 两个方向都不可少：锚点保证 CJK 不被低估，历史项则限制住溢出路径传入的合成 >= window 值（我验证过：只用锚点会让该路径重现 R1 的失败）。
开销常量用真实 tokenizer 实测：核心系统提示 4,839 + 8 个内置工具声明 3,669 − 压缩提示 705 = 7,803。真实环境工具更多，常量取偏小反而偏保守。
两点保留，以免被当成可直接套用的补丁。 它会让两个新测试失败（19,987 vs 期望的 11,987；第二个不再跳过）—— 但这两个夹具都是纯 ASCII 的 'x'.repeat(...)，其真实 token 数与 chars/4 相差极远，因此这些期望值本身就编码了正在被审视的那个假设。而 1.5/4.5 两个除数是 Qwen3 专用的。请把它当作「该区间可被救回」的证据，而非成品补丁 —— 估算器怎么选由你决定，并且它大概率应当与 #7963 共用，而不是各写一份。
|
@qwen-code /resolve |
4 participants
Closes #7960.
Summary
chatCompressionService.ts's compression side-query always requested a fixedmaxOutputTokens: COMPACT_MAX_OUTPUT_TOKENS(20,000), regardless of how much of the context window was already consumed by the prompt being compressed. On smaller--max-model-lendeployments this can pushpromptTokens + 20000past the model's context window, so the backend rejects the request with a400before the model ever generates anything — whichchatCompressionService.tsthen reports asCOMPRESSION_FAILED_EMPTY_SUMMARY, indistinguishable from the model genuinely producing an empty summary.Fix
computeCompactionOutputBudget(window, promptTokens)dynamically sizes the requested output budget tomin(COMPACT_MAX_OUTPUT_TOKENS, max(0, window - promptTokens - outputClampMargin(window))), sharing the same window-scaled safety margin (outputClampMargin, fromtokenLimits.ts) already used by the main-turn'sclampOutputTokensToWindow— so the two request-construction sites that decide "how much output can we ask for" no longer drift apart with two different margin policies.COMPACT_MIN_OUTPUT_TOKENS, the compression bails out early with the existingCompressionStatus.COMPRESSION_FAILED_OUTPUT_TRUNCATEDstatus instead of sending a request that's guaranteed to 400.Verification
--max-model-len 65536deployment). With the old fixed-20000 request: both reproduce the400 BadRequestError. With the new dynamically-computed budget: both succeed (200, non-empty well-formed<state_snapshot>, completion tokens well under the computed budget in both cases).chatCompressionService.test.ts: 94 tests passing, including 4 new tests coveringcomputeCompactionOutputBudgetdirectly (one reproducing the exact real-world numbers from the incident: 43,549 estimated vs 50,951 true prompt tokens).Screenshot / Demo
N/A — no user-facing change. This is an internal request-construction fix (how
maxOutputTokensis computed for the compression side-query); there is no new command, flag, or visible CLI behavior to demonstrate. See the Verification section above for before/after reproduction results.Notes for reviewers
COMPACT_OUTPUT_SAFETY_MARGINis kept as an exported constant for reference/back-compat but is no longer used in the budget computation itself (superseded by the window-scaledoutputClampMargin).