feat: truncate oversized inputs to fit instead of failing the summary - #1
Merged
Merged
Conversation
When an assembled prompt exceeds the model's input-token limit, the engine threw "Input token count exceeds model input limit" and aborted the run. Replace that hard failure with a head+tail truncate-to-fit: keep the document's opening and its conclusion, drop the middle, and proceed — a degraded summary of a truncated document beats no summary. - Add a shared clipHeadAndTail helper (head+tail windowing, boundary-aware, output bounded to the budget, head-only fallback for tiny budgets) and route the --max-extract-characters content budget through it so the closing section survives explicit truncation too. - Add fitUserTextToInputTokenBudget in the summary engine: shrink the prompt to the token budget (re-checking the real token count) and note it under --verbose instead of erroring. - Update the input-limit tests to assert truncate-and-proceed rather than the old throw; add unit tests for clipHeadAndTail and the fit helper. Inspired by NousResearch/hermes-agent#54843 (truncate-and-store over LLM summarization), adapted to a summarizer where the summary is the product. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
When an assembled prompt exceeds the model's input-token limit, the summary engine threw
Input token count exceeds model input limitand aborted the run. This replaces that hard failure with a head+tail truncate-to-fit: keep the document's opening and its conclusion, drop the middle, and proceed. A degraded summary of a truncated document beats no summary — which is exactly the inputs a summarizer is most useful for (long PDFs, multi-hour transcripts).Changes
clipHeadAndTailhelper (src/content/link-preview/content/cleaner.ts): head+tail windowing (~75/25), snaps cuts to nearby sentence/line boundaries, inserts a compact omission marker, guarantees output ≤ budget, and falls back to a head-only clip for tiny budgets.clipAtSentenceBoundaryis left unchanged.applyContentBudgetnow uses it, so the--max-extract-characterspath keeps the closing section instead of clipping head-only.fitUserTextToInputTokenBudget(src/run/summary-engine.ts): shrinks the prompt to the token budget (re-checking the realgpt-tokenizercount, tightening a few times if the estimate runs long) and logs the truncation under--verboseinstead of erroring.clipHeadAndTailand the fit helper.Because
buildTaggedPromptplaces<content>last, a head+tail clip of the assembled prompt naturally preserves the instructions (head) and the closing tag (tail), dropping only the content's middle.Why head+tail (not head-only)
For a summary the closing material — conclusions, results, recommendations — is often the highest-value passage, and the existing truncation everywhere was head-only (
slice(0, n)), discarding it.Verification
bun run check(format + lint + perf guard + full coverage) green: 1346 passed, 0 failed;bun run typecheckclean.Context
Inspired by NousResearch/hermes-agent#54843 (which replaced per-page LLM summarization with truncate-and-store in their agent's web-extract tool), adapted to a summarizer where the summary is the deliverable — so the borrowed idea is the truncation mechanism (graceful degradation + head+tail), not their "drop the summarizer" thesis.
🤖 Generated with Claude Code