feat(cli): prune large bash outputs with SWE-Pruner

[marius-kilocode]

Large test, build, and diagnostic commands can add substantial noisy shell output to every later model request. Experimental SWE-Pruner currently reduces large read and search results, but agent-invoked shell output bypasses that context optimization.

This extends the same task-aware pruning path to bash calls while preserving the existing size gates, hard-truncation exclusion, timeout, and fail-open behavior. Successful pruning replaces both the persisted tool result and the shell metadata preview so all clients consume a consistent reduced result, and completed bash cards expose the existing kept/total indicator.

Focus guidance remains tool-agnostic and explains both when to use pruning and when to preserve full output. The agent selects pruning for specific evidence in potentially large or noisy output, but omits it for broad exploration, complete audits, or output that may be needed later. The skimmer respects concrete evidence and noise criteria, preserves complete local evidence blocks and nearby context needed to interpret them, and retains the full result when most output is relevant. Experimental settings copy now reflects shell support across locales.

[marius-kilocode]

Measured benchmark results

I ran the same failing-test scenario with SWE-Pruner disabled and enabled, using the same main model and exact prompt in fresh sessions.

Metric	Disabled	Enabled	Change
Persisted bash output	7,955 chars	1,419 chars	-82.2%
Persisted bash output	127 lines	37 lines	-70.9%
Context introduced by the tool turn	1,790 tokens	510 tokens	-1,280 (-71.5%)
Total post-tool input context	19,459 tokens	18,453 tokens	-1,006 (-5.2%)

The fixture emitted 90 noisy passing traces and two intentional failures. In both runs, the main model reported both failing test names, both expected values, both received values, and the 90-passed/2-failed summary correctly.

The pruned result kept 35 of 127 lines. An isolated VS Code self-test also confirmed the enabled setting, the SWE-Pruner · kept 35 of 127 lines indicator, the omission marker in expanded shell output, both preserved failures, and no webview console errors.

[marius-kilocode]

Modeled cost and context impact

This is an estimate rather than measured billing. I modeled a representative long coding session using current Kilo catalog rates for Fable as the main model ($10/M input, $1/M cache reads, $12.50/M cache writes) and standard DeepSeek V4 Flash as the skimmer ($0.20/M input, $0.40/M output).

The session contained 92 completed bash calls, 16 non-truncated outputs that met the pruning thresholds, and about 200K eligible output characters. Each eligible output remained in context for an average of roughly 80 later model turns. Existing read/grep pruning had also reduced four read results and one grep result.

Using the measured bash retention ratio and a 60-80% selective-pruning rate for eligible shell output:

Estimate	Result
DeepSeek skimming cost across eligible bash calls	about $0.01
Combined read, grep, and bash saving	about $2.8-$3.5
Projected Fable session without pruning	about $55
Projected cost reduction with prompt caching	about 5-6.5%
Favorable upper bound	about 7.5%
Active context reduction near session end	about 35K-42K tokens

Estimated contribution to total pruning value was roughly 65-71% from bash, 28-34% from read, and less than 1% from grep. Grep output is already pattern-filtered, so the skimmer usually keeps most or all of it.

Prompt caching is included in these percentages. It reduces recurring dollar savings by roughly 10x because historical output is billed at the cache-read rate, but it does not reduce the context-window benefit. Without caching, both the baseline cost and pruning savings would be substantially higher.

The estimate has approximately ±20% uncertainty because destructively pruned read/grep output no longer retains the original text, so those token counts were reconstructed from kept/total line metadata.

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Incremental review of commits since the last pass. This update simplifies the focus-question guidance in SWE-Pruner: the tool-specific description() helper (with a bash-only example) is replaced by a single tool-agnostic DESCRIPTION constant, and SwePruner.extend() drops its now-unused tool parameter. The one call site in session/tools.ts (inside the existing kilocode_change block) and the test suite were updated to match; the removed test cases correctly correspond to the removed tool-specific behavior, and the remaining SwePruner.extend tests still cover both the schema-augmentation and non-object-passthrough paths. No behavioral, security, or fork-hygiene issues found in the changed lines.

Files Reviewed (3 files changed in this update)

• packages/opencode/src/kilocode/swe-pruner.ts
• packages/opencode/src/session/tools.ts
• packages/opencode/test/kilocode/swe-pruner.test.ts

Previous Review Summary (commit ed36326)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit ed36326)

Status: No Issues Found | Recommendation: Merge

Small, well-scoped change extending SWE-Pruner to bash output. The core logic (swe-pruner.ts) reuses the existing generic sweep/partition/parse pipeline for the new bash tool, correctly gates on the pre-existing hard-truncation flag, and syncs the bash-only metadata.output preview field so all clients see a consistent pruned result. The UI change in message-part.tsx mirrors the existing read/grep kept-lines indicator pattern exactly. New tests cover the success path, the hard-truncated bailout, the "keep everything" bailout, and interaction with the read-tool's dynamically-loaded-instructions tail — all exercising the real implementation rather than duplicating logic. The swe_pruner/swe_pruner_model config fields are unaffected (description text only) and remain inside their existing kilocode_change block, so no cloud schema mirror is needed. i18n updates are mechanical, consistent across all 22 locales, and the changeset is user-facing and concise.

Files Reviewed (26 files)

• .changeset/prune-bash-output.md
• packages/kilo-ui/src/components/message-part.tsx
• packages/kilo-vscode/tests/unit/kilo-ui-contract.test.ts
• packages/kilo-vscode/webview-ui/src/i18n/{ar,br,bs,da,de,en,es,fr,it,ja,ko,nl,no,pl,ru,th,tr,uk,zh,zht}.ts
• packages/opencode/src/config/config.ts
• packages/opencode/src/kilocode/swe-pruner.ts
• packages/opencode/src/session/tools.ts
• packages/opencode/test/kilocode/swe-pruner.test.ts

---

_{Reviewed by claude-sonnet-5-20260630 · Input: 26 · Output: 5.9K · Cached: 701.5K}

_{Review guidance: REVIEW.md from base branch main}

[marius-kilocode]

@Drilmo I extended this feature to the bash tool. I think it makes a lot of sense, especially for noisy CI/test output this can safe quite a lot of $. In my testing grep seems to be very close to break even and not very useful.

[marius-kilocode]

Follow-up: trace-based cost analysis

I revisited the earlier modeled cost estimate using traces from a real, multi-step coding workload. This analysis intentionally reports only normalized pruning economics, not workload identity, prompts, task names, run IDs, total spend, or evaluation results.

The earlier estimate projected a 5-6.5% cost reduction, with a favorable upper bound around 7.5%. The trace-based result went in the opposite direction: SWE-Pruner was approximately 1% more expensive all-in for this workload.

Normalized result

Component	Effect relative to observed primary-model cost
Gross primary-model input avoided	about 2.0% saved
DeepSeek V4 Flash skimming	about 0.2% added
Primary-model pruning-control overhead	about 2.9% added
Estimated all-in result	about 1.0% more expensive

The percentages are same-trajectory counterfactual estimates. They isolate direct token economics and do not claim that an enabled and disabled run would follow identical reasoning or tool paths.

Models and pricing

The primary coding model had expensive input and output tokens with a 10x prompt-cache discount: new input was billed at the full input rate, while the same historical context was billed at one tenth of that rate on later turns.

The skimmer was hardcoded to kilo/deepseek/deepseek-v4-flash, the fast/cheap DeepSeek V4 variant, not V4 Pro. The benchmark-time Kilo catalog listed $0.14/M input, $0.0028/M cached input, and $0.28/M output. The skimmer itself was therefore inexpensive and was not the reason savings disappeared. Its exact billed cost was not persisted, so the value above reconstructs token usage from the trace and catalog rates.

What the traces showed

• The primary model supplied context_focus_question on 2,083 read, grep, or bash calls.
• Only 251 outputs met the size gates and invoked the skimmer.
• Only 51 calls produced a smaller persisted result. The other 200 retained the full output because the skimmer returned ALL, produced unusable or too-broad ranges, timed out, or failed open. The trace does not distinguish these outcomes.
• Successful pruning removed an estimated 136K unique primary-model input tokens.
• Because those outputs remained in history, this also avoided about 2.12M later cached-input token occurrences.
• Gross primary-model context savings were approximately 2.0% of observed primary-model cost.
• Estimated DeepSeek spend was only approximately 0.2%.
• The primary-model control overhead was approximately 2.9%, outweighing both values.

Why the earlier model was optimistic

The earlier estimate primarily modeled eligible outputs with a high selective-pruning success rate and emphasized how retained context compounds across later turns. That compounding is real, but prompt caching changes its dollar value:

1. A newly returned tool result is uncached on the first following model request and is billed at the full input rate.
2. On later requests, the same result is normally inside the cached prefix and is billed at the much cheaper cache-read rate.
3. Pruning still preserves context-window capacity, but recurring dollar savings are roughly 10x smaller than they would be without caching.

The earlier estimate also omitted two important primary-model costs:

• The model must generate context_focus_question. In this trace that added approximately 47K output tokens, which are considerably more expensive than DeepSeek input tokens.
• Enabling pruning expands the read, grep, and bash schemas on every model turn, even when no output is large enough to skim. The generated focus questions then also become input context on later turns.

This means a focus question can cost money even when the tool output is below the 2,000 character / 50 line threshold and DeepSeek is never called. If DeepSeek is called but chooses not to prune, the feature pays the primary-model control overhead and skimmer cost while saving no context.

Conclusion

The cost issue is not DeepSeek V4 Flash pricing. It is that the current interface asks the expensive primary model to opt into pruning before output size is known, while the skimmer only produces useful pruning on a minority of eligible calls.

SWE-Pruner still provides a context-window benefit for very large noisy output, and a small number of sessions saved materially. As a general cost optimization, however, this implementation did not pay for itself in the observed workload. Future work should avoid per-call primary-model focus-question generation for small outputs, reduce always-on schema text, improve the skimmer success rate, and persist skimmer usage/outcome metadata so cost can be measured directly rather than reconstructed.

[kirillk]

I read the full paper (https://arxiv.org/pdf/2601.16746).

The short answer: the paper's approach is applicable, but Kilo's implementation diverges from the paper in the single most important dimension — the skimmer — and that divergence, combined with two factors the paper never models, explains why Kilo's economics came out the opposite of the paper's.

What the paper actually does (vs. what Kilo built)

Dimension	Paper (SWE-Pruner)	Kilo's implementation
The skimmer	A trained 0.6B encoder (Qwen3-Reranker-0.6B + CRF pruning head), trained on 61K line-labeled samples	A general-purpose chat LLM (DeepSeek V4 Flash) prompted to emit start-end line ranges
How it runs	One encoder forward pass, scores every line, <100ms even at 8K tokens, negligible cost	Autoregressive API generation, 15s timeout, real (if small) per-call cost
Primary target	read operations (cat/grep/head) — 76.1% of agent tokens	read + grep + bash (this PR added bash)
Focus/goal hint	context_focus_question param, generated per round	Identical — Kilo copied this interface faithfully
Reported result	23–38% token cut, +1.2–1.4pp success, −18–26% rounds	~2% gross context saved, ~1% more expensive all-in

The interface (context_focus_question, optional param, bypass when omitted) is copied exactly from the paper (§3.2, even the same grep_with_pruner pseudocode). Where Kilo departed is the engine behind prune().

Why the paper wins and Kilo doesn't — three concrete gaps

1. The skimmer is a trained model, not a prompted LLM. This is the crux. The paper's 0.6B CRF reranker:

• was trained on 61K samples for exactly this line-selection task → it rarely fails or returns "keep everything";
• runs as a single encoder pass with negligible cost and sub-100ms latency, so there's essentially no "skimmer cost" line item and no timeouts.

Kilo swapped that for DeepSeek prompted to output ranges. That directly produces the 51-of-251 success rate the retrospective flagged: a general chat model, untrained on the task, frequently replies ALL, over-broad ranges, or times out. The paper's skimmer wouldn't have those failure modes. This mismatch alone accounts for both the low success rate and much of the cost problem.

2. The paper never models prompt caching. Its cost tables (Table 1: cost −26.8%) count raw tokens. Kilo's retrospective found that with a 10× prompt-cache discount, the recurring dollar value of removed context shrinks ~10×, because historical tool output is billed at the cache-read rate anyway. So the paper's headline "39% token reduction → proportional cost saving" does not hold under caching — the token-window benefit survives (~2% in Kilo's trace), but the dollar benefit largely evaporates. The paper's benchmark agent (Mini SWE Agent) and setup don't involve the caching regime Kilo runs in.

3. The paper's win is read; Kilo's PR chased bash. The paper's entire motivation (Figure 2) is that read = 76.1% of tokens, execute (test/build) = 12.1%, edit = 11.8%. The validated win is pruning large file reads. Kilo extended pruning to arbitrary bash (test/CI output), which is the execute bucket the paper explicitly shows is small — and the retrospective found bash was where the always-on schema tax + per-call focus-question generation dominated, and where caching killed the savings. Kilo also confirmed grep is break-even (paper's method doesn't emphasize grep either).

So — is it applicable? Yes, but with corrections. Ideas:

Idea 1 (highest leverage): use the paper's actual trained skimmer, not a chat LLM. The authors open-sourced it (github.com/Ayanami1314/swe-pruner, Qwen3-Reranker-0.6B). Running the real trained encoder restores the paper's economics: near-zero per-call cost, no timeouts, and — critically — a much higher success rate than a prompted general model. The cost: you must host a 0.6B model locally or behind the Kilo gateway, which is heavy for a CLI/extension. But every one of the retrospective's complaints (0.2% skimmer cost is fine; the problem is success rate + control overhead) points back to the skimmer being the wrong kind of model.

Idea 2: refocus on read, drop/deprioritize bash and grep. The paper's evidence supports pruning large file reads. Kilo's own trace: read ≈ 28–34% of pruning value (positive), grep < 1% (break-even, already pattern-filtered), bash 65–71% but net-negative under caching. Matching the paper's actual scope (read-heavy) is likely net-positive even with a prompted skimmer.

Idea 3: make pruning post-hoc / size-gated first (the earlier discussion). The paper generates a goal hint every round too, but its reads are almost always large source files, so the "focus question wasted on tiny output" problem barely bites. Kilo's mixed bash workload had 2,083 focus questions but only 251 large enough to skim — 88% wasted primary-model output. Deferring the pruning decision until size is known (hook into the existing Truncate.output big-output branch) eliminates that waste and the always-on schema tax. The paper doesn't need this because of its workload; Kilo does because of its.

Idea 4: frame and measure it as a context-window optimization, not a cost optimization. Under caching, the honest claim is the paper's secondary benefit: the paper also reports −18–26% rounds and +1.2–1.4pp success from a cleaner context, independent of token price. That's the durable value. The retrospective already asks for persisted skimmer outcome metadata (why 200/251 didn't prune) — do that first so you can measure success rate directly and validate whether a trained skimmer moves it.

Bottom line: the paper is sound and directly relevant, but its results depend on (a) a trained 0.6B skimmer and (b) a read-dominated, non-cached measurement regime. Kilo reproduced the interface but substituted a prompted general LLM, extended to the wrong tool (bash), and runs under prompt caching. Fixing the skimmer (Idea 1) is the change most faithful to the paper and most likely to flip the economics; refocusing on read (Idea 2) is the cheapest partial fix without hosting a model.

[marius-kilocode]

@Drilmo I would go ahead and sundown SWE pruning. It hasn't shown to save any costs in practice. In our testing it even increased costs and made sessions more unreliable. Is that fine with you?