Skip to content

feat(bash): stream large output to tmpfile with filtering and GC #8953

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cgwalters wants to merge 3 commits into
base: dev
Choose a base branch
from
+555 −55
Open

feat(bash): stream large output to tmpfile with filtering and GC #8953

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f96d269
refactor: move formatSize to shared utility
cgwalters Jan 23, 2026
f5c7f2c
fix(bash): avoid O(n²) memory growth for large command output
cgwalters Jan 23, 2026
667ed6b
feat(bash): add output_filter parameter for build command filtering
cgwalters Jan 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions packages/opencode/src/cli/cmd/tui/routes/session/index.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ import { Prompt, type PromptRef } from "@tui/component/prompt"
import type { AssistantMessage, Part, ToolPart, UserMessage, TextPart, ReasoningPart } from "@opencode-ai/sdk/v2"
import { useLocal } from "@tui/context/local"
import { Locale } from "@/util/locale"
import { formatSize } from "@/util/format"
import type { Tool } from "@/tool/tool"
import type { ReadTool } from "@/tool/read"
import type { WriteTool } from "@/tool/write"
Expand Down Expand Up @@ -1608,6 +1609,14 @@ function Bash(props: ToolProps<typeof BashTool>) {
return [...lines().slice(0, 10), "…"].join("\n")
})

const filterInfo = createMemo(() => {
if (!props.metadata.filtered) return undefined
const total = formatSize(props.metadata.totalBytes ?? 0)
const omitted = formatSize(props.metadata.omittedBytes ?? 0)
const matches = props.metadata.matchCount ?? 0
return `Filtered: ${matches} match${matches === 1 ? "" : "es"} from ${total} (${omitted} omitted)`
})

const workdirDisplay = createMemo(() => {
const workdir = props.input.workdir
if (!workdir || workdir === ".") return undefined
Expand Down Expand Up @@ -1644,6 +1653,9 @@ function Bash(props: ToolProps<typeof BashTool>) {
<box gap={1}>
<text fg={theme.text}>$ {props.input.command}</text>
<text fg={theme.text}>{limited()}</text>
<Show when={filterInfo()}>
<text fg={theme.textMuted}>{filterInfo()}</text>
</Show>
<Show when={overflow()}>
<text fg={theme.textMuted}>{expanded() ? "Click to collapse" : "Click to expand"}</text>
</Show>
Expand Down
8 changes: 1 addition & 7 deletions packages/opencode/src/cli/cmd/uninstall.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import { UI } from "../ui"
import * as prompts from "@clack/prompts"
import { Installation } from "../../installation"
import { Global } from "../../global"
import { formatSize } from "../../util/format"
import { $ } from "bun"
import fs from "fs/promises"
import path from "path"
Expand Down Expand Up @@ -341,13 +342,6 @@ async function getDirectorySize(dir: string): Promise<number> {
return total
}

function formatSize(bytes: number): string {
if (bytes < 1024) return `${bytes} B`
if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`
if (bytes < 1024 * 1024 * 1024) return `${(bytes / (1024 * 1024)).toFixed(1)} MB`
return `${(bytes / (1024 * 1024 * 1024)).toFixed(1)} GB`
}

function shortenPath(p: string): string {
const home = os.homedir()
if (p.startsWith(home)) {
Expand Down
102 changes: 67 additions & 35 deletions packages/opencode/src/session/prompt.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ import { ListTool } from "../tool/ls"
import { FileTime } from "../file/time"
import { Flag } from "../flag/flag"
import { ulid } from "ulid"
import { spawn } from "child_process"

import { Command } from "../command"
import { $, fileURLToPath } from "bun"
import { ConfigMarkdown } from "../config/markdown"
Expand All @@ -44,7 +44,8 @@ import { SessionStatus } from "./status"
import { LLM } from "./llm"
import { iife } from "@/util/iife"
import { Shell } from "@/shell/shell"
import { Truncate } from "@/tool/truncation"
import { Truncate, StreamingOutput } from "@/tool/truncation"
import { spawn } from "child_process"

// @ts-ignore
globalThis.AI_SDK_LOG_WARNINGS = false
Expand Down Expand Up @@ -1475,39 +1476,31 @@ NOTE: At any point in time through this workflow you should feel free to ask the
const matchingInvocation = invocations[shellName] ?? invocations[""]
const args = matchingInvocation?.args

const streaming = new StreamingOutput()

const proc = spawn(shell, args, {
cwd: Instance.directory,
detached: process.platform !== "win32",
stdio: ["ignore", "pipe", "pipe"],
env: {
...process.env,
TERM: "dumb",
},
stdio: ["ignore", "pipe", "pipe"],
})

let output = ""

proc.stdout?.on("data", (chunk) => {
output += chunk.toString()
const append = (chunk: Buffer) => {
const preview = streaming.append(chunk)
if (part.state.status === "running") {
part.state.metadata = {
output: output,
output: preview,
description: "",
}
Session.updatePart(part)
}
})
}

proc.stderr?.on("data", (chunk) => {
output += chunk.toString()
if (part.state.status === "running") {
part.state.metadata = {
output: output,
description: "",
}
Session.updatePart(part)
}
})
proc.stdout?.on("data", append)
proc.stderr?.on("data", append)

let aborted = false
let exited = false
Expand All @@ -1526,33 +1519,72 @@ NOTE: At any point in time through this workflow you should feel free to ask the

abort.addEventListener("abort", abortHandler, { once: true })

await new Promise<void>((resolve) => {
proc.on("close", () => {
exited = true
await new Promise<void>((resolve, reject) => {
const cleanup = () => {
abort.removeEventListener("abort", abortHandler)
}

proc.once("exit", () => {
exited = true
cleanup()
resolve()
})

proc.once("error", (error) => {
exited = true
cleanup()
reject(error)
})

proc.once("close", () => {
exited = true
cleanup()
resolve()
})
})

streaming.close()

if (aborted) {
output += "\n\n" + ["<metadata>", "User aborted the command", "</metadata>"].join("\n")
streaming.appendMetadata("\n\n" + ["<metadata>", "User aborted the command", "</metadata>"].join("\n"))
}

msg.time.completed = Date.now()
await Session.updateMessage(msg)

if (part.state.status === "running") {
part.state = {
status: "completed",
time: {
...part.state.time,
end: Date.now(),
},
input: part.state.input,
title: "",
metadata: {
if (streaming.truncated) {
part.state = {
status: "completed",
time: {
...part.state.time,
end: Date.now(),
},
input: part.state.input,
title: "",
metadata: {
output: `[output streamed to file: ${streaming.totalBytes} bytes]`,
description: "",
outputPath: streaming.outputPath,
},
output: streaming.finalize(),
}
} else {
const output = streaming.inMemoryOutput
part.state = {
status: "completed",
time: {
...part.state.time,
end: Date.now(),
},
input: part.state.input,
title: "",
metadata: {
output,
description: "",
},
output,
description: "",
},
output,
}
}
await Session.updatePart(part)
}
Expand Down
99 changes: 87 additions & 12 deletions packages/opencode/src/tool/bash.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,19 +9,31 @@ import { lazy } from "@/util/lazy"
import { Language } from "web-tree-sitter"

import { $ } from "bun"
import { Filesystem } from "@/util/filesystem"
import { fileURLToPath } from "url"
import { Flag } from "@/flag/flag.ts"
import { Shell } from "@/shell/shell"

import { BashArity } from "@/permission/arity"
import { Truncate } from "./truncation"
import { Truncate, StreamingOutput } from "./truncation"

const MAX_METADATA_LENGTH = 30_000
const DEFAULT_TIMEOUT = Flag.OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS || 2 * 60 * 1000

export const log = Log.create({ service: "bash-tool" })

export interface BashMetadata {
output: string
exit: number | null
description: string
truncated?: boolean
outputPath?: string
filtered?: boolean
filterPattern?: string
matchCount?: number
totalBytes?: number
omittedBytes?: number
}

const resolveWasm = (asset: string) => {
if (asset.startsWith("file://")) return fileURLToPath(asset)
if (asset.startsWith("/") || /^[a-z]:/i.test(asset)) return asset
Expand Down Expand Up @@ -68,6 +80,12 @@ export const BashTool = Tool.define("bash", async () => {
`The working directory to run the command in. Defaults to ${Instance.directory}. Use this instead of 'cd' commands.`,
)
.optional(),
output_filter: z
.string()
.describe(
`Optional regex pattern to filter output. When set, full output streams to a file while lines matching the pattern are returned inline. Useful for build commands where you only care about warnings/errors. Example: "^(warning|error|WARN|ERROR):.*" to capture compiler diagnostics. The regex is matched against each line.`,
)
.optional(),
description: z
.string()
.describe(
Expand All @@ -80,6 +98,16 @@ export const BashTool = Tool.define("bash", async () => {
throw new Error(`Invalid timeout value: ${params.timeout}. Timeout must be a positive number.`)
}
const timeout = params.timeout ?? DEFAULT_TIMEOUT

// Parse output_filter regex if provided
let filter: RegExp | undefined
if (params.output_filter) {
try {
filter = new RegExp(params.output_filter)
} catch (e) {
throw new Error(`Invalid output_filter regex: ${params.output_filter}. ${e}`)
}
}
const tree = await parser().then((p) => p.parse(params.command))
if (!tree) {
throw new Error("Failed to parse command")
Expand Down Expand Up @@ -154,6 +182,8 @@ export const BashTool = Tool.define("bash", async () => {
})
}

const streaming = new StreamingOutput({ filter })

const proc = spawn(params.command, {
shell,
cwd,
Expand All @@ -164,8 +194,6 @@ export const BashTool = Tool.define("bash", async () => {
detached: process.platform !== "win32",
})

let output = ""

// Initialize metadata with empty output
ctx.metadata({
metadata: {
Expand All @@ -175,11 +203,12 @@ export const BashTool = Tool.define("bash", async () => {
})

const append = (chunk: Buffer) => {
output += chunk.toString()
const preview = streaming.append(chunk)
const display =
preview.length > MAX_METADATA_LENGTH ? preview.slice(0, MAX_METADATA_LENGTH) + "\n\n..." : preview
ctx.metadata({
metadata: {
// truncate the metadata to avoid GIANT blobs of data (has nothing to do w/ what agent can access)
output: output.length > MAX_METADATA_LENGTH ? output.slice(0, MAX_METADATA_LENGTH) + "\n\n..." : output,
output: display,
description: params.description,
},
})
Expand Down Expand Up @@ -228,29 +257,75 @@ export const BashTool = Tool.define("bash", async () => {
cleanup()
reject(error)
})

proc.once("close", () => {
exited = true
cleanup()
resolve()
})
})

const resultMetadata: string[] = []
streaming.close()

const resultMetadata: string[] = []
if (timedOut) {
resultMetadata.push(`bash tool terminated command after exceeding timeout ${timeout} ms`)
}

if (aborted) {
resultMetadata.push("User aborted the command")
}

if (resultMetadata.length > 0) {
output += "\n\n<bash_metadata>\n" + resultMetadata.join("\n") + "\n</bash_metadata>"
streaming.appendMetadata("\n\n<bash_metadata>\n" + resultMetadata.join("\n") + "\n</bash_metadata>")
}

// If using filter, return filtered lines
if (streaming.hasFilter) {
const output = streaming.truncated
? `${streaming.filteredOutput}\n${streaming.finalize(params.output_filter)}`
: streaming.finalize(params.output_filter)

return {
title: params.description,
metadata: {
output: streaming.filteredOutput || `[no matches for filter: ${params.output_filter}]`,
exit: proc.exitCode,
description: params.description,
truncated: streaming.truncated,
outputPath: streaming.outputPath,
filtered: true,
filterPattern: params.output_filter,
matchCount: streaming.matchCount,
totalBytes: streaming.totalBytes,
omittedBytes: streaming.omittedBytes,
} as BashMetadata,
output,
}
}

// If we streamed to a file (threshold exceeded), return truncated result
if (streaming.truncated) {
return {
title: params.description,
metadata: {
output: `[output streamed to file: ${streaming.totalBytes} bytes]`,
exit: proc.exitCode,
description: params.description,
truncated: true,
outputPath: streaming.outputPath,
totalBytes: streaming.totalBytes,
} as BashMetadata,
output: streaming.finalize(),
}
}

const output = streaming.inMemoryOutput
return {
title: params.description,
metadata: {
output: output.length > MAX_METADATA_LENGTH ? output.slice(0, MAX_METADATA_LENGTH) + "\n\n..." : output,
exit: proc.exitCode,
description: params.description,
},
} as BashMetadata,
output,
}
},
Expand Down
1 change: 1 addition & 0 deletions packages/opencode/src/tool/bash.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ Usage notes:
- You can specify an optional timeout in milliseconds. If not specified, commands will time out after 120000ms (2 minutes).
- It is very helpful if you write a clear, concise description of what this command does in 5-10 words.
- If the output exceeds ${maxLines} lines or ${maxBytes} bytes, it will be truncated and the full output will be written to a file. You can use Read with offset/limit to read specific sections or Grep to search the full content. Because of this, you do NOT need to use `head`, `tail`, or other truncation commands to limit output - just run the command directly.
- For build commands (make, cargo build, npm run build, tsc, etc.) that produce lots of output where you only care about warnings/errors, use the `output_filter` parameter with a regex like "^(warning|error|WARN|ERROR):". This streams full output to a file while returning only matching lines inline, saving you from having to grep the output afterward.

- Avoid using Bash with the `find`, `grep`, `cat`, `head`, `tail`, `sed`, `awk`, or `echo` commands, unless explicitly instructed or when these commands are truly necessary for the task. Instead, always prefer using the dedicated tools for these commands:
- File search: Use Glob (NOT find or ls)
Expand Down
Loading