Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions data/devlog/sliced-stream-suffixes-retain-whole-message.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
title: Sliced Stream Suffixes Retain The Whole Message
summary: |
A streamed reply that turns every chunk into its own timeline row by slicing the new suffix out of the accumulated message keeps every intermediate snapshot of that message alive. V8 shares a slice's storage with its source, so retained heap grows with the square of the stream length and a single 128 KB part can hold over 100 MB after garbage collection. Copy the suffix out instead of slicing it.
date: '2026-10-16 04:20:00 +1000'
tags:
- Node.js
- Memory
- Gotcha
draft: false
icon: Bug
---

A long reasoning stream takes the whole daemon down with `SIGABRT` and every agent on the host is interrupted. The reply itself is modest - one 128,000-byte part - yet a standalone reproduction of the timeline store retains about 123 MiB after an explicit garbage collection, and the projection it feeds contains one row. Double the message length and the retained memory roughly quadruples. This is [getpaseo/paseo#5882](https://github.com/getpaseo/paseo/issues/5882), fixed in [PR #6262](https://github.com/getpaseo/paseo/pull/6262).

## Why it happens

Streaming code often builds rows like this:

```js
const suffix = message.slice(offset) // shares `message`'s backing store
timeline.push({ text: suffix }) // and now every row pins the whole message
```

V8 can share a string's storage with the string it was sliced from. A 64-byte suffix is not 64 bytes of heap - it is a window onto the whole accumulated message as it stood when the chunk arrived. Keep every suffix in a growing store and the retained memory sums over all of those full-length snapshots, which is where the square comes from. A 128 KB stream of 64-byte chunks is 2,000 rows, each pinning up to 128 KB.

Anything that slices per delta is exposed. The fix here went into the one path that slices on every streamed chunk. Suffixes cut once per completed part hold at most one copy and are fine.

## The fix

Emit an independent copy of each suffix rather than a view onto the accumulated string, and copy in UTF-16 code units so a surrogate pair split across two chunks still arrives intact. After the change, memory grows with the length of the text, not its square.

If you are writing your own timeline or token stream display, the check is simple. Anywhere a row holds `message.slice(n)` where `message` keeps growing, that row is a snapshot of the whole message. Copy, or hold offsets and slice at render time from one live string.

Reported with the standalone reproduction by a contributor on the linked issue.
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
title: Stream End Without a Terminal Leaves the Run Running
summary: |
A provider event subscription that ends normally without its terminal event leaves the run marked running forever, because the consumer sees a clean stream end and returns without settling anything. A real run sat on a compaction step for nearly seven hours and later steering failed with "provider turn is not active". Treat a stream that ends without the terminal marker as an ingestion error.
date: '2026-10-16 04:30:00 +1000'
tags:
- Async
- State
- Gotcha
draft: false
icon: Bug
---

A delegated run stays marked "running" on a compaction step for six hours and forty-one minutes. The compaction it was stuck on finished in under three minutes and the native turn completed after it, but nothing ever recorded that. Steering the run fails with

```
provider turn is not active
```

and restarting the app is what finally clears it. The run never persisted a terminal state because none was ever seen. This is [pingdotgg/t3code#19127](https://github.com/pingdotgg/t3code/issues/19127), fixed in [PR #19131](https://github.com/pingdotgg/t3code/pull/19131).

## Why it happens

In this protocol the turn ends when a terminal event arrives, not when the stream closes. The consumer loop iterates its subscription, and when the subscription ended without that event it closed cleanly and returned. A normal end of stream looked like a normal end of work. Nothing failed the run, the attempt or the run-owned subagents, and the provider thread was never returned to idle, so every later call saw a run that was still current.

The hole is deterministic - any subscription that ends early, drops or routes wrong walks into it - and the symptom is a run that is only dead in the app's mind.

## The fix

Stream end without a terminal marker now raises an ingestion error instead of returning, and the existing failure finalisation does the rest. The shape of that change looks like this:

```js
for await (const event of subscription) {
if (event.type === 'turn.terminal') return finalize(event)
ingest(event)
}
// without this line the run stays "running" forever
throw new IngestionError('provider stream ended without a terminal')
```

The pattern generalises to any job whose completion is an event rather than the end of its stream. A loop that exits normally on stream end has to ask whether the completion marker ever arrived, and fail the job when it did not. Otherwise the cleanest possible failure mode is the worst one - a state machine convinced work is still in progress that nothing will ever finish.
1 change: 1 addition & 0 deletions data/devlog/sweep-status.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,4 @@ This file is `.md`, not `.mdx`, so contentlayer does not pick it up as an entry.
| 2026-10-15 | johnnyhuy/* and echohello-dev/* PRs and releases, getpaseo/paseo issues PRs and releases, openclaw/openclaw issues PRs and releases, openchamber/openchamber issues PRs and releases, pingdotgg/t3code issues PRs and releases, deepseek-ai/deepseek-harness releases, Next.js / Tailwind / ContentLayer2 / shadcn-ui releases | Five resolutions from after the previous row, one of them merged minutes before that row landed and missed there. OpenRouter ~anthropic/*-latest alias refs never get Anthropic cache markers - cacheRead stays 0 and every turn re-bills the full prompt at roughly eight times the per-turn input cost of the exact ref - fixed by normalising alias classification before cache boundaries are removed (openclaw #172642 / PR #172652). A first-touch macOS TCC consent prompt on a headless Mac parks the process in a syscall, so the run stays "running" with zero output and Stop does nothing until someone dismisses the dialog at the machine - grant Full Disk Access or pre-approve the protected folders (t3code #19776). git pathspecs resolve against the process working directory and not the repo root, so a repo-root-relative path fed to git run from a subdirectory selects nothing and an otherwise correct diff comes back with empty hunks (paseo #6867 / PR #6882). Inline markdown images at paths with spaces or non-ASCII fail with ENOENT because the percent-encoded destination is read from disk without decoding, and bare spaces end the URL under CommonMark (paseo #6871 / PR #6877, landed minutes before the previous row merged and missed there). Cursor free-plan sign-in loops back to Not authenticated because 403 plan_required is folded into a generic catalog failure and the fresh credential is cleared during verification - use a paid account or CURSOR_API_KEY (t3code #19766, duplicate of open #15075). Dropped - a Linux browser-import XDG_CONFIG_HOME report closed with no traceable fix; a quadratic code-block re-parse report closed with the repair still outstanding; a slow memory-CLI capture report closed into an open canonical issue; a --duration 0 validation fix too narrow for an entry. No new stable releases anywhere - Next.js 16.4.1 / 15.5.28, Tailwind 4.3.3, ContentLayer2 0.5.8, shadcn@4.21.5, paseo v0.11.2, OpenChamber v2.3.3, openclaw 2026.10.3 and pidex 2026.10.14-2 unchanged since the row above; openclaw 2026.10.5-beta.3, t3code nightlies and deepseek-harness alpha builds skipped as pre-releases. | tilde-aliases-skip-prompt-cache-markers, invisible-tcc-dialogs-block-headless-macs, git-pathspecs-resolve-from-cwd-not-repo-root, markdown-image-paths-need-decoding, free-plan-sign-in-loops-on-plan-required |
| 2026-10-15 | johnnyhuy/* and echohello-dev/* PRs and releases, getpaseo/paseo issues PRs and releases, openclaw/openclaw issues PRs and releases, openchamber/openchamber issues PRs and releases, pingdotgg/t3code issues PRs and releases, deepseek-ai/deepseek-harness releases, Next.js / Tailwind / ContentLayer2 / shadcn-ui releases | Four resolutions from after the previous row. A streamed reply that only mentions "tool call" or "function call" in ordinary prose latches the trace scrub and re-scans the whole accumulated reply on every delta - 83 KB streams in 4.3 s and the cost quadruples with each doubling - so the scrub now activates only on bounded trace-line heads (openclaw #172629 / PR #172631). Node's idle keep-alive close races pooled socket reuse on an MCP loopback server, so a tool call dies ECONNRESET before the handler and non-idempotent tools are lost with no signal to the caller - fixed with Connection: close on every loopback response (openclaw #172871 / PR #172980). A CLI bundled inside a macOS .app is invisible to a daemon that only walks PATH - name the binary in the provider's command (paseo #6990). Generated branch names like main/patch-1 nest under an existing ref and git refuses to create the ref in either direction - the allocator suffixes the conflicting segment instead (paseo #6830 / PR #6829). Dropped - a doctor manifest re-parse stall closed with the claim released and no fix proven (openclaw #172809); a managed update rejecting held deleted-agent stores landed a fix but stays too product-internal (openclaw #172740 / PR #172975); a context meter showing a previous model's window is a single display path (paseo #6795 / PR #6814); an ACP agent creation sending a mode to a provider that offers none is a duplicate of an already-merged fix awaiting release (paseo #6992 / #4747); TIFF attach rejection is a duplicate of an open gap (t3code #19800); paired environment themes are intended behaviour (t3code #19842); remaining openclaw UI and messaging reports too product-specific. No new stable releases anywhere - Next.js 16.4.1 / 15.5.28, Tailwind 4.3.3, ContentLayer2 0.5.8, shadcn@4.21.5, paseo v0.11.2, OpenChamber v2.3.3, openclaw 2026.10.3 and pidex 2026.10.14-2 unchanged since the row above; openclaw 2026.10.5-beta.4, t3code nightlies and deepseek-harness alpha builds skipped as pre-releases. | tool-prose-latches-quadratic-trace-scrub, idle-keep-alive-close-races-socket-reuse, app-bundled-clis-are-missing-from-daemon-path, git-refs-cannot-nest-under-existing-branches |
| 2026-10-16 | johnnyhuy/* and echohello-dev/* PRs and releases, getpaseo/paseo issues PRs and releases, openclaw/openclaw issues PRs and releases, openchamber/openchamber issues PRs and releases, pingdotgg/t3code issues PRs and releases, deepseek-ai/deepseek-harness releases, Next.js / Tailwind / ContentLayer2 / shadcn-ui releases | One release and three resolutions from after the previous row. OpenClaw 2026.9.33 refreshes the extended-stable line - the September code plus critical security updates, updates that finish on pnpm 12, npm and Windows instead of rolling back or wedging, work that survives a Gateway restart, and a streamed reply starting with a reply tag no longer re-parsing the whole accumulated reply on every delta (openclaw 2026.9.33, #166781 / PR #166782). T3 Code stable 0.0.45 accepts an opencode 2.x serverUrl because the version check is skipped whenever serverUrl is set and the only gate is a minimum version that 2.0.29 passes, so prompts 404 mid-thread and resume silently starts a different session - pin opencode 1.x (1.14.19) or take a nightly with the v2 driver (t3code #19881 / PR #14239). A credential helper bound by gh auth setup-git into a global gitconfig outside the container's persistent volumes disappears on every image update while the token itself survives, and the local commit succeeding lets the job report success while the remote falls behind - bind the helper on a persistent path and verify the remote ref (openclaw #173059). A Claude question permission answered in any shape but updatedInput.answers keyed by the question resolves as allow, delivers nothing, and the request is deleted before validation so the retry fails with nothing pending - PR #6936 makes malformed answers retryable and has merged but is not in v0.11.2 yet (paseo #6916 / PR #6936). Dropped - a remote model catalog that gained an input "video" enum value breaks every released client's refresh and recovery depends on the catalog owner regenerating compatible artifacts (openclaw #173027); a claude-stdio startup deadlock when more than 16 pre-init hook events arrive before initialize has a source-proven mechanism but no landed fix (openclaw #173002); a nested session.link parameter object destroyed in the tool path is a duplicate with the fix still open (openchamber #4959 / #4591); a context meter reading 0% on cached-token-heavy sessions is a display path already tracked as a duplicate (paseo #7013 / #4019). Nothing new from johnnyhuy/* or echohello-dev/* beyond the devlog PRs. No new stable releases elsewhere - Next.js 16.4.1 / 15.5.28, Tailwind 4.3.3, ContentLayer2 0.5.8, shadcn@4.21.5, paseo v0.11.2, OpenChamber v2.3.3, pidex 2026.10.14-2 and openclaw 2026.10.3 unchanged since the row above; openclaw 2026.10.5-beta.4, t3code nightlies and deepseek-harness alpha builds skipped as pre-releases. | openclaw-2026-9-33-brings-critical-fixes-to-extended-stable, server-url-skips-the-major-version-gate, container-updates-drop-global-git-config, malformed-question-answers-consume-the-request |
| 2026-10-16 | johnnyhuy/* and echohello-dev/* PRs and releases, getpaseo/paseo issues PRs and releases, openclaw/openclaw issues PRs and releases, openchamber/openchamber issues PRs and releases, pingdotgg/t3code issues PRs and releases, deepseek-ai/deepseek-harness releases, Next.js / Tailwind / ContentLayer2 / shadcn-ui releases | Four resolutions from after the previous row. A streamed part turned into its own timeline row by slicing the new suffix out of the accumulated message keeps every intermediate snapshot alive, because V8 shares a slice's storage with its source - 128 KB retains about 123 MiB after garbage collection and the heap grows with the square of the stream length - fixed by emitting an independent copy of each suffix in UTF-16 code units (paseo #5882 / PR #6262). On Windows, closing a validation runtime leaves PyInstaller children holding the executable and unpacked DLLs, so disposable-profile cleanup fails with EBUSY or EPERM and a fully successful install reports "Check disk space and directory access" - fixed by killing the process tree before the pipes close and retrying removal on Busy (t3code #18551 / PR #18472). A provider event subscription that ends without its terminal event closes the consumer cleanly and leaves the run running forever, so a real run sat on compaction for nearly seven hours and steering failed with "provider turn is not active" - stream end without a terminal now raises an ingestion error and failure finalisation settles the run (t3code #19127 / PR #19131). A composer draft keyed on runtime, directory and session is saved under the old directory and loads empty when a worktree move re-keys the session, dropping whatever was typed - the move now migrates the draft (openchamber #4833 / PR #4974). Paseo v0.11.3 shipped with a sponsor opt-in provider, search in the Add provider catalog and a plugin-provider Remove fix - too thin for an entry. Dropped - a spawn broker dead-PID map closed as already implemented with no new fix (openclaw #173222); parent-session cost excluding subagent usage drew a still-broken comment after its fix (openchamber #3275); a preview-host timeout eviction and a Cloudflare 403 in a third-party browser runtime closed without a landed fix (t3code #19925 / #19924); a Cursor Auto-review mapping lost in an SDK migration is too product-specific (t3code #19711); remaining openclaw update-failure and open reports too product-internal. Nothing new from johnnyhuy/* or echohello-dev/* beyond the devlog PRs. No new stable releases elsewhere - Next.js 16.4.1 / 15.5.28, Tailwind 4.3.3, ContentLayer2 0.5.8 and shadcn@4.21.5 unchanged since the row above; Next.js canaries, openclaw 2026.10.5-beta.4, t3code nightlies, deepseek-harness alpha builds and paseo pre-releases skipped as pre-releases. | sliced-stream-suffixes-retain-whole-message, temp-cleanup-fails-while-children-hold-handles, stream-end-without-a-terminal-leaves-the-run-running, worktree-moves-drop-the-composer-draft |
32 changes: 32 additions & 0 deletions data/devlog/temp-cleanup-fails-while-children-hold-handles.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
---
title: Temp Cleanup Fails While Children Hold Handles
summary: |
An installer that validates a runtime and then deletes its temporary profile can report a failure after a perfectly successful run, because the removal hits EBUSY on Windows while spawned children still hold the executable and unpacked DLLs. Kill the process tree before the pipes close and retry the removal, and a green validation stops looking like a disk fault.
date: '2026-10-16 04:25:00 +1000'
tags:
- Windows
- Processes
- Gotcha
draft: false
icon: Bug
---

The download and extraction succeed. Validation succeeds. Then the install fails at "Checking the downloaded runtime" with a message telling you to check disk space and directory access, and there is nothing wrong with either. This is [pingdotgg/t3code#18551](https://github.com/pingdotgg/t3code/issues/18551), fixed in [PR #18472](https://github.com/pingdotgg/t3code/pull/18472).

## Why it happens

The validation step runs the runtime from a disposable temporary profile. That runtime is itself a launcher, and its children - a PyInstaller-packaged process here - keep handles on the executable and the unpacked DLLs. Closing the validation wrapper does not wait for those children to exit, so when the installer removes the temporary directory the removal fails with `EBUSY` or `EPERM`. The run is marked failed and the error blames the disk.

Windows is where this bites hardest. A file with open handles cannot be deleted, and the delete fails immediately rather than waiting. A retry loop alone is nearly enough - the children exit on their own shortly after - but the timing window is wide enough to fail a whole install.

## The fix

Terminate the process tree before the wrapper's pipes close, while the launcher still exists for the tree kill to walk, then retry the cleanup:

```bash
taskkill /T /F /PID <launcher-pid>
```

The upstream fix does exactly that, and retries disposable-profile removal up to ten times at 100 ms intervals for the `Busy`, `PermissionDenied` and `Unknown` errors that linger while handles drain.

The general shape matters more than the product. Any installer, test harness or sandbox that spawns children out of a temp directory owns their exit before it owns the delete. Join child exit first, remove second, and never let a cleanup failure overstate itself as a corrupt download.
Loading