Skip to content

fix(coding-agent): single-dump kernel snapshots - #1945

Merged
snimu merged 2 commits into
mainfrom
fix/kernel-snapshot-double-dump
Sep 1, 2026
Merged

snimu merged 2 commits into
mainfrom
fix/kernel-snapshot-double-dump

Conversation

@snimu

@snimu snimu commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Saving kernel state — the namespace snapshot that lets a resumed session keep its variables — buffered the whole pickled namespace in memory up to three times over before writing it to disk: each variable was pickled into a dict of blobs, that dict was pickled again into a second in-memory buffer, and that buffer was copied once more for the file write. A snapshot runs shortly after every successful cell, so long sessions with sizeable variables repeatedly spike to roughly 4x the payload in transient RSS. This is the kernel-side memory spike tracked in ENG-5819 (kernel processes ballooning to 1.3-1.9GB and getting OOM-killed).

The fix deletes the in-memory aggregate stage: the dict of blobs now pickles directly into the staged temp file through a small size-capped writer (the same class replaces the old per-variable buffer). The on-disk format, the manifest, the max_bytes cap semantics, and the overflow fallback that keeps the largest prefix that fits are all unchanged; overflow retries rewind and rewrite the temp file instead of allocating fresh buffers. No host or protocol changes.

Measured on a 250MB namespace (25 x 10MB variables, fresh process per side, 3 snapshots, ru_maxrss):

peak RSS snapshot transient overhead latency
before (74c8d39) 1242 MB 963 MB (~3.9x payload) 0.11-0.20 s
after 529 MB 253 MB (~1.0x payload) 0.03-0.14 s

The remaining ~1x overhead is the per-variable blob dict, which pickling the aggregate genuinely needs. The snapshot still runs synchronously on the kernel loop; moving it off-loop is out of scope here because that ordering is what keeps the namespace stable while it pickles.

Validation: prime-agent-runtime python -m unittest test.test_repl 95/95 with no test changes (the cap and overflow-fallback behaviors are already pinned there); coding-agent kernel suites (repl-kernel-state-roundtrip against the real runtime, execute/abort/shutdown/startup/protocol-corruption, npm run test:kernel) green; root npm run check green.

Linear: ENG-5819


Note

Medium Risk
Touches the kernel snapshot write path used after every successful cell; behavior is intended to be equivalent but mistakes could truncate or mis-record state under size caps.

Overview
Fixes ENG-5819 by changing how _snapshot_state writes the aggregate namespace payload so the kernel no longer holds multiple full-size serialized copies in RAM on every debounced snapshot.

The in-memory _SnapshotBuffer path is replaced with _CappedWriter, which enforces per-dump byte limits while streaming dill output into a sink (still BytesIO for per-variable blobs, the staged temp file for the full payload dict). Aggregate overflow handling keeps the same largest-prefix binary search and skip reasons, but retries seek/truncate/redump on that temp file instead of re-pickling into fresh buffers.

Snapshot caps, manifest fields, on-disk format, and host protocol are unchanged; this is a memory/latency optimization inside the REPL kernel.

Reviewed by Cursor Bugbot for commit bffb62d. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Stream kernel snapshot payload directly to temp file instead of memory

  • Replaces the in-memory _SnapshotBuffer (BytesIO wrapper) in prime-agent-runtime/src/rlm/repl.py with _CappedWriter, a capped writer that streams writes to an arbitrary sink while enforcing a byte limit.
  • _snapshot_state now serializes the payload directly into the staged temp file using _CappedWriter instead of calling serialize() to build a full in-memory copy first. This reduces kernel memory spikes during namespace snapshots.
  • The prefix-truncation binary search rewrites candidate prefixes to the temp file via seek(0)/truncate(); bytes_written is sourced from the writer's running count rather than len(serialized).
  • Behavioral Change: _SnapshotBuffer is removed; any code importing it directly will break. Per-variable serialization now uses io.BytesIO + _CappedWriter instead of _SnapshotBuffer.

Macroscope summarized bffb62d.

The kernel snapshot pipeline serialized the aggregate payload dict into an
in-memory buffer, copied it out with getvalue(), and only then wrote the
file - on top of the per-variable blob dict, a ~3.9x-payload transient
memory spike on every debounced snapshot (measured 963MB overhead for a
250MB namespace). The aggregate dict now pickles straight into the staged
temp file through a size-capped pass-through writer (measured 253MB
overhead, ~1x payload); the strict max_bytes contract and the
largest-fitting-prefix fallback are preserved by rewinding the temp file
between prefix attempts. _SnapshotBuffer is replaced by the same
_CappedWriter over a plain BytesIO for the per-variable dumps.
@snimu
snimu requested a review from xeophon September 1, 2026 12:31
@snimu
snimu merged commit 1b5830f into main Sep 1, 2026
24 checks passed
@snimu
snimu deleted the fix/kernel-snapshot-double-dump branch September 1, 2026 12:48
olety added a commit to oneiron-dev/prime-agent that referenced this pull request Sep 1, 2026
Takes upstream's event-driven supervisor roster ledger + push (PrimeIntellect-ai#1897, PrimeIntellect-ai#1900),
direct TUI<->worker transport (ENG-5817), daemon startup/recovery hardening
(PrimeIntellect-ai#1929, PrimeIntellect-ai#1909), single-dump kernel snapshots (PrimeIntellect-ai#1945), empty-draft eviction
(PrimeIntellect-ai#1946), rlm_child_update suppression (PrimeIntellect-ai#1944), bash-skill preview (PrimeIntellect-ai#1911).

Fork laws re-expressed on the roster architecture:
- stable-target follow-up honesty kept (capability proof via worker hello,
  target_unavailable never not_found when ownership unproven)
- schema revision 26 (union of fork rev-24 stable-target + upstream
  rev-24/25 roster+transport); digest minted by the repo's own algorithm
- summary freshness reuse + single-flight + staleness + root-omission
  rejection restored on upstream's refresh pull
- adoption/recovery never fails a live worker on a slow or root-omitting
  catalog: get_state root seed + stale mark + bounded background rehydration
- repl.py keeps fork prune-on-aggregate-overflow
- delete handlers keep fork persistence reporting; eviction fence test keeps
  the stronger two-worker contention variant

Known test debt (deferred to post-Wave cleanup per owner): roster-era fixture
migrations in daemon-supervisor-monitor (2), plus un-triaged failures in
package-command-paths, agent-session-recursion, daemon-runtime-stress,
4600-supervisor-singleton, 4603-worker-recovery, 4606-update-restart-
coordinator, agents-view-roster. Production laws preserved; failures are
fixture-era artifacts or mechanism assertions to rewrite.
ketema added a commit to ketema/prime-agent that referenced this pull request Sep 1, 2026
- Direct session transport between TUI and worker (ENG-5817, PrimeIntellect-ai#1926)
- Event-driven supervisor agent roster with push subscriptions (PrimeIntellect-ai#1897, PrimeIntellect-ai#1900, PrimeIntellect-ai#1895)
- Hardened daemon startup, recovery ownership, and worker launch diagnostics (PrimeIntellect-ai#1929, PrimeIntellect-ai#1918)
- Python REPL runtime single-dump snapshots and bash preview tool (PrimeIntellect-ai#1945, PrimeIntellect-ai#1911)
- Non-blocking RLM subagent deletion and snapshot update suppression (PrimeIntellect-ai#1954, PrimeIntellect-ai#1944)
- Saved catalog loading on Agents View open (PrimeIntellect-ai#1960)
- Advanced Anthropic prompt caching marker across tool results (PrimeIntellect-ai#1927)
- TUI process replacement on update and empty draft eviction (PrimeIntellect-ai#1631, PrimeIntellect-ai#1946, PrimeIntellect-ai#1920)
paralin pushed a commit to paralin/prime-agent that referenced this pull request Sep 2, 2026
* fix(coding-agent): single-dump kernel snapshots (ENG-5819)

The kernel snapshot pipeline serialized the aggregate payload dict into an
in-memory buffer, copied it out with getvalue(), and only then wrote the
file - on top of the per-variable blob dict, a ~3.9x-payload transient
memory spike on every debounced snapshot (measured 963MB overhead for a
250MB namespace). The aggregate dict now pickles straight into the staged
temp file through a size-capped pass-through writer (measured 253MB
overhead, ~1x payload); the strict max_bytes contract and the
largest-fitting-prefix fallback are preserved by rewinding the temp file
between prefix attempts. _SnapshotBuffer is replaced by the same
_CappedWriter over a plain BytesIO for the per-variable dumps.

* chore(coding-agent): strip narrative comments from the snapshot dump path
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants