cmux-journal: one append-only journal per daemon, with group commit and ack after durable (R-DURABILITY S1) by lawrencecchen · Pull Request #33919 · manaflow-ai/cmux

lawrencecchen · 2026-11-03T14:01:36Z

Summary

Stage 1 of the R-DURABILITY rewrite (design: handoff/rewrites/R-DURABILITY/DESIGN.md, STATUS 9088). Today each daemon has many durable write paths (three SQLite databases, per-terminal tail and snapshot files, per-chat logs, session.json, about 30 small JSON files), each with its own writer, order, sync rule and recovery. That split is the common cause of REOPEN-F3 (#31643 and its 7 review rounds), acpmux g2 (#29311, #30120), the session.json vs log split, and the acked-but-unsynced changes the design lists (E9, E12-E15). The rewrite moves every acknowledged change of each daemon into ONE append-only journal with group commit, and a reply goes out only once its record is synced.

This PR adds only the journal crate, cmux-tui/crates/cmux-journal. Nothing uses it yet; the next stages wire it into acpmux (event logs, then chat metadata) and the cmux-tui daemon, each behind one switch.

Journal::append copies one record into memory and returns its LSN: no I/O, no wait. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch, or 1 s later.
Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux). The first barrier sync starts a 1 s window; then one batch (or one sync alone, when no batch comes) uses F_FULLFSYNC. So a power loss takes back at most 1 s of acked batches, always a suffix. On macOS a failed F_BARRIERFSYNC / F_FULLFSYNC falls back to fsync only when the volume does not support the call (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY); a call that a signal interrupts (EINTR) is made again; any other error is a failed sync.
Sync markers: after each successful sync that covered records, the writer appends a marker entry (no payload) that vouches for them; a seal and a rotation sync the last marker too.
Recovery is the one start path (clean exit, crash, kill -9, OS crash). It fully syncs the last segment, checks every entry's CRC and LSN, and hands the records after the caller's compaction point to the caller's replay. It trusts the last segment only up to its last marker: the records after it may sit in pages that a failed sync dropped, so recovery writes them again into a new segment (written and fully synced as <name>.seg.new, then renamed, folder synced) before open returns. The journal always goes on in a segment this open made. A torn tail stays in its file. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted). No LSN is ever used twice: every LSN of every segment, live or quarantined, counts at every open; a header that does not read is read for its LSNs from its first record.
A failed write is cut back to its size before the write and written again in the same segment. A failed sync is never retried on that segment (after a failed fsync the pages can be clean and lost): every record after the durable LSN goes into a new segment. A panic on the writer thread closes the journal: appends fail (Closed), waits end (WriterEnded), listeners are dropped.
Storage is the file system seam. The sim feature's SimStorage simulates a page cache over a disk (4 KiB pages), Linux write-back errors (a failed sync drops pages that a later sync never writes), one ordered log of names, process crashes, OS crashes and power losses, also inside recovery. Later stages use it for store-level crash tests of the daemons.

OWNERS.md gets a row for the crate.

Review fixes (independent review FIX at `89293fb`, comment 6610468202)

finding	fix	regression test (red on the code before the fix)
F1 a crash inside quarantine reuses quarantined LSNs	quarantined LSNs count at every open; an unfinished quarantine is reported again	`a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn` (every operation of the open is a crash point once)
F2 a damaged last header reuses its LSNs	the segment is read for its LSNs from its first record; the bytes after a cut are scanned too	`a_damaged_header_in_the_last_segment_never_reuses_its_lsns`
F3 a listener panic hangs waiters	a guard closes the journal when the writer thread ends	`a_panic_on_the_writer_thread_closes_the_journal`
F4 a restart after a failed sync trusts the failed segment	sync markers; recovery writes the unvouched records again; never goes on in an old segment	`a_restart_after_a_failed_sync_keeps_every_later_ack`, and in the crash sim `a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record`
F5 the sim could not find F1 or F4	`SimStorage` rewritten (page cache, write-back errors, partial failed writes, ordered names, crash inside recovery, aimed faults)	4 sim self-tests
F6 one fd per segment at open; no retirement	one file open at a time; retirement is an S2 API (before the first consumer)	`recovery_holds_at_most_two_files_open`
F7 two docs contradict the code	format.rs and lib.rs corrected

Also: the EINTR retry (a93684835c4).

Re-review fixes (FIX at `3498975`, comment 6620113993)

finding	fix	regression test (red on the code before the fix)
G1 the copy re-read the file, unchecked; an evicted dropped page gave zeros under a vouching marker	recovery stages the bytes its checked read returned	`recovery_copies_the_records_it_checked_when_the_cache_drops_them` (the sim now evicts dropped pages after a read)
G2 a crash at the open's last step lost the damage report	the damage goes to `DAMAGE` (whole, synced) before the new segment hides it; every open reports it until `Journal::clear_damage`	the F1 regression asserts the report (`lost_through` 40) at every crash point
G3 a symlinked state folder failed every open	folder syncs follow symlinks (no `O_NOFOLLOW`)	`a_journal_in_a_symlinked_folder_opens_and_round_trips` (real file system)
G4 every open read every quarantined segment	the quarantine is read only while no live segment is numbered above it	`an_open_after_a_finished_quarantine_reads_no_quarantined_segment`
G5 `sim.rs` over the function limit	split: `sim/machine.rs` (the machine model)
G6 one tracked arithmetic hit	`checked_div`

Third-review fixes (FIX at `cb893c6`, comment 6621681018)

finding	fix	regression test (red on the code before the fix)
G7 the quarantine scan was skipped behind a newest segment whose header is damaged; a header-less quarantined file was scanned from too low a base	skip only after a segment whose header checked out is numbered above the quarantine; scans start from the running floor	`a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn` (A), `a_damaged_newest_segment_after_a_quarantine_reports_its_records` (B)
N1 one loss reported twice (after a crash, or by two steps of one open)	a loss inside another is recorded and reported once, as the larger range	the F1 regression asserts that no reported range lies inside another
N2 docs	`DAMAGE` named; long doc lines wrapped
recovery.rs near its limit	split: `recovery/damage.rs`, `recovery/scan.rs`

Fourth-review fixes (FIX at `215c8f0`, comment 6622740919)

finding	fix	regression test (red on the code before the fix)
G8 a recovery copy is numbered above the quarantine but its records start below the quarantined LSNs; with the empty segment after it damaged, nothing that is read carried the floor	each quarantined file is named `<segment>.<n>.<floor>`; every open takes the quarantine's floor from the names alone (no file read, no header trusted)	`a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn` (the reviewer's three steps)
N3 an absorbed loss dropped its files' names	the kept loss takes the absorbed one's files and reason	G7 B and the F1 test assert that every quarantined file is named by a reported loss
N4 DAMAGE was written after the moves	the quarantine is planned (measured and named), the loss recorded, then the files move	the F1 test asserts, at every crash point, that the loss through 40 names the damaged segment

Fifth-review fixes (FIX at `9c47594`, comment 6623794646)

finding	fix	regression test (red on the code before the fix)
G9 an open that gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which the recovery copy then hides from every later open) kept that floor only in the header of the empty segment it made; with that header damaged, the next open gave those LSNs out again (`clear_damage` also removes the DAMAGE line, and torn records are not damage at all)	the open makes the empty file `FLOOR.<lsn>` and syncs the folder before it makes a segment; every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine names and this open's moves carry their floors already: no file is made for them	`a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again`, `torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again`

Testing

Two test files (crate cmux-journal):

src/cmux_journal_tests.rs (filter cmux_journal_tests, 35 tests): the record and segment format; group commit (400 acked records from 8 clients with a 20 ms sync take fewer than 200 syncs); an ack is not durable before its sync ends; listeners in order; stream records; an idle journal does no storage operation; the full-sync window; backlog and size limits; seal; the lock; rotation; failed writes and syncs; a failure at any of 11 operations keeps every acked record; torn tails; quarantine; compaction points; replay errors; the review regressions above; a round trip on the real file system, also in a symlinked folder.
src/crash_sim_tests.rs (filter crash_sim_tests, 9 tests): 200 seeds each of process crashes, OS crashes, power losses and a mix, over several epochs per seed (appends, a crash at a random operation or after an aimed failed sync, an open that may crash inside recovery). After each open: every record byte for byte, every record an earlier open replayed still there, the epoch's records a prefix of its appends, every acked record kept (fully durable after a power loss), no damage. One run forces the F4 order on every seed. 4 tests check the simulated machine. CMUX_JOURNAL_CRASH_SEEDS=N runs more; CMUX_JOURNAL_CRASH_SEED=S reruns one seed.

Red on the code before the fix (the tests' commits on top of 89293fb):

cmux_journal_tests at 263da86: fleet c4178647 (hq run 41720702331): 22 passed, 5 failed (the 5 review regressions), each also failed alone.
crash_sim_tests at 4dfdffc: fleet ccb22f4d (hq run 41746999434): a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record failed 200 of 200 seeds ("records are missing before the next segment" after the OS crash), and the mixed run failed 5 seeds; both failed again alone.
Re-review regressions at 644e77a: fleet de35f6d0 (hq run 41759247817): 26 passed, 4 failed (G1: [1], wanted [1, 2, 3], with "a synced record is damaged"; G2: damage: [] after a crash at op 69; G3: "sync the parent folder: Not a directory"; G4: a quarantined segment was read), each also failed alone.
Third-review regressions at 936b8ac: fleet 27d011ca (hq run 41767569544): 29 passed, 3 failed (G7 A: next append Ok(15), wanted Ok(41); G7 B: "the loss of 41..43 is not reported"; the F1 test's N1 check at op 64), each also failed alone.
Fourth-review regressions at 4da4678: fleet 0166dc1b (owned Mac aws-m4pro-4; the Blacksmith run 79bcd79e died on a full runner disk before its tests): 30 passed, 3 failed (G8: Ok(6), wanted Ok(11); the F1 test and G7 B: a quarantined file not named), each also failed alone.
Fifth-review regressions at 65cead0: fleet 3372e3af (owned Mac aws-m4pro-8): 33 passed, 2 failed (G9 skip: the next append got Ok(5), wanted Ok(12); G9 torn: Ok(4), wanted Ok(6): LSNs given out again), each also failed alone.

Green at the head 51cfdee:

fleet 30b36898 (hq run 41783356193): all cmux_journal_tests: fmt, clippy (--workspace --all-targets -D warnings) and tests pass, cmux-journal 35 passed, 0 failed, 0 flaky, cmux-tui-rust-check: green.
fleet f95c6d2b (hq run 41783540718): test crash_sim_tests: 9 passed, cmux-tui-rust-check: green.
fleet 8aa92453 (owned Mac aws-m4pro-5): test cmux_journal_tests at the head: 35 passed, the real-file-system tests with F_BARRIERFSYNC/F_FULLFSYNC and both G9 tests included, cmux-tui-rust-check: green.
Earlier heads: 9c47594 (9fcb00fb 33/33, 61d2d510 9/9; owned Macs at b589044: 681db47a 33/33, d49e26c6 9/9), 215c8f0 (4e7c3e2a 32/32, c249f6ab 9/9), cb893c6 (36b0dc5d 30/30, d86543e0 9/9), 3498975 (6bee46e4 27/27, a439eb2e 9/9).

Observed / Not run

Observed: the fleet steps above (Linux: Blacksmith 8-vCPU Ubuntu 24.04; macOS: owned Macs aws-m4pro-*).
Found on the way: the first crash-sim red step (fleet 184129d8 at 263da86) passed on the old code. Its random faults reached the F4 order in 1 of 800 runs, and that run passed when it ran alone. The forced run and the aimed faults (commit 4dfdffc) make it fail every time on the old code.
macOS: the crate's tests ran on owned Macs (above), the real-file-system tests included; the crash sim last ran on a Mac at b589044 (G9 does not change the simulator or the crash runs). The 100,000-seed acceptance run (DESIGN.md A2) is a fleet measurement, not part of this PR.

Changelog

none

Proof

No capture: a Rust crate with no user-visible change and no consumer yet.

…nd ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner).

cursor · 2026-11-03T14:01:44Z

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

github-actions · 2026-11-03T14:01:51Z

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 51cfdee: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr33919 -f ref=rewrite/r-durability-s1-journal-crate -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); NEW: green on the base 3a43646; do not merge until green; see lane-rules

Shipped in:

cmux NEXT nightly 1.0.0-nightly.4182499138501 (e0aa712b3b)
cmux NEXT nightly 1.0.0-nightly.4184087390301 (08052d331e)
cmux NEXT nightly 1.0.0-nightly.4184224384601 (a3abfc4160)
cmux NEXT nightly 1.0.0-nightly.4184409870901 (78ee0e108e)

blacksmith-sh · 2026-11-03T14:02:30Z

Found 6 test failures on Blacksmith runners:

Failures

Test	View Logs
`/tmp/tmp1cck10fx/summary-fail.json (live):/summary-fail.json (live):`	View Logs
`/tmp/tmpuu0hgxl6/summary-missing.json (live):/summary-missing.json (live):`	View Logs
`[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/` `[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows`	View Logs
`[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames`	View Logs
`[tabs] debug.tab_switch timed 0 of 40 switches/` `[tabs] debug.tab_switch timed 0 of 40 switches`	View Logs
`[tabs] the window draws no frames/[tabs] the window draws no frames`	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

lawrencecchen · 2026-11-03T14:08:29Z

Red checks at 26e4e71, classified (author):

Fleet panic ratchet, job d71d7873a88966156e46dc43 (hq run https://github.com/manaflow-ai/cmuxterm-hq/actions/runs/41648190909): BASE red. The RED lines are acpmux: clippy::indexing_slicing 693 > baseline 686, acpmux: clippy::string_slice 95 > baseline 94 and cmux-host: clippy::indexing_slicing 1 > baseline 0. This PR changes neither crate. The run lists no line for cmux-journal (0 hits of every deny and tracked lint). The base 5dee1ab is inside the tip's red window for this check (tip watcher: red since 92bf57a, STATUS 8868 / 9054; green again at 6dedb9c, STATUS 9064; fix base red: acpmux github_status, cmux-host's repositories and the away recap read and write JSON without an index (panic ratchet after #33154, #33116) #33569 merged c8f5046).
acpmux tests (lib, hub_integration) (https://github.com/manaflow-ai/cmux/actions/runs/41649674910/job/125964195756): BASE red. It is the workflow-scope test test_the_workflow_runs_for_every_included_file_outside_cmux_tui: cmux-tui-spec.yml's paths miss skills/cmux-customization/*, an input of acpmux's tests. This PR touches no skill and no workflow.

…clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red.

cursor · 2026-11-03T14:19:44Z

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

… segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee.

Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c8: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5.

lawrencecchen · 2026-11-03T15:09:10Z

Checks at the head 0027391 (author):

Own checks that cover this crate, all green at the head:

clippy (changed cmux-tui packages) passed: it checked cmux-journal with --all-targets -D warnings (https://github.com/manaflow-ai/cmux/actions/runs/41657162550/job/125990431592).
rustfmt, cargo check (x86_64-pc-windows-gnu), build / build aarch64-apple-darwin, x86_64-apple-darwin, both linux-musl builds, cmux-next fast tier (cargo), cmux-next checks (god files, concurrency, crash safety, l10n), cmux-next crash ratchet: pass.

Fleet at 36f09c8 (job 7f6bfd5afb225cd1cee9d0f1, hq run https://github.com/manaflow-ai/cmuxterm-hq/actions/runs/41652730061): 24 of 25 cmux_journal_tests passed, the three 200-seed crash simulations included (process crash, OS crash, power loss). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault (it appended in the restarted storage and read the first one back), fixed in 0027391. The full Linux run at this head is job 4ff5caa33c4c76e0e14ca1fb.

Red checks at the head, classified:

action export areas and autofix repairs: BASE/infra. Both stop at "generator merge snapshot is stale: it kept the PR side for a base input: Packages/macOS/CmuxNext/Sources/CmuxNextApp/AgentShareConsentFeed.swift". This PR changes no Swift file; that file moved on the base after this branch's merge base.
nightly-guards / workflow-guard-tests / release-notary (and its wrapper Guard status): BASE. JSONDecodeError and test_r2_cas.py: --renew-every must leave at least three renewals inside --ttl: release tooling this PR does not touch.
Fleet clippy step (cmux-tui-rust-check.sh all) also stops at the base error acpmux/src/login_env/watch.rs:15 unused import notify::Watcher (not in this PR); the PR's own clippy job above lints cmux-journal clean.

lawrencecchen · 2026-11-03T15:10:17Z

Fleet result at the head 0027391 (author): job 4ff5caa33c4c76e0e14ca1fb, cmux-tui-rust-check.sh all cmux_journal_tests on Linux (hq run https://github.com/manaflow-ai/cmuxterm-hq/actions/runs/41657293781).

fmt: pass.
cmux-journal: 25 passed, 0 failed, 0 flaky (25 PASS lines, no gate-notice: FLAKY). This includes the three crash simulations (200 seeds each: process crash, OS crash, power loss).
clippy: no finding in cmux-journal (it was checked with --keep-going). The step is red only for the base error crates/acpmux/src/login_env/watch.rs:15 unused import notify::Watcher, which this PR does not touch; the PR's own clippy (changed cmux-tui packages) job lints cmux-journal clean.

…rate Merge-main commit by scripts/merge-main.sh. Merged by scripts/merge-main.sh: origin/feat-cmux-next at 3a43646. Merge-main-previous-head: 0027391 Merge-main-base: 3a43646

lawrencecchen · 2026-11-03T16:05:00Z

Fleet result at the merged head 89293fb (author): job 06ab7177b653f7f39de661b1, cmux-tui-rust-check.sh all cmux_journal_tests on Linux (hq run https://github.com/manaflow-ai/cmuxterm-hq/actions/runs/41663350495): cmux-tui-rust-check: green.

fmt pass; clippy pass (the base acpmux import error is gone with feat 3a43646, which has base red: the notify Watcher import is macOS-only (unused on Linux after #33406) #33664); workspace tests pass.
cmux-journal: 25 passed, 0 failed, 0 flaky (25 PASS lines), the three 200-seed crash simulations included.

This head is feat 3a43646 merged in with scripts/merge-main.sh --base feat-cmux-next (plain merge, on the lead's order). The PR's files are byte-identical to 0027391: git diff 002739125579 89293fb15f3 -- cmux-tui/crates/cmux-journal plans/cmux-next/OWNERS.md is empty.

lawrencecchen · 2026-11-03T17:08:07Z

Review (r-durability-reviewer, subagent, correctness first): FIX at 89293fb
reviewer-session: 41b835b6-daba-4bc8-bd95-8fc3272cbe8f
author-session: r-durability-owner
perf claim: none

Scope: the crate at 89293fb. It is byte-identical to the crate at 0027391 (git diff 002739125579 89293fb15f39 -- cmux-tui/crates/cmux-journal plans/cmux-next/OWNERS.md is empty, and the merge of feat 3a43646 adds no other PR file). I read every file of the crate before the PR body. I also read the deltas 26e4e71..36f09c8..0027391. The design is sound: one writer, group commit, ack after the sync, and the "previous last LSN" chain. The defects below are local and fixable. Line numbers are at 89293fb.

Findings

F1 (major): a crash during quarantine gives the quarantined LSNs to new records

recovery.rs:187-189 calls quarantine. recovery.rs:436-451 renames the segments one by one. recovery.rs:216-226 makes the segment that skips their LSNs only after all renames.

Scenario: S1 holds 1..10. S2 holds 11..20, with media damage at record 15. S3 holds 21..30 and S4 holds 31..40, all acked. The open finds "records are missing before the next segment" at S2 and moves S3 and S4 to quarantine/. highest_lsn reads each moved segment in full before its rename, so the window from the first rename to the durable new segment includes reading every later segment (up to 64 MiB each) plus a full sync. A kill -9 in that window, or an OS crash that keeps the renames, gives this result: the next open lists only S1 and S2. It reads the damage in S2 as a torn tail (torn_tail_bytes, no Damage::Corrupt) and sets next_lsn = 15. New records take LSNs 15..40, which quarantine/ still holds. Nobody is told that acked records 15..40 are lost.

Fix: on every open, include the LSNs that quarantine/ holds in the LSN floor (list quarantine/ and take the max of highest_lsn). When quarantine/ holds segments numbered above the last listed segment, report Damage::Corrupt again. Regression test: crash inside the open's quarantine step, reopen, then assert that the next append gets 41 and that the damage is reported.

F2 (major): a damaged header in the last segment gives its LSNs to new records

recovery.rs:277-289 sets lost_through to the previous segment's first_lsn - 1. recovery.rs:459 returns 0 for an unreadable header.

Scenario: S1 holds 1..10. S2 is the active segment (size > 56) and holds 11..20, all acked. One byte of S2's header changes (media damage). read_headers marks S2 bad, quarantine moves it, and highest_lsn gives 0. lost_through is then 10, and the report says lost_after == lost_through == 10 (nothing lost). next_lsn is 11, so new records reuse 11..20 while quarantine/ holds the acked originals. The held doc commit 477b48f calls these LSNs "unknown" and lost_through "a lower bound". That is not a fix. The LSNs are known: each record carries its LSN under its own CRC.

Fix: when the header cannot be read, scan the records from offset SEGMENT_HEADER_LEN. Take the first CRC-valid record's LSN as the expected start, and return the highest LSN read. Regression test: corrupt a header byte of the last segment, reopen, and assert that lost_through is 20 and that the next append gets 21. (Optional: the same scan of the bytes after a cut in the last segment would also keep acked LSNs from coming back after media damage there.)

F3 (major): a panic on the writer thread stops durability, and nobody is told

writer.rs:118-130 sets ended only on a normal return. writer.rs:321-326 runs the daemon's listeners on the writer thread with no unwind guard. DESIGN 3.1 says "A panic in the writer ends the process". No code makes that true: the hook in cmux-panic-log writes a log and calls the previous hook, and the workspace has no panic = "abort".

Scenario: a listener calls .lock().unwrap() on a poisoned reply-funnel mutex and panics. The writer thread ends, and ended and closed stay false. append keeps returning LSNs that are never written. Every reply held for durable_lsn waits forever. wait_durable waits to its timeout instead of returning WriterEnded, and seal fails only at its timeout.

Fix: in run(), hold a guard whose Drop sets ended = true and closed = true and notifies durable. Then abort the process when the thread is panicking, as the design says. (Or put catch_unwind around each listener call and abort.) Regression test: register a listener that panics, then assert that wait_durable returns WriterEnded and append returns Closed. For the abort, use a child-process test.

F4 (major): the failed-sync rule does not survive a restart during the backoff

writer.rs:347-359: after a failed sync, fail() only sets retry_at. The replacement segment is made on the retry, retry_min (100 ms) to retry_max (5 s) later. recovery.rs:143-149 then fully syncs the last segment, and recovery.rs:211-218 goes on in it.

The crate's own failure model (writer.rs:13-18) says that a failed sync can leave the pages clean and lost, and that a second sync can report success. Scenario: the daemon is killed or crashes inside the backoff, which is likely when a disk fails and the person restarts cmux. The next open syncs the failed segment on a new descriptor. On Linux, the writeback error was already reported to the writer's descriptor, so this fsync returns 0. The open replays the unsynced tail from the page cache as "fully synced" and appends new acked records into the same segment. When those pages are lost later, the next open stops at the hole. Every acked record after the hole becomes a "torn tail" and is lost. If a newer segment exists, those records are quarantined as "records are missing".

Fix: make the replacement segment at once when a sync fails, before the backoff. Its header names durable, and it and the folder are synced. Then the failed segment's tail is fenced off on disk while the writer waits. Use the backoff only for later retries. The regression test needs the sim mode from F5, so that it fails on the current code.

F5 (minor): the crash simulation cannot find F1 or F4

cmux_journal_tests.rs:497 arms the crash only after the first open. The open after the crash runs on a fresh storage with no crash point (sim.rs:283-287). fail_after (sim.rs:116-121) fails an operation before it has an effect. So a failed append never leaves partial bytes, and a failed sync never loses pages. Thus the 3 × 200 seeds never crash inside recovery. The "cut back a partial write" path also runs only as a no-op.

Add these: a crash point inside the recovery open (crash, reopen, crash again); a failed append that writes a prefix; and a failed sync after which an OS crash keeps only the bytes of the last good sync. Also, the sim keeps every folder across a crash (sim.rs:283), but FsStorage never syncs the parent of a journal folder it makes (recovery.rs:129, storage.rs:66-72). Sync the parent once after create_dir_all makes the folder, and let the sim lose an unsynced folder. Last, the sim keeps or loses a rename between two folders per folder (sim.rs:300-316). That can lose or duplicate a quarantined segment, which a journaling file system does not do (too pessimistic for future quarantine crash tests).

F6 (minor): recovery holds one open file per segment, and segments are never retired

recovery.rs:269-275 keeps each header's handle in segments until recover returns. lib.rs:32: nothing deletes a segment, and there is no API to retire compacted segments. So the segment count only grows. A long-lived journal eventually reaches the descriptor limit at open (the macOS default soft limit is 256), and the daemon cannot start. Fix: close each handle after its header is read, and open a segment again only to read its records. Add the retirement API before the first consumer.

F7 (nit): two doc sentences contradict the code

format.rs:14 says "a writer starts a new segment after any failed write or sync". But a failed write is cut back and retried in the same segment (writer.rs:11-12). lib.rs:32 says nothing shortens a segment that holds a record. But writer.rs:251 truncates after a failed write, which can remove whole (unacked) records that a partial write left. Correct the two sentences.

Held follow-ups: put the EINTR retry (202e746) in with these fixes. The recovery-limits doc (477b48f) does not fix F2.

What I checked and found correct

Wakeups: append changes pending under the lock before notify_one, and the writer checks pending under the lock before it waits. durable_lsn and ended change under the lock before notify_all, so wait_durable and seal do not lose a wakeup. No lock is held across a listener call. Every wake predicate in next_job matches an action in work(), so there is no busy loop. Stop, seal and drop end, also during a backoff.
Durability: durable_lsn moves only after a sync of the one segment that holds every retained record. full_durable_lsn moves only after a Full sync. Each LSN is given once, under the lock. trusted_end keeps a failed segment's second copy out of the journal. I traced the chain for rotation, Ahead, a torn tail, the reuse of a torn header, and a second open right after an open that made a segment; the writer goes on in that segment correctly.
Crash-test oracles: Process needs every written record, Os needs every acked record, and PowerLoss needs every fully synced record. All three need prefix order and no damage. These are the right oracles for the model. The gaps are in F5.
Cargo.lock: the cmux-journal entry is in order (before cmux-json), with crc32fast, libc and tempfile (one version of each in the lock). cargo clippy --locked and cargo test --locked ran on the fleet.
Toolchain: File::try_lock and TryLockError need Rust 1.89, and the workspace needs 1.91. libc 0.2.189 defines F_BARRIERFSYNC (85), F_FULLFSYNC (51), ENOTSUP (45), EOPNOTSUPP (102), EINVAL and ENOTTY for Apple.
File size limits: the largest file has 639 lines and 38 fns (tests). The library files have at most 475 lines and 37 fns.

Evidence

Fleet 06ab7177b653f7f39de661b1 at 89293fb (hq run 41663350495): fmt pass, clippy pass (workspace, --all-targets -D warnings), workspace tests pass, cmux-journal pass 25 0 0, Summary [2.029s] 25 tests run: 25 passed, cmux-tui-rust-check: green. The log has no gate-notice line.
Fleet 4ff5caa33c4c76e0e14ca1fb at 0027391 (hq run 41657293781): cmux-journal pass 25 0 0. clippy is red only on the base acpmux/src/login_env/watch.rs:15 unused import.
Fleet 7f6bfd5afb225cd1cee9d0f1 at 36f09c8 (hq run 41652730061): red from this PR. clippy type_complexity at cmux_journal_tests.rs:490. a_compaction_point_past_the_end_never_reuses_an_lsn failed in the full run and alone (left [1], right [1, 10]). The fixes at 0027391 removed both.
Fleet 1a494d9372ec2bb7d71e8e1e at 26e4e71 (hq run 41647886718): red from this PR. E0689 at cmux_journal_tests.rs:101 and dead_code Active::seq. The tests did not build.
Panic ratchet: fleet d71d7873a88966156e46dc43 at 26e4e71 (hq run 41648190909) printed rust-panic-ratchet: RED only for acpmux (indexing_slicing 693 > 686, string_slice 95 > 94) and cmux-host (indexing_slicing 1 > 0), with no cmux-journal line. At 0027391, the PR check clippy (changed cmux-tui packages) passed with the whole-workspace ratchet (86 crates checked, 36 at deny, no RED). So the crate has 0 deny hits.
GitHub checks at 89293fb are red only from the base: acpmux protocol IR is current (a stale acpmux-protocol.json; feat fixed it in acpmux: regenerate the protocol IR for mcp_connect_request/answer (base red) #34000, after the merged base 3a43646) and clippy (changed cmux-tui packages) (acpmux: clippy::indexing_slicing 703 > baseline 686). At 0027391, action export areas and autofix repairs (a stale generator merge snapshot on AgentShareConsentFeed.swift, a base input) and nightly-guards / release-notary (signing-runner pool snapshot HTTP 502) were not from this PR. The PR changes no acpmux file.
Not run: I ran no build or test. No check compiles the macOS or Windows code of this crate (no consumer yet; the Windows check builds only cmux-tui-core, cmux-tui and acpmux). I only read those paths.

fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r).

…1 review regressions (red) The S1 review (FIX at 89293fb) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open

lawrencecchen · 2026-11-04T04:41:37Z

Review (r-durability-reviewer, subagent, correctness first): FIX at 215c8f0
reviewer-session: 41b835b6-daba-4bc8-bd95-8fc3272cbe8f
author-session: r-durability-owner
perf claim: none

Scope: the repair delta cb893c6..215c8f0 (3 commits, only cmux-tui/crates/cmux-journal/). I read the code before the author's notes. Line numbers are at 215c8f0.

G7 scenarios A and B, N1 and N2 are fixed, and the three new regressions fail on the old code for the reported reasons. The split of recovery.rs moved code without changing it: the only code lines that differ are the G7 and N1 changes. One gap remains in the G7 fix (G8 below). It is the same class as G7, and its root is a false claim in the comment at recovery.rs:279-283.

Findings

G8 (major impact, rare; the G7 class): a recovery copy is numbered above the quarantine, but its first LSN is below the quarantined LSNs

recovery.rs:203-205 skips the quarantine scan when the newest segment whose header checks out is numbered above every quarantined one. The comment at recovery.rs:279-283 says that such a segment "was made after their quarantine with a first LSN above all of theirs". That is false for the copy of step 4. The copy is numbered above the quarantine (recovery.rs:305, 383), but its first LSN is vouched + 1 (recovery.rs:384), below the quarantined LSNs. Only the empty segment E that follows the copy (recovery.rs:402-403) carries the floor. When E's header is damaged, nothing that is read carries it.

Scenario (one kill -9 and two damaged headers, the same rarity as G7 A):

Acked record 1 is synced. Records 2..5 are written and not synced (Stream records), and a kill -9 comes. Open Fix find focus and align search overlay #1 copies 2..5 into segment 2 (C1), and the writer goes on in C1 with acked records 6..10.
One byte of C1's header is damaged. Open Disambiguate Sentry environment between production and dev #2 moves C1 to quarantine/ (its records go to 10) and records the loss (5, 10] in DAMAGE. Segment 1 is again the newest segment that reads, so its records after its last marker (2..5) are copied again, into segment 3 (C2). Then the empty segment 4 (E, first LSN 11) is made. The daemon exits before any append.
One byte of E's header is damaged. Open Release v1.13.0 #3 reads E as a torn creation (reuse). checked_max_seq is 3 (C2), above q_max_seq (2), so the quarantine is not read. The floor is C2's end, 5, and the next append gets 6. New records take LSNs 6..10, which quarantine/ still holds, and the DAMAGE report (5, 10] now overlaps live LSNs.

When E holds records and its header and first record are damaged, quarantine() scans E with base 5 (scan.rs:183). The window (5, 5 + span + 1] (scan.rs:131) misses E's LSNs when the quarantine held more records than E's span, so their loss is not reported either.

Fix (either one):

(a) preferred: make the quarantine floor durable without depending on any header and without reading any file. quarantine() already computes each moved file's highest LSN before the renames, so put it into the quarantine name (for example <segment>.<n>.<highest>, with list_quarantined parsing it). At every open, take the floor as the maximum over the names, and pass it as the base of every scan. This keeps G4 (a normal open reads no quarantined file) and closes the G7 class.
(b) or run quarantined_floor at every open again, with the running floor as base. G4's cost then comes back, but only after a damage event.

Regression test: the three steps above. In step 1, use a long stream_sync_delay, wait until written_lsn is 5, then restart(Crash::Process). Damage header byte 20 of segment 2, then of segment 4. At open #3, the next append must get 11.

N3 (nit): a dropped loss takes its `moved` names with it

distinct_losses (recovery.rs:142-166) drops an entry whose range lies inside another, and that entry's moved list and reason go with it. In scenario G7 B, the earlier entry (k, 40] with moved = segments 3..N lies inside the new entry (k, 43] with moved = [L]. DAMAGE is then rewritten with the new entry only (recovery.rs:361), so the record no longer names the files that hold k+1..40 for a salvage. Fix: when one loss absorbs another, merge the moved names into the kept entry and join the reasons.

N4 (nit): DAMAGE is written after the renames

A crash between the moves into quarantine/ and write_damage leaves the next open to rebuild the loss from a half-done quarantine. One shape still loses the report: the moved segment holds no LSN above the damaged segment's last record, and that segment's last marker never reached the disk (a failed sync dropped it). The next open then reads the damage as a torn tail, and nobody is told that acked records were lost. This needs several faults at once. Fix: record the loss before the renames. quarantine() already measures every file first, so split it into a measure step and a move step, and write DAMAGE between them. Most of the N1 deduplication is then no longer needed.

What I checked and found correct

G7 A and B: checked_max_seq is read right after read_headers, before the truncation at a hole. quarantined_floor and quarantine pass the running floor as the base of each next file. I traced both scenarios: the next append gets 41 in A, and in B the loss is reported through 43 and the next append gets 44.
The "unfinished quarantine" report is now made only when the scan raised the floor. It is skipped only when the quarantine holds no LSN above what is already counted, so no LSN is at risk when it is skipped.
N1: distinct_losses drops only ranges that another range contains, so the reported LSN ranges still cover every loss (N3 is about the names only). DAMAGE is rewritten only when its set of losses changes.
N2: the module doc names DAMAGE, and the wrapped doc lines are fixed. The lines still over 100 columns are a test doc comment, a test message string and the table in format.rs; rustfmt does not wrap them.
The split: comparing code lines before and after, the only differences are the G7 and N1 changes and the new use lines. All files are under the limits; the largest is recovery.rs at 674 lines.

Evidence

Fleet 4e7c3e2ae4880f0820962059 at 215c8f0 (hq run 41769509781): fmt pass, clippy pass, workspace tests pass, cmux-journal pass 32 0 0, Summary [8.099s] 32 tests run: 32 passed, cmux-tui-rust-check: green. The log has no gate-notice line.
Fleet c249f6abf8307d5a25997423 at 215c8f0 (hq run 41769684371, crash_sim_tests): cmux-journal pass 9 0 0, 9 tests run: 9 passed, green.
Red on the old code: fleet 27d011ca3d29991efc5c0e89 at 936b8ac (hq run 41767569544): 32 tests run: 29 passed, 3 failed. Each failure failed in the full run and failed alone. G7 A: Ok(15), wanted Ok(41). G7 B: "the loss of 41..43 is not reported". The F1 test's N1 check failed at op 64. The steps at db9eb7b were cancelled; that head fails its own N1 test, which 215c8f0 fixes.
GitHub checks at 215c8f0: 92 pass, 10 pending, 1 red. tooling-tests fails because webviews/bench/replayLogs.test.ts is outside the type check's roots. That file came from feat bench(web-ui): transcript replay harness: acpmux logs at top speed, shift, per-token slope, long tasks, heap, idle #34382, and feat fixed it in fix(webviews): move replayLogs.test.ts under test/ (tooling-tests base red from #34382) #34695. This PR changes only the journal crate, its workspace registration and the OWNERS row. clippy (changed cmux-tui packages), cargo check (x86_64-pc-windows-gnu), rustfmt and the god-file check pass.
Not run: I ran no build or test. No check compiles this crate for macOS or Windows.

Fourth review FIX at 215c8f0 (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report.

… the moves (G8, N3, N4) Fourth review FIX at 215c8f0 (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format.

…essary_sort_by) Fleet d9f22bed at b589044: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`).

lawrencecchen · 2026-11-04T05:57:56Z

Review (r-durability-reviewer, subagent, correctness first): FIX at 9c47594
reviewer-session: 41b835b6-daba-4bc8-bd95-8fc3272cbe8f
author-session: r-durability-owner
perf claim: none

Scope: the repair delta 215c8f0..9c47594 (3 commits, only cmux-tui/crates/cmux-journal/). I read the code before the author's notes. Line numbers are at 9c47594.

G8, N3 and N4 are fixed. The quarantine floor is now read from the file names at every open, so no header can hide it, and a normal open reads no quarantined file. DAMAGE is written before the moves (recovery.rs:397 before 401). Merged losses keep every file name. The three new regressions fail on the old code for the reported reasons and pass at the head. One gap of the same class remains (G9). Its root is the one behind G7 and G8: when an open skips LSNs, the new empty segment's header is the only place that stores the skip. The names now store the quarantine's part, but not the part found in the last segment.

Findings

G9 (major impact, rare; the G7/G8 class): the floor from a damaged record in the last segment is lost once a copy follows it

The last segment's bytes after its first bad entry raise the floor (recovery.rs:290-293). That scan runs only while the segment is the last one that reads. When the same open also copies records (recovery.rs:385), the copy becomes the segment after it, so later opens read the damaged segment only through the copy's prev_last (recovery.rs:258) and never scan it again. The skipped LSNs are then stored only in the header of the empty segment E (recovery.rs:441-442), not in DAMAGE and not in a name.

Scenario (two media faults, the same rarity as G8, and no crash):

In segment 1: acked record 1 and its marker, then records 2..10 and acked record 11, synced as one run with marker 11 after it (a batch of several records: group commit or Stream records). Then media damage in record 5.
Open Fix find focus and align search overlay #1 reads 1..4, finds marker 11 after the damage, and reports "a synced record is damaged" (4, 11] (recovery.rs:295-302). The floor is 11. Records 2..4 come after the last marker that reads (marker 1), so they are copied into segment 2 (C, first LSN 2). Then E (segment 3, first LSN 12) is made. The daemon exits before any append.
One byte of E's header is damaged. Open Disambiguate Sentry environment between production and dev #2 reads E as a torn creation (recovery.rs:550). Segment 1 is read only through C's prev_last (1), and C ends whole at 4, so nothing is scanned. The floor is 4 and the next append gets 5. New records take LSNs 5..11, which were acked and which DAMAGE reports as lost.

The same path also loses torn_lsns (records after a tear that no marker vouches for, mostly Stream bytes that clients already saw). The crate docs say those LSNs are never used again.

Fix: store the floor of every open whose next LSN is above through + 1 independently of E's header. DAMAGE is not enough on its own: clear_damage moves it, and torn LSNs are not damage. For example, make an empty file named FLOOR.<lsn> (create, then folder sync) before E, and at every open take the maximum over these names from the listing, with no file read, as the quarantine names already do.

Regression test: write record 1 (Ack, synced), records 2..10 as Stream with a long stream_sync_delay, then record 11 as Ack. Seal, then damage a payload byte of record 5 in segment 1. Open #1 must report (4, 11] and give 12; drop it with no append. Damage header byte 20 of segment 3. At open #2, the next append must get 12; the current code gives 5.

What I checked and found correct

G8: Quarantined::parse reads the names that plan_quarantine makes (<segment>.seg.<attempt>.<floor>) and ignores damage-report.N and foreign names. The floor in a name is the running maximum after that file. bad.lost_through is never above plan.highest, because the moved next segment's header gives at least its prev_last. A damaged file is scanned from the floor counted so far. Its LSNs continue from that floor when the floor is complete (G9 is the case where it is not), so the scan window holds them. I traced the author's three steps: open Release v1.13.0 #3 gives 11.
The "unfinished quarantine" report needs q_floor > before_quarantine. A finished quarantine followed by a readable segment never meets it, so a crash is still never reported as damage.
N4: a crash among the moves plans the same names again (same attempt, same running floor), so DAMAGE names files that exist after the moves finish. A re-found loss merges into the recorded one, and the reason is appended only once.
N3: every dropped loss has a kept loss that holds it (the first of the largest equal ranges), and its names and reason move there without duplicates.
I agree with keeping the beyond-cut scan of the last segment on its own base: those records continue from read.end, and a higher base would hide its markers and its torn_lsns.
All files are under the limits; the largest is recovery.rs at 706 lines. Cargo.toml and Cargo.lock did not change.

Evidence

Fleet 9fcb00fb6fdb5ad8ba1225d4 at 9c47594 (hq run 41778707948): fmt pass, clippy pass, workspace tests pass, cmux-journal pass 33 0 0, Summary [8.099s] 33 tests run: 33 passed, cmux-tui-rust-check: green. The log has no gate-notice line.
Fleet 61d2d51021a833af076f777b at 9c47594 (hq run 41778859948, crash_sim_tests): cmux-journal pass 9 0 0, 9 tests run: 9 passed, green.
macOS, first run of this crate: fleet 681db47af5beeb3b6a5c5c0a (aws-m4pro-6, aarch64-apple-darwin) at b589044 has 33 tests run: 33 passed, including the_real_file_system_round_trips_and_keeps_a_torn_tail_out and a_journal_in_a_symlinked_folder_opens_and_round_trips. Those tests run the F_BARRIERFSYNC/F_FULLFSYNC path and the folder sync on APFS. Fleet d49e26c6d5eb0b65faf56741 (aws-m4pro-9) has crash_sim_tests 9/9. b589044 differs from the head only by the sort_by_key change.
Red on the old code: fleet 0166dc1b2cd518f3ae22b218 (aws-m4pro-4) at 4da4678 has 33 tests run: 30 passed, 3 failed, and each failure failed in the full run and failed alone. G8: Ok(6), wanted Ok(11). N3 and N4: 00000000000000000003.seg.0 is not named. Fleet d9f22bed1af80954230a6c99 at b589044 found the sort_by_key clippy hit at recovery.rs:478, which 9c47594 fixes.
GitHub checks at 9c47594: 86 pass, 10 pending, 4 red, all from the base. clippy (changed cmux-tui packages) stops at cmux-git/tests/sync_barrier.rs:108 (redundant clone; feat fix(cloud): wait for repo sync before native agent tools #34782, fix(cloud): wait for repository seeding before agent startup and writes #34491). cmux-next Rust notices generate cannot update Native/DiffSidecar/Cargo.lock under --locked; DiffSidecar depends on cmux-git, which changed on feat. cmux-next CI guard group fails "Lint every workflow". backend migrations applied fails its migration plan. The notices, CI guard group and clippy checks are also red on feat commit 93703fe. This PR changes none of those files: only the journal crate, its workspace registration and the OWNERS row. The panic ratchet printed no RED and no cmux-journal line.
Not run: I ran no build or test. No check compiles this crate for Windows (the Windows cargo check covers other crates only).

Fifth review FIX at 9c47594 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4.

Fifth review FIX at 9c47594 (PR #33919 comment 6623794646); the regression tests are the commit before this one. When an open gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which a copy then hides from every later open), the only record of that floor was the header of the empty segment it makes; a damaged header lost it, and DAMAGE is cleared by `clear_damage` (torn records are not damage at all). Now such an open makes the empty file `FLOOR.<lsn>` and syncs its folder before any new segment, and every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine's names and this open's moves say their floors already: no file is made for them.

lawrencecchen · 2026-11-04T06:43:34Z

Review (r-durability-reviewer, subagent, correctness first): SHIP at 51cfdee
reviewer-session: 41b835b6-daba-4bc8-bd95-8fc3272cbe8f
author-session: r-durability-owner
perf claim: none

Scope: the repair delta 9c47594..51cfdee (2 commits, only cmux-tui/crates/cmux-journal/), and one more full read of recover() at this head, because six rounds of changes could interact. I read the code before the author's notes. Line numbers are at 51cfdee.

Findings

None. G9 is fixed, and I found no new defect. Every finding of the earlier reviews (F1-F7, G1-G9, N1-N4) is fixed. Each behavior finding has a regression test that failed on the code before its fix; the others were docs, a file split, a lint, or the simulation itself (which now has its own tests).

What I checked

G9: an open now makes the empty file FLOOR.<lsn> and syncs the folder when its floor is above everything else that stores one (recovery.rs:413-419). Those other stores are the kept segments (journal_end), the caller's from, earlier FLOOR names, the quarantine names and this open's planned moves. I checked the condition: bad.lost_through is never above plan.highest or journal_end, so the file is made exactly when the last segment's damaged or torn bytes raise the floor. Every open takes the highest FLOOR name from the listing (recovery.rs:482, 320), with no file read.
The floor class is closed. Every part of the next LSN's floor is now stored outside the new empty segment's header: kept segments, the caller's compaction point, the quarantine names and FLOOR names. A damaged copy or writer segment is planned into quarantine from that floor, and its LSNs continue from it, so the scan window holds them.
Order and crashes: record the damage (403), move into quarantine (407), write FLOOR (419), then make the copy and the new segment. A crash before FLOOR is durable leaves no new segment, so the next open rescans the same last segment and finds the same floor. A crash after FLOOR makes no second file (said includes named_floor). FLOOR files are not touched by clear_damage, and no other file name matches them.
The full recover() at this head: the read loop, hole, beyond scan, quarantine plan, distinct_losses, the copy and the new segment's numbering (including reuse) all agree with what I traced in the earlier rounds. The writer, the format and the sim are unchanged since my last read.
Tests: the two G9 regressions (a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again, torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again) follow the reported scenario and also cover torn LSNs. No existing assertion was loosened.
Limits: all files are under the god-file limits; the largest are recovery.rs (738 lines, 16 fns) and cmux_journal_tests.rs (989 lines, 47 fns, test limits). Cargo.toml and Cargo.lock did not change in this delta.

Evidence

Fleet 30b36898d34f7a05d23fdeed at 51cfdee (hq run 41783356193, all cmux_journal_tests): fmt pass, clippy pass (--workspace --all-targets -D warnings), workspace tests pass, cmux-journal pass 35 0 0, Summary [8.093s] 35 tests run: 35 passed, cmux-tui-rust-check: green. Both G9 tests PASS. The log has no gate-notice line.
Fleet f95c6d2bc002144ae0d1966c at 51cfdee (hq run 41783540718, crash_sim_tests): cmux-journal pass 9 0 0, 9 tests run: 9 passed, green.
Red on the old code: fleet 3372e3af3c9e6c348a736643 (aws-m4pro-8, aarch64-apple-darwin) at 65cead0 has 35 tests run: 33 passed, 2 failed, and each failure failed in the full run and failed alone. G9: Ok(5), wanted Ok(12). Torn LSNs: Ok(4), wanted Ok(6).
Panic ratchet: the PR check clippy (changed cmux-tui packages) ran the step "Rust panic ratchet on changed cmux-tui packages" with conclusion success. It printed rust-panic-ratchet: 90 crates checked, 40 at deny, no RED and no cmux-journal line, so the crate has 0 deny hits and 0 tracked hits.
macOS at this head: fleet 8aa924539821229f009dc556 (aws-m4pro-5, aarch64-apple-darwin) has 35 tests run: 35 passed, cmux-tui-rust-check: green. That includes the_real_file_system_round_trips_and_keeps_a_torn_tail_out and a_journal_in_a_symlinked_folder_opens_and_round_trips, which run the F_BARRIERFSYNC/F_FULLFSYNC path and the folder sync on APFS, and both G9 tests.
GitHub checks at 51cfdee: 83 pass, 19 pending, 3 red, all from the base. clippy (changed cmux-tui packages) fails only its clippy step, at cmux-git/tests/sync_barrier.rs:108 (redundant clone; feat fix(cloud): wait for repo sync before native agent tools #34782, fix(cloud): wait for repository seeding before agent startup and writes #34491). cmux-next Rust notices generate cannot update Native/DiffSidecar/Cargo.lock under --locked; it was also red on feat 93703fe. cmux-next checks fails check-scrollbars (12 hits over a ceiling of 10, all in Swift and webviews files); it was also red on feat 0a2b815 and e1869ab. This PR's diff against its merge base is only the journal crate, its workspace registration and the OWNERS row.

Not verified

I ran no build or test myself; everything above is fleet or CI output.
No check compiles this crate for Windows (the Windows cargo check builds other crates). I only read its Windows code paths.
The crate has no consumer yet. The ack gate, the reply funnel and the daemon's use of Closed and WriterEnded (DESIGN 3.2, decision 9.8) are reviewed in the stages that add them.

lawrencecchen · 2026-11-04T06:51:58Z

Red checks on 51cfdee are base findings, none in this PR's files: (1) clippy: redundant_clone in crates/cmux-git/tests/sync_barrier.rs:108; (2) Rust notices generate: Native/DiffSidecar/Cargo.lock needs an update under --locked; (3) cmux-next checks: "No fleet host names in gallery fixtures" and "Base-red classifier (per test)" (its "Pushed commits add no violations" step passed). Each is also red on feat-cmux-next. Review receipt: SHIP at the exact head (comment 6624489297).

…ournal (R-DURABILITY S2) (#34901) * cmux-journal: one append-only journal per daemon, with group commit and ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner). * cmux-journal: type a test range; drop an unread segment field (fleet clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red. * cmux-journal: a failed F_FULLFSYNC is an error, not a plain fsync; no segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee. * cmux-journal: the past-the-end test reads back the storage it wrote to Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c8: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5. * cmux-journal: a sync a signal interrupts is made again (macOS) fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r). * cmux-journal: crash sim of write-back errors and crash in recovery; S1 review regressions (red) The S1 review (FIX at 89293fb) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open * cmux-journal: the F2 regression damages the header of the last of several segments (red) The reviewer's F2 scenario has two segments: S1 holds 1..10, S2 (the last) holds 11..20, and S2's header is damaged. The test now rotates segments (300-byte segments, 20 acked records) and damages the last one's header, so the report must say lost_through 20 and the next append must take 21. On the code before the fix the moved segment's LSNs read as 0, so the next append reuses one. * cmux-journal: the crash sim forces the F4 order (failed sync, kill, OS crash) (red) On the code before the S1 fix, the crash-sim step was green (fleet 184129d8 at 263da86): 6 of 7 runs passed, and the mixed run failed only seed 140 in the full run and passed when it ran alone. The random faults reached the order the review found (F4) in 1 of 800 runs. So these tests were no red/green proof. The order: the sync of a write that spans pages fails and the disk gets none of them (Linux keeps them clean in the cache, so a later sync reports success); the process is killed in the writer's backoff; the next open reads the failed segment from the cache, replays its records and goes on; an OS crash or a power loss then takes those pages, and records an open already replayed are gone. - sim.rs: `fail_next_sync(pages, kind)` fails the next sync that has at least that many dirty pages, and the disk gets none of them. `crash_after_failure(n)` makes the n-th operation after the next injected failure the crash point. - crash_sim_tests.rs: a new run forces the order on every seed (a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record): epoch 0 aims the fault, the kill is a process crash at the writer's retry, the open after it does not crash, epoch 1 ends in an OS crash, and a third open checks that the records the second open replayed are all there. In the random runs, one epoch in four aims the fault, with the crash 1 to 3 operations after the failed sync and a crash of any kind after it. - A sim test checks the aimed fault itself (the_sim_fails_the_aimed_sync_and_crashes_in_the_backoff). Expected on this commit (old journal code): the forced run fails every seed; the mixed run fails most runs. The fix commit follows. * cmux-journal: fix the S1 review findings F1-F7 (markers, recovery copy, LSN floor, writer end) Review FIX at 89293fb (PR #33919 comment 6610468202). The regression tests are the two commits before this one. - Sync markers (format.rs): an entry with flags 1 and no payload, whose LSN field says "every record of this segment before me is synced". The writer appends one after each successful sync that covered records (after the durable listeners, so no reply waits for it), and syncs it fully at a seal and at a rotation. - F4, recovery (step 4 and 6): the last segment is trusted only up to its last marker. Its records after the marker may sit in pages a failed sync dropped (a later sync on another descriptor reports success without them), so recovery writes them again into a new segment: header, records and a marker, fully synced under `<name>.seg.new`, renamed into place, folder synced. The journal always goes on in a segment this open made, never in one an earlier process wrote. A restart in the writer's backoff after a failed sync is then safe. - F1, quarantine: every LSN a quarantined segment holds counts in the LSN floor at every open, and a quarantine that no new segment followed (a crash in the middle of an earlier recovery) is reported as damage again. The highest LSN of every moved segment is read before the first rename. - F2, highest LSN: a segment whose header is damaged is read from its first record (each record's checksum covers its LSN), then the bytes after the first bad entry are scanned for records and markers that still check out. The same scan runs after a cut in the last segment: a marker there is media damage (reported), records there are a torn write (`torn_lsns`); their LSNs are never used again. - F3, writer end: a guard closes the journal when the writer thread ends, also by a panic: appends fail with Closed, waits end with WriterEnded, the listeners are dropped (so a channel a held reply waits on closes), and stats.failing says why. The crate does not abort the process; the daemon owns that policy. - F6: recovery closes each segment after its header is read and opens one again only to read it. (Retirement of compacted segments is an S2 API, before the first consumer.) - F5: recovery syncs the parent folder after it makes the journal folder. - F7: format.rs and lib.rs now say what the code does: a failed write is cut back and retried in the same segment; the cut removes only bytes never synced and never acknowledged; nothing shortens a segment past a synced record. Docs: journal.rs says when open's replayed records are durable (on Ok) and what a stopped writer does. * cmux-journal: split the crash sim's machine model into sim/machine.rs (review G5) sim.rs had 924 lines and 63 functions, over the 60-function limit for a new Rust file (the god-file check warned "not in baseline: split it"). The machine model (each file's page cache and disk, the ordered log of names, and what an OS crash or a power loss leaves) moves to sim/machine.rs (297 lines, 19 functions); the process view and the Storage implementation stay in sim.rs (639 lines, 30 functions). No behavior change: the items move as they are, made visible to the sim module. * cmux-journal: re-review regressions G1-G4 and page eviction in the crash sim (red) Re-review FIX at 3498975 (PR #33919 comment 6620113993). Tests and sim only; the fix follows. - sim: the pages a failed sync dropped stay in the cache until an eviction; `evict_dropped_pages_after_read(true)` makes each read evict the dropped pages it read, so a second read of them sees the disk's bytes. - G1 recovery_copies_the_records_it_checked_when_the_cache_drops_them: a sync fails and drops record 2's pages, a kill in the backoff, the next open reads record 2 from the cache (replayed) and the pages are evicted after that read; then an OS crash. Record 2 and the record 3 acked after the restart must survive. On this code recovery's second, unchecked read stages zeros under a vouching marker, and the last open keeps only record 1 and reports damage. - G2: the F1 regression asserts that every open after the damage reports the loss (lost_through 40). On this code a crash at the open's last step (the folder sync after the new segment's rename) loses the report for good. - G3 a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system): a journal in a symlinked state folder, and a journal folder that is a symlink. On this code the parent folder's sync opens with O_NOFOLLOW (ELOOP). - G4 an_open_after_a_finished_quarantine_reads_no_quarantined_segment: on this code every open reads every quarantined segment in full. * cmux-journal: fix the re-review findings G1-G4 and G6 Re-review FIX at 3498975 (PR #33919 comment 6620113993); the regression tests are the commit before this one (red: fleet de35f6d0). - G1: recovery writes again exactly the bytes its checked read returned. The last segment's entries after its last marker are kept from the checked read (header and payload) and staged from memory; the second, unchecked read of the file is gone. A page that a failed sync dropped can be evicted between two reads, and the second read then returned the disk's zeros, which the copy kept under a vouching marker. - G2: the damage an open finds is written to `DAMAGE` (whole, fully synced, renamed into place, folder synced) before the new segment that hides it from later opens. Every open reports the recorded damage first, until the daemon calls the new `Journal::clear_damage`, which moves the record into `quarantine/` (kept, never deleted). - G3: a folder sync follows a symlink in its path (no O_NOFOLLOW): a state folder, or the journal folder itself, may be a symlink. The sync is for the folder that holds the entries. - G4: the quarantined segments are read for their LSNs only while no live segment is numbered above them (an unfinished quarantine); once one is, its first LSN already carries their floor. - G6: `checked_div` in the scan of the bytes after a cut (the crate's one tracked arithmetic hit). - G5 (the sim split) is the commit before the regressions. * cmux-journal: third-review regressions G7 (A, B) and N1 (red) Third review FIX at cb893c6 (PR #33919 comment 6621681018). Tests only. - G7 A a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn: after a finished quarantine, one byte of the empty new segment's header changes. On this code the next open skips the quarantine scan (the listing counts the damaged segment) and the next append reuses a quarantined LSN. - G7 B a_damaged_newest_segment_after_a_quarantine_reports_its_records: the newest segment holds acked 41..43, and its header and its first record are damaged. On this code the moved segment is scanned with a base far below 41, so the loss is not reported through 43 and the next append is not 44. - N1: the F1 regression also asserts that an open reports each loss once (no reported LSN range lies inside another). On this code a crash after the quarantine's renames makes the next open report the same loss twice. * cmux-journal: fix the third-review findings G7, N1, N2; split recovery.rs Third review FIX at cb893c6 (PR #33919 comment 6621681018); the regression tests are the commit before this one. - G7: the quarantine scan is skipped only when a segment whose header checked out is numbered above every quarantined segment (the folder listing also counted a newest segment with a damaged header, which carries no floor). The scans start from the running floor: `quarantined_floor` and `quarantine` pass the highest LSN counted so far as the base of the next file's scan, so a quarantined file with no readable header and no readable first record is scanned for the LSNs right above what is counted. - The "unfinished quarantine" report is made only when the quarantine holds LSNs nothing else counted (an old quarantine behind a segment that a crash tore at its creation holds none, and a crash is not damage). Its reason now also names a damaged newest header. - N1: a loss whose LSN range lies inside a recorded one is the same loss found again after a crash: it is neither recorded nor reported twice. - N2: recovery.rs's docs name `DAMAGE`; two doc lines over 100 columns are wrapped (lib.rs, sim.rs). - recovery.rs was at 987 of its 1000 lines: the damage record moves to recovery/damage.rs, the entry reader and the LSN scans to recovery/scan.rs (moved as they are). * cmux-journal: report each loss once, also when one open finds it twice (N1) The red run of the N1 regression (fleet 27d011ca, op 64 of the F1 test) shows the case the previous commit did not cover: a crash after the quarantine's renames and before DAMAGE is written; the next open finds both the damaged synced record of the last segment (7..14) and the unfinished quarantine (7..40), and reported both. Now every loss inside another (recorded or found by this open) is dropped, the larger range kept; DAMAGE is rewritten only when its set of losses changed. * cmux-journal: fourth-review regressions G8, N3, N4 (red) Fourth review FIX at 215c8f0 (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report. * cmux-journal: the quarantine's floor in its file names; DAMAGE before the moves (G8, N3, N4) Fourth review FIX at 215c8f0 (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format. * cmux-journal: sort the quarantine list with sort_by_key (clippy unnecessary_sort_by) Fleet d9f22bed at b589044: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`). * cmux-journal: Journal::on_written, a listener per written batch (for S2's projector) R-DURABILITY S2 (acpmux event logs through the journal) projects each chat's log line into its ndjson file once the journal has WRITTEN the batch that holds it, so readers see stream lines at once. The listener runs on the writer thread after the write and before the sync, once per batch that moved the written LSN: a batch written again into a new segment after a failed sync is not heard twice. When the writer stops, these listeners are dropped too. Test: heard while the batch's sync still runs, once. Port of 145faa6 (branch rewrite/r-durability-s2-acpmux-logs) onto the S1 review fixes. * cmux-journal: Journal::retire_through, the one delete (segments a compaction point covers) R-DURABILITY S2: a daemon's projector compacts the journal into its own files (acpmux: the chat ndjson logs, then a synced checkpoint). Without a delete the segment count only grows (review finding F6). `retire_through(compacted)` deletes the segments that hold no record after the caller's compaction point: a segment goes when the next segment's "previous last LSN" is at or before it, so the newest segment never goes. The point is capped at the durable LSN. The deletes and a folder sync run on the caller's thread (the projector, never a request path). A delete that fails leaves its segment live for a later call. Why a crash during a retirement is safe: the caller reopens from a point at or after `compacted`, and recovery reads nothing from a segment the point covers (it skips it on its next segment's header), so any subset of the deletes may reach the disk. A point older than the retired segments is reported as Damage::Missing. Quarantined segments are never deleted. - Storage::remove_file (FsStorage: std::fs::remove_file; SimStorage: one more change in the ordered log of names, durable at the folder sync). - The live segment list: recovery's, then each segment the writer makes. - JournalStats::segments_retired. - Tests: only covered segments go, and a second call deletes none; an OS crash after it keeps every record after the point and an older point is reported; a crash at every operation of a retirement (process, OS crash, power loss) keeps every record after the point and never reuses an LSN; the newest segment stays; a point in the middle of what this writer wrote. This adds a delete of journal data, so its PR is risk:high. * acpmux: move MemoryStore and the store settings into their own files store.rs (2139 lines, 120 fns) and config.rs (1909 lines, 97 fns) are over the god-file limit, and the journal store (R-DURABILITY S2) must not grow them. MemoryStore moves to store/memory.rs and StoreMode, StoreConfig and its defaults to config/store_config.rs; both stay exported where they were (crate::store::MemoryStore, crate::config::{StoreConfig, StoreMode}). No behavior change: the code moves as it is. * acpmux: the event log's journal switch and its crash tests (red) R-DURABILITY S2 (DESIGN.md 3.3b). Today no event-log append is synced: a line a client was told about is on the disk only after the next flush (shutdown), so an OS crash or a power loss takes back every line since (E12). This commit adds only the switch, a seam and the tests; the journal store follows. - `store.journal` (StoreConfig::journal, default false). - `Store::wait_durable(timeout)`: tests and tools wait until every appended record is durable. The default returns at once (the Local and Memory stores never sync an append), which is today's behavior. - `store::open_with_journal` (tests): the store with the journal on a given `cmux_journal::Storage`. Here it ignores the journal and opens LocalStore. - acpmux depends on cmux-journal (dev: with the `sim` feature). - store/journal_store_tests.rs, red on this commit: - a_line_the_store_says_is_durable_survives_an_os_crash (lines 2..20 are lost: nothing synced them); - a_deleted_chat_stays_deleted_after_a_crash and a_line_the_log_refused_is_never_written_later pass here, and must keep passing with the journal store (no replay may bring them back); - a_kill_keeps_every_line_once passes here and must keep passing (a replay after a kill must write no line twice). * acpmux: the event log through one journal, behind `store.journal` (R-DURABILITY S2) DESIGN.md 3.3b. With `store.journal` on, each event-log append is one record in `<home>/journal/` (cmux-journal: group commit, one sync per batch; a power loss takes back at most the last second, always a suffix), then the same line goes to the chat's ndjson as before. Readers, segments and file formats do not change. Default off. - store/journal_store.rs (JournalStore over LocalStore): - append: under one lock, the journal record (a memory copy, no I/O), then the log line; a line the log refuses gets a `cancel` record. Agent stream lines (`in` session/update, claude.*, codex.*) are Class::Stream, every other line Class::Ack. - checkpoint (`<home>/journal-projected.json`): every 30 s while the journal grew, past 64 MiB, at flush (shutdown) and before a retention commit. It syncs each log written since (LocalStore::sync_log, with no lock held while the disk works) and the sessions folder, writes the LSN (temp file, sync, rename, folder sync, every error checked), then retires the journal before it. A log close that failed its sync freezes the checkpoint until a restart (LocalStore::sync_failed). - recovery: the records after the checkpoint; for each chat, the log is cut right after its last readable line below the first journaled seq (later segments removed, the boundary one shortened, their kind-index copies dropped), and the journaled lines are written again in order. `forget` deletes a chat again; `cancel` keeps a refused line out. - delete: `forget` record, folder removal, then a bounded wait until the record is durable. - seal: the logs refuse writes and the journal syncs fully. - LocalStore: `append` is serialize + `append_line`; `sync_log`; a failed sync at a log's close is noted (`sync_failed`). - store::open opens the JournalStore when `store.journal` is on. Not yet (S2c): replies and events do not wait for durability; the daemon does not stop when the journal closes; the writer thread keeps the process's Utility I/O policy. * acpmux: journal store tests for the seal and a chat made again after its delete (red) - a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay: a chat is deleted (its forget record), made again under the same id (a transfer back: session.json and new lines), then a kill -9. On this code the replay deletes the folder again and its session.json is gone. - a_line_after_the_seal_is_refused_and_never_written_later: coverage of the seal (it passes here; the fix also closes a race the test cannot force). * acpmux: journal store: the seal under the appends' lock; a replay keeps a chat made again - seal takes the appends' lock: an append lands in both the journal and the logs, or in neither. Before, an append that raced the seal could leave a journal record whose line the sealed logs refused (its cancel then failed on the closed journal), and the next start would write that line. - append and delete check the logs' seal first, under the same lock, and do not touch the journal after it (no error per append at shutdown). - replay deletes a forgotten chat again only when no line follows its forget record: otherwise the chat was made again under the same id, and its folder holds that chat's own files. * cmux-journal: fifth-review regressions G9 (red) Fifth review FIX at 9c47594 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4. * cmux-journal: name a skipped LSN floor in a FLOOR.<lsn> file (G9) Fifth review FIX at 9c47594 (PR #33919 comment 6623794646); the regression tests are the commit before this one. When an open gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which a copy then hides from every later open), the only record of that floor was the header of the empty segment it makes; a damaged header lost it, and DAMAGE is cleared by `clear_damage` (torn records are not damage at all). Now such an open makes the empty file `FLOOR.<lsn>` and syncs its folder before any new segment, and every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine's names and this open's moves say their floors already: no file is made for them. * acpmux: journal store regressions from the S2 review (red) Review FIX at ab0e953 (PR #34901 comment 6624439824). Each test fails on this commit and passes after the fix that follows it: - 1a lines_written_with_the_switch_off_survive_turning_it_on_again - 1b a_kill_while_the_journal_syncs_keeps_the_lines_the_log_got - 1c a_line_flush_session_made_durable_survives_losing_its_record - 2 a_folder_put_back_after_a_delete_is_not_deleted_by_the_replay - 3 a_kill_in_a_delete_leaves_the_chat_whole_or_gone - 4 a_delete_does_not_wait_for_the_journal_sync - 5 a_failed_log_sync_freezes_the_checkpoint, a_checkpoint_that_keeps_failing_asks_the_daemon_to_stop - 6 one_chat_that_cannot_be_repaired_does_not_stop_the_open - 7 the_journal_storage_refuses_a_lost_home - the repair keeps a log's lines after a hole, also through a crash in it: an_os_crash_hole_is_repaired_and_the_lines_after_it_kept, a_crash_inside_a_repair_loses_no_line - nits: an_unreadable_checkpoint_fails_the_open, the_open_waits_for_the_previous_daemon_to_let_go_of_the_journal, a_crashed_checkpoint_temporary_is_removed_at_open, a_crash_right_after_a_checkpoint_reports_no_damage - the catch-up with the switch off: switching_off_after_a_crash_keeps_every_acknowledged_line - cmux-journal: check_append_says_what_append_would_do So they compile here, this commit adds the seams they use, with no behavior: `Store::stop_reason` (default None), `Journal::check_append` (always Ok), `FencedStorage` (never asks its fence), the open report and a test stop point after a repair's cut. * acpmux: the checkpoint freeze test fails a log sync, not a folder open (red) The freeze guards against a failed fsync (a later fsync of the same file may report success for pages the kernel dropped), not against a folder the store cannot open: a chmod 000 folder is an open error, which the fix retries instead of freezing. The test now fails the log sync itself through a test seam, `FAIL_NEXT_LOG_SYNC`. Red on the code before the fix, like the other review regressions (fleet d8e7037d, 1c22175e, 63604b26 at e030f4f: 17 of 17 new acpmux tests and the crate's check_append test fail). * acpmux: the journal store repairs logs, never rebuilds them (S2 review) Review FIX at ab0e953 (PR #34901 comment 6624439824); the regression tests are the two commits before this one. DESIGN.md 3.3c. - 1a/1b/1c: recovery no longer cuts a chat's log at its first journaled line and writes back only the journal's lines. It merges the log's lines and the journal's by seq: a line in both must be equal, a line only the log has stays, a line only the journal has goes in its place. Only bytes that do not read (a torn tail, a hole of zeros) are dropped. When the merge must change the log before its end, the lines only the log holds from there are journaled and made durable first, so a crash inside the repair loses no line. A log that moved on without the journal (a line in both that differs, lines out of order) is left as it is, and the journal's lines go to journal-unapplied/. - Append writes the log first, then the record, under one lock, after `Journal::check_append` (new in cmux-journal: the append's refusals, asked before any write). So a record always has its log line, and the `cancel` record is gone. - 2/3/4: the replay never deletes a folder and never makes a missing one again (a forget only drops the chat's earlier journaled lines). Delete appends its forget under the lock and removes the folder off it, with no wait for the record. - 5: a failed fsync in a checkpoint freezes it (a folder that does not open is retried); a frozen checkpoint returns an error, and the store asks the daemon to stop for a restart (`Store::stop_reason`, wired in daemon.rs), whose replay writes the logs again. Ten failed checkpoints in a row ask the same, so the journal past the checkpoint stays bounded. - 6: an error in one chat keeps its journaled lines in journal-unapplied/ and the open goes on. - 7: the journal's storage refuses writes under a lost home (`FencedStorage`, `config::refuse_lost_home`). - With store.journal off, a journal left by a run with it on is replayed, checkpointed and retired first (`catch_up`). - Nits: the checkpoint is never past the journal's durable end (no `Ahead` after a normal crash); damage is cleared after the next checkpoint; a kind-index copy goes before its segment is cut; an unreadable checkpoint fails the open; the open waits up to 5 s for the journal lock; crashed checkpoint temporaries are removed; `on_written` leaves cmux-journal (no user); README lists store.journal. * acpmux: split the journal store under its size budget (a move) journal_store.rs grew to 1085 lines and 71 functions with the review fix (budget 1000 and 60), and the fix grew store.rs and daemon.rs, which are over budget on the base. A pure move, no behavior change: - store/journal_store/{checkpoint,fenced,repair}.rs: the checkpoint and its thread, `FencedStorage`, the replay and the repair. - store/log_sync.rs: `LocalStore::sync_log`, `SyncFailed` and the test seam `FAIL_NEXT_LOG_SYNC`, out of store.rs (2139 lines on the base, 2113 now). - daemon/stop_causes.rs: `AbortOnDrop`, `next_signal` and `store_asks_to_stop`, out of daemon.rs (2052 on the base, 2046 now). check-no-godfiles.sh --base 51cfdee on every changed file: no file over budget grew, and no new file is over budget. * acpmux: the unrepairable-chat test waits for line 1 to be durable before its checkpoint At 03fccde (fleet 32df3c72 on aws-m4pro-9, 144d4ccc on Linux) the test kept chat-b's lines [1, 2, 3] aside, not [2, 3]: it flushed right after appending line 1, and the checkpoint is now never past the journal's durable end (the review's Ahead nit), so line 1 was still past it. The fix's behavior is right; the test now waits until line 1 is durable, so its checkpoint covers it. On the code before the fix the test still fails (the open stops at chat-b's link). * acpmux: the journal lock wait's deadline without an unchecked Instant addition `Instant + Duration` can panic on overflow (clippy::arithmetic_side_effects, tracked by the panic ratchet). The wait now has no deadline only in the case that cannot happen (an Instant 5 s ahead overflowing), where it stops waiting at once. * acpmux: an agent chunk is a Stream record, test (red) The hub's tap logs an ACP session/update under its sessionUpdate kind (hub/tap.rs: `kind = su.to_owned()`), so a record's kind is `agent_message_chunk`, `tool_call`, ... and never `session/update`; the message's method is. The journal store's `streamed` checked the kind, so every agent chunk was journaled as Ack: a sync per batch at the token rate, and in S2c each token's event waited for that sync (the S2c I/O A/B: STRESS token to client 2.2 ms -> 20.3 ms p50 with the journal on). Red here: the chunk reads back as Ack. * acpmux: an agent chunk is a Stream record (the record's message names session/update) The test is the commit before this one. `streamed` now reads the logged message's method: the hub's tap logs a session/update under its sessionUpdate kind (hub/tap.rs), so the kind never said it. Agent output is again Stream (synced with the next acknowledged batch or a second later), as DESIGN.md 3.3b says. * acpmux: review 2 of #34901, tests (red): a crash model that follows real log syncs The logs' OS crash model now follows the syncs that ran (store/sync_model.rs, tests only): a segment keeps its length at its last sync that counts; a failed sync loses what was written since the one before, and a later sync does not bring it back; a cut ends a loss it removes. Every log segment sync goes through log_sync::sync_segment (same calls as before), where the model and FAIL_NEXT_LOG_SYNC hook in. Red tests, one per finding: - 1a an_os_crash_after_a_restart_from_a_kill_keeps_every_acked_line: the restart's first checkpoint retires records whose log lines no sync made durable; expected [1..5], the head gives [1]. - 1b an_os_crash_after_a_restart_from_a_failed_sync_keeps_every_acked_line: the same after a frozen checkpoint and a restart; expected [1..5]. - 2 a_hole_that_starts_inside_a_line_is_repaired (the reviewer's test): a torn line is read as line 6 of a log that moved on; expected [1..9], the head gives [1, 2, 3, 4, 5, 9]. - 3 a_failed_sync_in_flush_session_freezes_the_checkpoint: a failed sync in flush_session does not freeze the checkpoint. * acpmux: review 2 of #34901: a restart writes the replayed lines again; a torn line is not a line; flush_session freezes Finding 1 (high): the repair now writes every merged line again from the repair start, even when the log already has them all. A killed daemon's log writes may be in the page cache only, and after a failed sync the kernel may have dropped pages that still read back and that a later sync reports as written; the first checkpoint after the open then syncs bytes this process wrote before it retires their records. The lines only the log has from the repair start are journaled and made durable first, as before a cut. The work is bounded by the checkpoint interval. Finding 2: a piece of the log is a line only when it parses as one JSON value (piece_seq). A hole that starts inside a line keeps that line's {"seq":N, head and runs into a later line's tail; it was read as a line of a log that moved on, and the chat's journaled lines went aside. Now its bytes go and the lines are written again. Finding 3: a failed sync in flush_session freezes the checkpoint (its file sync is a SyncFailed now), as a failed sync in a checkpoint or at a log's close does. Nits: cut_at syncs events/ after its removals (and the kind-index folder after its copies go), so a removed segment cannot come back beside the lines written again; the README says that with the switch on an event line over 64 MiB is refused and its log does not get it. Tests: the four red tests of d2c6386.

…daemon stops for a restart when the journal cannot go on (R-DURABILITY S2c) (#35400) * cmux-journal: one append-only journal per daemon, with group commit and ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner). * cmux-journal: type a test range; drop an unread segment field (fleet clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red. * cmux-journal: a failed F_FULLFSYNC is an error, not a plain fsync; no segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee. * cmux-journal: the past-the-end test reads back the storage it wrote to Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c8: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5. * cmux-journal: a sync a signal interrupts is made again (macOS) fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r). * cmux-journal: crash sim of write-back errors and crash in recovery; S1 review regressions (red) The S1 review (FIX at 89293fb) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open * cmux-journal: the F2 regression damages the header of the last of several segments (red) The reviewer's F2 scenario has two segments: S1 holds 1..10, S2 (the last) holds 11..20, and S2's header is damaged. The test now rotates segments (300-byte segments, 20 acked records) and damages the last one's header, so the report must say lost_through 20 and the next append must take 21. On the code before the fix the moved segment's LSNs read as 0, so the next append reuses one. * cmux-journal: the crash sim forces the F4 order (failed sync, kill, OS crash) (red) On the code before the S1 fix, the crash-sim step was green (fleet 184129d8 at 263da86): 6 of 7 runs passed, and the mixed run failed only seed 140 in the full run and passed when it ran alone. The random faults reached the order the review found (F4) in 1 of 800 runs. So these tests were no red/green proof. The order: the sync of a write that spans pages fails and the disk gets none of them (Linux keeps them clean in the cache, so a later sync reports success); the process is killed in the writer's backoff; the next open reads the failed segment from the cache, replays its records and goes on; an OS crash or a power loss then takes those pages, and records an open already replayed are gone. - sim.rs: `fail_next_sync(pages, kind)` fails the next sync that has at least that many dirty pages, and the disk gets none of them. `crash_after_failure(n)` makes the n-th operation after the next injected failure the crash point. - crash_sim_tests.rs: a new run forces the order on every seed (a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record): epoch 0 aims the fault, the kill is a process crash at the writer's retry, the open after it does not crash, epoch 1 ends in an OS crash, and a third open checks that the records the second open replayed are all there. In the random runs, one epoch in four aims the fault, with the crash 1 to 3 operations after the failed sync and a crash of any kind after it. - A sim test checks the aimed fault itself (the_sim_fails_the_aimed_sync_and_crashes_in_the_backoff). Expected on this commit (old journal code): the forced run fails every seed; the mixed run fails most runs. The fix commit follows. * cmux-journal: fix the S1 review findings F1-F7 (markers, recovery copy, LSN floor, writer end) Review FIX at 89293fb (PR #33919 comment 6610468202). The regression tests are the two commits before this one. - Sync markers (format.rs): an entry with flags 1 and no payload, whose LSN field says "every record of this segment before me is synced". The writer appends one after each successful sync that covered records (after the durable listeners, so no reply waits for it), and syncs it fully at a seal and at a rotation. - F4, recovery (step 4 and 6): the last segment is trusted only up to its last marker. Its records after the marker may sit in pages a failed sync dropped (a later sync on another descriptor reports success without them), so recovery writes them again into a new segment: header, records and a marker, fully synced under `<name>.seg.new`, renamed into place, folder synced. The journal always goes on in a segment this open made, never in one an earlier process wrote. A restart in the writer's backoff after a failed sync is then safe. - F1, quarantine: every LSN a quarantined segment holds counts in the LSN floor at every open, and a quarantine that no new segment followed (a crash in the middle of an earlier recovery) is reported as damage again. The highest LSN of every moved segment is read before the first rename. - F2, highest LSN: a segment whose header is damaged is read from its first record (each record's checksum covers its LSN), then the bytes after the first bad entry are scanned for records and markers that still check out. The same scan runs after a cut in the last segment: a marker there is media damage (reported), records there are a torn write (`torn_lsns`); their LSNs are never used again. - F3, writer end: a guard closes the journal when the writer thread ends, also by a panic: appends fail with Closed, waits end with WriterEnded, the listeners are dropped (so a channel a held reply waits on closes), and stats.failing says why. The crate does not abort the process; the daemon owns that policy. - F6: recovery closes each segment after its header is read and opens one again only to read it. (Retirement of compacted segments is an S2 API, before the first consumer.) - F5: recovery syncs the parent folder after it makes the journal folder. - F7: format.rs and lib.rs now say what the code does: a failed write is cut back and retried in the same segment; the cut removes only bytes never synced and never acknowledged; nothing shortens a segment past a synced record. Docs: journal.rs says when open's replayed records are durable (on Ok) and what a stopped writer does. * cmux-journal: split the crash sim's machine model into sim/machine.rs (review G5) sim.rs had 924 lines and 63 functions, over the 60-function limit for a new Rust file (the god-file check warned "not in baseline: split it"). The machine model (each file's page cache and disk, the ordered log of names, and what an OS crash or a power loss leaves) moves to sim/machine.rs (297 lines, 19 functions); the process view and the Storage implementation stay in sim.rs (639 lines, 30 functions). No behavior change: the items move as they are, made visible to the sim module. * cmux-journal: re-review regressions G1-G4 and page eviction in the crash sim (red) Re-review FIX at 3498975 (PR #33919 comment 6620113993). Tests and sim only; the fix follows. - sim: the pages a failed sync dropped stay in the cache until an eviction; `evict_dropped_pages_after_read(true)` makes each read evict the dropped pages it read, so a second read of them sees the disk's bytes. - G1 recovery_copies_the_records_it_checked_when_the_cache_drops_them: a sync fails and drops record 2's pages, a kill in the backoff, the next open reads record 2 from the cache (replayed) and the pages are evicted after that read; then an OS crash. Record 2 and the record 3 acked after the restart must survive. On this code recovery's second, unchecked read stages zeros under a vouching marker, and the last open keeps only record 1 and reports damage. - G2: the F1 regression asserts that every open after the damage reports the loss (lost_through 40). On this code a crash at the open's last step (the folder sync after the new segment's rename) loses the report for good. - G3 a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system): a journal in a symlinked state folder, and a journal folder that is a symlink. On this code the parent folder's sync opens with O_NOFOLLOW (ELOOP). - G4 an_open_after_a_finished_quarantine_reads_no_quarantined_segment: on this code every open reads every quarantined segment in full. * cmux-journal: fix the re-review findings G1-G4 and G6 Re-review FIX at 3498975 (PR #33919 comment 6620113993); the regression tests are the commit before this one (red: fleet de35f6d0). - G1: recovery writes again exactly the bytes its checked read returned. The last segment's entries after its last marker are kept from the checked read (header and payload) and staged from memory; the second, unchecked read of the file is gone. A page that a failed sync dropped can be evicted between two reads, and the second read then returned the disk's zeros, which the copy kept under a vouching marker. - G2: the damage an open finds is written to `DAMAGE` (whole, fully synced, renamed into place, folder synced) before the new segment that hides it from later opens. Every open reports the recorded damage first, until the daemon calls the new `Journal::clear_damage`, which moves the record into `quarantine/` (kept, never deleted). - G3: a folder sync follows a symlink in its path (no O_NOFOLLOW): a state folder, or the journal folder itself, may be a symlink. The sync is for the folder that holds the entries. - G4: the quarantined segments are read for their LSNs only while no live segment is numbered above them (an unfinished quarantine); once one is, its first LSN already carries their floor. - G6: `checked_div` in the scan of the bytes after a cut (the crate's one tracked arithmetic hit). - G5 (the sim split) is the commit before the regressions. * cmux-journal: third-review regressions G7 (A, B) and N1 (red) Third review FIX at cb893c6 (PR #33919 comment 6621681018). Tests only. - G7 A a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn: after a finished quarantine, one byte of the empty new segment's header changes. On this code the next open skips the quarantine scan (the listing counts the damaged segment) and the next append reuses a quarantined LSN. - G7 B a_damaged_newest_segment_after_a_quarantine_reports_its_records: the newest segment holds acked 41..43, and its header and its first record are damaged. On this code the moved segment is scanned with a base far below 41, so the loss is not reported through 43 and the next append is not 44. - N1: the F1 regression also asserts that an open reports each loss once (no reported LSN range lies inside another). On this code a crash after the quarantine's renames makes the next open report the same loss twice. * cmux-journal: fix the third-review findings G7, N1, N2; split recovery.rs Third review FIX at cb893c6 (PR #33919 comment 6621681018); the regression tests are the commit before this one. - G7: the quarantine scan is skipped only when a segment whose header checked out is numbered above every quarantined segment (the folder listing also counted a newest segment with a damaged header, which carries no floor). The scans start from the running floor: `quarantined_floor` and `quarantine` pass the highest LSN counted so far as the base of the next file's scan, so a quarantined file with no readable header and no readable first record is scanned for the LSNs right above what is counted. - The "unfinished quarantine" report is made only when the quarantine holds LSNs nothing else counted (an old quarantine behind a segment that a crash tore at its creation holds none, and a crash is not damage). Its reason now also names a damaged newest header. - N1: a loss whose LSN range lies inside a recorded one is the same loss found again after a crash: it is neither recorded nor reported twice. - N2: recovery.rs's docs name `DAMAGE`; two doc lines over 100 columns are wrapped (lib.rs, sim.rs). - recovery.rs was at 987 of its 1000 lines: the damage record moves to recovery/damage.rs, the entry reader and the LSN scans to recovery/scan.rs (moved as they are). * cmux-journal: report each loss once, also when one open finds it twice (N1) The red run of the N1 regression (fleet 27d011ca, op 64 of the F1 test) shows the case the previous commit did not cover: a crash after the quarantine's renames and before DAMAGE is written; the next open finds both the damaged synced record of the last segment (7..14) and the unfinished quarantine (7..40), and reported both. Now every loss inside another (recorded or found by this open) is dropped, the larger range kept; DAMAGE is rewritten only when its set of losses changed. * cmux-journal: fourth-review regressions G8, N3, N4 (red) Fourth review FIX at 215c8f0 (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report. * cmux-journal: the quarantine's floor in its file names; DAMAGE before the moves (G8, N3, N4) Fourth review FIX at 215c8f0 (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format. * cmux-journal: sort the quarantine list with sort_by_key (clippy unnecessary_sort_by) Fleet d9f22bed at b589044: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`). * cmux-journal: Journal::on_written, a listener per written batch (for S2's projector) R-DURABILITY S2 (acpmux event logs through the journal) projects each chat's log line into its ndjson file once the journal has WRITTEN the batch that holds it, so readers see stream lines at once. The listener runs on the writer thread after the write and before the sync, once per batch that moved the written LSN: a batch written again into a new segment after a failed sync is not heard twice. When the writer stops, these listeners are dropped too. Test: heard while the batch's sync still runs, once. Port of 145faa6 (branch rewrite/r-durability-s2-acpmux-logs) onto the S1 review fixes. * cmux-journal: Journal::retire_through, the one delete (segments a compaction point covers) R-DURABILITY S2: a daemon's projector compacts the journal into its own files (acpmux: the chat ndjson logs, then a synced checkpoint). Without a delete the segment count only grows (review finding F6). `retire_through(compacted)` deletes the segments that hold no record after the caller's compaction point: a segment goes when the next segment's "previous last LSN" is at or before it, so the newest segment never goes. The point is capped at the durable LSN. The deletes and a folder sync run on the caller's thread (the projector, never a request path). A delete that fails leaves its segment live for a later call. Why a crash during a retirement is safe: the caller reopens from a point at or after `compacted`, and recovery reads nothing from a segment the point covers (it skips it on its next segment's header), so any subset of the deletes may reach the disk. A point older than the retired segments is reported as Damage::Missing. Quarantined segments are never deleted. - Storage::remove_file (FsStorage: std::fs::remove_file; SimStorage: one more change in the ordered log of names, durable at the folder sync). - The live segment list: recovery's, then each segment the writer makes. - JournalStats::segments_retired. - Tests: only covered segments go, and a second call deletes none; an OS crash after it keeps every record after the point and an older point is reported; a crash at every operation of a retirement (process, OS crash, power loss) keeps every record after the point and never reuses an LSN; the newest segment stays; a point in the middle of what this writer wrote. This adds a delete of journal data, so its PR is risk:high. * acpmux: move MemoryStore and the store settings into their own files store.rs (2139 lines, 120 fns) and config.rs (1909 lines, 97 fns) are over the god-file limit, and the journal store (R-DURABILITY S2) must not grow them. MemoryStore moves to store/memory.rs and StoreMode, StoreConfig and its defaults to config/store_config.rs; both stay exported where they were (crate::store::MemoryStore, crate::config::{StoreConfig, StoreMode}). No behavior change: the code moves as it is. * acpmux: the event log's journal switch and its crash tests (red) R-DURABILITY S2 (DESIGN.md 3.3b). Today no event-log append is synced: a line a client was told about is on the disk only after the next flush (shutdown), so an OS crash or a power loss takes back every line since (E12). This commit adds only the switch, a seam and the tests; the journal store follows. - `store.journal` (StoreConfig::journal, default false). - `Store::wait_durable(timeout)`: tests and tools wait until every appended record is durable. The default returns at once (the Local and Memory stores never sync an append), which is today's behavior. - `store::open_with_journal` (tests): the store with the journal on a given `cmux_journal::Storage`. Here it ignores the journal and opens LocalStore. - acpmux depends on cmux-journal (dev: with the `sim` feature). - store/journal_store_tests.rs, red on this commit: - a_line_the_store_says_is_durable_survives_an_os_crash (lines 2..20 are lost: nothing synced them); - a_deleted_chat_stays_deleted_after_a_crash and a_line_the_log_refused_is_never_written_later pass here, and must keep passing with the journal store (no replay may bring them back); - a_kill_keeps_every_line_once passes here and must keep passing (a replay after a kill must write no line twice). * acpmux: the event log through one journal, behind `store.journal` (R-DURABILITY S2) DESIGN.md 3.3b. With `store.journal` on, each event-log append is one record in `<home>/journal/` (cmux-journal: group commit, one sync per batch; a power loss takes back at most the last second, always a suffix), then the same line goes to the chat's ndjson as before. Readers, segments and file formats do not change. Default off. - store/journal_store.rs (JournalStore over LocalStore): - append: under one lock, the journal record (a memory copy, no I/O), then the log line; a line the log refuses gets a `cancel` record. Agent stream lines (`in` session/update, claude.*, codex.*) are Class::Stream, every other line Class::Ack. - checkpoint (`<home>/journal-projected.json`): every 30 s while the journal grew, past 64 MiB, at flush (shutdown) and before a retention commit. It syncs each log written since (LocalStore::sync_log, with no lock held while the disk works) and the sessions folder, writes the LSN (temp file, sync, rename, folder sync, every error checked), then retires the journal before it. A log close that failed its sync freezes the checkpoint until a restart (LocalStore::sync_failed). - recovery: the records after the checkpoint; for each chat, the log is cut right after its last readable line below the first journaled seq (later segments removed, the boundary one shortened, their kind-index copies dropped), and the journaled lines are written again in order. `forget` deletes a chat again; `cancel` keeps a refused line out. - delete: `forget` record, folder removal, then a bounded wait until the record is durable. - seal: the logs refuse writes and the journal syncs fully. - LocalStore: `append` is serialize + `append_line`; `sync_log`; a failed sync at a log's close is noted (`sync_failed`). - store::open opens the JournalStore when `store.journal` is on. Not yet (S2c): replies and events do not wait for durability; the daemon does not stop when the journal closes; the writer thread keeps the process's Utility I/O policy. * acpmux: journal store tests for the seal and a chat made again after its delete (red) - a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay: a chat is deleted (its forget record), made again under the same id (a transfer back: session.json and new lines), then a kill -9. On this code the replay deletes the folder again and its session.json is gone. - a_line_after_the_seal_is_refused_and_never_written_later: coverage of the seal (it passes here; the fix also closes a race the test cannot force). * acpmux: journal store: the seal under the appends' lock; a replay keeps a chat made again - seal takes the appends' lock: an append lands in both the journal and the logs, or in neither. Before, an append that raced the seal could leave a journal record whose line the sealed logs refused (its cancel then failed on the closed journal), and the next start would write that line. - append and delete check the logs' seal first, under the same lock, and do not touch the journal after it (no error per append at shutdown). - replay deletes a forgotten chat again only when no line follows its forget record: otherwise the chat was made again under the same id, and its folder holds that chat's own files. * cmux-journal: fifth-review regressions G9 (red) Fifth review FIX at 9c47594 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4. * cmux-journal: name a skipped LSN floor in a FLOOR.<lsn> file (G9) Fifth review FIX at 9c47594 (PR #33919 comment 6623794646); the regression tests are the commit before this one. When an open gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which a copy then hides from every later open), the only record of that floor was the header of the empty segment it makes; a damaged header lost it, and DAMAGE is cleared by `clear_damage` (torn records are not damage at all). Now such an open makes the empty file `FLOOR.<lsn>` and syncs its folder before any new segment, and every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine's names and this open's moves say their floors already: no file is made for them. * acpmux: journal store regressions from the S2 review (red) Review FIX at ab0e953 (PR #34901 comment 6624439824). Each test fails on this commit and passes after the fix that follows it: - 1a lines_written_with_the_switch_off_survive_turning_it_on_again - 1b a_kill_while_the_journal_syncs_keeps_the_lines_the_log_got - 1c a_line_flush_session_made_durable_survives_losing_its_record - 2 a_folder_put_back_after_a_delete_is_not_deleted_by_the_replay - 3 a_kill_in_a_delete_leaves_the_chat_whole_or_gone - 4 a_delete_does_not_wait_for_the_journal_sync - 5 a_failed_log_sync_freezes_the_checkpoint, a_checkpoint_that_keeps_failing_asks_the_daemon_to_stop - 6 one_chat_that_cannot_be_repaired_does_not_stop_the_open - 7 the_journal_storage_refuses_a_lost_home - the repair keeps a log's lines after a hole, also through a crash in it: an_os_crash_hole_is_repaired_and_the_lines_after_it_kept, a_crash_inside_a_repair_loses_no_line - nits: an_unreadable_checkpoint_fails_the_open, the_open_waits_for_the_previous_daemon_to_let_go_of_the_journal, a_crashed_checkpoint_temporary_is_removed_at_open, a_crash_right_after_a_checkpoint_reports_no_damage - the catch-up with the switch off: switching_off_after_a_crash_keeps_every_acknowledged_line - cmux-journal: check_append_says_what_append_would_do So they compile here, this commit adds the seams they use, with no behavior: `Store::stop_reason` (default None), `Journal::check_append` (always Ok), `FencedStorage` (never asks its fence), the open report and a test stop point after a repair's cut. * acpmux: the checkpoint freeze test fails a log sync, not a folder open (red) The freeze guards against a failed fsync (a later fsync of the same file may report success for pages the kernel dropped), not against a folder the store cannot open: a chmod 000 folder is an open error, which the fix retries instead of freezing. The test now fails the log sync itself through a test seam, `FAIL_NEXT_LOG_SYNC`. Red on the code before the fix, like the other review regressions (fleet d8e7037d, 1c22175e, 63604b26 at e030f4f: 17 of 17 new acpmux tests and the crate's check_append test fail). * acpmux: the journal store repairs logs, never rebuilds them (S2 review) Review FIX at ab0e953 (PR #34901 comment 6624439824); the regression tests are the two commits before this one. DESIGN.md 3.3c. - 1a/1b/1c: recovery no longer cuts a chat's log at its first journaled line and writes back only the journal's lines. It merges the log's lines and the journal's by seq: a line in both must be equal, a line only the log has stays, a line only the journal has goes in its place. Only bytes that do not read (a torn tail, a hole of zeros) are dropped. When the merge must change the log before its end, the lines only the log holds from there are journaled and made durable first, so a crash inside the repair loses no line. A log that moved on without the journal (a line in both that differs, lines out of order) is left as it is, and the journal's lines go to journal-unapplied/. - Append writes the log first, then the record, under one lock, after `Journal::check_append` (new in cmux-journal: the append's refusals, asked before any write). So a record always has its log line, and the `cancel` record is gone. - 2/3/4: the replay never deletes a folder and never makes a missing one again (a forget only drops the chat's earlier journaled lines). Delete appends its forget under the lock and removes the folder off it, with no wait for the record. - 5: a failed fsync in a checkpoint freezes it (a folder that does not open is retried); a frozen checkpoint returns an error, and the store asks the daemon to stop for a restart (`Store::stop_reason`, wired in daemon.rs), whose replay writes the logs again. Ten failed checkpoints in a row ask the same, so the journal past the checkpoint stays bounded. - 6: an error in one chat keeps its journaled lines in journal-unapplied/ and the open goes on. - 7: the journal's storage refuses writes under a lost home (`FencedStorage`, `config::refuse_lost_home`). - With store.journal off, a journal left by a run with it on is replayed, checkpointed and retired first (`catch_up`). - Nits: the checkpoint is never past the journal's durable end (no `Ahead` after a normal crash); damage is cleared after the next checkpoint; a kind-index copy goes before its segment is cut; an unreadable checkpoint fails the open; the open waits up to 5 s for the journal lock; crashed checkpoint temporaries are removed; `on_written` leaves cmux-journal (no user); README lists store.journal. * acpmux: split the journal store under its size budget (a move) journal_store.rs grew to 1085 lines and 71 functions with the review fix (budget 1000 and 60), and the fix grew store.rs and daemon.rs, which are over budget on the base. A pure move, no behavior change: - store/journal_store/{checkpoint,fenced,repair}.rs: the checkpoint and its thread, `FencedStorage`, the replay and the repair. - store/log_sync.rs: `LocalStore::sync_log`, `SyncFailed` and the test seam `FAIL_NEXT_LOG_SYNC`, out of store.rs (2139 lines on the base, 2113 now). - daemon/stop_causes.rs: `AbortOnDrop`, `next_signal` and `store_asks_to_stop`, out of daemon.rs (2052 on the base, 2046 now). check-no-godfiles.sh --base 51cfdee on every changed file: no file over budget grew, and no new file is over budget. * acpmux: the unrepairable-chat test waits for line 1 to be durable before its checkpoint At 03fccde (fleet 32df3c72 on aws-m4pro-9, 144d4ccc on Linux) the test kept chat-b's lines [1, 2, 3] aside, not [2, 3]: it flushed right after appending line 1, and the checkpoint is now never past the journal's durable end (the review's Ahead nit), so line 1 was still past it. The fix's behavior is right; the test now waits until line 1 is durable, so its checkpoint covers it. On the code before the fix the test still fails (the open stops at chat-b's link). * acpmux: replies and events wait for durability, tests (red, S2c) DESIGN.md 3.3d. With the event journal on and every journal sync taking 400 ms (`SimStorage::set_sync_delay`): - an_acked_change_is_answered_once_its_record_is_durable: a rename's reply must come no sooner than one sync after its `renamed` record. Red here: the reply goes at once. - agent_output_is_not_held_behind_another_chats_acked_change: agent output to a connection attached to its chat goes at once while another chat's acknowledged change waits for its sync. Green here (nothing waits); it guards the fix against holding agent output behind changes the connection does not hear. * acpmux: replies and events wait until what they show is durable (S2c) DESIGN.md 3.3d; the tests are the commit before this one. With `store.journal` on, each connection gets an outbound gate (server/durable_gate.rs) between it and its transport: - Every line a connection queues raises the gate's `needs` to the last acknowledged LSN (`Store::durability().acked`, set in the journal store's append before the hub publishes anything of the record), except the lines of a `Stream` record's event (agent output, DESIGN C4): the fan-out delivers those in a thread-local stream scope. - The gate's task takes every queued line at once, reads `needs`, waits (async) until the journal's durable LSN reaches it, then sends the batch in order: one wait per batch, as the journal's group commit. - The durable LSN is a `tokio::sync::watch` fed by `Journal::on_durable`. The listener owns a guard: a writer that ends while the store is not being dropped (a panic on its thread) asks the daemon to stop (`Store::stop_reason`, decision 9.8), and the watch closes, so each gate drops what it holds and ends, which closes its connection. - Without the journal, `Store::durability()` is None: no gate, and `Conn::raise_gate` does nothing. Moves, so server/mod.rs and store.rs stay under their size on the base: the outbound queue helpers to server/outbound.rs, `open_with_journal` to the journal store, and `streamed` to store/durability.rs. * acpmux: the journal lock wait's deadline without an unchecked Instant addition `Instant + Duration` can panic on overflow (clippy::arithmetic_side_effects, tracked by the panic ratchet). The wait now has no deadline only in the case that cannot happen (an Instant 5 s ahead overflowing), where it stops waiting at once. * acpmux: a failed log sync stops the real daemon for a restart, test (red) Lead rule 2026-11-04 (3): a daemon-level test of the store's "stop for a restart" before S2c merges, red first. With store.journal on and every log sync failing (a debug build's ACPMUX_TEST_FAIL_LOG_SYNC; the checkpoint every 200 ms through ACPMUX_TEST_JOURNAL_CHECKPOINT_MS), a rename of a chat on disk makes its log dirty, the checkpoint's sync fails and freezes it, and the store asks the daemon to stop. The test asserts: the WebSocket client gets the restart close (1012), the socket client's pending `_acpmux/wait` ends with the connection (no hang), and the daemon exits with status 74. Red here: the daemon stops and sends 1012, but exits 0 (no exit status says that it stopped to be repaired). This commit adds only the two debug-build seams (release builds have none) and the test. * acpmux: a daemon its store stopped for a restart exits with status 74 The test is the commit before this one. `daemon::run` returned Ok and the process exited 0 for every stop, so a supervisor or a person could not tell a stop to be repaired from a normal one; with systemd `Restart=on-failure` it would not even start again. Now the store's stop (`store_asks_to_stop`: a failed log sync froze the event journal's checkpoint, or the journal's writer ended) marks itself, sets the restart handoff (WebSocket clients get 1012, as before), and the process exits with `daemon::EXIT_REPAIR` (74, EX_IOERR). The units prevent only 75, so launchd KeepAlive and systemd Restart=always start it again; the app does not read the daemon's exit status. README: store.journal names the status. * acpmux: the restart test's request in flight goes through the acpmux client Lead rule (3): the client sees a typed error, not a hang. The socket client is now the acpmux client library (acpmux::client::Client, as the CLI uses it), and the test asserts what the CLI makes of the request that the stop ends: `daemon_closed`, retryable (cli::errors::classify). The first red (fleet 41f2d037, hq run 41804640229, at 80c5fb2) failed only on the exit status (0, wanted 74), with 1012 and the end of the connection passing. * acpmux: the daemon's disk tier with the journal on, tests (red) Lead rule 2026-11-04 (2): a reply the person waits for never waits on a throttled disk sync. The fleet A/B (STATUS 11554) found that at IOPOL_UTILITY an acknowledged journal append waits about 16 ms p50 under another process's disk stress (0.7 to 1.1 ms at the default tier). - the_daemon_with_the_journal_keeps_the_default_disk_tier: with store.journal on, the daemon process keeps the default tier; without it, utility as before (each case in a child of the test binary). - a_background_thread_yields_its_disk_work: a dedicated background thread (the checkpointer) works at utility; a thread beside it does not. The API they use is stubbed here (`qos::enter_daemon` ignores the journal; `qos::background_thread` does nothing), so both fail on macOS. * acpmux: the restart test borrows no &str twice (clippy needless_borrows_for_generic_args) Found by fleet 7f8afd9d (hq run: the S2c server step at 62b4f68); the step's other clippy red is the base red in link_push_bench.rs. * acpmux: with the journal on, the daemon keeps the default disk tier (S2c) The tests are 994f741. Lead rule 2026-11-04: a reply the person waits for never waits on a throttled disk sync. Measured on two owned Macs (STATUS 11554): one acknowledged append to durable at IOPOL_UTILITY under another app's disk stress took 16.3-16.7 ms p50 and 37-45 ms p95, against 0.7-1.1 ms p50 and 1.5-1.8 ms p95 at the default tier. - qos::enter_daemon(journal): with store.journal on, the daemon process keeps the default disk tier; without it, utility as before. The switch is read alone from config.json (config::store_journal_enabled), before any thread starts, with no harness discovery. - qos::background_thread: the journal store's checkpointer lowers its own disk I/O to utility for its whole life (a thread that does nothing else: a thread-scope policy drops it out of QoS for good). * acpmux: an agent chunk is a Stream record, test (red) The hub's tap logs an ACP session/update under its sessionUpdate kind (hub/tap.rs: `kind = su.to_owned()`), so a record's kind is `agent_message_chunk`, `tool_call`, ... and never `session/update`; the message's method is. The journal store's `streamed` checked the kind, so every agent chunk was journaled as Ack: a sync per batch at the token rate, and in S2c each token's event waited for that sync (the S2c I/O A/B: STRESS token to client 2.2 ms -> 20.3 ms p50 with the journal on). Red here: the chunk reads back as Ack. * acpmux: an agent chunk is a Stream record (the record's message names session/update) The test is the commit before this one. `streamed` now reads the logged message's method: the hub's tap logs a session/update under its sessionUpdate kind (hub/tap.rs), so the kind never said it. Agent output is again Stream (synced with the next acknowledged batch or a second later), as DESIGN.md 3.3b says. * acpmux: review 2 of #34901, tests (red): a crash model that follows real log syncs The logs' OS crash model now follows the syncs that ran (store/sync_model.rs, tests only): a segment keeps its length at its last sync that counts; a failed sync loses what was written since the one before, and a later sync does not bring it back; a cut ends a loss it removes. Every log segment sync goes through log_sync::sync_segment (same calls as before), where the model and FAIL_NEXT_LOG_SYNC hook in. Red tests, one per finding: - 1a an_os_crash_after_a_restart_from_a_kill_keeps_every_acked_line: the restart's first checkpoint retires records whose log lines no sync made durable; expected [1..5], the head gives [1]. - 1b an_os_crash_after_a_restart_from_a_failed_sync_keeps_every_acked_line: the same after a frozen checkpoint and a restart; expected [1..5]. - 2 a_hole_that_starts_inside_a_line_is_repaired (the reviewer's test): a torn line is read as line 6 of a log that moved on; expected [1..9], the head gives [1, 2, 3, 4, 5, 9]. - 3 a_failed_sync_in_flush_session_freezes_the_checkpoint: a failed sync in flush_session does not freeze the checkpoint. * acpmux: review 2 of #34901: a restart writes the replayed lines again; a torn line is not a line; flush_session freezes Finding 1 (high): the repair now writes every merged line again from the repair start, even when the log already has them all. A killed daemon's log writes may be in the page cache only, and after a failed sync the kernel may have dropped pages that still read back and that a later sync reports as written; the first checkpoint after the open then syncs bytes this process wrote before it retires their records. The lines only the log has from the repair start are journaled and made durable first, as before a cut. The work is bounded by the checkpoint interval. Finding 2: a piece of the log is a line only when it parses as one JSON value (piece_seq). A hole that starts inside a line keeps that line's {"seq":N, head and runs into a later line's tail; it was read as a line of a log that moved on, and the chat's journaled lines went aside. Now its bytes go and the lines are written again. Finding 3: a failed sync in flush_session freezes the checkpoint (its file sync is a SyncFailed now), as a failed sync in a checkpoint or at a log's close does. Nits: cut_at syncs events/ after its removals (and the kind-index folder after its copies go), so a removed segment cannot come back beside the lines written again; the README says that with the switch on an event line over 64 MiB is refused and its log does not get it. Tests: the four red tests of d2c6386. * acpmux: the qos journal tests module is macOS-only (its Linux build held only an unused import: clippy red, review of #35400 F0) * acpmux: review of #35400 F1 and F2, tests (red) - a_read_that_shows_a_line_covers_its_record (F1): a read sees a log line between its log write and its record (the appends' lock held there), and the reply built from it goes out with no wait for that record. - a_journal_that_syncs_nothing_asks_the_daemon_to_stop (F2): with a journal that makes nothing durable, every reply waits without a limit and nothing asks the daemon to stop. Test seams only: PAUSE_AFTER_LOG_WRITE, TEST_STALL_MS. * acpmux: review of #35400 F1, the streaming read, test (red): session/load's replay sends a line before its record * acpmux: the test seam PAUSE_AFTER_LOG_WRITE is reachable from the server tests * acpmux: review of #35400: a read covers the records of the lines it shows; a stalled journal stops the daemon F1: the append writes a line to the log, then appends its record and sets the acknowledged LSN, all under the appends' lock; reads take no lock. A read could show a line before its record existed, and the reply built from it went out at once. Now every journal-store read of the logs ends with a fence on the appends' lock (`Inner::fenced`): a line the read saw has its record appended and its LSN acknowledged when the read returns, so the reply's gate covers it. The one read that sends as it reads (`Hub::scan_events`, session/load's replay) stops at the chat's last_seq read first. F2: a journal writer that keeps failing (it retries without end) held every reply on every connection with no limit. A connection's gate now waits at most 10 s (`STALL_LIMIT`); past it, `Durability::stalled` asks the daemon to stop for a restart, as a failed log sync does. Tests: the 3 of 3916f84 and c553a19. * threading-model: the acpmux event journal's threads and bounds (writer, checkpointer, per-connection gate; review of #35400) * acpmux: the stall test's journal syncs take 3 s, not an hour (a sync under way kept its delay and hung the test's end)

…35483) * cmux-journal: one append-only journal per daemon, with group commit and ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner). * cmux-journal: type a test range; drop an unread segment field (fleet clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71e3f9 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red. * cmux-journal: a failed F_FULLFSYNC is an error, not a plain fsync; no segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee. * cmux-journal: the past-the-end test reads back the storage it wrote to Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c87b8d: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5c53e. * cmux-journal: a sync a signal interrupts is made again (macOS) fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r). * cmux-journal: crash sim of write-back errors and crash in recovery; S1 review regressions (red) The S1 review (FIX at 89293fb15f39) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open * cmux-journal: the F2 regression damages the header of the last of several segments (red) The reviewer's F2 scenario has two segments: S1 holds 1..10, S2 (the last) holds 11..20, and S2's header is damaged. The test now rotates segments (300-byte segments, 20 acked records) and damages the last one's header, so the report must say lost_through 20 and the next append must take 21. On the code before the fix the moved segment's LSNs read as 0, so the next append reuses one. * cmux-journal: the crash sim forces the F4 order (failed sync, kill, OS crash) (red) On the code before the S1 fix, the crash-sim step was green (fleet 184129d8 at 263da86f223): 6 of 7 runs passed, and the mixed run failed only seed 140 in the full run and passed when it ran alone. The random faults reached the order the review found (F4) in 1 of 800 runs. So these tests were no red/green proof. The order: the sync of a write that spans pages fails and the disk gets none of them (Linux keeps them clean in the cache, so a later sync reports success); the process is killed in the writer's backoff; the next open reads the failed segment from the cache, replays its records and goes on; an OS crash or a power loss then takes those pages, and records an open already replayed are gone. - sim.rs: `fail_next_sync(pages, kind)` fails the next sync that has at least that many dirty pages, and the disk gets none of them. `crash_after_failure(n)` makes the n-th operation after the next injected failure the crash point. - crash_sim_tests.rs: a new run forces the order on every seed (a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record): epoch 0 aims the fault, the kill is a process crash at the writer's retry, the open after it does not crash, epoch 1 ends in an OS crash, and a third open checks that the records the second open replayed are all there. In the random runs, one epoch in four aims the fault, with the crash 1 to 3 operations after the failed sync and a crash of any kind after it. - A sim test checks the aimed fault itself (the_sim_fails_the_aimed_sync_and_crashes_in_the_backoff). Expected on this commit (old journal code): the forced run fails every seed; the mixed run fails most runs. The fix commit follows. * cmux-journal: fix the S1 review findings F1-F7 (markers, recovery copy, LSN floor, writer end) Review FIX at 89293fb15f39 (PR #33919 comment 6610468202). The regression tests are the two commits before this one. - Sync markers (format.rs): an entry with flags 1 and no payload, whose LSN field says "every record of this segment before me is synced". The writer appends one after each successful sync that covered records (after the durable listeners, so no reply waits for it), and syncs it fully at a seal and at a rotation. - F4, recovery (step 4 and 6): the last segment is trusted only up to its last marker. Its records after the marker may sit in pages a failed sync dropped (a later sync on another descriptor reports success without them), so recovery writes them again into a new segment: header, records and a marker, fully synced under `<name>.seg.new`, renamed into place, folder synced. The journal always goes on in a segment this open made, never in one an earlier process wrote. A restart in the writer's backoff after a failed sync is then safe. - F1, quarantine: every LSN a quarantined segment holds counts in the LSN floor at every open, and a quarantine that no new segment followed (a crash in the middle of an earlier recovery) is reported as damage again. The highest LSN of every moved segment is read before the first rename. - F2, highest LSN: a segment whose header is damaged is read from its first record (each record's checksum covers its LSN), then the bytes after the first bad entry are scanned for records and markers that still check out. The same scan runs after a cut in the last segment: a marker there is media damage (reported), records there are a torn write (`torn_lsns`); their LSNs are never used again. - F3, writer end: a guard closes the journal when the writer thread ends, also by a panic: appends fail with Closed, waits end with WriterEnded, the listeners are dropped (so a channel a held reply waits on closes), and stats.failing says why. The crate does not abort the process; the daemon owns that policy. - F6: recovery closes each segment after its header is read and opens one again only to read it. (Retirement of compacted segments is an S2 API, before the first consumer.) - F5: recovery syncs the parent folder after it makes the journal folder. - F7: format.rs and lib.rs now say what the code does: a failed write is cut back and retried in the same segment; the cut removes only bytes never synced and never acknowledged; nothing shortens a segment past a synced record. Docs: journal.rs says when open's replayed records are durable (on Ok) and what a stopped writer does. * cmux-journal: split the crash sim's machine model into sim/machine.rs (review G5) sim.rs had 924 lines and 63 functions, over the 60-function limit for a new Rust file (the god-file check warned "not in baseline: split it"). The machine model (each file's page cache and disk, the ordered log of names, and what an OS crash or a power loss leaves) moves to sim/machine.rs (297 lines, 19 functions); the process view and the Storage implementation stay in sim.rs (639 lines, 30 functions). No behavior change: the items move as they are, made visible to the sim module. * cmux-journal: re-review regressions G1-G4 and page eviction in the crash sim (red) Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993). Tests and sim only; the fix follows. - sim: the pages a failed sync dropped stay in the cache until an eviction; `evict_dropped_pages_after_read(true)` makes each read evict the dropped pages it read, so a second read of them sees the disk's bytes. - G1 recovery_copies_the_records_it_checked_when_the_cache_drops_them: a sync fails and drops record 2's pages, a kill in the backoff, the next open reads record 2 from the cache (replayed) and the pages are evicted after that read; then an OS crash. Record 2 and the record 3 acked after the restart must survive. On this code recovery's second, unchecked read stages zeros under a vouching marker, and the last open keeps only record 1 and reports damage. - G2: the F1 regression asserts that every open after the damage reports the loss (lost_through 40). On this code a crash at the open's last step (the folder sync after the new segment's rename) loses the report for good. - G3 a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system): a journal in a symlinked state folder, and a journal folder that is a symlink. On this code the parent folder's sync opens with O_NOFOLLOW (ELOOP). - G4 an_open_after_a_finished_quarantine_reads_no_quarantined_segment: on this code every open reads every quarantined segment in full. * cmux-journal: fix the re-review findings G1-G4 and G6 Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993); the regression tests are the commit before this one (red: fleet de35f6d0). - G1: recovery writes again exactly the bytes its checked read returned. The last segment's entries after its last marker are kept from the checked read (header and payload) and staged from memory; the second, unchecked read of the file is gone. A page that a failed sync dropped can be evicted between two reads, and the second read then returned the disk's zeros, which the copy kept under a vouching marker. - G2: the damage an open finds is written to `DAMAGE` (whole, fully synced, renamed into place, folder synced) before the new segment that hides it from later opens. Every open reports the recorded damage first, until the daemon calls the new `Journal::clear_damage`, which moves the record into `quarantine/` (kept, never deleted). - G3: a folder sync follows a symlink in its path (no O_NOFOLLOW): a state folder, or the journal folder itself, may be a symlink. The sync is for the folder that holds the entries. - G4: the quarantined segments are read for their LSNs only while no live segment is numbered above them (an unfinished quarantine); once one is, its first LSN already carries their floor. - G6: `checked_div` in the scan of the bytes after a cut (the crate's one tracked arithmetic hit). - G5 (the sim split) is the commit before the regressions. * cmux-journal: third-review regressions G7 (A, B) and N1 (red) Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018). Tests only. - G7 A a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn: after a finished quarantine, one byte of the empty new segment's header changes. On this code the next open skips the quarantine scan (the listing counts the damaged segment) and the next append reuses a quarantined LSN. - G7 B a_damaged_newest_segment_after_a_quarantine_reports_its_records: the newest segment holds acked 41..43, and its header and its first record are damaged. On this code the moved segment is scanned with a base far below 41, so the loss is not reported through 43 and the next append is not 44. - N1: the F1 regression also asserts that an open reports each loss once (no reported LSN range lies inside another). On this code a crash after the quarantine's renames makes the next open report the same loss twice. * cmux-journal: fix the third-review findings G7, N1, N2; split recovery.rs Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018); the regression tests are the commit before this one. - G7: the quarantine scan is skipped only when a segment whose header checked out is numbered above every quarantined segment (the folder listing also counted a newest segment with a damaged header, which carries no floor). The scans start from the running floor: `quarantined_floor` and `quarantine` pass the highest LSN counted so far as the base of the next file's scan, so a quarantined file with no readable header and no readable first record is scanned for the LSNs right above what is counted. - The "unfinished quarantine" report is made only when the quarantine holds LSNs nothing else counted (an old quarantine behind a segment that a crash tore at its creation holds none, and a crash is not damage). Its reason now also names a damaged newest header. - N1: a loss whose LSN range lies inside a recorded one is the same loss found again after a crash: it is neither recorded nor reported twice. - N2: recovery.rs's docs name `DAMAGE`; two doc lines over 100 columns are wrapped (lib.rs, sim.rs). - recovery.rs was at 987 of its 1000 lines: the damage record moves to recovery/damage.rs, the entry reader and the LSN scans to recovery/scan.rs (moved as they are). * cmux-journal: report each loss once, also when one open finds it twice (N1) The red run of the N1 regression (fleet 27d011ca, op 64 of the F1 test) shows the case the previous commit did not cover: a crash after the quarantine's renames and before DAMAGE is written; the next open finds both the damaged synced record of the last segment (7..14) and the unfinished quarantine (7..40), and reported both. Now every loss inside another (recorded or found by this open) is dropped, the larger range kept; DAMAGE is rewritten only when its set of losses changed. * cmux-journal: fourth-review regressions G8, N3, N4 (red) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report. * cmux-journal: the quarantine's floor in its file names; DAMAGE before the moves (G8, N3, N4) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format. * cmux-journal: sort the quarantine list with sort_by_key (clippy unnecessary_sort_by) Fleet d9f22bed at b589044b852: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`). * cmux-journal: Journal::on_written, a listener per written batch (for S2's projector) R-DURABILITY S2 (acpmux event logs through the journal) projects each chat's log line into its ndjson file once the journal has WRITTEN the batch that holds it, so readers see stream lines at once. The listener runs on the writer thread after the write and before the sync, once per batch that moved the written LSN: a batch written again into a new segment after a failed sync is not heard twice. When the writer stops, these listeners are dropped too. Test: heard while the batch's sync still runs, once. Port of 145faa6faa8 (branch rewrite/r-durability-s2-acpmux-logs) onto the S1 review fixes. * cmux-journal: Journal::retire_through, the one delete (segments a compaction point covers) R-DURABILITY S2: a daemon's projector compacts the journal into its own files (acpmux: the chat ndjson logs, then a synced checkpoint). Without a delete the segment count only grows (review finding F6). `retire_through(compacted)` deletes the segments that hold no record after the caller's compaction point: a segment goes when the next segment's "previous last LSN" is at or before it, so the newest segment never goes. The point is capped at the durable LSN. The deletes and a folder sync run on the caller's thread (the projector, never a request path). A delete that fails leaves its segment live for a later call. Why a crash during a retirement is safe: the caller reopens from a point at or after `compacted`, and recovery reads nothing from a segment the point covers (it skips it on its next segment's header), so any subset of the deletes may reach the disk. A point older than the retired segments is reported as Damage::Missing. Quarantined segments are never deleted. - Storage::remove_file (FsStorage: std::fs::remove_file; SimStorage: one more change in the ordered log of names, durable at the folder sync). - The live segment list: recovery's, then each segment the writer makes. - JournalStats::segments_retired. - Tests: only covered segments go, and a second call deletes none; an OS crash after it keeps every record after the point and an older point is reported; a crash at every operation of a retirement (process, OS crash, power loss) keeps every record after the point and never reuses an LSN; the newest segment stays; a point in the middle of what this writer wrote. This adds a delete of journal data, so its PR is risk:high. * acpmux: move MemoryStore and the store settings into their own files store.rs (2139 lines, 120 fns) and config.rs (1909 lines, 97 fns) are over the god-file limit, and the journal store (R-DURABILITY S2) must not grow them. MemoryStore moves to store/memory.rs and StoreMode, StoreConfig and its defaults to config/store_config.rs; both stay exported where they were (crate::store::MemoryStore, crate::config::{StoreConfig, StoreMode}). No behavior change: the code moves as it is. * acpmux: the event log's journal switch and its crash tests (red) R-DURABILITY S2 (DESIGN.md 3.3b). Today no event-log append is synced: a line a client was told about is on the disk only after the next flush (shutdown), so an OS crash or a power loss takes back every line since (E12). This commit adds only the switch, a seam and the tests; the journal store follows. - `store.journal` (StoreConfig::journal, default false). - `Store::wait_durable(timeout)`: tests and tools wait until every appended record is durable. The default returns at once (the Local and Memory stores never sync an append), which is today's behavior. - `store::open_with_journal` (tests): the store with the journal on a given `cmux_journal::Storage`. Here it ignores the journal and opens LocalStore. - acpmux depends on cmux-journal (dev: with the `sim` feature). - store/journal_store_tests.rs, red on this commit: - a_line_the_store_says_is_durable_survives_an_os_crash (lines 2..20 are lost: nothing synced them); - a_deleted_chat_stays_deleted_after_a_crash and a_line_the_log_refused_is_never_written_later pass here, and must keep passing with the journal store (no replay may bring them back); - a_kill_keeps_every_line_once passes here and must keep passing (a replay after a kill must write no line twice). * acpmux: the event log through one journal, behind `store.journal` (R-DURABILITY S2) DESIGN.md 3.3b. With `store.journal` on, each event-log append is one record in `<home>/journal/` (cmux-journal: group commit, one sync per batch; a power loss takes back at most the last second, always a suffix), then the same line goes to the chat's ndjson as before. Readers, segments and file formats do not change. Default off. - store/journal_store.rs (JournalStore over LocalStore): - append: under one lock, the journal record (a memory copy, no I/O), then the log line; a line the log refuses gets a `cancel` record. Agent stream lines (`in` session/update, claude.*, codex.*) are Class::Stream, every other line Class::Ack. - checkpoint (`<home>/journal-projected.json`): every 30 s while the journal grew, past 64 MiB, at flush (shutdown) and before a retention commit. It syncs each log written since (LocalStore::sync_log, with no lock held while the disk works) and the sessions folder, writes the LSN (temp file, sync, rename, folder sync, every error checked), then retires the journal before it. A log close that failed its sync freezes the checkpoint until a restart (LocalStore::sync_failed). - recovery: the records after the checkpoint; for each chat, the log is cut right after its last readable line below the first journaled seq (later segments removed, the boundary one shortened, their kind-index copies dropped), and the journaled lines are written again in order. `forget` deletes a chat again; `cancel` keeps a refused line out. - delete: `forget` record, folder removal, then a bounded wait until the record is durable. - seal: the logs refuse writes and the journal syncs fully. - LocalStore: `append` is serialize + `append_line`; `sync_log`; a failed sync at a log's close is noted (`sync_failed`). - store::open opens the JournalStore when `store.journal` is on. Not yet (S2c): replies and events do not wait for durability; the daemon does not stop when the journal closes; the writer thread keeps the process's Utility I/O policy. * acpmux: journal store tests for the seal and a chat made again after its delete (red) - a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay: a chat is deleted (its forget record), made again under the same id (a transfer back: session.json and new lines), then a kill -9. On this code the replay deletes the folder again and its session.json is gone. - a_line_after_the_seal_is_refused_and_never_written_later: coverage of the seal (it passes here; the fix also closes a race the test cannot force). * acpmux: journal store: the seal under the appends' lock; a replay keeps a chat made again - seal takes the appends' lock: an append lands in both the journal and the logs, or in neither. Before, an append that raced the seal could leave a journal record whose line the sealed logs refused (its cancel then failed on the closed journal), and the next start would write that line. - append and delete check the logs' seal first, under the same lock, and do not touch the journal after it (no error per append at shutdown). - replay deletes a forgotten chat again only when no line follows its forget record: otherwise the chat was made again under the same id, and its folder holds that chat's own files. * cmux-journal: fifth-review regressions G9 (red) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4. * cmux-journal: name a skipped LSN floor in a FLOOR.<lsn> file (G9) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646); the regression tests are the commit before this one. When an open gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which a copy then hides from every later open), the only record of that floor was the header of the empty segment it makes; a damaged header lost it, and DAMAGE is cleared by `clear_damage` (torn records are not damage at all). Now such an open makes the empty file `FLOOR.<lsn>` and syncs its folder before any new segment, and every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine's names and this open's moves say their floors already: no file is made for them. * acpmux: journal store regressions from the S2 review (red) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824). Each test fails on this commit and passes after the fix that follows it: - 1a lines_written_with_the_switch_off_survive_turning_it_on_again - 1b a_kill_while_the_journal_syncs_keeps_the_lines_the_log_got - 1c a_line_flush_session_made_durable_survives_losing_its_record - 2 a_folder_put_back_after_a_delete_is_not_deleted_by_the_replay - 3 a_kill_in_a_delete_leaves_the_chat_whole_or_gone - 4 a_delete_does_not_wait_for_the_journal_sync - 5 a_failed_log_sync_freezes_the_checkpoint, a_checkpoint_that_keeps_failing_asks_the_daemon_to_stop - 6 one_chat_that_cannot_be_repaired_does_not_stop_the_open - 7 the_journal_storage_refuses_a_lost_home - the repair keeps a log's lines after a hole, also through a crash in it: an_os_crash_hole_is_repaired_and_the_lines_after_it_kept, a_crash_inside_a_repair_loses_no_line - nits: an_unreadable_checkpoint_fails_the_open, the_open_waits_for_the_previous_daemon_to_let_go_of_the_journal, a_crashed_checkpoint_temporary_is_removed_at_open, a_crash_right_after_a_checkpoint_reports_no_damage - the catch-up with the switch off: switching_off_after_a_crash_keeps_every_acknowledged_line - cmux-journal: check_append_says_what_append_would_do So they compile here, this commit adds the seams they use, with no behavior: `Store::stop_reason` (default None), `Journal::check_append` (always Ok), `FencedStorage` (never asks its fence), the open report and a test stop point after a repair's cut. * acpmux: the checkpoint freeze test fails a log sync, not a folder open (red) The freeze guards against a failed fsync (a later fsync of the same file may report success for pages the kernel dropped), not against a folder the store cannot open: a chmod 000 folder is an open error, which the fix retries instead of freezing. The test now fails the log sync itself through a test seam, `FAIL_NEXT_LOG_SYNC`. Red on the code before the fix, like the other review regressions (fleet d8e7037d, 1c22175e, 63604b26 at e030f4fe402: 17 of 17 new acpmux tests and the crate's check_append test fail). * acpmux: the journal store repairs logs, never rebuilds them (S2 review) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824); the regression tests are the two commits before this one. DESIGN.md 3.3c. - 1a/1b/1c: recovery no longer cuts a chat's log at its first journaled line and writes back only the journal's lines. It merges the log's lines and the journal's by seq: a line in both must be equal, a line only the log has stays, a line only the journal has goes in its place. Only bytes that do not read (a torn tail, a hole of zeros) are dropped. When the merge must change the log before its end, the lines only the log holds from there are journaled and made durable first, so a crash inside the repair loses no line. A log that moved on without the journal (a line in both that differs, lines out of order) is left as it is, and the journal's lines go to journal-unapplied/. - Append writes the log first, then the record, under one lock, after `Journal::check_append` (new in cmux-journal: the append's refusals, asked before any write). So a record always has its log line, and the `cancel` record is gone. - 2/3/4: the replay never deletes a folder and never makes a missing one again (a forget only drops the chat's earlier journaled lines). Delete appends its forget under the lock and removes the folder off it, with no wait for the record. - 5: a failed fsync in a checkpoint freezes it (a folder that does not open is retried); a frozen checkpoint returns an error, and the store asks the daemon to stop for a restart (`Store::stop_reason`, wired in daemon.rs), whose replay writes the logs again. Ten failed checkpoints in a row ask the same, so the journal past the checkpoint stays bounded. - 6: an error in one chat keeps its journaled lines in journal-unapplied/ and the open goes on. - 7: the journal's storage refuses writes under a lost home (`FencedStorage`, `config::refuse_lost_home`). - With store.journal off, a journal left by a run with it on is replayed, checkpointed and retired first (`catch_up`). - Nits: the checkpoint is never past the journal's durable end (no `Ahead` after a normal crash); damage is cleared after the next checkpoint; a kind-index copy goes before its segment is cut; an unreadable checkpoint fails the open; the open waits up to 5 s for the journal lock; crashed checkpoint temporaries are removed; `on_written` leaves cmux-journal (no user); README lists store.journal. * acpmux: split the journal store under its size budget (a move) journal_store.rs grew to 1085 lines and 71 functions with the review fix (budget 1000 and 60), and the fix grew store.rs and daemon.rs, which are over budget on the base. A pure move, no behavior change: - store/journal_store/{checkpoint,fenced,repair}.rs: the checkpoint and its thread, `FencedStorage`, the replay and the repair. - store/log_sync.rs: `LocalStore::sync_log`, `SyncFailed` and the test seam `FAIL_NEXT_LOG_SYNC`, out of store.rs (2139 lines on the base, 2113 now). - daemon/stop_causes.rs: `AbortOnDrop`, `next_signal` and `store_asks_to_stop`, out of daemon.rs (2052 on the base, 2046 now). check-no-godfiles.sh --base 51cfdee876b on every changed file: no file over budget grew, and no new file is over budget. * acpmux: the unrepairable-chat test waits for line 1 to be durable before its checkpoint At 03fccdef69c (fleet 32df3c72 on aws-m4pro-9, 144d4ccc on Linux) the test kept chat-b's lines [1, 2, 3] aside, not [2, 3]: it flushed right after appending line 1, and the checkpoint is now never past the journal's durable end (the review's Ahead nit), so line 1 was still past it. The fix's behavior is right; the test now waits until line 1 is durable, so its checkpoint covers it. On the code before the fix the test still fails (the open stops at chat-b's link). * acpmux: replies and events wait for durability, tests (red, S2c) DESIGN.md 3.3d. With the event journal on and every journal sync taking 400 ms (`SimStorage::set_sync_delay`): - an_acked_change_is_answered_once_its_record_is_durable: a rename's reply must come no sooner than one sync after its `renamed` record. Red here: the reply goes at once. - agent_output_is_not_held_behind_another_chats_acked_change: agent output to a connection attached to its chat goes at once while another chat's acknowledged change waits for its sync. Green here (nothing waits); it guards the fix against holding agent output behind changes the connection does not hear. * acpmux: replies and events wait until what they show is durable (S2c) DESIGN.md 3.3d; the tests are the commit before this one. With `store.journal` on, each connection gets an outbound gate (server/durable_gate.rs) between it and its transport: - Every line a connection queues raises the gate's `needs` to the last acknowledged LSN (`Store::durability().acked`, set in the journal store's append before the hub publishes anything of the record), except the lines of a `Stream` record's event (agent output, DESIGN C4): the fan-out delivers those in a thread-local stream scope. - The gate's task takes every queued line at once, reads `needs`, waits (async) until the journal's durable LSN reaches it, then sends the batch in order: one wait per batch, as the journal's group commit. - The durable LSN is a `tokio::sync::watch` fed by `Journal::on_durable`. The listener owns a guard: a writer that ends while the store is not being dropped (a panic on its thread) asks the daemon to stop (`Store::stop_reason`, decision 9.8), and the watch closes, so each gate drops what it holds and ends, which closes its connection. - Without the journal, `Store::durability()` is None: no gate, and `Conn::raise_gate` does nothing. Moves, so server/mod.rs and store.rs stay under their size on the base: the outbound queue helpers to server/outbound.rs, `open_with_journal` to the journal store, and `streamed` to store/durability.rs. * acpmux: the journal lock wait's deadline without an unchecked Instant addition `Instant + Duration` can panic on overflow (clippy::arithmetic_side_effects, tracked by the panic ratchet). The wait now has no deadline only in the case that cannot happen (an Instant 5 s ahead overflowing), where it stops waiting at once. * acpmux: a failed log sync stops the real daemon for a restart, test (red) Lead rule 2026-11-04 (3): a daemon-level test of the store's "stop for a restart" before S2c merges, red first. With store.journal on and every log sync failing (a debug build's ACPMUX_TEST_FAIL_LOG_SYNC; the checkpoint every 200 ms through ACPMUX_TEST_JOURNAL_CHECKPOINT_MS), a rename of a chat on disk makes its log dirty, the checkpoint's sync fails and freezes it, and the store asks the daemon to stop. The test asserts: the WebSocket client gets the restart close (1012), the socket client's pending `_acpmux/wait` ends with the connection (no hang), and the daemon exits with status 74. Red here: the daemon stops and sends 1012, but exits 0 (no exit status says that it stopped to be repaired). This commit adds only the two debug-build seams (release builds have none) and the test. * acpmux: a daemon its store stopped for a restart exits with status 74 The test is the commit before this one. `daemon::run` returned Ok and the process exited 0 for every stop, so a supervisor or a person could not tell a stop to be repaired from a normal one; with systemd `Restart=on-failure` it would not even start again. Now the store's stop (`store_asks_to_stop`: a failed log sync froze the event journal's checkpoint, or the journal's writer ended) marks itself, sets the restart handoff (WebSocket clients get 1012, as before), and the process exits with `daemon::EXIT_REPAIR` (74, EX_IOERR). The units prevent only 75, so launchd KeepAlive and systemd Restart=always start it again; the app does not read the daemon's exit status. README: store.journal names the status. * acpmux: the restart test's request in flight goes through the acpmux client Lead rule (3): the client sees a typed error, not a hang. The socket client is now the acpmux client library (acpmux::client::Client, as the CLI uses it), and the test asserts what the CLI makes of the request that the stop ends: `daemon_closed`, retryable (cli::errors::classify). The first red (fleet 41f2d037, hq run 41804640229, at 80c5fb2b7f8) failed only on the exit status (0, wanted 74), with 1012 and the end of the connection passing. * acpmux: the daemon's disk tier with the journal on, tests (red) Lead rule 2026-11-04 (2): a reply the person waits for never waits on a throttled disk sync. The fleet A/B (STATUS 11554) found that at IOPOL_UTILITY an acknowledged journal append waits about 16 ms p50 under another process's disk stress (0.7 to 1.1 ms at the default tier). - the_daemon_with_the_journal_keeps_the_default_disk_tier: with store.journal on, the daemon process keeps the default tier; without it, utility as before (each case in a child of the test binary). - a_background_thread_yields_its_disk_work: a dedicated background thread (the checkpointer) works at utility; a thread beside it does not. The API they use is stubbed here (`qos::enter_daemon` ignores the journal; `qos::background_thread` does nothing), so both fail on macOS. * acpmux: the restart test borrows no &str twice (clippy needless_borrows_for_generic_args) Found by fleet 7f8afd9d (hq run: the S2c server step at 62b4f68ef8b); the step's other clippy red is the base red in link_push_bench.rs. * acpmux: with the journal on, the daemon keeps the default disk tier (S2c) The tests are 994f74119fb. Lead rule 2026-11-04: a reply the person waits for never waits on a throttled disk sync. Measured on two owned Macs (STATUS 11554): one acknowledged append to durable at IOPOL_UTILITY under another app's disk stress took 16.3-16.7 ms p50 and 37-45 ms p95, against 0.7-1.1 ms p50 and 1.5-1.8 ms p95 at the default tier. - qos::enter_daemon(journal): with store.journal on, the daemon process keeps the default disk tier; without it, utility as before. The switch is read alone from config.json (config::store_journal_enabled), before any thread starts, with no harness discovery. - qos::background_thread: the journal store's checkpointer lowers its own disk I/O to utility for its whole life (a thread that does nothing else: a thread-scope policy drops it out of QoS for good). * acpmux: an agent chunk is a Stream record, test (red) The hub's tap logs an ACP session/update under its sessionUpdate kind (hub/tap.rs: `kind = su.to_owned()`), so a record's kind is `agent_message_chunk`, `tool_call`, ... and never `session/update`; the message's method is. The journal store's `streamed` checked the kind, so every agent chunk was journaled as Ack: a sync per batch at the token rate, and in S2c each token's event waited for that sync (the S2c I/O A/B: STRESS token to client 2.2 ms -> 20.3 ms p50 with the journal on). Red here: the chunk reads back as Ack. * acpmux: an agent chunk is a Stream record (the record's message names session/update) The test is the commit before this one. `streamed` now reads the logged message's method: the hub's tap logs a session/update under its sessionUpdate kind (hub/tap.rs), so the kind never said it. Agent output is again Stream (synced with the next acknowledged batch or a second later), as DESIGN.md 3.3b says. * acpmux: review 2 of #34901, tests (red): a crash model that follows real log syncs The logs' OS crash model now follows the syncs that ran (store/sync_model.rs, tests only): a segment keeps its length at its last sync that counts; a failed sync loses what was written since the one before, and a later sync does not bring it back; a cut ends a loss it removes. Every log segment sync goes through log_sync::sync_segment (same calls as before), where the model and FAIL_NEXT_LOG_SYNC hook in. Red tests, one per finding: - 1a an_os_crash_after_a_restart_from_a_kill_keeps_every_acked_line: the restart's first checkpoint retires records whose log lines no sync made durable; expected [1..5], the head gives [1]. - 1b an_os_crash_after_a_restart_from_a_failed_sync_keeps_every_acked_line: the same after a frozen checkpoint and a restart; expected [1..5]. - 2 a_hole_that_starts_inside_a_line_is_repaired (the reviewer's test): a torn line is read as line 6 of a log that moved on; expected [1..9], the head gives [1, 2, 3, 4, 5, 9]. - 3 a_failed_sync_in_flush_session_freezes_the_checkpoint: a failed sync in flush_session does not freeze the checkpoint. * acpmux: review 2 of #34901: a restart writes the replayed lines again; a torn line is not a line; flush_session freezes Finding 1 (high): the repair now writes every merged line again from the repair start, even when the log already has them all. A killed daemon's log writes may be in the page cache only, and after a failed sync the kernel may have dropped pages that still read back and that a later sync reports as written; the first checkpoint after the open then syncs bytes this process wrote before it retires their records. The lines only the log has from the repair start are journaled and made durable first, as before a cut. The work is bounded by the checkpoint interval. Finding 2: a piece of the log is a line only when it parses as one JSON value (piece_seq). A hole that starts inside a line keeps that line's {"seq":N, head and runs into a later line's tail; it was read as a line of a log that moved on, and the chat's journaled lines went aside. Now its bytes go and the lines are written again. Finding 3: a failed sync in flush_session freezes the checkpoint (its file sync is a SyncFailed now), as a failed sync in a checkpoint or at a log's close does. Nits: cut_at syncs events/ after its removals (and the kind-index folder after its copies go), so a removed segment cannot come back beside the lines written again; the README says that with the switch on an event line over 64 MiB is refused and its log does not get it. Tests: the four red tests of d2c63860ece. * acpmux: chat metadata through the event journal, tests (red, R-DURABILITY S3a) DESIGN.md 3.3e. Red on S2c: - last_seq_never_passes_the_log_after_an_os_crash (E13): session.json is synced at each save while the agent's lines wait for the journal's next sync, so after an OS crash the metadata names lines the log lost. - a_chat_made_after_the_checkpoint_survives_an_os_crash_that_loses_its_folder: nothing syncs the sessions folder when a chat's folder is made, and the replay never makes a folder again, so an acknowledged new chat is lost. - a_metadata_change_is_written_to_session_json_by_the_checkpoint: today each change writes and syncs session.json on the request path. Guards (green before and after): a chat made before the checkpoint and deleted is not made again; the replay never writes an older image over session.json. * acpmux: chat metadata through the event journal (R-DURABILITY S3a) DESIGN.md 3.3e, behind `store.journal` (default off). A metadata change is a `meta` record (kind 4: chat id, a `created` flag, the whole image) instead of a session.json write with two syncs on the request path; the reply waits for the record through S2c's gate, one group commit for a burst. - Overlay: each chat's newest image until a checkpoint writes session.json; load, load_startup and list read it first (list under the projection lock, so a checkpoint cannot drop an entry between its two reads). - Checkpoint: waits on its own thread until every record through its LSN is durable, then writes each image it read (session.json carries the image's LSN as `journalLsn`), then the logs and the sessions folder, as before. An image so names only lines the journal holds (E13). A chat deleted or made again meanwhile is passed (its overlay entry is gone or new). - Replay: per chat after its last `forget`, the newest image, written when newer than the file's `journalLsn`. A missing folder is made again only for a chat a replayed `created` record made; a chat made before the checkpoint whose folder is gone keeps its image in journal-unapplied/. - flush_session also waits until the chat's records are durable. - Delete drops the overlay entry with its `forget`, and removes the folder under the folders lock a checkpoint's write also takes. Tests: the 5 of 0505e9bee04 (3 red there). * acpmux: S3a, tests (red): folders handed out of the store Code outside the store reads and moves chat folders directly (transfer_in puts the live folder aside and installs a transferred chat; session_dir hands the folder to the transfer, the session files view and others). With the metadata in the overlay until a checkpoint: - a_folder_replaced_outside_the_store_keeps_its_own_metadata: a read, the checkpoint and the replay must not put the old folder's image in the new one. - session_dir_hands_out_a_folder_whose_metadata_is_written. - flush_session_writes_the_chats_metadata (the trait says log and metadata). - the_checkpoint_never_makes_a_folder_removed_outside_the_store_again. Red on 71bc09a1e63 (S3a's first implementation). * acpmux: S3a red tests: a replay after a folder swap; flush_session's chat has a log (as its callers' chats do) * acpmux: S3a: each metadata image is bound to its chat folder; session_dir and flush_session write it first Code outside the store reads and moves chat folders directly (a transfer puts the live folder aside and installs a transferred chat under the same id). So a `meta` record now carries its folder's identity (file id and birth time): - reads use an overlay image only while the folder at the path is its folder; - the checkpoint writes an image only into its own folder, never makes a missing one, and keeps the image of a removed or replaced folder in journal-unapplied/; - the replay writes an image only into its own folder (or a folder a replayed `created` record made), else keeps it aside. `session_dir` (the transfer, the session files view) and `flush_session` (the trait: "log and metadata to stable storage") write the chat's image first, once its record is durable. The list reads the overlay before the files (a checkpoint writes a file before it drops the entry), so it needs no lock. Tests: the 5 of 0280ac37f5e and 0e617489731. * acpmux: the qos journal tests module is macOS-only (its Linux build held only an unused import: clippy red, review of #35400 F0) * acpmux: review of #35400 F1 and F2, tests (red) - a_read_that_shows_a_line_covers_its_record (F1): a read sees a log line between its log write and its record (the appends' lock held there), and the reply built from it goes out with no wait for that record. - a_journal_that_syncs_nothing_asks_the_daemon_to_stop (F2): with a journal that makes nothing durable, every reply waits without a limit and nothing asks the daemon to stop. Test seams only: PAUSE_AFTER_LOG_WRITE, TEST_STALL_MS. * acpmux: review of #35400 F1, the streaming read, test (red): session/load's replay sends a line before its record * acpmux: the test seam PAUSE_AFTER_LOG_WRITE is reachable from the server tests * acpmux: review of #35400: a read covers the records of the lines it shows; a stalled journal stops the daemon F1: the append writes a line to the log, then appends its record and sets the acknowledged LSN, all under the appends' lock; reads take no lock. A read could show a line before its record existed, and the reply built from it went out at once. Now every journal-store read of the logs ends with a fence on the appends' lock (`Inner::fenced`): a line the read saw has its record appended and its LSN acknowledged when the read returns, so the reply's gate covers it. The one read that sends as it reads (`Hub::scan_events`, session/load's replay) stops at the chat's last_seq read first. F2: a journal writer that keeps failing (it retries without end) held every reply on every connection with no limit. A connection's gate now waits at most 10 s (`STALL_LIMIT`); past it, `Durability::stalled` asks the daemon to stop for a restart, as a failed log sync does. Tests: the 3 of 3916f84dd68 and c553a19fd8b. * threading-model: the acpmux event journal's threads and bounds (writer, checkpointer, per-connection gate; review of #35400) * acpmux: the stall test's journal syncs take 3 s, not an hour (a sync under way kept its delay and hung the test's end) * acpmux: S3a review (#35483) red tests: switch-off metadata and deletes, a late settle, a failed session.json folder sync, a purge after the checkpoint F1: metadata written with the switch off survives turning it on again; a chat deleted with the switch off after a failed catch-up is not made again. F2: a late settle never writes an older image over a checkpointed one (the reviewer's test and hold). F3: a failed folder sync after a metadata write freezes the checkpoint. F4: a chat made after the checkpoint and deleted is not made again after a kill. Test seams only; the code is the S3a head merged with feat. * acpmux: S3a review (#35483) red tests: a new chat is acknowledged only once its folder is durable; session_path never waits The root of F4 and of F1's delete case: the replay makes a chat's folder again, so a chat removed by any path comes back. The fix will never make a folder again, which needs an acknowledged chat's folder to survive an OS crash: the test models what each folder's last sync kept (sync_model) and asserts the gate's durability passes a new chat only after its sessions entry is synced. P3 1 of the review: a caller that needs only the path never waits for the journal (Store::session_path, a stub with session_dir's behavior here). * acpmux: S3a review (#35483) fixes: the replay never makes a folder again; switch-off changes are kept; no late or unsynced metadata write F1: each meta record carries the stamp (file id, modification time, length) of the session.json its save saw. The replay writes an image over a file the journal did not write only when one of the chat's records saw that file; otherwise the file moved on without the journal (a run with store.journal off whose catch-up did not replay, or an older build) and the image goes to journal-unapplied/, as S2 does for a log. F1 (deletes) and F4 at the root: the replay never makes a chat's folder again. A save that made a folder notes its record's LSN; the new acpmux-journal-folders thread (the daemon's disk tier) syncs sessions/ at once, and the durable LSN the S2c gate waits for stays below that record until then (journal_store/new_folders.rs). So an acknowledged chat's folder survives an OS crash, and a folder that is gone was removed: a purge whose forget a kill took back, a run with the switch off, the transfer recovery. The created flag and the replay's folder remake are gone. F2: write_through does not write a file that holds the same or a newer journalLsn. F3: session.json is written with its own syncs (file, then folder); a failed sync is SyncFailed and freezes the checkpoint; the replay and the kept-aside images return their errors. P3 1: Store::session_path gives the path with no lock, wait or write; an agent's start, the fixed-id check, the temporary-folder sweep, the host probe and the transfer recovery's root use it. The replaced test (a chat made after the checkpoint survives an OS crash that loses its folder) asserted a remake that cannot be right any more; the new test asserts the guarantee that makes the loss impossible, then the crash. Red (7 of 17 journal_meta_tests, each for its reason; the other 10 pass): macOS 5bc434a9 and 624d5df4, Linux a2207ff0 and 638a44ef. * acpmux: S3a delta review (N1) red tests: a change to a chat whose session.json another journal wrote reaches it A folder moved in from another home (build adoption), checkpointed and replayed, and a journal made again from nothing: the other journal's journalLsn must not hide a change made here. The first test is the reviewer's (fleet 4d0d98bc and 071ad7c4 at e944a7c21de). * acpmux: S3a delta review (N1): journalLsn is bound to its journal's id; a file another journal wrote follows the stamp rule session.json now carries journalId (16 hex digits, a string) beside journalLsn. The checkpoint's "held" check (F2) and the replay's rule compare an LSN only for a file with this journal's id: a file from another journal (a chat folder build adoption moved in from another home, or a home whose journal was made again) counts as one this journal did not write, so the stamp rule decides, as for a run with the switch off. cmux-journal: Journal::journal_id (the id already in every segment header; kept across opens and retirements), with a unit test. Red at 7c8cb957745: fleet c29b659d (Linux) and 47570494 (macOS) running; the reviewer's red at e944a7c21de: 4d0d98bc (macOS) and 071ad7c4 (Linux).

…a hole's last zero splits, an idle checkpointer sleeps (#35562) * cmux-journal: one append-only journal per daemon, with group commit and ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner). * cmux-journal: type a test range; drop an unread segment field (fleet clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71e3f9 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red. * cmux-journal: a failed F_FULLFSYNC is an error, not a plain fsync; no segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee. * cmux-journal: the past-the-end test reads back the storage it wrote to Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c87b8d: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5c53e. * cmux-journal: a sync a signal interrupts is made again (macOS) fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r). * cmux-journal: crash sim of write-back errors and crash in recovery; S1 review regressions (red) The S1 review (FIX at 89293fb15f39) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open * cmux-journal: the F2 regression damages the header of the last of several segments (red) The reviewer's F2 scenario has two segments: S1 holds 1..10, S2 (the last) holds 11..20, and S2's header is damaged. The test now rotates segments (300-byte segments, 20 acked records) and damages the last one's header, so the report must say lost_through 20 and the next append must take 21. On the code before the fix the moved segment's LSNs read as 0, so the next append reuses one. * cmux-journal: the crash sim forces the F4 order (failed sync, kill, OS crash) (red) On the code before the S1 fix, the crash-sim step was green (fleet 184129d8 at 263da86f223): 6 of 7 runs passed, and the mixed run failed only seed 140 in the full run and passed when it ran alone. The random faults reached the order the review found (F4) in 1 of 800 runs. So these tests were no red/green proof. The order: the sync of a write that spans pages fails and the disk gets none of them (Linux keeps them clean in the cache, so a later sync reports success); the process is killed in the writer's backoff; the next open reads the failed segment from the cache, replays its records and goes on; an OS crash or a power loss then takes those pages, and records an open already replayed are gone. - sim.rs: `fail_next_sync(pages, kind)` fails the next sync that has at least that many dirty pages, and the disk gets none of them. `crash_after_failure(n)` makes the n-th operation after the next injected failure the crash point. - crash_sim_tests.rs: a new run forces the order on every seed (a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record): epoch 0 aims the fault, the kill is a process crash at the writer's retry, the open after it does not crash, epoch 1 ends in an OS crash, and a third open checks that the records the second open replayed are all there. In the random runs, one epoch in four aims the fault, with the crash 1 to 3 operations after the failed sync and a crash of any kind after it. - A sim test checks the aimed fault itself (the_sim_fails_the_aimed_sync_and_crashes_in_the_backoff). Expected on this commit (old journal code): the forced run fails every seed; the mixed run fails most runs. The fix commit follows. * cmux-journal: fix the S1 review findings F1-F7 (markers, recovery copy, LSN floor, writer end) Review FIX at 89293fb15f39 (PR #33919 comment 6610468202). The regression tests are the two commits before this one. - Sync markers (format.rs): an entry with flags 1 and no payload, whose LSN field says "every record of this segment before me is synced". The writer appends one after each successful sync that covered records (after the durable listeners, so no reply waits for it), and syncs it fully at a seal and at a rotation. - F4, recovery (step 4 and 6): the last segment is trusted only up to its last marker. Its records after the marker may sit in pages a failed sync dropped (a later sync on another descriptor reports success without them), so recovery writes them again into a new segment: header, records and a marker, fully synced under `<name>.seg.new`, renamed into place, folder synced. The journal always goes on in a segment this open made, never in one an earlier process wrote. A restart in the writer's backoff after a failed sync is then safe. - F1, quarantine: every LSN a quarantined segment holds counts in the LSN floor at every open, and a quarantine that no new segment followed (a crash in the middle of an earlier recovery) is reported as damage again. The highest LSN of every moved segment is read before the first rename. - F2, highest LSN: a segment whose header is damaged is read from its first record (each record's checksum covers its LSN), then the bytes after the first bad entry are scanned for records and markers that still check out. The same scan runs after a cut in the last segment: a marker there is media damage (reported), records there are a torn write (`torn_lsns`); their LSNs are never used again. - F3, writer end: a guard closes the journal when the writer thread ends, also by a panic: appends fail with Closed, waits end with WriterEnded, the listeners are dropped (so a channel a held reply waits on closes), and stats.failing says why. The crate does not abort the process; the daemon owns that policy. - F6: recovery closes each segment after its header is read and opens one again only to read it. (Retirement of compacted segments is an S2 API, before the first consumer.) - F5: recovery syncs the parent folder after it makes the journal folder. - F7: format.rs and lib.rs now say what the code does: a failed write is cut back and retried in the same segment; the cut removes only bytes never synced and never acknowledged; nothing shortens a segment past a synced record. Docs: journal.rs says when open's replayed records are durable (on Ok) and what a stopped writer does. * cmux-journal: split the crash sim's machine model into sim/machine.rs (review G5) sim.rs had 924 lines and 63 functions, over the 60-function limit for a new Rust file (the god-file check warned "not in baseline: split it"). The machine model (each file's page cache and disk, the ordered log of names, and what an OS crash or a power loss leaves) moves to sim/machine.rs (297 lines, 19 functions); the process view and the Storage implementation stay in sim.rs (639 lines, 30 functions). No behavior change: the items move as they are, made visible to the sim module. * cmux-journal: re-review regressions G1-G4 and page eviction in the crash sim (red) Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993). Tests and sim only; the fix follows. - sim: the pages a failed sync dropped stay in the cache until an eviction; `evict_dropped_pages_after_read(true)` makes each read evict the dropped pages it read, so a second read of them sees the disk's bytes. - G1 recovery_copies_the_records_it_checked_when_the_cache_drops_them: a sync fails and drops record 2's pages, a kill in the backoff, the next open reads record 2 from the cache (replayed) and the pages are evicted after that read; then an OS crash. Record 2 and the record 3 acked after the restart must survive. On this code recovery's second, unchecked read stages zeros under a vouching marker, and the last open keeps only record 1 and reports damage. - G2: the F1 regression asserts that every open after the damage reports the loss (lost_through 40). On this code a crash at the open's last step (the folder sync after the new segment's rename) loses the report for good. - G3 a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system): a journal in a symlinked state folder, and a journal folder that is a symlink. On this code the parent folder's sync opens with O_NOFOLLOW (ELOOP). - G4 an_open_after_a_finished_quarantine_reads_no_quarantined_segment: on this code every open reads every quarantined segment in full. * cmux-journal: fix the re-review findings G1-G4 and G6 Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993); the regression tests are the commit before this one (red: fleet de35f6d0). - G1: recovery writes again exactly the bytes its checked read returned. The last segment's entries after its last marker are kept from the checked read (header and payload) and staged from memory; the second, unchecked read of the file is gone. A page that a failed sync dropped can be evicted between two reads, and the second read then returned the disk's zeros, which the copy kept under a vouching marker. - G2: the damage an open finds is written to `DAMAGE` (whole, fully synced, renamed into place, folder synced) before the new segment that hides it from later opens. Every open reports the recorded damage first, until the daemon calls the new `Journal::clear_damage`, which moves the record into `quarantine/` (kept, never deleted). - G3: a folder sync follows a symlink in its path (no O_NOFOLLOW): a state folder, or the journal folder itself, may be a symlink. The sync is for the folder that holds the entries. - G4: the quarantined segments are read for their LSNs only while no live segment is numbered above them (an unfinished quarantine); once one is, its first LSN already carries their floor. - G6: `checked_div` in the scan of the bytes after a cut (the crate's one tracked arithmetic hit). - G5 (the sim split) is the commit before the regressions. * cmux-journal: third-review regressions G7 (A, B) and N1 (red) Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018). Tests only. - G7 A a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn: after a finished quarantine, one byte of the empty new segment's header changes. On this code the next open skips the quarantine scan (the listing counts the damaged segment) and the next append reuses a quarantined LSN. - G7 B a_damaged_newest_segment_after_a_quarantine_reports_its_records: the newest segment holds acked 41..43, and its header and its first record are damaged. On this code the moved segment is scanned with a base far below 41, so the loss is not reported through 43 and the next append is not 44. - N1: the F1 regression also asserts that an open reports each loss once (no reported LSN range lies inside another). On this code a crash after the quarantine's renames makes the next open report the same loss twice. * cmux-journal: fix the third-review findings G7, N1, N2; split recovery.rs Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018); the regression tests are the commit before this one. - G7: the quarantine scan is skipped only when a segment whose header checked out is numbered above every quarantined segment (the folder listing also counted a newest segment with a damaged header, which carries no floor). The scans start from the running floor: `quarantined_floor` and `quarantine` pass the highest LSN counted so far as the base of the next file's scan, so a quarantined file with no readable header and no readable first record is scanned for the LSNs right above what is counted. - The "unfinished quarantine" report is made only when the quarantine holds LSNs nothing else counted (an old quarantine behind a segment that a crash tore at its creation holds none, and a crash is not damage). Its reason now also names a damaged newest header. - N1: a loss whose LSN range lies inside a recorded one is the same loss found again after a crash: it is neither recorded nor reported twice. - N2: recovery.rs's docs name `DAMAGE`; two doc lines over 100 columns are wrapped (lib.rs, sim.rs). - recovery.rs was at 987 of its 1000 lines: the damage record moves to recovery/damage.rs, the entry reader and the LSN scans to recovery/scan.rs (moved as they are). * cmux-journal: report each loss once, also when one open finds it twice (N1) The red run of the N1 regression (fleet 27d011ca, op 64 of the F1 test) shows the case the previous commit did not cover: a crash after the quarantine's renames and before DAMAGE is written; the next open finds both the damaged synced record of the last segment (7..14) and the unfinished quarantine (7..40), and reported both. Now every loss inside another (recorded or found by this open) is dropped, the larger range kept; DAMAGE is rewritten only when its set of losses changed. * cmux-journal: fourth-review regressions G8, N3, N4 (red) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report. * cmux-journal: the quarantine's floor in its file names; DAMAGE before the moves (G8, N3, N4) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format. * cmux-journal: sort the quarantine list with sort_by_key (clippy unnecessary_sort_by) Fleet d9f22bed at b589044b852: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`). * cmux-journal: Journal::on_written, a listener per written batch (for S2's projector) R-DURABILITY S2 (acpmux event logs through the journal) projects each chat's log line into its ndjson file once the journal has WRITTEN the batch that holds it, so readers see stream lines at once. The listener runs on the writer thread after the write and before the sync, once per batch that moved the written LSN: a batch written again into a new segment after a failed sync is not heard twice. When the writer stops, these listeners are dropped too. Test: heard while the batch's sync still runs, once. Port of 145faa6faa8 (branch rewrite/r-durability-s2-acpmux-logs) onto the S1 review fixes. * cmux-journal: Journal::retire_through, the one delete (segments a compaction point covers) R-DURABILITY S2: a daemon's projector compacts the journal into its own files (acpmux: the chat ndjson logs, then a synced checkpoint). Without a delete the segment count only grows (review finding F6). `retire_through(compacted)` deletes the segments that hold no record after the caller's compaction point: a segment goes when the next segment's "previous last LSN" is at or before it, so the newest segment never goes. The point is capped at the durable LSN. The deletes and a folder sync run on the caller's thread (the projector, never a request path). A delete that fails leaves its segment live for a later call. Why a crash during a retirement is safe: the caller reopens from a point at or after `compacted`, and recovery reads nothing from a segment the point covers (it skips it on its next segment's header), so any subset of the deletes may reach the disk. A point older than the retired segments is reported as Damage::Missing. Quarantined segments are never deleted. - Storage::remove_file (FsStorage: std::fs::remove_file; SimStorage: one more change in the ordered log of names, durable at the folder sync). - The live segment list: recovery's, then each segment the writer makes. - JournalStats::segments_retired. - Tests: only covered segments go, and a second call deletes none; an OS crash after it keeps every record after the point and an older point is reported; a crash at every operation of a retirement (process, OS crash, power loss) keeps every record after the point and never reuses an LSN; the newest segment stays; a point in the middle of what this writer wrote. This adds a delete of journal data, so its PR is risk:high. * acpmux: move MemoryStore and the store settings into their own files store.rs (2139 lines, 120 fns) and config.rs (1909 lines, 97 fns) are over the god-file limit, and the journal store (R-DURABILITY S2) must not grow them. MemoryStore moves to store/memory.rs and StoreMode, StoreConfig and its defaults to config/store_config.rs; both stay exported where they were (crate::store::MemoryStore, crate::config::{StoreConfig, StoreMode}). No behavior change: the code moves as it is. * acpmux: the event log's journal switch and its crash tests (red) R-DURABILITY S2 (DESIGN.md 3.3b). Today no event-log append is synced: a line a client was told about is on the disk only after the next flush (shutdown), so an OS crash or a power loss takes back every line since (E12). This commit adds only the switch, a seam and the tests; the journal store follows. - `store.journal` (StoreConfig::journal, default false). - `Store::wait_durable(timeout)`: tests and tools wait until every appended record is durable. The default returns at once (the Local and Memory stores never sync an append), which is today's behavior. - `store::open_with_journal` (tests): the store with the journal on a given `cmux_journal::Storage`. Here it ignores the journal and opens LocalStore. - acpmux depends on cmux-journal (dev: with the `sim` feature). - store/journal_store_tests.rs, red on this commit: - a_line_the_store_says_is_durable_survives_an_os_crash (lines 2..20 are lost: nothing synced them); - a_deleted_chat_stays_deleted_after_a_crash and a_line_the_log_refused_is_never_written_later pass here, and must keep passing with the journal store (no replay may bring them back); - a_kill_keeps_every_line_once passes here and must keep passing (a replay after a kill must write no line twice). * acpmux: the event log through one journal, behind `store.journal` (R-DURABILITY S2) DESIGN.md 3.3b. With `store.journal` on, each event-log append is one record in `<home>/journal/` (cmux-journal: group commit, one sync per batch; a power loss takes back at most the last second, always a suffix), then the same line goes to the chat's ndjson as before. Readers, segments and file formats do not change. Default off. - store/journal_store.rs (JournalStore over LocalStore): - append: under one lock, the journal record (a memory copy, no I/O), then the log line; a line the log refuses gets a `cancel` record. Agent stream lines (`in` session/update, claude.*, codex.*) are Class::Stream, every other line Class::Ack. - checkpoint (`<home>/journal-projected.json`): every 30 s while the journal grew, past 64 MiB, at flush (shutdown) and before a retention commit. It syncs each log written since (LocalStore::sync_log, with no lock held while the disk works) and the sessions folder, writes the LSN (temp file, sync, rename, folder sync, every error checked), then retires the journal before it. A log close that failed its sync freezes the checkpoint until a restart (LocalStore::sync_failed). - recovery: the records after the checkpoint; for each chat, the log is cut right after its last readable line below the first journaled seq (later segments removed, the boundary one shortened, their kind-index copies dropped), and the journaled lines are written again in order. `forget` deletes a chat again; `cancel` keeps a refused line out. - delete: `forget` record, folder removal, then a bounded wait until the record is durable. - seal: the logs refuse writes and the journal syncs fully. - LocalStore: `append` is serialize + `append_line`; `sync_log`; a failed sync at a log's close is noted (`sync_failed`). - store::open opens the JournalStore when `store.journal` is on. Not yet (S2c): replies and events do not wait for durability; the daemon does not stop when the journal closes; the writer thread keeps the process's Utility I/O policy. * acpmux: journal store tests for the seal and a chat made again after its delete (red) - a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay: a chat is deleted (its forget record), made again under the same id (a transfer back: session.json and new lines), then a kill -9. On this code the replay deletes the folder again and its session.json is gone. - a_line_after_the_seal_is_refused_and_never_written_later: coverage of the seal (it passes here; the fix also closes a race the test cannot force). * acpmux: journal store: the seal under the appends' lock; a replay keeps a chat made again - seal takes the appends' lock: an append lands in both the journal and the logs, or in neither. Before, an append that raced the seal could leave a journal record whose line the sealed logs refused (its cancel then failed on the closed journal), and the next start would write that line. - append and delete check the logs' seal first, under the same lock, and do not touch the journal after it (no error per append at shutdown). - replay deletes a forgotten chat again only when no line follows its forget record: otherwise the chat was made again under the same id, and its folder holds that chat's own files. * cmux-journal: fifth-review regressions G9 (red) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4. * cmux-journal: name a skipped LSN floor in a FLOOR.<lsn> file (G9) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646); the regression tests are the commit before this one. When an open gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which a copy then hides from every later open), the only record of that floor was the header of the empty segment it makes; a damaged header lost it, and DAMAGE is cleared by `clear_damage` (torn records are not damage at all). Now such an open makes the empty file `FLOOR.<lsn>` and syncs its folder before any new segment, and every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine's names and this open's moves say their floors already: no file is made for them. * acpmux: journal store regressions from the S2 review (red) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824). Each test fails on this commit and passes after the fix that follows it: - 1a lines_written_with_the_switch_off_survive_turning_it_on_again - 1b a_kill_while_the_journal_syncs_keeps_the_lines_the_log_got - 1c a_line_flush_session_made_durable_survives_losing_its_record - 2 a_folder_put_back_after_a_delete_is_not_deleted_by_the_replay - 3 a_kill_in_a_delete_leaves_the_chat_whole_or_gone - 4 a_delete_does_not_wait_for_the_journal_sync - 5 a_failed_log_sync_freezes_the_checkpoint, a_checkpoint_that_keeps_failing_asks_the_daemon_to_stop - 6 one_chat_that_cannot_be_repaired_does_not_stop_the_open - 7 the_journal_storage_refuses_a_lost_home - the repair keeps a log's lines after a hole, also through a crash in it: an_os_crash_hole_is_repaired_and_the_lines_after_it_kept, a_crash_inside_a_repair_loses_no_line - nits: an_unreadable_checkpoint_fails_the_open, the_open_waits_for_the_previous_daemon_to_let_go_of_the_journal, a_crashed_checkpoint_temporary_is_removed_at_open, a_crash_right_after_a_checkpoint_reports_no_damage - the catch-up with the switch off: switching_off_after_a_crash_keeps_every_acknowledged_line - cmux-journal: check_append_says_what_append_would_do So they compile here, this commit adds the seams they use, with no behavior: `Store::stop_reason` (default None), `Journal::check_append` (always Ok), `FencedStorage` (never asks its fence), the open report and a test stop point after a repair's cut. * acpmux: the checkpoint freeze test fails a log sync, not a folder open (red) The freeze guards against a failed fsync (a later fsync of the same file may report success for pages the kernel dropped), not against a folder the store cannot open: a chmod 000 folder is an open error, which the fix retries instead of freezing. The test now fails the log sync itself through a test seam, `FAIL_NEXT_LOG_SYNC`. Red on the code before the fix, like the other review regressions (fleet d8e7037d, 1c22175e, 63604b26 at e030f4fe402: 17 of 17 new acpmux tests and the crate's check_append test fail). * acpmux: the journal store repairs logs, never rebuilds them (S2 review) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824); the regression tests are the two commits before this one. DESIGN.md 3.3c. - 1a/1b/1c: recovery no longer cuts a chat's log at its first journaled line and writes back only the journal's lines. It merges the log's lines and the journal's by seq: a line in both must be equal, a line only the log has stays, a line only the journal has goes in its place. Only bytes that do not read (a torn tail, a hole of zeros) are dropped. When the merge must change the log before its end, the lines only the log holds from there are journaled and made durable first, so a crash inside the repair loses no line. A log that moved on without the journal (a line in both that differs, lines out of order) is left as it is, and the journal's lines go to journal-unapplied/. - Append writes the log first, then the record, under one lock, after `Journal::check_append` (new in cmux-journal: the append's refusals, asked before any write). So a record always has its log line, and the `cancel` record is gone. - 2/3/4: the replay never deletes a folder and never makes a missing one again (a forget only drops the chat's earlier journaled lines). Delete appends its forget under the lock and removes the folder off it, with no wait for the record. - 5: a failed fsync in a checkpoint freezes it (a folder that does not open is retried); a frozen checkpoint returns an error, and the store asks the daemon to stop for a restart (`Store::stop_reason`, wired in daemon.rs), whose replay writes the logs again. Ten failed checkpoints in a row ask the same, so the journal past the checkpoint stays bounded. - 6: an error in one chat keeps its journaled lines in journal-unapplied/ and the open goes on. - 7: the journal's storage refuses writes under a lost home (`FencedStorage`, `config::refuse_lost_home`). - With store.journal off, a journal left by a run with it on is replayed, checkpointed and retired first (`catch_up`). - Nits: the checkpoint is never past the journal's durable end (no `Ahead` after a normal crash); damage is cleared after the next checkpoint; a kind-index copy goes before its segment is cut; an unreadable checkpoint fails the open; the open waits up to 5 s for the journal lock; crashed checkpoint temporaries are removed; `on_written` leaves cmux-journal (no user); README lists store.journal. * acpmux: split the journal store under its size budget (a move) journal_store.rs grew to 1085 lines and 71 functions with the review fix (budget 1000 and 60), and the fix grew store.rs and daemon.rs, which are over budget on the base. A pure move, no behavior change: - store/journal_store/{checkpoint,fenced,repair}.rs: the checkpoint and its thread, `FencedStorage`, the replay and the repair. - store/log_sync.rs: `LocalStore::sync_log`, `SyncFailed` and the test seam `FAIL_NEXT_LOG_SYNC`, out of store.rs (2139 lines on the base, 2113 now). - daemon/stop_causes.rs: `AbortOnDrop`, `next_signal` and `store_asks_to_stop`, out of daemon.rs (2052 on the base, 2046 now). check-no-godfiles.sh --base 51cfdee876b on every changed file: no file over budget grew, and no new file is over budget. * acpmux: the unrepairable-chat test waits for line 1 to be durable before its checkpoint At 03fccdef69c (fleet 32df3c72 on aws-m4pro-9, 144d4ccc on Linux) the test kept chat-b's lines [1, 2, 3] aside, not [2, 3]: it flushed right after appending line 1, and the checkpoint is now never past the journal's durable end (the review's Ahead nit), so line 1 was still past it. The fix's behavior is right; the test now waits until line 1 is durable, so its checkpoint covers it. On the code before the fix the test still fails (the open stops at chat-b's link). * acpmux: replies and events wait for durability, tests (red, S2c) DESIGN.md 3.3d. With the event journal on and every journal sync taking 400 ms (`SimStorage::set_sync_delay`): - an_acked_change_is_answered_once_its_record_is_durable: a rename's reply must come no sooner than one sync after its `renamed` record. Red here: the reply goes at once. - agent_output_is_not_held_behind_another_chats_acked_change: agent output to a connection attached to its chat goes at once while another chat's acknowledged change waits for its sync. Green here (nothing waits); it guards the fix against holding agent output behind changes the connection does not hear. * acpmux: replies and events wait until what they show is durable (S2c) DESIGN.md 3.3d; the tests are the commit before this one. With `store.journal` on, each connection gets an outbound gate (server/durable_gate.rs) between it and its transport: - Every line a connection queues raises the gate's `needs` to the last acknowledged LSN (`Store::durability().acked`, set in the journal store's append before the hub publishes anything of the record), except the lines of a `Stream` record's event (agent output, DESIGN C4): the fan-out delivers those in a thread-local stream scope. - The gate's task takes every queued line at once, reads `needs`, waits (async) until the journal's durable LSN reaches it, then sends the batch in order: one wait per batch, as the journal's group commit. - The durable LSN is a `tokio::sync::watch` fed by `Journal::on_durable`. The listener owns a guard: a writer that ends while the store is not being dropped (a panic on its thread) asks the daemon to stop (`Store::stop_reason`, decision 9.8), and the watch closes, so each gate drops what it holds and ends, which closes its connection. - Without the journal, `Store::durability()` is None: no gate, and `Conn::raise_gate` does nothing. Moves, so server/mod.rs and store.rs stay under their size on the base: the outbound queue helpers to server/outbound.rs, `open_with_journal` to the journal store, and `streamed` to store/durability.rs. * acpmux: the journal lock wait's deadline without an unchecked Instant addition `Instant + Duration` can panic on overflow (clippy::arithmetic_side_effects, tracked by the panic ratchet). The wait now has no deadline only in the case that cannot happen (an Instant 5 s ahead overflowing), where it stops waiting at once. * acpmux: a failed log sync stops the real daemon for a restart, test (red) Lead rule 2026-11-04 (3): a daemon-level test of the store's "stop for a restart" before S2c merges, red first. With store.journal on and every log sync failing (a debug build's ACPMUX_TEST_FAIL_LOG_SYNC; the checkpoint every 200 ms through ACPMUX_TEST_JOURNAL_CHECKPOINT_MS), a rename of a chat on disk makes its log dirty, the checkpoint's sync fails and freezes it, and the store asks the daemon to stop. The test asserts: the WebSocket client gets the restart close (1012), the socket client's pending `_acpmux/wait` ends with the connection (no hang), and the daemon exits with status 74. Red here: the daemon stops and sends 1012, but exits 0 (no exit status says that it stopped to be repaired). This commit adds only the two debug-build seams (release builds have none) and the test. * acpmux: a daemon its store stopped for a restart exits with status 74 The test is the commit before this one. `daemon::run` returned Ok and the process exited 0 for every stop, so a supervisor or a person could not tell a stop to be repaired from a normal one; with systemd `Restart=on-failure` it would not even start again. Now the store's stop (`store_asks_to_stop`: a failed log sync froze the event journal's checkpoint, or the journal's writer ended) marks itself, sets the restart handoff (WebSocket clients get 1012, as before), and the process exits with `daemon::EXIT_REPAIR` (74, EX_IOERR). The units prevent only 75, so launchd KeepAlive and systemd Restart=always start it again; the app does not read the daemon's exit status. README: store.journal names the status. * acpmux: the restart test's request in flight goes through the acpmux client Lead rule (3): the client sees a typed error, not a hang. The socket client is now the acpmux client library (acpmux::client::Client, as the CLI uses it), and the test asserts what the CLI makes of the request that the stop ends: `daemon_closed`, retryable (cli::errors::classify). The first red (fleet 41f2d037, hq run 41804640229, at 80c5fb2b7f8) failed only on the exit status (0, wanted 74), with 1012 and the end of the connection passing. * acpmux: the daemon's disk tier with the journal on, tests (red) Lead rule 2026-11-04 (2): a reply the person waits for never waits on a throttled disk sync. The fleet A/B (STATUS 11554) found that at IOPOL_UTILITY an acknowledged journal append waits about 16 ms p50 under another process's disk stress (0.7 to 1.1 ms at the default tier). - the_daemon_with_the_journal_keeps_the_default_disk_tier: with store.journal on, the daemon process keeps the default tier; without it, utility as before (each case in a child of the test binary). - a_background_thread_yields_its_disk_work: a dedicated background thread (the checkpointer) works at utility; a thread beside it does not. The API they use is stubbed here (`qos::enter_daemon` ignores the journal; `qos::background_thread` does nothing), so both fail on macOS. * acpmux: the restart test borrows no &str twice (clippy needless_borrows_for_generic_args) Found by fleet 7f8afd9d (hq run: the S2c server step at 62b4f68ef8b); the step's other clippy red is the base red in link_push_bench.rs. * acpmux: with the journal on, the daemon keeps the default disk tier (S2c) The tests are 994f74119fb. Lead rule 2026-11-04: a reply the person waits for never waits on a throttled disk sync. Measured on two owned Macs (STATUS 11554): one acknowledged append to durable at IOPOL_UTILITY under another app's disk stress took 16.3-16.7 ms p50 and 37-45 ms p95, against 0.7-1.1 ms p50 and 1.5-1.8 ms p95 at the default tier. - qos::enter_daemon(journal): with store.journal on, the daemon process keeps the default disk tier; without it, utility as before. The switch is read alone from config.json (config::store_journal_enabled), before any thread starts, with no harness discovery. - qos::background_thread: the journal store's checkpointer lowers its own disk I/O to utility for its whole life (a thread that does nothing else: a thread-scope policy drops it out of QoS for good). * acpmux: an agent chunk is a Stream record, test (red) The hub's tap logs an ACP session/update under its sessionUpdate kind (hub/tap.rs: `kind = su.to_owned()`), so a record's kind is `agent_message_chunk`, `tool_call`, ... and never `session/update`; the message's method is. The journal store's `streamed` checked the kind, so every agent chunk was journaled as Ack: a sync per batch at the token rate, and in S2c each token's event waited for that sync (the S2c I/O A/B: STRESS token to client 2.2 ms -> 20.3 ms p50 with the journal on). Red here: the chunk reads back as Ack. * acpmux: an agent chunk is a Stream record (the record's message names session/update) The test is the commit before this one. `streamed` now reads the logged message's method: the hub's tap logs a session/update under its sessionUpdate kind (hub/tap.rs), so the kind never said it. Agent output is again Stream (synced with the next acknowledged batch or a second later), as DESIGN.md 3.3b says. * acpmux: review 2 of #34901, tests (red): a crash model that follows real log syncs The logs' OS crash model now follows the syncs that ran (store/sync_model.rs, tests only): a segment keeps its length at its last sync that counts; a failed sync loses what was written since the one before, and a later sync does not bring it back; a cut ends a loss it removes. Every log segment sync goes through log_sync::sync_segment (same calls as before), where the model and FAIL_NEXT_LOG_SYNC hook in. Red tests, one per finding: - 1a an_os_crash_after_a_restart_from_a_kill_keeps_every_acked_line: the restart's first checkpoint retires records whose log lines no sync made durable; expected [1..5], the head gives [1]. - 1b an_os_crash_after_a_restart_from_a_failed_sync_keeps_every_acked_line: the same after a frozen checkpoint and a restart; expected [1..5]. - 2 a_hole_that_starts_inside_a_line_is_repaired (the reviewer's test): a torn line is read as line 6 of a log that moved on; expected [1..9], the head gives [1, 2, 3, 4, 5, 9]. - 3 a_failed_sync_in_flush_session_freezes_the_checkpoint: a failed sync in flush_session does not freeze the checkpoint. * acpmux: review 2 of #34901: a restart writes the replayed lines again; a torn line is not a line; flush_session freezes Finding 1 (high): the repair now writes every merged line again from the repair start, even when the log already has them all. A killed daemon's log writes may be in the page cache only, and after a failed sync the kernel may have dropped pages that still read back and that a later sync reports as written; the first checkpoint after the open then syncs bytes this process wrote before it retires their records. The lines only the log has from the repair start are journaled and made durable first, as before a cut. The work is bounded by the checkpoint interval. Finding 2: a piece of the log is a line only when it parses as one JSON value (piece_seq). A hole that starts inside a line keeps that line's {"seq":N, head and runs into a later line's tail; it was read as a line of a log that moved on, and the chat's journaled lines went aside. Now its bytes go and the lines are written again. Finding 3: a failed sync in flush_session freezes the checkpoint (its file sync is a SyncFailed now), as a failed sync in a checkpoint or at a log's close does. Nits: cut_at syncs events/ after its removals (and the kind-index folder after its copies go), so a removed segment cannot come back beside the lines written again; the README says that with the switch on an event line over 64 MiB is refused and its log does not get it. Tests: the four red tests of d2c63860ece. * acpmux: chat metadata through the event journal, tests (red, R-DURABILITY S3a) DESIGN.md 3.3e. Red on S2c: - last_seq_never_passes_the_log_after_an_os_crash (E13): session.json is synced at each save while the agent's lines wait for the journal's next sync, so after an OS crash the metadata names lines the log lost. - a_chat_made_after_the_checkpoint_survives_an_os_crash_that_loses_its_folder: nothing syncs the sessions folder when a chat's folder is made, and the replay never makes a folder again, so an acknowledged new chat is lost. - a_metadata_change_is_written_to_session_json_by_the_checkpoint: today each change writes and syncs session.json on the request path. Guards (green before and after): a chat made before the checkpoint and deleted is not made again; the replay never writes an older image over session.json. * acpmux: chat metadata through the event journal (R-DURABILITY S3a) DESIGN.md 3.3e, behind `store.journal` (default off). A metadata change is a `meta` record (kind 4: chat id, a `created` flag, the whole image) instead of a session.json write with two syncs on the request path; the reply waits for the record through S2c's gate, one group commit for a burst. - Overlay: each chat's newest image until a checkpoint writes session.json; load, load_startup and list read it first (list under the projection lock, so a checkpoint cannot drop an entry between its two reads). - Checkpoint: waits on its own thread until every record through its LSN is durable, then writes each image it read (session.json carries the image's LSN as `journalLsn`), then the logs and the sessions folder, as before. An image so names only lines the journal holds (E13). A chat deleted or made again meanwhile is passed (its overlay entry is gone or new). - Replay: per chat after its last `forget`, the newest image, written when newer than the file's `journalLsn`. A missing folder is made again only for a chat a replayed `created` record made; a chat made before the checkpoint whose folder is gone keeps its image in journal-unapplied/. - flush_session also waits until the chat's records are durable. - Delete drops the overlay entry with its `forget`, and removes the folder under the folders lock a checkpoint's write also takes. Tests: the 5 of 0505e9bee04 (3 red there). * acpmux: S3a, tests (red): folders handed out of the store Code outside the store reads and moves chat folders directly (transfer_in puts the live folder aside and installs a transferred chat; session_dir hands the folder to the transfer, the session files view and others). With the metadata in the overlay until a checkpoint: - a_folder_replaced_outside_the_store_keeps_its_own_metadata: a read, the checkpoint and the replay must not put the old folder's image in the new one. - session_dir_hands_out_a_folder_whose_metadata_is_written. - flush_session_writes_the_chats_metadata (the trait says log and metadata). - the_checkpoint_never_makes_a_folder_removed_outside_the_store_again. Red on 71bc09a1e63 (S3a's first implementation). * acpmux: S3a red tests: a replay after a folder swap; flush_session's chat has a log (as its callers' chats do) * acpmux: S3a: each metadata image is bound to its chat folder; session_dir and flush_session write it first Code outside the store reads and moves chat folders directly (a transfer puts the live folder aside and installs a transferred chat under the same id). So a `meta` record now carries its folder's identity (file id and birth time): - reads use an overlay image only while the folder at the path is its folder; - the checkpoint writes an image only into its own folder, never makes a missing one, and keeps the image of a removed or replaced folder in journal-unapplied/; - the replay writes an image only into its own folder (or a folder a replayed `created` record made), else keeps it aside. `session_dir` (the transfer, the session files view) and `flush_session` (the trait: "log and metadata to stable storage") write the chat's image first, once its record is durable. The list reads the overlay before the files (a checkpoint writes a file before it drops the entry), so it needs no lock. Tests: the 5 of 0280ac37f5e and 0e617489731. * acpmux: S2 follow-ups from review 3 of #34901, tests (red) - a_chat_whose_repair_failed_is_repaired_at_the_next_start (follow-up 1): the open's checkpoint retires the records of a chat whose repair failed, so no later start puts its lines back (they are only in journal-unapplied/). - a_hole_that_ends_at_a_line_start_keeps_that_line (follow-up 2): a torn line, zeros, then a whole line only the log holds: the piece does not parse and the whole line is dropped. - an_idle_store_does_not_wake_its_checkpointer (follow-up 4, cmux-ad m9): the checkpointer wakes every period with nothing to do. Test seams only (no behavior change): CHECKPOINTER_WAKES and TEST_CHECKPOINT_EVERY_MS. * acpmux: S2 follow-ups from review 3 of #34901: a failed repair holds its records; a hole's last zero splits; an idle checkpointer sleeps - Follow-up 1: a chat whose repair fails at the open (its log cannot be read or written now) keeps its lines in journal-unapplied/ as before, and the checkpoint now stays below its first journaled LSN (`Order::held`), so the next start repairs it again. Bounded: past 256 MiB of journal after it, the hold goes and the lines stay only in journal-unapplied/ (an error says so). - Follow-up 2: a piece splits at its last zero byte (a whole JSON line holds none), so a hole that ends at a line start, after a torn line, keeps that line (it mattered for a line only the log holds). - Follow-up 4 (cmux-ad m9): with nothing to checkpoint, the checkpointer sleeps until a change wakes it (`Inner::touched`, one atomic read per append), then checkpoints one period later; before, it woke every 30 s. Tests: the 3 of d147057bf85. * acpmux: the qos journal tests module is macOS-only (its Linux build held only an unused import: clippy red, review of #35400 F0) * acpmux: review of #35400 F1 and F2, tests (red) - a_read_that_shows_a_line_covers_its_record (F1): a read sees a log line between its log write and its record (the appends' lock held there), and the reply built from it goes out with no wait for that record. - a_journal_that_syncs_nothing_asks_the_daemon_to_stop (F2): with a journal that makes nothing durable, every reply waits without a limit and nothing asks the daemon to stop. Test seams only: PAUSE_AFTER_LOG_WRITE, TEST_STALL_MS. * acpmux: review of #35400 F1, the streaming read, test (red): session/load's replay sends a line before its record * acpmux: the test seam PAUSE_AFTER_LOG_WRITE is reachable from the server tests * acpmux: review of #35400: a read covers the records of the lines it shows; a stalled journal stops the daemon F1: the append writes a line to the log, then appends its record and sets the acknowledged LSN, all under the appends' lock; reads take no lock. A read could show a line before its record existed, and the reply built from it went out at once. Now every journal-store read of the logs ends with a fence on the appends' lock (`Inner::fenced`): a line the read saw has its record appended and its LSN acknowledged when the read returns, so the reply's gate covers it. The one read that sends as it reads (`Hub::scan_events`, session/load's replay) stops at the chat's last_seq read first. F2: a journal writer that keeps failing (it retries without end) held every reply on every connection with no limit. A connection's gate now waits at most 10 s (`STALL_LIMIT`); past it, `Durability::stalled` asks the daemon to stop for a restart, as a failed log sync does. Tests: the 3 of 3916f84dd68 and c553a19fd8b. * threading-model: the acpmux event journal's threads and bounds (writer, checkpointer, per-connection gate; review of #35400) * acpmux: the stall test's journal syncs take 3 s, not an hour (a sync under way kept its delay and hung the test's end) * acpmux: the idle checkpointer's period, test (red): a change seen by its second look must not start a checkpoint at once - a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period: the checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies when it gets the timer lock. The second look saw the change, so the checkpointer is in its period wait by then, and the notify ends that wait: a checkpoint runs at once on the checkpointer's thread. This is the cause of a_failed_log_sync_freezes_the_checkpoint failing in the full store:: run of fleet 0577ac62 (that early checkpoint took the line whose sync the test fails, on a thread where the failure is not set). The test holds both threads at the two points, so the order is exact. Test seams only (no behavior change): the checkpointer's test seams are now by the store's home (period, wake count, the two holds), so tests that run as threads of one process under cargo test do not see each other's. * acpmux: the idle checkpointer's period ends only on its own state, not on a wake meant for its idle wait The checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies once it gets the timer lock. Its change shows in the second look, so the checkpointer has gone to its period wait by then, and that notify ended the period at once: a checkpoint ran right away on the checkpointer's thread (a burst no longer shared one checkpoint), and a spurious wakeup could do the same. In the full store:: run of fleet 0577ac62 such a checkpoint took the line whose sync a_failed_log_sync_freezes_the_checkpoint fails, so that test's own flush had nothing to do and returned Ok. The period wait now ends only on the period, `wanted` or `stop` (`wait_timeout_while`): each of the checkpointer's two waits ends only on the state its waker sets before it notifies under the timer lock. No sleep, no retry. The idle wait is unchanged. Test: a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period (edca3b4823d). * acpmux: S3a review (#35483) red tests: switch-off metadata and deletes, a late settle, a failed session.json folder sync, a purge after the checkpoint F1: metadata written with the switch off survives turning it on again; a chat deleted with the switch off after a failed catch-up is not made again. F2: a late settle never writes an older image over a checkpointed one (the reviewer's test and hold). F3: a failed folder sync after a metadata write freezes the checkpoint. F4: a chat made after the checkpoint and deleted is not made again after a kill. Test seams only; the code is the S3a head merged with feat. * acpmux: S3a review (#35483) red tests: a new chat is acknowledged only once its folder is durable; session_path never waits The root of F4 and of F1's delete case: the replay makes a chat's folder again, so a chat removed by any path comes back. The fix will never make a folder again, which needs an acknowledged chat's folder to survive an OS crash: the test models what each folder's last sync kept (sync_model) and asserts the gate's durability passes a new chat only after its sessions entry is synced. P3 1 of the review: a caller that needs only the path never waits for the journal (Store::session_path, a stub with session_dir's behavior here). * acpmux: S3a review (#35483) fixes: the replay never makes a folder again; switch-off changes are kept; no late or unsynced metadata write F1: each meta record carries the stamp (file id, modification time, length) of the session.json its save saw. The replay writes an image over a file the journal did not write only when one of the chat's records saw that file; otherwise the file moved on without the journal (a run with store.journal off whose catch-up did not replay, or an older build) and the image goes to journal-unapplied/, as S2 does for a log. F1 (deletes) and F4 at the root: the replay never makes a chat's folder again. A save that made a folder notes its record's LSN; the new acpmux-journal-folders thread (the daemon's disk tier) syncs sessions/ at once, and the durable LSN the S2c gate waits for stays below that record until then (journal_store/new_folders.rs). So an acknowledged chat's folder survives an OS crash, and a folder that is gone was removed: a purge whose forget a kill took back, a run with the switch off, the transfer recovery. The created flag and the replay's folder remake are gone. F2: write_through does not write a file that holds the same or a newer journalLsn. F3: session.json is written with its own syncs (file, then folder); a failed sync is SyncFailed and freezes the checkpoint; the replay and the kept-aside images return their errors. P3 1: Store::session_path gives the path with no lock, wait or write; an agent's start, the fixed-id check, the temporary-folder sweep, the host probe and the transfer recovery's root use it. The replaced test (a chat made after the checkpoint survives an OS crash that loses its folder) asserted a remake that cannot be right any more; the new test asserts the guarantee that makes the loss impossible, then the crash. Red (7 of 17 journal_meta_tests, each for its reason; the other 10 pass): macOS 5bc434a9 and 624d5df4, Linux a2207ff0 and 638a44ef. * acpmux: review of #35562 red tests: a failed repair holds no other chat's checkpoint; a checkpoint after it writes only durable images F1 (the reviewer's test, renamed): a checkpoint after a failed repair writes only durable metadata images. F2/F3 at the root: a failed repair does not hold the checkpoint of other chats; a retention rewrite of another chat after a failed repair stays after a restart. Guards: the reviewer's two switch-off tests (green here since the S3a fix 0d246364d90: the stamp rule and no folder remake), and a chat deleted and made again after its failed repair gets none of the old chat's lines. * acpmux: review of #35562: a failed repair keeps its chat's lines in a repair file; no hold of the global checkpoint The hold (Order::held, 49d7e5cef83) kept every chat's records past the checkpoint while one chat's repair failed. That let a checkpoint write metadata images whose records were not durable (F1), made each start with the switch off replay the whole window again and undid a retention rewrite (F2), and grew the journal with each restart (F3, the 256 MiB bound counted one run only). Now a chat whose log cannot be read or written at the open keeps its lines in journal-unapplied/<id>.<lsn>.repair.ndjson (a header with the chat id, then the lines, synced). The next open journals them again, waits until they are durable, removes the file and repairs the chat with them as with any journaled line (a line in both is the same bytes, so a file journaled twice after a crash changes nothing). The checkpoint of every chat goes on as before S2's follow-ups. A delete removes its chat's repair file after the folder, so a chat made again under the same id gets none of the old lines. Red at 23870301064 (macOS a8be0684): the 3 fail for their reasons ("the checkpoint stayed at 1 below chat-a's records (the journal ends at 5)"; "the checkpoint wrote an image to session.json before its record was durable"; "the open kept a copy of a rewritten log aside"); the guards pass; 339 of 342. * acpmux: S3a delta review (N1) red tests: a change to a chat whose session.json another journal wrote reaches it A folder moved in from another home (build adoption), checkpointed and replayed, and a journal made again from nothing: the other journal's journalLsn must not hide a change made here. The first test is the reviewer's (fleet 4d0d98bc and 071ad7c4 at e944a7c21de). * acpmux: S3a delta review (N1): journalLsn is bound to its journal's id; a file another journal wrote follows the stamp rule session.json now carries journalId (16 hex digits, a string) beside journalLsn. The checkpoint's "held" check (F2) and the replay's rule compare an LSN only for a file with this journal's id: a file from another journal (a chat folder build adoption moved in from another home, or a home whose journal was made again) counts as one this journal did not write, so the stamp rule decides, as for a run with the switch off. cmux-journal: Journal::journal_id (the id already in every segment header; kept across opens and retirements), with a unit test. Red at 7c8cb957745: fleet c29b659d (Linux) and 47570494 (macOS) running; the reviewer's red at e944a7c21de: 4d0d98bc (macOS) and 071ad7c4 (Linux). * acpmux: delta review of #35562 red tests: a repair file and its folder; a torn file; a failed delete; a file larger than the journal's queue D1 (the reviewer's test, renamed): a failed repair's kept lines survive a folder that is aside at the next open; and they never go into a folder that replaced the one they were kept for. D2 (the reviewer's test, renamed): a torn repair file does not stop the chat's repair. D3: a delete that succeeds after a failed one removes the chat's repair file (test seam FAIL_NEXT_FOLDER_REMOVAL). D4: a repair file larger than max_pending_bytes is applied. * acpmux: delta review of #35562: a repair file applies only to the folder it was kept for; written whole; torn lines passed; a full queue waits; a failed delete keeps its file D1 (P2): the open took a repair file even when the chat's folder was not there (a transfer held it aside at a kill, and the hub put it back after the store's open), and the lines went nowhere. The file's header now has the folder's identity (meta.rs FolderId, the same one S3a binds metadata images to); take_repairs journals a file only when the folder at the chat's path has that identity. Otherwise the file stays: a later open applies it when that folder is back, and else it is a person's. A failed repair with no folder to bind to keeps its lines as a plain kept file, as before S2's follow-ups. D2 (P3): kept files are written whole (a temporary copy, synced, renamed, folder synced); the open removes a temporary a crash left; bytes in a repair file that are not a whole JSON line are passed. D3 (P3): delete keeps the chat's repair-file entry until the file is removed, so a delete tried again after a failed folder removal still removes it. D4 (P3): the open's repair appends wait for the writer when the journal's queue is full (append_waiting), so a repair file larger than max_pending_bytes is applied; journal_lines uses it too. Red at c0a20e09f1f: macOS fleet 54945ca4, 5 of 41 journal_store_tests fail, each for its reason; Linux fleet 393040c6 running. * acpmux: the delete test seam goes before the append pause's doc comment, not between it and its item (clippy unused_doc_comments, fleet 56f7ec6c)

…URABILITY S3b) (#35565) * cmux-journal: one append-only journal per daemon, with group commit and ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner). * cmux-journal: type a test range; drop an unread segment field (fleet clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71e3f9 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red. * cmux-journal: a failed F_FULLFSYNC is an error, not a plain fsync; no segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee. * cmux-journal: the past-the-end test reads back the storage it wrote to Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c87b8d: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5c53e. * cmux-journal: a sync a signal interrupts is made again (macOS) fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r). * cmux-journal: crash sim of write-back errors and crash in recovery; S1 review regressions (red) The S1 review (FIX at 89293fb15f39) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open * cmux-journal: the F2 regression damages the header of the last of several segments (red) The reviewer's F2 scenario has two segments: S1 holds 1..10, S2 (the last) holds 11..20, and S2's header is damaged. The test now rotates segments (300-byte segments, 20 acked records) and damages the last one's header, so the report must say lost_through 20 and the next append must take 21. On the code before the fix the moved segment's LSNs read as 0, so the next append reuses one. * cmux-journal: the crash sim forces the F4 order (failed sync, kill, OS crash) (red) On the code before the S1 fix, the crash-sim step was green (fleet 184129d8 at 263da86f223): 6 of 7 runs passed, and the mixed run failed only seed 140 in the full run and passed when it ran alone. The random faults reached the order the review found (F4) in 1 of 800 runs. So these tests were no red/green proof. The order: the sync of a write that spans pages fails and the disk gets none of them (Linux keeps them clean in the cache, so a later sync reports success); the process is killed in the writer's backoff; the next open reads the failed segment from the cache, replays its records and goes on; an OS crash or a power loss then takes those pages, and records an open already replayed are gone. - sim.rs: `fail_next_sync(pages, kind)` fails the next sync that has at least that many dirty pages, and the disk gets none of them. `crash_after_failure(n)` makes the n-th operation after the next injected failure the crash point. - crash_sim_tests.rs: a new run forces the order on every seed (a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record): epoch 0 aims the fault, the kill is a process crash at the writer's retry, the open after it does not crash, epoch 1 ends in an OS crash, and a third open checks that the records the second open replayed are all there. In the random runs, one epoch in four aims the fault, with the crash 1 to 3 operations after the failed sync and a crash of any kind after it. - A sim test checks the aimed fault itself (the_sim_fails_the_aimed_sync_and_crashes_in_the_backoff). Expected on this commit (old journal code): the forced run fails every seed; the mixed run fails most runs. The fix commit follows. * cmux-journal: fix the S1 review findings F1-F7 (markers, recovery copy, LSN floor, writer end) Review FIX at 89293fb15f39 (PR #33919 comment 6610468202). The regression tests are the two commits before this one. - Sync markers (format.rs): an entry with flags 1 and no payload, whose LSN field says "every record of this segment before me is synced". The writer appends one after each successful sync that covered records (after the durable listeners, so no reply waits for it), and syncs it fully at a seal and at a rotation. - F4, recovery (step 4 and 6): the last segment is trusted only up to its last marker. Its records after the marker may sit in pages a failed sync dropped (a later sync on another descriptor reports success without them), so recovery writes them again into a new segment: header, records and a marker, fully synced under `<name>.seg.new`, renamed into place, folder synced. The journal always goes on in a segment this open made, never in one an earlier process wrote. A restart in the writer's backoff after a failed sync is then safe. - F1, quarantine: every LSN a quarantined segment holds counts in the LSN floor at every open, and a quarantine that no new segment followed (a crash in the middle of an earlier recovery) is reported as damage again. The highest LSN of every moved segment is read before the first rename. - F2, highest LSN: a segment whose header is damaged is read from its first record (each record's checksum covers its LSN), then the bytes after the first bad entry are scanned for records and markers that still check out. The same scan runs after a cut in the last segment: a marker there is media damage (reported), records there are a torn write (`torn_lsns`); their LSNs are never used again. - F3, writer end: a guard closes the journal when the writer thread ends, also by a panic: appends fail with Closed, waits end with WriterEnded, the listeners are dropped (so a channel a held reply waits on closes), and stats.failing says why. The crate does not abort the process; the daemon owns that policy. - F6: recovery closes each segment after its header is read and opens one again only to read it. (Retirement of compacted segments is an S2 API, before the first consumer.) - F5: recovery syncs the parent folder after it makes the journal folder. - F7: format.rs and lib.rs now say what the code does: a failed write is cut back and retried in the same segment; the cut removes only bytes never synced and never acknowledged; nothing shortens a segment past a synced record. Docs: journal.rs says when open's replayed records are durable (on Ok) and what a stopped writer does. * cmux-journal: split the crash sim's machine model into sim/machine.rs (review G5) sim.rs had 924 lines and 63 functions, over the 60-function limit for a new Rust file (the god-file check warned "not in baseline: split it"). The machine model (each file's page cache and disk, the ordered log of names, and what an OS crash or a power loss leaves) moves to sim/machine.rs (297 lines, 19 functions); the process view and the Storage implementation stay in sim.rs (639 lines, 30 functions). No behavior change: the items move as they are, made visible to the sim module. * cmux-journal: re-review regressions G1-G4 and page eviction in the crash sim (red) Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993). Tests and sim only; the fix follows. - sim: the pages a failed sync dropped stay in the cache until an eviction; `evict_dropped_pages_after_read(true)` makes each read evict the dropped pages it read, so a second read of them sees the disk's bytes. - G1 recovery_copies_the_records_it_checked_when_the_cache_drops_them: a sync fails and drops record 2's pages, a kill in the backoff, the next open reads record 2 from the cache (replayed) and the pages are evicted after that read; then an OS crash. Record 2 and the record 3 acked after the restart must survive. On this code recovery's second, unchecked read stages zeros under a vouching marker, and the last open keeps only record 1 and reports damage. - G2: the F1 regression asserts that every open after the damage reports the loss (lost_through 40). On this code a crash at the open's last step (the folder sync after the new segment's rename) loses the report for good. - G3 a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system): a journal in a symlinked state folder, and a journal folder that is a symlink. On this code the parent folder's sync opens with O_NOFOLLOW (ELOOP). - G4 an_open_after_a_finished_quarantine_reads_no_quarantined_segment: on this code every open reads every quarantined segment in full. * cmux-journal: fix the re-review findings G1-G4 and G6 Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993); the regression tests are the commit before this one (red: fleet de35f6d0). - G1: recovery writes again exactly the bytes its checked read returned. The last segment's entries after its last marker are kept from the checked read (header and payload) and staged from memory; the second, unchecked read of the file is gone. A page that a failed sync dropped can be evicted between two reads, and the second read then returned the disk's zeros, which the copy kept under a vouching marker. - G2: the damage an open finds is written to `DAMAGE` (whole, fully synced, renamed into place, folder synced) before the new segment that hides it from later opens. Every open reports the recorded damage first, until the daemon calls the new `Journal::clear_damage`, which moves the record into `quarantine/` (kept, never deleted). - G3: a folder sync follows a symlink in its path (no O_NOFOLLOW): a state folder, or the journal folder itself, may be a symlink. The sync is for the folder that holds the entries. - G4: the quarantined segments are read for their LSNs only while no live segment is numbered above them (an unfinished quarantine); once one is, its first LSN already carries their floor. - G6: `checked_div` in the scan of the bytes after a cut (the crate's one tracked arithmetic hit). - G5 (the sim split) is the commit before the regressions. * cmux-journal: third-review regressions G7 (A, B) and N1 (red) Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018). Tests only. - G7 A a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn: after a finished quarantine, one byte of the empty new segment's header changes. On this code the next open skips the quarantine scan (the listing counts the damaged segment) and the next append reuses a quarantined LSN. - G7 B a_damaged_newest_segment_after_a_quarantine_reports_its_records: the newest segment holds acked 41..43, and its header and its first record are damaged. On this code the moved segment is scanned with a base far below 41, so the loss is not reported through 43 and the next append is not 44. - N1: the F1 regression also asserts that an open reports each loss once (no reported LSN range lies inside another). On this code a crash after the quarantine's renames makes the next open report the same loss twice. * cmux-journal: fix the third-review findings G7, N1, N2; split recovery.rs Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018); the regression tests are the commit before this one. - G7: the quarantine scan is skipped only when a segment whose header checked out is numbered above every quarantined segment (the folder listing also counted a newest segment with a damaged header, which carries no floor). The scans start from the running floor: `quarantined_floor` and `quarantine` pass the highest LSN counted so far as the base of the next file's scan, so a quarantined file with no readable header and no readable first record is scanned for the LSNs right above what is counted. - The "unfinished quarantine" report is made only when the quarantine holds LSNs nothing else counted (an old quarantine behind a segment that a crash tore at its creation holds none, and a crash is not damage). Its reason now also names a damaged newest header. - N1: a loss whose LSN range lies inside a recorded one is the same loss found again after a crash: it is neither recorded nor reported twice. - N2: recovery.rs's docs name `DAMAGE`; two doc lines over 100 columns are wrapped (lib.rs, sim.rs). - recovery.rs was at 987 of its 1000 lines: the damage record moves to recovery/damage.rs, the entry reader and the LSN scans to recovery/scan.rs (moved as they are). * cmux-journal: report each loss once, also when one open finds it twice (N1) The red run of the N1 regression (fleet 27d011ca, op 64 of the F1 test) shows the case the previous commit did not cover: a crash after the quarantine's renames and before DAMAGE is written; the next open finds both the damaged synced record of the last segment (7..14) and the unfinished quarantine (7..40), and reported both. Now every loss inside another (recorded or found by this open) is dropped, the larger range kept; DAMAGE is rewritten only when its set of losses changed. * cmux-journal: fourth-review regressions G8, N3, N4 (red) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report. * cmux-journal: the quarantine's floor in its file names; DAMAGE before the moves (G8, N3, N4) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format. * cmux-journal: sort the quarantine list with sort_by_key (clippy unnecessary_sort_by) Fleet d9f22bed at b589044b852: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`). * cmux-journal: Journal::on_written, a listener per written batch (for S2's projector) R-DURABILITY S2 (acpmux event logs through the journal) projects each chat's log line into its ndjson file once the journal has WRITTEN the batch that holds it, so readers see stream lines at once. The listener runs on the writer thread after the write and before the sync, once per batch that moved the written LSN: a batch written again into a new segment after a failed sync is not heard twice. When the writer stops, these listeners are dropped too. Test: heard while the batch's sync still runs, once. Port of 145faa6faa8 (branch rewrite/r-durability-s2-acpmux-logs) onto the S1 review fixes. * cmux-journal: Journal::retire_through, the one delete (segments a compaction point covers) R-DURABILITY S2: a daemon's projector compacts the journal into its own files (acpmux: the chat ndjson logs, then a synced checkpoint). Without a delete the segment count only grows (review finding F6). `retire_through(compacted)` deletes the segments that hold no record after the caller's compaction point: a segment goes when the next segment's "previous last LSN" is at or before it, so the newest segment never goes. The point is capped at the durable LSN. The deletes and a folder sync run on the caller's thread (the projector, never a request path). A delete that fails leaves its segment live for a later call. Why a crash during a retirement is safe: the caller reopens from a point at or after `compacted`, and recovery reads nothing from a segment the point covers (it skips it on its next segment's header), so any subset of the deletes may reach the disk. A point older than the retired segments is reported as Damage::Missing. Quarantined segments are never deleted. - Storage::remove_file (FsStorage: std::fs::remove_file; SimStorage: one more change in the ordered log of names, durable at the folder sync). - The live segment list: recovery's, then each segment the writer makes. - JournalStats::segments_retired. - Tests: only covered segments go, and a second call deletes none; an OS crash after it keeps every record after the point and an older point is reported; a crash at every operation of a retirement (process, OS crash, power loss) keeps every record after the point and never reuses an LSN; the newest segment stays; a point in the middle of what this writer wrote. This adds a delete of journal data, so its PR is risk:high. * acpmux: move MemoryStore and the store settings into their own files store.rs (2139 lines, 120 fns) and config.rs (1909 lines, 97 fns) are over the god-file limit, and the journal store (R-DURABILITY S2) must not grow them. MemoryStore moves to store/memory.rs and StoreMode, StoreConfig and its defaults to config/store_config.rs; both stay exported where they were (crate::store::MemoryStore, crate::config::{StoreConfig, StoreMode}). No behavior change: the code moves as it is. * acpmux: the event log's journal switch and its crash tests (red) R-DURABILITY S2 (DESIGN.md 3.3b). Today no event-log append is synced: a line a client was told about is on the disk only after the next flush (shutdown), so an OS crash or a power loss takes back every line since (E12). This commit adds only the switch, a seam and the tests; the journal store follows. - `store.journal` (StoreConfig::journal, default false). - `Store::wait_durable(timeout)`: tests and tools wait until every appended record is durable. The default returns at once (the Local and Memory stores never sync an append), which is today's behavior. - `store::open_with_journal` (tests): the store with the journal on a given `cmux_journal::Storage`. Here it ignores the journal and opens LocalStore. - acpmux depends on cmux-journal (dev: with the `sim` feature). - store/journal_store_tests.rs, red on this commit: - a_line_the_store_says_is_durable_survives_an_os_crash (lines 2..20 are lost: nothing synced them); - a_deleted_chat_stays_deleted_after_a_crash and a_line_the_log_refused_is_never_written_later pass here, and must keep passing with the journal store (no replay may bring them back); - a_kill_keeps_every_line_once passes here and must keep passing (a replay after a kill must write no line twice). * acpmux: the event log through one journal, behind `store.journal` (R-DURABILITY S2) DESIGN.md 3.3b. With `store.journal` on, each event-log append is one record in `<home>/journal/` (cmux-journal: group commit, one sync per batch; a power loss takes back at most the last second, always a suffix), then the same line goes to the chat's ndjson as before. Readers, segments and file formats do not change. Default off. - store/journal_store.rs (JournalStore over LocalStore): - append: under one lock, the journal record (a memory copy, no I/O), then the log line; a line the log refuses gets a `cancel` record. Agent stream lines (`in` session/update, claude.*, codex.*) are Class::Stream, every other line Class::Ack. - checkpoint (`<home>/journal-projected.json`): every 30 s while the journal grew, past 64 MiB, at flush (shutdown) and before a retention commit. It syncs each log written since (LocalStore::sync_log, with no lock held while the disk works) and the sessions folder, writes the LSN (temp file, sync, rename, folder sync, every error checked), then retires the journal before it. A log close that failed its sync freezes the checkpoint until a restart (LocalStore::sync_failed). - recovery: the records after the checkpoint; for each chat, the log is cut right after its last readable line below the first journaled seq (later segments removed, the boundary one shortened, their kind-index copies dropped), and the journaled lines are written again in order. `forget` deletes a chat again; `cancel` keeps a refused line out. - delete: `forget` record, folder removal, then a bounded wait until the record is durable. - seal: the logs refuse writes and the journal syncs fully. - LocalStore: `append` is serialize + `append_line`; `sync_log`; a failed sync at a log's close is noted (`sync_failed`). - store::open opens the JournalStore when `store.journal` is on. Not yet (S2c): replies and events do not wait for durability; the daemon does not stop when the journal closes; the writer thread keeps the process's Utility I/O policy. * acpmux: journal store tests for the seal and a chat made again after its delete (red) - a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay: a chat is deleted (its forget record), made again under the same id (a transfer back: session.json and new lines), then a kill -9. On this code the replay deletes the folder again and its session.json is gone. - a_line_after_the_seal_is_refused_and_never_written_later: coverage of the seal (it passes here; the fix also closes a race the test cannot force). * acpmux: journal store: the seal under the appends' lock; a replay keeps a chat made again - seal takes the appends' lock: an append lands in both the journal and the logs, or in neither. Before, an append that raced the seal could leave a journal record whose line the sealed logs refused (its cancel then failed on the closed journal), and the next start would write that line. - append and delete check the logs' seal first, under the same lock, and do not touch the journal after it (no error per append at shutdown). - replay deletes a forgotten chat again only when no line follows its forget record: otherwise the chat was made again under the same id, and its folder holds that chat's own files. * cmux-journal: fifth-review regressions G9 (red) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4. * cmux-journal: name a skipped LSN floor in a FLOOR.<lsn> file (G9) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646); the regression tests are the commit before this one. When an open gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which a copy then hides from every later open), the only record of that floor was the header of the empty segment it makes; a damaged header lost it, and DAMAGE is cleared by `clear_damage` (torn records are not damage at all). Now such an open makes the empty file `FLOOR.<lsn>` and syncs its folder before any new segment, and every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine's names and this open's moves say their floors already: no file is made for them. * acpmux: journal store regressions from the S2 review (red) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824). Each test fails on this commit and passes after the fix that follows it: - 1a lines_written_with_the_switch_off_survive_turning_it_on_again - 1b a_kill_while_the_journal_syncs_keeps_the_lines_the_log_got - 1c a_line_flush_session_made_durable_survives_losing_its_record - 2 a_folder_put_back_after_a_delete_is_not_deleted_by_the_replay - 3 a_kill_in_a_delete_leaves_the_chat_whole_or_gone - 4 a_delete_does_not_wait_for_the_journal_sync - 5 a_failed_log_sync_freezes_the_checkpoint, a_checkpoint_that_keeps_failing_asks_the_daemon_to_stop - 6 one_chat_that_cannot_be_repaired_does_not_stop_the_open - 7 the_journal_storage_refuses_a_lost_home - the repair keeps a log's lines after a hole, also through a crash in it: an_os_crash_hole_is_repaired_and_the_lines_after_it_kept, a_crash_inside_a_repair_loses_no_line - nits: an_unreadable_checkpoint_fails_the_open, the_open_waits_for_the_previous_daemon_to_let_go_of_the_journal, a_crashed_checkpoint_temporary_is_removed_at_open, a_crash_right_after_a_checkpoint_reports_no_damage - the catch-up with the switch off: switching_off_after_a_crash_keeps_every_acknowledged_line - cmux-journal: check_append_says_what_append_would_do So they compile here, this commit adds the seams they use, with no behavior: `Store::stop_reason` (default None), `Journal::check_append` (always Ok), `FencedStorage` (never asks its fence), the open report and a test stop point after a repair's cut. * acpmux: the checkpoint freeze test fails a log sync, not a folder open (red) The freeze guards against a failed fsync (a later fsync of the same file may report success for pages the kernel dropped), not against a folder the store cannot open: a chmod 000 folder is an open error, which the fix retries instead of freezing. The test now fails the log sync itself through a test seam, `FAIL_NEXT_LOG_SYNC`. Red on the code before the fix, like the other review regressions (fleet d8e7037d, 1c22175e, 63604b26 at e030f4fe402: 17 of 17 new acpmux tests and the crate's check_append test fail). * acpmux: the journal store repairs logs, never rebuilds them (S2 review) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824); the regression tests are the two commits before this one. DESIGN.md 3.3c. - 1a/1b/1c: recovery no longer cuts a chat's log at its first journaled line and writes back only the journal's lines. It merges the log's lines and the journal's by seq: a line in both must be equal, a line only the log has stays, a line only the journal has goes in its place. Only bytes that do not read (a torn tail, a hole of zeros) are dropped. When the merge must change the log before its end, the lines only the log holds from there are journaled and made durable first, so a crash inside the repair loses no line. A log that moved on without the journal (a line in both that differs, lines out of order) is left as it is, and the journal's lines go to journal-unapplied/. - Append writes the log first, then the record, under one lock, after `Journal::check_append` (new in cmux-journal: the append's refusals, asked before any write). So a record always has its log line, and the `cancel` record is gone. - 2/3/4: the replay never deletes a folder and never makes a missing one again (a forget only drops the chat's earlier journaled lines). Delete appends its forget under the lock and removes the folder off it, with no wait for the record. - 5: a failed fsync in a checkpoint freezes it (a folder that does not open is retried); a frozen checkpoint returns an error, and the store asks the daemon to stop for a restart (`Store::stop_reason`, wired in daemon.rs), whose replay writes the logs again. Ten failed checkpoints in a row ask the same, so the journal past the checkpoint stays bounded. - 6: an error in one chat keeps its journaled lines in journal-unapplied/ and the open goes on. - 7: the journal's storage refuses writes under a lost home (`FencedStorage`, `config::refuse_lost_home`). - With store.journal off, a journal left by a run with it on is replayed, checkpointed and retired first (`catch_up`). - Nits: the checkpoint is never past the journal's durable end (no `Ahead` after a normal crash); damage is cleared after the next checkpoint; a kind-index copy goes before its segment is cut; an unreadable checkpoint fails the open; the open waits up to 5 s for the journal lock; crashed checkpoint temporaries are removed; `on_written` leaves cmux-journal (no user); README lists store.journal. * acpmux: split the journal store under its size budget (a move) journal_store.rs grew to 1085 lines and 71 functions with the review fix (budget 1000 and 60), and the fix grew store.rs and daemon.rs, which are over budget on the base. A pure move, no behavior change: - store/journal_store/{checkpoint,fenced,repair}.rs: the checkpoint and its thread, `FencedStorage`, the replay and the repair. - store/log_sync.rs: `LocalStore::sync_log`, `SyncFailed` and the test seam `FAIL_NEXT_LOG_SYNC`, out of store.rs (2139 lines on the base, 2113 now). - daemon/stop_causes.rs: `AbortOnDrop`, `next_signal` and `store_asks_to_stop`, out of daemon.rs (2052 on the base, 2046 now). check-no-godfiles.sh --base 51cfdee876b on every changed file: no file over budget grew, and no new file is over budget. * acpmux: the unrepairable-chat test waits for line 1 to be durable before its checkpoint At 03fccdef69c (fleet 32df3c72 on aws-m4pro-9, 144d4ccc on Linux) the test kept chat-b's lines [1, 2, 3] aside, not [2, 3]: it flushed right after appending line 1, and the checkpoint is now never past the journal's durable end (the review's Ahead nit), so line 1 was still past it. The fix's behavior is right; the test now waits until line 1 is durable, so its checkpoint covers it. On the code before the fix the test still fails (the open stops at chat-b's link). * acpmux: replies and events wait for durability, tests (red, S2c) DESIGN.md 3.3d. With the event journal on and every journal sync taking 400 ms (`SimStorage::set_sync_delay`): - an_acked_change_is_answered_once_its_record_is_durable: a rename's reply must come no sooner than one sync after its `renamed` record. Red here: the reply goes at once. - agent_output_is_not_held_behind_another_chats_acked_change: agent output to a connection attached to its chat goes at once while another chat's acknowledged change waits for its sync. Green here (nothing waits); it guards the fix against holding agent output behind changes the connection does not hear. * acpmux: replies and events wait until what they show is durable (S2c) DESIGN.md 3.3d; the tests are the commit before this one. With `store.journal` on, each connection gets an outbound gate (server/durable_gate.rs) between it and its transport: - Every line a connection queues raises the gate's `needs` to the last acknowledged LSN (`Store::durability().acked`, set in the journal store's append before the hub publishes anything of the record), except the lines of a `Stream` record's event (agent output, DESIGN C4): the fan-out delivers those in a thread-local stream scope. - The gate's task takes every queued line at once, reads `needs`, waits (async) until the journal's durable LSN reaches it, then sends the batch in order: one wait per batch, as the journal's group commit. - The durable LSN is a `tokio::sync::watch` fed by `Journal::on_durable`. The listener owns a guard: a writer that ends while the store is not being dropped (a panic on its thread) asks the daemon to stop (`Store::stop_reason`, decision 9.8), and the watch closes, so each gate drops what it holds and ends, which closes its connection. - Without the journal, `Store::durability()` is None: no gate, and `Conn::raise_gate` does nothing. Moves, so server/mod.rs and store.rs stay under their size on the base: the outbound queue helpers to server/outbound.rs, `open_with_journal` to the journal store, and `streamed` to store/durability.rs. * acpmux: the journal lock wait's deadline without an unchecked Instant addition `Instant + Duration` can panic on overflow (clippy::arithmetic_side_effects, tracked by the panic ratchet). The wait now has no deadline only in the case that cannot happen (an Instant 5 s ahead overflowing), where it stops waiting at once. * acpmux: a failed log sync stops the real daemon for a restart, test (red) Lead rule 2026-11-04 (3): a daemon-level test of the store's "stop for a restart" before S2c merges, red first. With store.journal on and every log sync failing (a debug build's ACPMUX_TEST_FAIL_LOG_SYNC; the checkpoint every 200 ms through ACPMUX_TEST_JOURNAL_CHECKPOINT_MS), a rename of a chat on disk makes its log dirty, the checkpoint's sync fails and freezes it, and the store asks the daemon to stop. The test asserts: the WebSocket client gets the restart close (1012), the socket client's pending `_acpmux/wait` ends with the connection (no hang), and the daemon exits with status 74. Red here: the daemon stops and sends 1012, but exits 0 (no exit status says that it stopped to be repaired). This commit adds only the two debug-build seams (release builds have none) and the test. * acpmux: a daemon its store stopped for a restart exits with status 74 The test is the commit before this one. `daemon::run` returned Ok and the process exited 0 for every stop, so a supervisor or a person could not tell a stop to be repaired from a normal one; with systemd `Restart=on-failure` it would not even start again. Now the store's stop (`store_asks_to_stop`: a failed log sync froze the event journal's checkpoint, or the journal's writer ended) marks itself, sets the restart handoff (WebSocket clients get 1012, as before), and the process exits with `daemon::EXIT_REPAIR` (74, EX_IOERR). The units prevent only 75, so launchd KeepAlive and systemd Restart=always start it again; the app does not read the daemon's exit status. README: store.journal names the status. * acpmux: the restart test's request in flight goes through the acpmux client Lead rule (3): the client sees a typed error, not a hang. The socket client is now the acpmux client library (acpmux::client::Client, as the CLI uses it), and the test asserts what the CLI makes of the request that the stop ends: `daemon_closed`, retryable (cli::errors::classify). The first red (fleet 41f2d037, hq run 41804640229, at 80c5fb2b7f8) failed only on the exit status (0, wanted 74), with 1012 and the end of the connection passing. * acpmux: the daemon's disk tier with the journal on, tests (red) Lead rule 2026-11-04 (2): a reply the person waits for never waits on a throttled disk sync. The fleet A/B (STATUS 11554) found that at IOPOL_UTILITY an acknowledged journal append waits about 16 ms p50 under another process's disk stress (0.7 to 1.1 ms at the default tier). - the_daemon_with_the_journal_keeps_the_default_disk_tier: with store.journal on, the daemon process keeps the default tier; without it, utility as before (each case in a child of the test binary). - a_background_thread_yields_its_disk_work: a dedicated background thread (the checkpointer) works at utility; a thread beside it does not. The API they use is stubbed here (`qos::enter_daemon` ignores the journal; `qos::background_thread` does nothing), so both fail on macOS. * acpmux: the restart test borrows no &str twice (clippy needless_borrows_for_generic_args) Found by fleet 7f8afd9d (hq run: the S2c server step at 62b4f68ef8b); the step's other clippy red is the base red in link_push_bench.rs. * acpmux: with the journal on, the daemon keeps the default disk tier (S2c) The tests are 994f74119fb. Lead rule 2026-11-04: a reply the person waits for never waits on a throttled disk sync. Measured on two owned Macs (STATUS 11554): one acknowledged append to durable at IOPOL_UTILITY under another app's disk stress took 16.3-16.7 ms p50 and 37-45 ms p95, against 0.7-1.1 ms p50 and 1.5-1.8 ms p95 at the default tier. - qos::enter_daemon(journal): with store.journal on, the daemon process keeps the default disk tier; without it, utility as before. The switch is read alone from config.json (config::store_journal_enabled), before any thread starts, with no harness discovery. - qos::background_thread: the journal store's checkpointer lowers its own disk I/O to utility for its whole life (a thread that does nothing else: a thread-scope policy drops it out of QoS for good). * acpmux: an agent chunk is a Stream record, test (red) The hub's tap logs an ACP session/update under its sessionUpdate kind (hub/tap.rs: `kind = su.to_owned()`), so a record's kind is `agent_message_chunk`, `tool_call`, ... and never `session/update`; the message's method is. The journal store's `streamed` checked the kind, so every agent chunk was journaled as Ack: a sync per batch at the token rate, and in S2c each token's event waited for that sync (the S2c I/O A/B: STRESS token to client 2.2 ms -> 20.3 ms p50 with the journal on). Red here: the chunk reads back as Ack. * acpmux: an agent chunk is a Stream record (the record's message names session/update) The test is the commit before this one. `streamed` now reads the logged message's method: the hub's tap logs a session/update under its sessionUpdate kind (hub/tap.rs), so the kind never said it. Agent output is again Stream (synced with the next acknowledged batch or a second later), as DESIGN.md 3.3b says. * acpmux: review 2 of #34901, tests (red): a crash model that follows real log syncs The logs' OS crash model now follows the syncs that ran (store/sync_model.rs, tests only): a segment keeps its length at its last sync that counts; a failed sync loses what was written since the one before, and a later sync does not bring it back; a cut ends a loss it removes. Every log segment sync goes through log_sync::sync_segment (same calls as before), where the model and FAIL_NEXT_LOG_SYNC hook in. Red tests, one per finding: - 1a an_os_crash_after_a_restart_from_a_kill_keeps_every_acked_line: the restart's first checkpoint retires records whose log lines no sync made durable; expected [1..5], the head gives [1]. - 1b an_os_crash_after_a_restart_from_a_failed_sync_keeps_every_acked_line: the same after a frozen checkpoint and a restart; expected [1..5]. - 2 a_hole_that_starts_inside_a_line_is_repaired (the reviewer's test): a torn line is read as line 6 of a log that moved on; expected [1..9], the head gives [1, 2, 3, 4, 5, 9]. - 3 a_failed_sync_in_flush_session_freezes_the_checkpoint: a failed sync in flush_session does not freeze the checkpoint. * acpmux: review 2 of #34901: a restart writes the replayed lines again; a torn line is not a line; flush_session freezes Finding 1 (high): the repair now writes every merged line again from the repair start, even when the log already has them all. A killed daemon's log writes may be in the page cache only, and after a failed sync the kernel may have dropped pages that still read back and that a later sync reports as written; the first checkpoint after the open then syncs bytes this process wrote before it retires their records. The lines only the log has from the repair start are journaled and made durable first, as before a cut. The work is bounded by the checkpoint interval. Finding 2: a piece of the log is a line only when it parses as one JSON value (piece_seq). A hole that starts inside a line keeps that line's {"seq":N, head and runs into a later line's tail; it was read as a line of a log that moved on, and the chat's journaled lines went aside. Now its bytes go and the lines are written again. Finding 3: a failed sync in flush_session freezes the checkpoint (its file sync is a SyncFailed now), as a failed sync in a checkpoint or at a log's close does. Nits: cut_at syncs events/ after its removals (and the kind-index folder after its copies go), so a removed segment cannot come back beside the lines written again; the README says that with the switch on an event line over 64 MiB is refused and its log does not get it. Tests: the four red tests of d2c63860ece. * acpmux: chat metadata through the event journal, tests (red, R-DURABILITY S3a) DESIGN.md 3.3e. Red on S2c: - last_seq_never_passes_the_log_after_an_os_crash (E13): session.json is synced at each save while the agent's lines wait for the journal's next sync, so after an OS crash the metadata names lines the log lost. - a_chat_made_after_the_checkpoint_survives_an_os_crash_that_loses_its_folder: nothing syncs the sessions folder when a chat's folder is made, and the replay never makes a folder again, so an acknowledged new chat is lost. - a_metadata_change_is_written_to_session_json_by_the_checkpoint: today each change writes and syncs session.json on the request path. Guards (green before and after): a chat made before the checkpoint and deleted is not made again; the replay never writes an older image over session.json. * acpmux: chat metadata through the event journal (R-DURABILITY S3a) DESIGN.md 3.3e, behind `store.journal` (default off). A metadata change is a `meta` record (kind 4: chat id, a `created` flag, the whole image) instead of a session.json write with two syncs on the request path; the reply waits for the record through S2c's gate, one group commit for a burst. - Overlay: each chat's newest image until a checkpoint writes session.json; load, load_startup and list read it first (list under the projection lock, so a checkpoint cannot drop an entry between its two reads). - Checkpoint: waits on its own thread until every record through its LSN is durable, then writes each image it read (session.json carries the image's LSN as `journalLsn`), then the logs and the sessions folder, as before. An image so names only lines the journal holds (E13). A chat deleted or made again meanwhile is passed (its overlay entry is gone or new). - Replay: per chat after its last `forget`, the newest image, written when newer than the file's `journalLsn`. A missing folder is made again only for a chat a replayed `created` record made; a chat made before the checkpoint whose folder is gone keeps its image in journal-unapplied/. - flush_session also waits until the chat's records are durable. - Delete drops the overlay entry with its `forget`, and removes the folder under the folders lock a checkpoint's write also takes. Tests: the 5 of 0505e9bee04 (3 red there). * acpmux: S3a, tests (red): folders handed out of the store Code outside the store reads and moves chat folders directly (transfer_in puts the live folder aside and installs a transferred chat; session_dir hands the folder to the transfer, the session files view and others). With the metadata in the overlay until a checkpoint: - a_folder_replaced_outside_the_store_keeps_its_own_metadata: a read, the checkpoint and the replay must not put the old folder's image in the new one. - session_dir_hands_out_a_folder_whose_metadata_is_written. - flush_session_writes_the_chats_metadata (the trait says log and metadata). - the_checkpoint_never_makes_a_folder_removed_outside_the_store_again. Red on 71bc09a1e63 (S3a's first implementation). * acpmux: S3a red tests: a replay after a folder swap; flush_session's chat has a log (as its callers' chats do) * acpmux: S3a: each metadata image is bound to its chat folder; session_dir and flush_session write it first Code outside the store reads and moves chat folders directly (a transfer puts the live folder aside and installs a transferred chat under the same id). So a `meta` record now carries its folder's identity (file id and birth time): - reads use an overlay image only while the folder at the path is its folder; - the checkpoint writes an image only into its own folder, never makes a missing one, and keeps the image of a removed or replaced folder in journal-unapplied/; - the replay writes an image only into its own folder (or a folder a replayed `created` record made), else keeps it aside. `session_dir` (the transfer, the session files view) and `flush_session` (the trait: "log and metadata to stable storage") write the chat's image first, once its record is durable. The list reads the overlay before the files (a checkpoint writes a file before it drops the entry), so it needs no lock. Tests: the 5 of 0280ac37f5e and 0e617489731. * acpmux: S2 follow-ups from review 3 of #34901, tests (red) - a_chat_whose_repair_failed_is_repaired_at_the_next_start (follow-up 1): the open's checkpoint retires the records of a chat whose repair failed, so no later start puts its lines back (they are only in journal-unapplied/). - a_hole_that_ends_at_a_line_start_keeps_that_line (follow-up 2): a torn line, zeros, then a whole line only the log holds: the piece does not parse and the whole line is dropped. - an_idle_store_does_not_wake_its_checkpointer (follow-up 4, cmux-ad m9): the checkpointer wakes every period with nothing to do. Test seams only (no behavior change): CHECKPOINTER_WAKES and TEST_CHECKPOINT_EVERY_MS. * acpmux: S2 follow-ups from review 3 of #34901: a failed repair holds its records; a hole's last zero splits; an idle checkpointer sleeps - Follow-up 1: a chat whose repair fails at the open (its log cannot be read or written now) keeps its lines in journal-unapplied/ as before, and the checkpoint now stays below its first journaled LSN (`Order::held`), so the next start repairs it again. Bounded: past 256 MiB of journal after it, the hold goes and the lines stay only in journal-unapplied/ (an error says so). - Follow-up 2: a piece splits at its last zero byte (a whole JSON line holds none), so a hole that ends at a line start, after a torn line, keeps that line (it mattered for a line only the log holds). - Follow-up 4 (cmux-ad m9): with nothing to checkpoint, the checkpointer sleeps until a change wakes it (`Inner::touched`, one atomic read per append), then checkpoints one period later; before, it woke every 30 s. Tests: the 3 of d147057bf85. * acpmux: a metadata change logs its line before its metadata, tests (red, R-DURABILITY S3b) DESIGN.md 3.3e, E6: with the event journal, records survive a crash as a prefix in append order. So a change whose line reaches the store before any metadata image that holds it is never, after a crash, in the chat's metadata without its line in the log. A recording store checks the order of the store calls: - a_tag_change_logs_its_line_before_its_metadata: red (tags saves first). - a_rules_change_logs_its_line_before_its_metadata: red (rules saves first). - a_rename_logs_its_line_before_its_metadata: green (a guard). * acpmux: the qos journal tests module is macOS-only (its Linux build held only an unused import: clippy red, review of #35400 F0) * acpmux: review of #35400 F1 and F2, tests (red) - a_read_that_shows_a_line_covers_its_record (F1): a read sees a log line between its log write and its record (the appends' lock held there), and the reply built from it goes out with no wait for that record. - a_journal_that_syncs_nothing_asks_the_daemon_to_stop (F2): with a journal that makes nothing durable, every reply waits without a limit and nothing asks the daemon to stop. Test seams only: PAUSE_AFTER_LOG_WRITE, TEST_STALL_MS. * acpmux: review of #35400 F1, the streaming read, test (red): session/load's replay sends a line before its record * acpmux: the test seam PAUSE_AFTER_LOG_WRITE is reachable from the server tests * acpmux: review of #35400: a read covers the records of the lines it shows; a stalled journal stops the daemon F1: the append writes a line to the log, then appends its record and sets the acknowledged LSN, all under the appends' lock; reads take no lock. A read could show a line before its record existed, and the reply built from it went out at once. Now every journal-store read of the logs ends with a fence on the appends' lock (`Inner::fenced`): a line the read saw has its record appended and its LSN acknowledged when the read returns, so the reply's gate covers it. The one read that sends as it reads (`Hub::scan_events`, session/load's replay) stops at the chat's last_seq read first. F2: a journal writer that keeps failing (it retries without end) held every reply on every connection with no limit. A connection's gate now waits at most 10 s (`STALL_LIMIT`); past it, `Durability::stalled` asks the daemon to stop for a restart, as a failed log sync does. Tests: the 3 of 3916f84dd68 and c553a19fd8b. * acpmux: S3b red tests: the recording store implements delete (a required Store method) * threading-model: the acpmux event journal's threads and bounds (writer, checkpointer, per-connection gate; review of #35400) * acpmux: the stall test's journal syncs take 3 s, not an hour (a sync under way kept its delay and hung the test's end) * acpmux: a tags or rules change logs its line before its metadata (R-DURABILITY S3b) DESIGN.md 3.3e (revised): with the event journal, records survive a crash as a prefix in append order, so a change whose line reaches the store before any metadata image that holds it is never, after a crash, in the chat's metadata without its line in the log (E6). `renamed`, `policy` and `mode` already did this; `tags` and `rules` saved first. Their reply still waits for the save. Tests: the 3 of 5bb3a27a2da and 7a4f02a56f3 (red at 7a4f02a56f3, fleet 88309e91: tags ["save:tagged", "line:tags"]; rules ["line:permission_chat_allowance", "save:ruled", "line:rules"]). * acpmux: the idle checkpointer's period, test (red): a change seen by its second look must not start a checkpoint at once - a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period: the checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies when it gets the timer lock. The second look saw the change, so the checkpointer is in its period wait by then, and the notify ends that wait: a checkpoint runs at once on the checkpointer's thread. This is the cause of a_failed_log_sync_freezes_the_checkpoint failing in the full store:: run of fleet 0577ac62 (that early checkpoint took the line whose sync the test fails, on a thread where the failure is not set). The test holds both threads at the two points, so the order is exact. Test seams only (no behavior change): the checkpointer's test seams are now by the store's home (period, wake count, the two holds), so tests that run as threads of one process under cargo test do not see each other's. * acpmux: the idle checkpointer's period ends only on its own state, not on a wake meant for its idle wait The checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies once it gets the timer lock. Its change shows in the second look, so the checkpointer has gone to its period wait by then, and that notify ended the period at once: a checkpoint ran right away on the checkpointer's thread (a burst no longer shared one checkpoint), and a spurious wakeup could do the same. In the full store:: run of fleet 0577ac62 such a checkpoint took the line whose sync a_failed_log_sync_freezes_the_checkpoint fails, so that test's own flush had nothing to do and returned Ok. The period wait now ends only on the period, `wanted` or `stop` (`wait_timeout_while`): each of the checkpointer's two waits ends only on the state its waker sets before it notifies under the timer lock. No sleep, no retry. The idle wait is unchanged. Test: a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period (edca3b4823d). * acpmux: S3a review (#35483) red tests: switch-off metadata and deletes, a late settle, a failed session.json folder sync, a purge after the checkpoint F1: metadata written with the switch off survives turning it on again; a chat deleted with the switch off after a failed catch-up is not made again. F2: a late settle never writes an older image over a checkpointed one (the reviewer's test and hold). F3: a failed folder sync after a metadata write freezes the checkpoint. F4: a chat made after the checkpoint and deleted is not made again after a kill. Test seams only; the code is the S3a head merged with feat. * acpmux: S3a review (#35483) red tests: a new chat is acknowledged only once its folder is durable; session_path never waits The root of F4 and of F1's delete case: the replay makes a chat's folder again, so a chat removed by any path comes back. The fix will never make a folder again, which needs an acknowledged chat's folder to survive an OS crash: the test models what each folder's last sync kept (sync_model) and asserts the gate's durability passes a new chat only after its sessions entry is synced. P3 1 of the review: a caller that needs only the path never waits for the journal (Store::session_path, a stub with session_dir's behavior here). * acpmux: S3a review (#35483) fixes: the replay never makes a folder again; switch-off changes are kept; no late or unsynced metadata write F1: each meta record carries the stamp (file id, modification time, length) of the session.json its save saw. The replay writes an image over a file the journal did not write only when one of the chat's records saw that file; otherwise the file moved on without the journal (a run with store.journal off whose catch-up did not replay, or an older build) and the image goes to journal-unapplied/, as S2 does for a log. F1 (deletes) and F4 at the root: the replay never makes a chat's folder again. A save that made a folder notes its record's LSN; the new acpmux-journal-folders thread (the daemon's disk tier) syncs sessions/ at once, and the durable LSN the S2c gate waits for stays below that record until then (journal_store/new_folders.rs). So an acknowledged chat's folder survives an OS crash, and a folder that is gone was removed: a purge whose forget a kill took back, a run with the switch off, the transfer recovery. The created flag and the replay's folder remake are gone. F2: write_through does not write a file that holds the same or a newer journalLsn. F3: session.json is written with its own syncs (file, then folder); a failed sync is SyncFailed and freezes the checkpoint; the replay and the kept-aside images return their errors. P3 1: Store::session_path gives the path with no lock, wait or write; an agent's start, the fixed-id check, the temporary-folder sweep, the host probe and the transfer recovery's root use it. The replaced test (a chat made after the checkpoint survives an OS crash that loses its folder) asserted a remake that cannot be right any more; the new test asserts the guarantee that makes the loss impossible, then the crash. Red (7 of 17 journal_meta_tests, each for its reason; the other 10 pass): macOS 5bc434a9 and 624d5df4, Linux a2207ff0 and 638a44ef. * acpmux: review of #35562 red tests: a failed repair holds no other chat's checkpoint; a checkpoint after it writes only durable images F1 (the reviewer's test, renamed): a checkpoint after a failed repair writes only durable metadata images. F2/F3 at the root: a failed repair does not hold the checkpoint of other chats; a retention rewrite of another chat after a failed repair stays after a restart. Guards: the reviewer's two switch-off tests (green here since the S3a fix 0d246364d90: the stamp rule and no folder remake), and a chat deleted and made again after its failed repair gets none of the old chat's lines. * acpmux: review of #35562: a failed repair keeps its chat's lines in a repair file; no hold of the global checkpoint The hold (Order::held, 49d7e5cef83) kept every chat's records past the checkpoint while one chat's repair failed. That let a checkpoint write metadata images whose records were not durable (F1), made each start with the switch off replay the whole window again and undid a retention rewrite (F2), and grew the journal with each restart (F3, the 256 MiB bound counted one run only). Now a chat whose log cannot be read or written at the open keeps its lines in journal-unapplied/<id>.<lsn>.repair.ndjson (a header with the chat id, then the lines, synced). The next open journals them again, waits until they are durable, removes the file and repairs the chat with them as with any journaled line (a line in both is the same bytes, so a file journaled twice after a crash changes nothing). The checkpoint of every chat goes on as before S2's follow-ups. A delete removes its chat's repair file after the folder, so a chat made again under the same id gets none of the old lines. Red at 23870301064 (macOS a8be0684): the 3 fail for their reasons ("the checkpoint stayed at 1 below chat-a's records (the journal ends at 5)"; "the checkpoint wrote an image to session.json before its record was durable"; "the open kept a copy of a rewritten log aside"); the guards pass; 339 of 342. * acpmux: S3a delta review (N1) red tests: a change to a chat whose session.json another journal wrote reaches it A folder moved in from another home (build adoption), checkpointed and replayed, and a journal made again from nothing: the other journal's journalLsn must not hide a change made here. The first test is the reviewer's (fleet 4d0d98bc and 071ad7c4 at e944a7c21de). * acpmux: S3a delta review (N1): journalLsn is bound to its journal's id; a file another journal wrote follows the stamp rule session.json now carries journalId (16 hex digits, a string) beside journalLsn. The checkpoint's "held" check (F2) and the replay's rule compare an LSN only for a file with this journal's id: a file from another journal (a chat folder build adoption moved in from another home, or a home whose journal was made again) counts as one this journal did not write, so the stamp rule decides, as for a run with the switch off. cmux-journal: Journal::journal_id (the id already in every segment header; kept across opens and retirements), with a unit test. Red at 7c8cb957745: fleet c29b659d (Linux) and 47570494 (macOS) running; the reviewer's red at e944a7c21de: 4d0d98bc (macOS) and 071ad7c4 (Linux).

…ion never waits for a checkpoint in flight (R-DURABILITY, follow-up 4 of review 3 of #34901) (#35866) * cmux-journal: one append-only journal per daemon, with group commit and ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner). * cmux-journal: type a test range; drop an unread segment field (fleet clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71e3f9 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red. * cmux-journal: a failed F_FULLFSYNC is an error, not a plain fsync; no segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee. * cmux-journal: the past-the-end test reads back the storage it wrote to Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c87b8d: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5c53e. * cmux-journal: a sync a signal interrupts is made again (macOS) fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r). * cmux-journal: crash sim of write-back errors and crash in recovery; S1 review regressions (red) The S1 review (FIX at 89293fb15f39) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open * cmux-journal: the F2 regression damages the header of the last of several segments (red) The reviewer's F2 scenario has two segments: S1 holds 1..10, S2 (the last) holds 11..20, and S2's header is damaged. The test now rotates segments (300-byte segments, 20 acked records) and damages the last one's header, so the report must say lost_through 20 and the next append must take 21. On the code before the fix the moved segment's LSNs read as 0, so the next append reuses one. * cmux-journal: the crash sim forces the F4 order (failed sync, kill, OS crash) (red) On the code before the S1 fix, the crash-sim step was green (fleet 184129d8 at 263da86f223): 6 of 7 runs passed, and the mixed run failed only seed 140 in the full run and passed when it ran alone. The random faults reached the order the review found (F4) in 1 of 800 runs. So these tests were no red/green proof. The order: the sync of a write that spans pages fails and the disk gets none of them (Linux keeps them clean in the cache, so a later sync reports success); the process is killed in the writer's backoff; the next open reads the failed segment from the cache, replays its records and goes on; an OS crash or a power loss then takes those pages, and records an open already replayed are gone. - sim.rs: `fail_next_sync(pages, kind)` fails the next sync that has at least that many dirty pages, and the disk gets none of them. `crash_after_failure(n)` makes the n-th operation after the next injected failure the crash point. - crash_sim_tests.rs: a new run forces the order on every seed (a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record): epoch 0 aims the fault, the kill is a process crash at the writer's retry, the open after it does not crash, epoch 1 ends in an OS crash, and a third open checks that the records the second open replayed are all there. In the random runs, one epoch in four aims the fault, with the crash 1 to 3 operations after the failed sync and a crash of any kind after it. - A sim test checks the aimed fault itself (the_sim_fails_the_aimed_sync_and_crashes_in_the_backoff). Expected on this commit (old journal code): the forced run fails every seed; the mixed run fails most runs. The fix commit follows. * cmux-journal: fix the S1 review findings F1-F7 (markers, recovery copy, LSN floor, writer end) Review FIX at 89293fb15f39 (PR #33919 comment 6610468202). The regression tests are the two commits before this one. - Sync markers (format.rs): an entry with flags 1 and no payload, whose LSN field says "every record of this segment before me is synced". The writer appends one after each successful sync that covered records (after the durable listeners, so no reply waits for it), and syncs it fully at a seal and at a rotation. - F4, recovery (step 4 and 6): the last segment is trusted only up to its last marker. Its records after the marker may sit in pages a failed sync dropped (a later sync on another descriptor reports success without them), so recovery writes them again into a new segment: header, records and a marker, fully synced under `<name>.seg.new`, renamed into place, folder synced. The journal always goes on in a segment this open made, never in one an earlier process wrote. A restart in the writer's backoff after a failed sync is then safe. - F1, quarantine: every LSN a quarantined segment holds counts in the LSN floor at every open, and a quarantine that no new segment followed (a crash in the middle of an earlier recovery) is reported as damage again. The highest LSN of every moved segment is read before the first rename. - F2, highest LSN: a segment whose header is damaged is read from its first record (each record's checksum covers its LSN), then the bytes after the first bad entry are scanned for records and markers that still check out. The same scan runs after a cut in the last segment: a marker there is media damage (reported), records there are a torn write (`torn_lsns`); their LSNs are never used again. - F3, writer end: a guard closes the journal when the writer thread ends, also by a panic: appends fail with Closed, waits end with WriterEnded, the listeners are dropped (so a channel a held reply waits on closes), and stats.failing says why. The crate does not abort the process; the daemon owns that policy. - F6: recovery closes each segment after its header is read and opens one again only to read it. (Retirement of compacted segments is an S2 API, before the first consumer.) - F5: recovery syncs the parent folder after it makes the journal folder. - F7: format.rs and lib.rs now say what the code does: a failed write is cut back and retried in the same segment; the cut removes only bytes never synced and never acknowledged; nothing shortens a segment past a synced record. Docs: journal.rs says when open's replayed records are durable (on Ok) and what a stopped writer does. * cmux-journal: split the crash sim's machine model into sim/machine.rs (review G5) sim.rs had 924 lines and 63 functions, over the 60-function limit for a new Rust file (the god-file check warned "not in baseline: split it"). The machine model (each file's page cache and disk, the ordered log of names, and what an OS crash or a power loss leaves) moves to sim/machine.rs (297 lines, 19 functions); the process view and the Storage implementation stay in sim.rs (639 lines, 30 functions). No behavior change: the items move as they are, made visible to the sim module. * cmux-journal: re-review regressions G1-G4 and page eviction in the crash sim (red) Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993). Tests and sim only; the fix follows. - sim: the pages a failed sync dropped stay in the cache until an eviction; `evict_dropped_pages_after_read(true)` makes each read evict the dropped pages it read, so a second read of them sees the disk's bytes. - G1 recovery_copies_the_records_it_checked_when_the_cache_drops_them: a sync fails and drops record 2's pages, a kill in the backoff, the next open reads record 2 from the cache (replayed) and the pages are evicted after that read; then an OS crash. Record 2 and the record 3 acked after the restart must survive. On this code recovery's second, unchecked read stages zeros under a vouching marker, and the last open keeps only record 1 and reports damage. - G2: the F1 regression asserts that every open after the damage reports the loss (lost_through 40). On this code a crash at the open's last step (the folder sync after the new segment's rename) loses the report for good. - G3 a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system): a journal in a symlinked state folder, and a journal folder that is a symlink. On this code the parent folder's sync opens with O_NOFOLLOW (ELOOP). - G4 an_open_after_a_finished_quarantine_reads_no_quarantined_segment: on this code every open reads every quarantined segment in full. * cmux-journal: fix the re-review findings G1-G4 and G6 Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993); the regression tests are the commit before this one (red: fleet de35f6d0). - G1: recovery writes again exactly the bytes its checked read returned. The last segment's entries after its last marker are kept from the checked read (header and payload) and staged from memory; the second, unchecked read of the file is gone. A page that a failed sync dropped can be evicted between two reads, and the second read then returned the disk's zeros, which the copy kept under a vouching marker. - G2: the damage an open finds is written to `DAMAGE` (whole, fully synced, renamed into place, folder synced) before the new segment that hides it from later opens. Every open reports the recorded damage first, until the daemon calls the new `Journal::clear_damage`, which moves the record into `quarantine/` (kept, never deleted). - G3: a folder sync follows a symlink in its path (no O_NOFOLLOW): a state folder, or the journal folder itself, may be a symlink. The sync is for the folder that holds the entries. - G4: the quarantined segments are read for their LSNs only while no live segment is numbered above them (an unfinished quarantine); once one is, its first LSN already carries their floor. - G6: `checked_div` in the scan of the bytes after a cut (the crate's one tracked arithmetic hit). - G5 (the sim split) is the commit before the regressions. * cmux-journal: third-review regressions G7 (A, B) and N1 (red) Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018). Tests only. - G7 A a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn: after a finished quarantine, one byte of the empty new segment's header changes. On this code the next open skips the quarantine scan (the listing counts the damaged segment) and the next append reuses a quarantined LSN. - G7 B a_damaged_newest_segment_after_a_quarantine_reports_its_records: the newest segment holds acked 41..43, and its header and its first record are damaged. On this code the moved segment is scanned with a base far below 41, so the loss is not reported through 43 and the next append is not 44. - N1: the F1 regression also asserts that an open reports each loss once (no reported LSN range lies inside another). On this code a crash after the quarantine's renames makes the next open report the same loss twice. * cmux-journal: fix the third-review findings G7, N1, N2; split recovery.rs Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018); the regression tests are the commit before this one. - G7: the quarantine scan is skipped only when a segment whose header checked out is numbered above every quarantined segment (the folder listing also counted a newest segment with a damaged header, which carries no floor). The scans start from the running floor: `quarantined_floor` and `quarantine` pass the highest LSN counted so far as the base of the next file's scan, so a quarantined file with no readable header and no readable first record is scanned for the LSNs right above what is counted. - The "unfinished quarantine" report is made only when the quarantine holds LSNs nothing else counted (an old quarantine behind a segment that a crash tore at its creation holds none, and a crash is not damage). Its reason now also names a damaged newest header. - N1: a loss whose LSN range lies inside a recorded one is the same loss found again after a crash: it is neither recorded nor reported twice. - N2: recovery.rs's docs name `DAMAGE`; two doc lines over 100 columns are wrapped (lib.rs, sim.rs). - recovery.rs was at 987 of its 1000 lines: the damage record moves to recovery/damage.rs, the entry reader and the LSN scans to recovery/scan.rs (moved as they are). * cmux-journal: report each loss once, also when one open finds it twice (N1) The red run of the N1 regression (fleet 27d011ca, op 64 of the F1 test) shows the case the previous commit did not cover: a crash after the quarantine's renames and before DAMAGE is written; the next open finds both the damaged synced record of the last segment (7..14) and the unfinished quarantine (7..40), and reported both. Now every loss inside another (recorded or found by this open) is dropped, the larger range kept; DAMAGE is rewritten only when its set of losses changed. * cmux-journal: fourth-review regressions G8, N3, N4 (red) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report. * cmux-journal: the quarantine's floor in its file names; DAMAGE before the moves (G8, N3, N4) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format. * cmux-journal: sort the quarantine list with sort_by_key (clippy unnecessary_sort_by) Fleet d9f22bed at b589044b852: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`). * cmux-journal: Journal::on_written, a listener per written batch (for S2's projector) R-DURABILITY S2 (acpmux event logs through the journal) projects each chat's log line into its ndjson file once the journal has WRITTEN the batch that holds it, so readers see stream lines at once. The listener runs on the writer thread after the write and before the sync, once per batch that moved the written LSN: a batch written again into a new segment after a failed sync is not heard twice. When the writer stops, these listeners are dropped too. Test: heard while the batch's sync still runs, once. Port of 145faa6faa8 (branch rewrite/r-durability-s2-acpmux-logs) onto the S1 review fixes. * cmux-journal: Journal::retire_through, the one delete (segments a compaction point covers) R-DURABILITY S2: a daemon's projector compacts the journal into its own files (acpmux: the chat ndjson logs, then a synced checkpoint). Without a delete the segment count only grows (review finding F6). `retire_through(compacted)` deletes the segments that hold no record after the caller's compaction point: a segment goes when the next segment's "previous last LSN" is at or before it, so the newest segment never goes. The point is capped at the durable LSN. The deletes and a folder sync run on the caller's thread (the projector, never a request path). A delete that fails leaves its segment live for a later call. Why a crash during a retirement is safe: the caller reopens from a point at or after `compacted`, and recovery reads nothing from a segment the point covers (it skips it on its next segment's header), so any subset of the deletes may reach the disk. A point older than the retired segments is reported as Damage::Missing. Quarantined segments are never deleted. - Storage::remove_file (FsStorage: std::fs::remove_file; SimStorage: one more change in the ordered log of names, durable at the folder sync). - The live segment list: recovery's, then each segment the writer makes. - JournalStats::segments_retired. - Tests: only covered segments go, and a second call deletes none; an OS crash after it keeps every record after the point and an older point is reported; a crash at every operation of a retirement (process, OS crash, power loss) keeps every record after the point and never reuses an LSN; the newest segment stays; a point in the middle of what this writer wrote. This adds a delete of journal data, so its PR is risk:high. * acpmux: move MemoryStore and the store settings into their own files store.rs (2139 lines, 120 fns) and config.rs (1909 lines, 97 fns) are over the god-file limit, and the journal store (R-DURABILITY S2) must not grow them. MemoryStore moves to store/memory.rs and StoreMode, StoreConfig and its defaults to config/store_config.rs; both stay exported where they were (crate::store::MemoryStore, crate::config::{StoreConfig, StoreMode}). No behavior change: the code moves as it is. * acpmux: the event log's journal switch and its crash tests (red) R-DURABILITY S2 (DESIGN.md 3.3b). Today no event-log append is synced: a line a client was told about is on the disk only after the next flush (shutdown), so an OS crash or a power loss takes back every line since (E12). This commit adds only the switch, a seam and the tests; the journal store follows. - `store.journal` (StoreConfig::journal, default false). - `Store::wait_durable(timeout)`: tests and tools wait until every appended record is durable. The default returns at once (the Local and Memory stores never sync an append), which is today's behavior. - `store::open_with_journal` (tests): the store with the journal on a given `cmux_journal::Storage`. Here it ignores the journal and opens LocalStore. - acpmux depends on cmux-journal (dev: with the `sim` feature). - store/journal_store_tests.rs, red on this commit: - a_line_the_store_says_is_durable_survives_an_os_crash (lines 2..20 are lost: nothing synced them); - a_deleted_chat_stays_deleted_after_a_crash and a_line_the_log_refused_is_never_written_later pass here, and must keep passing with the journal store (no replay may bring them back); - a_kill_keeps_every_line_once passes here and must keep passing (a replay after a kill must write no line twice). * acpmux: the event log through one journal, behind `store.journal` (R-DURABILITY S2) DESIGN.md 3.3b. With `store.journal` on, each event-log append is one record in `<home>/journal/` (cmux-journal: group commit, one sync per batch; a power loss takes back at most the last second, always a suffix), then the same line goes to the chat's ndjson as before. Readers, segments and file formats do not change. Default off. - store/journal_store.rs (JournalStore over LocalStore): - append: under one lock, the journal record (a memory copy, no I/O), then the log line; a line the log refuses gets a `cancel` record. Agent stream lines (`in` session/update, claude.*, codex.*) are Class::Stream, every other line Class::Ack. - checkpoint (`<home>/journal-projected.json`): every 30 s while the journal grew, past 64 MiB, at flush (shutdown) and before a retention commit. It syncs each log written since (LocalStore::sync_log, with no lock held while the disk works) and the sessions folder, writes the LSN (temp file, sync, rename, folder sync, every error checked), then retires the journal before it. A log close that failed its sync freezes the checkpoint until a restart (LocalStore::sync_failed). - recovery: the records after the checkpoint; for each chat, the log is cut right after its last readable line below the first journaled seq (later segments removed, the boundary one shortened, their kind-index copies dropped), and the journaled lines are written again in order. `forget` deletes a chat again; `cancel` keeps a refused line out. - delete: `forget` record, folder removal, then a bounded wait until the record is durable. - seal: the logs refuse writes and the journal syncs fully. - LocalStore: `append` is serialize + `append_line`; `sync_log`; a failed sync at a log's close is noted (`sync_failed`). - store::open opens the JournalStore when `store.journal` is on. Not yet (S2c): replies and events do not wait for durability; the daemon does not stop when the journal closes; the writer thread keeps the process's Utility I/O policy. * acpmux: journal store tests for the seal and a chat made again after its delete (red) - a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay: a chat is deleted (its forget record), made again under the same id (a transfer back: session.json and new lines), then a kill -9. On this code the replay deletes the folder again and its session.json is gone. - a_line_after_the_seal_is_refused_and_never_written_later: coverage of the seal (it passes here; the fix also closes a race the test cannot force). * acpmux: journal store: the seal under the appends' lock; a replay keeps a chat made again - seal takes the appends' lock: an append lands in both the journal and the logs, or in neither. Before, an append that raced the seal could leave a journal record whose line the sealed logs refused (its cancel then failed on the closed journal), and the next start would write that line. - append and delete check the logs' seal first, under the same lock, and do not touch the journal after it (no error per append at shutdown). - replay deletes a forgotten chat again only when no line follows its forget record: otherwise the chat was made again under the same id, and its folder holds that chat's own files. * cmux-journal: fifth-review regressions G9 (red) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4. * cmux-journal: name a skipped LSN floor in a FLOOR.<lsn> file (G9) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646); the regression tests are the commit before this one. When an open gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which a copy then hides from every later open), the only record of that floor was the header of the empty segment it makes; a damaged header lost it, and DAMAGE is cleared by `clear_damage` (torn records are not damage at all). Now such an open makes the empty file `FLOOR.<lsn>` and syncs its folder before any new segment, and every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine's names and this open's moves say their floors already: no file is made for them. * acpmux: journal store regressions from the S2 review (red) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824). Each test fails on this commit and passes after the fix that follows it: - 1a lines_written_with_the_switch_off_survive_turning_it_on_again - 1b a_kill_while_the_journal_syncs_keeps_the_lines_the_log_got - 1c a_line_flush_session_made_durable_survives_losing_its_record - 2 a_folder_put_back_after_a_delete_is_not_deleted_by_the_replay - 3 a_kill_in_a_delete_leaves_the_chat_whole_or_gone - 4 a_delete_does_not_wait_for_the_journal_sync - 5 a_failed_log_sync_freezes_the_checkpoint, a_checkpoint_that_keeps_failing_asks_the_daemon_to_stop - 6 one_chat_that_cannot_be_repaired_does_not_stop_the_open - 7 the_journal_storage_refuses_a_lost_home - the repair keeps a log's lines after a hole, also through a crash in it: an_os_crash_hole_is_repaired_and_the_lines_after_it_kept, a_crash_inside_a_repair_loses_no_line - nits: an_unreadable_checkpoint_fails_the_open, the_open_waits_for_the_previous_daemon_to_let_go_of_the_journal, a_crashed_checkpoint_temporary_is_removed_at_open, a_crash_right_after_a_checkpoint_reports_no_damage - the catch-up with the switch off: switching_off_after_a_crash_keeps_every_acknowledged_line - cmux-journal: check_append_says_what_append_would_do So they compile here, this commit adds the seams they use, with no behavior: `Store::stop_reason` (default None), `Journal::check_append` (always Ok), `FencedStorage` (never asks its fence), the open report and a test stop point after a repair's cut. * acpmux: the checkpoint freeze test fails a log sync, not a folder open (red) The freeze guards against a failed fsync (a later fsync of the same file may report success for pages the kernel dropped), not against a folder the store cannot open: a chmod 000 folder is an open error, which the fix retries instead of freezing. The test now fails the log sync itself through a test seam, `FAIL_NEXT_LOG_SYNC`. Red on the code before the fix, like the other review regressions (fleet d8e7037d, 1c22175e, 63604b26 at e030f4fe402: 17 of 17 new acpmux tests and the crate's check_append test fail). * acpmux: the journal store repairs logs, never rebuilds them (S2 review) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824); the regression tests are the two commits before this one. DESIGN.md 3.3c. - 1a/1b/1c: recovery no longer cuts a chat's log at its first journaled line and writes back only the journal's lines. It merges the log's lines and the journal's by seq: a line in both must be equal, a line only the log has stays, a line only the journal has goes in its place. Only bytes that do not read (a torn tail, a hole of zeros) are dropped. When the merge must change the log before its end, the lines only the log holds from there are journaled and made durable first, so a crash inside the repair loses no line. A log that moved on without the journal (a line in both that differs, lines out of order) is left as it is, and the journal's lines go to journal-unapplied/. - Append writes the log first, then the record, under one lock, after `Journal::check_append` (new in cmux-journal: the append's refusals, asked before any write). So a record always has its log line, and the `cancel` record is gone. - 2/3/4: the replay never deletes a folder and never makes a missing one again (a forget only drops the chat's earlier journaled lines). Delete appends its forget under the lock and removes the folder off it, with no wait for the record. - 5: a failed fsync in a checkpoint freezes it (a folder that does not open is retried); a frozen checkpoint returns an error, and the store asks the daemon to stop for a restart (`Store::stop_reason`, wired in daemon.rs), whose replay writes the logs again. Ten failed checkpoints in a row ask the same, so the journal past the checkpoint stays bounded. - 6: an error in one chat keeps its journaled lines in journal-unapplied/ and the open goes on. - 7: the journal's storage refuses writes under a lost home (`FencedStorage`, `config::refuse_lost_home`). - With store.journal off, a journal left by a run with it on is replayed, checkpointed and retired first (`catch_up`). - Nits: the checkpoint is never past the journal's durable end (no `Ahead` after a normal crash); damage is cleared after the next checkpoint; a kind-index copy goes before its segment is cut; an unreadable checkpoint fails the open; the open waits up to 5 s for the journal lock; crashed checkpoint temporaries are removed; `on_written` leaves cmux-journal (no user); README lists store.journal. * acpmux: split the journal store under its size budget (a move) journal_store.rs grew to 1085 lines and 71 functions with the review fix (budget 1000 and 60), and the fix grew store.rs and daemon.rs, which are over budget on the base. A pure move, no behavior change: - store/journal_store/{checkpoint,fenced,repair}.rs: the checkpoint and its thread, `FencedStorage`, the replay and the repair. - store/log_sync.rs: `LocalStore::sync_log`, `SyncFailed` and the test seam `FAIL_NEXT_LOG_SYNC`, out of store.rs (2139 lines on the base, 2113 now). - daemon/stop_causes.rs: `AbortOnDrop`, `next_signal` and `store_asks_to_stop`, out of daemon.rs (2052 on the base, 2046 now). check-no-godfiles.sh --base 51cfdee876b on every changed file: no file over budget grew, and no new file is over budget. * acpmux: the unrepairable-chat test waits for line 1 to be durable before its checkpoint At 03fccdef69c (fleet 32df3c72 on aws-m4pro-9, 144d4ccc on Linux) the test kept chat-b's lines [1, 2, 3] aside, not [2, 3]: it flushed right after appending line 1, and the checkpoint is now never past the journal's durable end (the review's Ahead nit), so line 1 was still past it. The fix's behavior is right; the test now waits until line 1 is durable, so its checkpoint covers it. On the code before the fix the test still fails (the open stops at chat-b's link). * acpmux: replies and events wait for durability, tests (red, S2c) DESIGN.md 3.3d. With the event journal on and every journal sync taking 400 ms (`SimStorage::set_sync_delay`): - an_acked_change_is_answered_once_its_record_is_durable: a rename's reply must come no sooner than one sync after its `renamed` record. Red here: the reply goes at once. - agent_output_is_not_held_behind_another_chats_acked_change: agent output to a connection attached to its chat goes at once while another chat's acknowledged change waits for its sync. Green here (nothing waits); it guards the fix against holding agent output behind changes the connection does not hear. * acpmux: replies and events wait until what they show is durable (S2c) DESIGN.md 3.3d; the tests are the commit before this one. With `store.journal` on, each connection gets an outbound gate (server/durable_gate.rs) between it and its transport: - Every line a connection queues raises the gate's `needs` to the last acknowledged LSN (`Store::durability().acked`, set in the journal store's append before the hub publishes anything of the record), except the lines of a `Stream` record's event (agent output, DESIGN C4): the fan-out delivers those in a thread-local stream scope. - The gate's task takes every queued line at once, reads `needs`, waits (async) until the journal's durable LSN reaches it, then sends the batch in order: one wait per batch, as the journal's group commit. - The durable LSN is a `tokio::sync::watch` fed by `Journal::on_durable`. The listener owns a guard: a writer that ends while the store is not being dropped (a panic on its thread) asks the daemon to stop (`Store::stop_reason`, decision 9.8), and the watch closes, so each gate drops what it holds and ends, which closes its connection. - Without the journal, `Store::durability()` is None: no gate, and `Conn::raise_gate` does nothing. Moves, so server/mod.rs and store.rs stay under their size on the base: the outbound queue helpers to server/outbound.rs, `open_with_journal` to the journal store, and `streamed` to store/durability.rs. * acpmux: the journal lock wait's deadline without an unchecked Instant addition `Instant + Duration` can panic on overflow (clippy::arithmetic_side_effects, tracked by the panic ratchet). The wait now has no deadline only in the case that cannot happen (an Instant 5 s ahead overflowing), where it stops waiting at once. * acpmux: a failed log sync stops the real daemon for a restart, test (red) Lead rule 2026-11-04 (3): a daemon-level test of the store's "stop for a restart" before S2c merges, red first. With store.journal on and every log sync failing (a debug build's ACPMUX_TEST_FAIL_LOG_SYNC; the checkpoint every 200 ms through ACPMUX_TEST_JOURNAL_CHECKPOINT_MS), a rename of a chat on disk makes its log dirty, the checkpoint's sync fails and freezes it, and the store asks the daemon to stop. The test asserts: the WebSocket client gets the restart close (1012), the socket client's pending `_acpmux/wait` ends with the connection (no hang), and the daemon exits with status 74. Red here: the daemon stops and sends 1012, but exits 0 (no exit status says that it stopped to be repaired). This commit adds only the two debug-build seams (release builds have none) and the test. * acpmux: a daemon its store stopped for a restart exits with status 74 The test is the commit before this one. `daemon::run` returned Ok and the process exited 0 for every stop, so a supervisor or a person could not tell a stop to be repaired from a normal one; with systemd `Restart=on-failure` it would not even start again. Now the store's stop (`store_asks_to_stop`: a failed log sync froze the event journal's checkpoint, or the journal's writer ended) marks itself, sets the restart handoff (WebSocket clients get 1012, as before), and the process exits with `daemon::EXIT_REPAIR` (74, EX_IOERR). The units prevent only 75, so launchd KeepAlive and systemd Restart=always start it again; the app does not read the daemon's exit status. README: store.journal names the status. * acpmux: the restart test's request in flight goes through the acpmux client Lead rule (3): the client sees a typed error, not a hang. The socket client is now the acpmux client library (acpmux::client::Client, as the CLI uses it), and the test asserts what the CLI makes of the request that the stop ends: `daemon_closed`, retryable (cli::errors::classify). The first red (fleet 41f2d037, hq run 41804640229, at 80c5fb2b7f8) failed only on the exit status (0, wanted 74), with 1012 and the end of the connection passing. * acpmux: the daemon's disk tier with the journal on, tests (red) Lead rule 2026-11-04 (2): a reply the person waits for never waits on a throttled disk sync. The fleet A/B (STATUS 11554) found that at IOPOL_UTILITY an acknowledged journal append waits about 16 ms p50 under another process's disk stress (0.7 to 1.1 ms at the default tier). - the_daemon_with_the_journal_keeps_the_default_disk_tier: with store.journal on, the daemon process keeps the default tier; without it, utility as before (each case in a child of the test binary). - a_background_thread_yields_its_disk_work: a dedicated background thread (the checkpointer) works at utility; a thread beside it does not. The API they use is stubbed here (`qos::enter_daemon` ignores the journal; `qos::background_thread` does nothing), so both fail on macOS. * acpmux: the restart test borrows no &str twice (clippy needless_borrows_for_generic_args) Found by fleet 7f8afd9d (hq run: the S2c server step at 62b4f68ef8b); the step's other clippy red is the base red in link_push_bench.rs. * acpmux: with the journal on, the daemon keeps the default disk tier (S2c) The tests are 994f74119fb. Lead rule 2026-11-04: a reply the person waits for never waits on a throttled disk sync. Measured on two owned Macs (STATUS 11554): one acknowledged append to durable at IOPOL_UTILITY under another app's disk stress took 16.3-16.7 ms p50 and 37-45 ms p95, against 0.7-1.1 ms p50 and 1.5-1.8 ms p95 at the default tier. - qos::enter_daemon(journal): with store.journal on, the daemon process keeps the default disk tier; without it, utility as before. The switch is read alone from config.json (config::store_journal_enabled), before any thread starts, with no harness discovery. - qos::background_thread: the journal store's checkpointer lowers its own disk I/O to utility for its whole life (a thread that does nothing else: a thread-scope policy drops it out of QoS for good). * acpmux: an agent chunk is a Stream record, test (red) The hub's tap logs an ACP session/update under its sessionUpdate kind (hub/tap.rs: `kind = su.to_owned()`), so a record's kind is `agent_message_chunk`, `tool_call`, ... and never `session/update`; the message's method is. The journal store's `streamed` checked the kind, so every agent chunk was journaled as Ack: a sync per batch at the token rate, and in S2c each token's event waited for that sync (the S2c I/O A/B: STRESS token to client 2.2 ms -> 20.3 ms p50 with the journal on). Red here: the chunk reads back as Ack. * acpmux: an agent chunk is a Stream record (the record's message names session/update) The test is the commit before this one. `streamed` now reads the logged message's method: the hub's tap logs a session/update under its sessionUpdate kind (hub/tap.rs), so the kind never said it. Agent output is again Stream (synced with the next acknowledged batch or a second later), as DESIGN.md 3.3b says. * acpmux: review 2 of #34901, tests (red): a crash model that follows real log syncs The logs' OS crash model now follows the syncs that ran (store/sync_model.rs, tests only): a segment keeps its length at its last sync that counts; a failed sync loses what was written since the one before, and a later sync does not bring it back; a cut ends a loss it removes. Every log segment sync goes through log_sync::sync_segment (same calls as before), where the model and FAIL_NEXT_LOG_SYNC hook in. Red tests, one per finding: - 1a an_os_crash_after_a_restart_from_a_kill_keeps_every_acked_line: the restart's first checkpoint retires records whose log lines no sync made durable; expected [1..5], the head gives [1]. - 1b an_os_crash_after_a_restart_from_a_failed_sync_keeps_every_acked_line: the same after a frozen checkpoint and a restart; expected [1..5]. - 2 a_hole_that_starts_inside_a_line_is_repaired (the reviewer's test): a torn line is read as line 6 of a log that moved on; expected [1..9], the head gives [1, 2, 3, 4, 5, 9]. - 3 a_failed_sync_in_flush_session_freezes_the_checkpoint: a failed sync in flush_session does not freeze the checkpoint. * acpmux: review 2 of #34901: a restart writes the replayed lines again; a torn line is not a line; flush_session freezes Finding 1 (high): the repair now writes every merged line again from the repair start, even when the log already has them all. A killed daemon's log writes may be in the page cache only, and after a failed sync the kernel may have dropped pages that still read back and that a later sync reports as written; the first checkpoint after the open then syncs bytes this process wrote before it retires their records. The lines only the log has from the repair start are journaled and made durable first, as before a cut. The work is bounded by the checkpoint interval. Finding 2: a piece of the log is a line only when it parses as one JSON value (piece_seq). A hole that starts inside a line keeps that line's {"seq":N, head and runs into a later line's tail; it was read as a line of a log that moved on, and the chat's journaled lines went aside. Now its bytes go and the lines are written again. Finding 3: a failed sync in flush_session freezes the checkpoint (its file sync is a SyncFailed now), as a failed sync in a checkpoint or at a log's close does. Nits: cut_at syncs events/ after its removals (and the kind-index folder after its copies go), so a removed segment cannot come back beside the lines written again; the README says that with the switch on an event line over 64 MiB is refused and its log does not get it. Tests: the four red tests of d2c63860ece. * acpmux: chat metadata through the event journal, tests (red, R-DURABILITY S3a) DESIGN.md 3.3e. Red on S2c: - last_seq_never_passes_the_log_after_an_os_crash (E13): session.json is synced at each save while the agent's lines wait for the journal's next sync, so after an OS crash the metadata names lines the log lost. - a_chat_made_after_the_checkpoint_survives_an_os_crash_that_loses_its_folder: nothing syncs the sessions folder when a chat's folder is made, and the replay never makes a folder again, so an acknowledged new chat is lost. - a_metadata_change_is_written_to_session_json_by_the_checkpoint: today each change writes and syncs session.json on the request path. Guards (green before and after): a chat made before the checkpoint and deleted is not made again; the replay never writes an older image over session.json. * acpmux: chat metadata through the event journal (R-DURABILITY S3a) DESIGN.md 3.3e, behind `store.journal` (default off). A metadata change is a `meta` record (kind 4: chat id, a `created` flag, the whole image) instead of a session.json write with two syncs on the request path; the reply waits for the record through S2c's gate, one group commit for a burst. - Overlay: each chat's newest image until a checkpoint writes session.json; load, load_startup and list read it first (list under the projection lock, so a checkpoint cannot drop an entry between its two reads). - Checkpoint: waits on its own thread until every record through its LSN is durable, then writes each image it read (session.json carries the image's LSN as `journalLsn`), then the logs and the sessions folder, as before. An image so names only lines the journal holds (E13). A chat deleted or made again meanwhile is passed (its overlay entry is gone or new). - Replay: per chat after its last `forget`, the newest image, written when newer than the file's `journalLsn`. A missing folder is made again only for a chat a replayed `created` record made; a chat made before the checkpoint whose folder is gone keeps its image in journal-unapplied/. - flush_session also waits until the chat's records are durable. - Delete drops the overlay entry with its `forget`, and removes the folder under the folders lock a checkpoint's write also takes. Tests: the 5 of 0505e9bee04 (3 red there). * acpmux: S3a, tests (red): folders handed out of the store Code outside the store reads and moves chat folders directly (transfer_in puts the live folder aside and installs a transferred chat; session_dir hands the folder to the transfer, the session files view and others). With the metadata in the overlay until a checkpoint: - a_folder_replaced_outside_the_store_keeps_its_own_metadata: a read, the checkpoint and the replay must not put the old folder's image in the new one. - session_dir_hands_out_a_folder_whose_metadata_is_written. - flush_session_writes_the_chats_metadata (the trait says log and metadata). - the_checkpoint_never_makes_a_folder_removed_outside_the_store_again. Red on 71bc09a1e63 (S3a's first implementation). * acpmux: S3a red tests: a replay after a folder swap; flush_session's chat has a log (as its callers' chats do) * acpmux: S3a: each metadata image is bound to its chat folder; session_dir and flush_session write it first Code outside the store reads and moves chat folders directly (a transfer puts the live folder aside and installs a transferred chat under the same id). So a `meta` record now carries its folder's identity (file id and birth time): - reads use an overlay image only while the folder at the path is its folder; - the checkpoint writes an image only into its own folder, never makes a missing one, and keeps the image of a removed or replaced folder in journal-unapplied/; - the replay writes an image only into its own folder (or a folder a replayed `created` record made), else keeps it aside. `session_dir` (the transfer, the session files view) and `flush_session` (the trait: "log and metadata to stable storage") write the chat's image first, once its record is durable. The list reads the overlay before the files (a checkpoint writes a file before it drops the entry), so it needs no lock. Tests: the 5 of 0280ac37f5e and 0e617489731. * acpmux: S2 follow-ups from review 3 of #34901, tests (red) - a_chat_whose_repair_failed_is_repaired_at_the_next_start (follow-up 1): the open's checkpoint retires the records of a chat whose repair failed, so no later start puts its lines back (they are only in journal-unapplied/). - a_hole_that_ends_at_a_line_start_keeps_that_line (follow-up 2): a torn line, zeros, then a whole line only the log holds: the piece does not parse and the whole line is dropped. - an_idle_store_does_not_wake_its_checkpointer (follow-up 4, cmux-ad m9): the checkpointer wakes every period with nothing to do. Test seams only (no behavior change): CHECKPOINTER_WAKES and TEST_CHECKPOINT_EVERY_MS. * acpmux: S2 follow-ups from review 3 of #34901: a failed repair holds its records; a hole's last zero splits; an idle checkpointer sleeps - Follow-up 1: a chat whose repair fails at the open (its log cannot be read or written now) keeps its lines in journal-unapplied/ as before, and the checkpoint now stays below its first journaled LSN (`Order::held`), so the next start repairs it again. Bounded: past 256 MiB of journal after it, the hold goes and the lines stay only in journal-unapplied/ (an error says so). - Follow-up 2: a piece splits at its last zero byte (a whole JSON line holds none), so a hole that ends at a line start, after a torn line, keeps that line (it mattered for a line only the log holds). - Follow-up 4 (cmux-ad m9): with nothing to checkpoint, the checkpointer sleeps until a change wakes it (`Inner::touched`, one atomic read per append), then checkpoints one period later; before, it woke every 30 s. Tests: the 3 of d147057bf85. * acpmux: a metadata change logs its line before its metadata, tests (red, R-DURABILITY S3b) DESIGN.md 3.3e, E6: with the event journal, records survive a crash as a prefix in append order. So a change whose line reaches the store before any metadata image that holds it is never, after a crash, in the chat's metadata without its line in the log. A recording store checks the order of the store calls: - a_tag_change_logs_its_line_before_its_metadata: red (tags saves first). - a_rules_change_logs_its_line_before_its_metadata: red (rules saves first). - a_rename_logs_its_line_before_its_metadata: green (a guard). * acpmux: the qos journal tests module is macOS-only (its Linux build held only an unused import: clippy red, review of #35400 F0) * acpmux: review of #35400 F1 and F2, tests (red) - a_read_that_shows_a_line_covers_its_record (F1): a read sees a log line between its log write and its record (the appends' lock held there), and the reply built from it goes out with no wait for that record. - a_journal_that_syncs_nothing_asks_the_daemon_to_stop (F2): with a journal that makes nothing durable, every reply waits without a limit and nothing asks the daemon to stop. Test seams only: PAUSE_AFTER_LOG_WRITE, TEST_STALL_MS. * acpmux: review of #35400 F1, the streaming read, test (red): session/load's replay sends a line before its record * acpmux: the test seam PAUSE_AFTER_LOG_WRITE is reachable from the server tests * acpmux: review of #35400: a read covers the records of the lines it shows; a stalled journal stops the daemon F1: the append writes a line to the log, then appends its record and sets the acknowledged LSN, all under the appends' lock; reads take no lock. A read could show a line before its record existed, and the reply built from it went out at once. Now every journal-store read of the logs ends with a fence on the appends' lock (`Inner::fenced`): a line the read saw has its record appended and its LSN acknowledged when the read returns, so the reply's gate covers it. The one read that sends as it reads (`Hub::scan_events`, session/load's replay) stops at the chat's last_seq read first. F2: a journal writer that keeps failing (it retries without end) held every reply on every connection with no limit. A connection's gate now waits at most 10 s (`STALL_LIMIT`); past it, `Durability::stalled` asks the daemon to stop for a restart, as a failed log sync does. Tests: the 3 of 3916f84dd68 and c553a19fd8b. * acpmux: S3b red tests: the recording store implements delete (a required Store method) * threading-model: the acpmux event journal's threads and bounds (writer, checkpointer, per-connection gate; review of #35400) * acpmux: the stall test's journal syncs take 3 s, not an hour (a sync under way kept its delay and hung the test's end) * acpmux: a tags or rules change logs its line before its metadata (R-DURABILITY S3b) DESIGN.md 3.3e (revised): with the event journal, records survive a crash as a prefix in append order, so a change whose line reaches the store before any metadata image that holds it is never, after a crash, in the chat's metadata without its line in the log (E6). `renamed`, `policy` and `mode` already did this; `tags` and `rules` saved first. Their reply still waits for the save. Tests: the 3 of 5bb3a27a2da and 7a4f02a56f3 (red at 7a4f02a56f3, fleet 88309e91: tags ["save:tagged", "line:tags"]; rules ["line:permission_chat_allowance", "save:ruled", "line:rules"]). * acpmux: an open that replays many chats flushes the drive a bounded number of times, test (red) Measured (fleet ead05e82, owned Mac aws-m4pro-5): the event journal store's open after a kill takes 29 ms per replayed chat on macOS (8.8 s for 300 chats, past the 8 s start budget; Linux 0.6 ms per chat, 681725fb). Each replayed chat costs about 7 F_FULLFSYNC (every sync_all/sync_data on macOS flushes the drive's cache): the rewrite's truncate and folder syncs, session.json's write_atomic, the checkpoint's log and folder syncs. Test seams only (no behavior change): FULL_SYNCS counts the store's full syncs (`log_sync::sync_full`, `sync_segment`, the folder syncs, session.json). * acpmux: the journal store's checkpoint and recovery sync in write order and flush the drive once Measured (fleet ead05e82, owned Mac aws-m4pro-5): an open after a kill took 29 ms per replayed chat on macOS (8.8 s for 300 chats, past the 8 s start budget), because every sync was an F_FULLFSYNC (std's sync_all and sync_data on macOS), about 7 per chat. Now the checkpoint's log and folder syncs, the repair's cut, and session.json's write sync with F_BARRIERFSYNC (durable against a crash of any process and a kernel panic, in write order; Linux fdatasync as before), and the checkpoint file's own syncs stay full: its F_FULLFSYNC flushes the drive's whole cache once per checkpoint, before any journal record is retired, so everything synced before it is durable against a power loss too (DESIGN.md 3.4: the checkpoint and recovery still end fully durable). cmux-journal: `sync_file(file, SyncLevel)` is public (the crate's tested F_BARRIERFSYNC/F_FULLFSYNC call with its ENOTSUP fallback), so acpmux uses the same sync. Test: an_open_that_replays_many_chats_flushes_the_drive_a_bounded_number_of_times (red at b2240f1b050). * acpmux: the open after a kill syncs nothing per replayed chat, test (red) At 15cfa1421c4 (ordered syncs) an open after a kill still took 6.1 s for 300 replayed chats on an owned Mac (fleet 59175d6b): about 8 ordered syncs per chat at about 2 ms each under load, plus a full flush per log evicted past 64 open logs. The test counts every sync the store makes during the open (seam ALL_SYNCS, tests only) and wants a constant, not one per chat; then the checkpoint (flush) makes every line durable through an OS crash. * acpmux: the open-syncs test counts the opening thread's syncs only (the checkpointer's thread is not the open's cost) * acpmux: the open after a kill syncs nothing per replayed chat; a freeze leaves a durable marker At 15cfa1421c4 an open after a kill still took 6.1 s for 300 replayed chats on an owned Mac (59175d6b), and it synced 283 times for 40 chats (90d87e10): every replayed chat's log was cut and written again (review 2 of #34901, finding 1, the forced rewrite) and its session.json written, then the first checkpoint ran before the open returned. Now: - The repair leaves a log that already holds every merged line, in order and untorn, as it is, and records its segments; the next checkpoint syncs them (ordered) before it retires their records, so finding 1a stays closed. - A freeze writes a durable marker (`journal-frozen`): the next open writes every replayed chat's lines again (finding 1b: the kernel may have dropped pages that still read back); the first checkpoint after it removes the marker, unless a chat's repair failed (the hold keeps it for the next start). - Replayed metadata images go to the overlay; the checkpoint writes session.json. - The open's first checkpoint runs on the checkpointer's thread at once, after the open returns (the daemon binds its socket after the open). Tests: an_open_after_a_kill_syncs_nothing_per_chat_and_the_checkpoint_makes_it_durable (red at 6def18f92c5), a_freeze_marker_makes_the_next_open_write_the_lines_again_then_goes; the restart-then-OS-crash tests of review 2 now flush after the reopen (the open's checkpoint is no longer inside it). * acpmux: the idle checkpointer's period, test (red): a change seen by its second look must not start a checkpoint at once - a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period: the checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies when it gets the timer lock. The second look saw the change, so the checkpointer is in its period wait by then, and the notify ends that wait: a checkpoint runs at once on the checkpointer's thread. This is the cause of a_failed_log_sync_freezes_the_checkpoint failing in the full store:: run of fleet 0577ac62 (that early checkpoint took the line whose sync the test fails, on a thread where the failure is not set). The test holds both threads at the two points, so the order is exact. Test seams only (no behavior change): the checkpointer's test seams are now by the store's home (period, wake count, the two holds), so tests that run as threads of one process under cargo test do not see each other's. * acpmux: the idle checkpointer's period ends only on its own state, not on a wake meant for its idle wait The checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies once it gets the timer lock. Its change shows in the second look, so the checkpointer has gone to its period wait by then, and that notify ended the period at once: a checkpoint ran right away on the checkpointer's thread (a burst no longer shared one checkpoint), and a spurious wakeup could do the same. In the full store:: run of fleet 0577ac62 such a checkpoint took the line whose sync a_failed_log_sync_freezes_the_checkpoint fails, so that test's own flush had nothing to do and returned Ok. The period wait now ends only on the period, `wanted` or `stop` (`wait_timeout_while`): each of the checkpointer's two waits ends only on the state its waker sets before it notifies under the timer lock. No sleep, no retry. The idle wait is unchanged. Test: a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period (edca3b4823d). * acpmux: S3a review (#35483) red tests: switch-off metadata and deletes, a late settle, a failed session.json folder sync, a purge after the checkpoint F1: metadata written with the switch off survives turning it on again; a chat deleted with the switch off after a failed catch-up is not made again. F2: a late settle never writes an older image over a checkpointed one (the reviewer's test and hold). F3: a failed folder sync after a metadata write freezes the checkpoint. F4: a chat made after the checkpoint and deleted is not made again after a kill. Test seams only; the code is the S3a head merged with feat. * acpmux: S3a review (#35483) red tests: a new chat is acknowledged only once its folder is durable; session_path never waits The root of F4 and of F1's delete case: the replay makes a chat's folder again, so a chat removed by any path comes back. The fix will never make a folder again, which needs an acknowledged chat's folder to survive an OS crash: the test models what each folder's last sync kept (sync_model) and asserts the gate's durability passes a new chat only after its sessions entry is synced. P3 1 of the review: a caller that needs only the path never waits for the journal (Store::session_path, a stub with session_dir's behavior here). * acpmux: S3a review (#35483) fixes: the replay never makes a folder again; switch-off changes are kept; no late or unsynced metadata write F1: each meta record carries the stamp (file id, modification time, length) of the session.json its save saw. The replay writes an image over a file the journal did not write only when one of the chat's records saw that file; otherwise the file moved on without the journal (a run with store.journal off whose catch-up did not replay, or an older build) and the image goes to journal-unapplied/, as S2 does for a log. F1 (deletes) and F4 at the root: the replay never makes a chat's folder again. A save that made a folder notes its record's LSN; the new acpmux-journal-folders thread (the daemon's disk tier) syncs sessions/ at once, and the durable LSN the S2c gate waits for stays below that record until then (journal_store/new_folders.rs). So an acknowledged chat's folder survives an OS crash, and a folder that is gone was removed: a purge whose forget a kill took back, a run with the switch off, the transfer recovery. The created flag and the replay's folder remake are gone. F2: write_through does not write a file that holds the same or a newer journalLsn. F3: session.json is written with its own syncs (file, then folder); a failed sync is SyncFailed and freezes the checkpoint; the replay and the kept-aside images return their errors. P3 1: Store::session_path gives the path with no lock, wait or write; an agent's start, the fixed-id check, the temporary-folder sweep, the host probe and the transfer recovery's root use it. The replaced test (a chat made after the checkpoint survives an OS crash that loses its folder) asserted a remake that cannot be right any more; the new test asserts the guarantee that makes the loss impossible, then the crash. Red (7 of 17 journal_meta_tests, each for its reason; the other 10 pass): macOS 5bc434a9 and 624d5df4, Linux a2207ff0 and 638a44ef. * acpmux: review of #35562 red tests: a failed repair holds no other chat's checkpoint; a checkpoint after it writes only durable images F1 (the reviewer's test, renamed): a checkpoint after a failed repair writes only durable metadata images. F2/F3 at the root: a failed repair does not hold the checkpoint of other chats; a retention rewrite of another chat after a failed repair stays after a restart. Guards: the reviewer's two switch-off tests (green here since the S3a fix 0d2…

github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Nov 3, 2026

lawrencecchen temporarily deployed to ci-routing-agent November 3, 2026 14:19 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to sentry-dsyms November 3, 2026 14:31 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts November 3, 2026 14:31 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts November 3, 2026 14:34 — with GitHub Actions Inactive

lawrencecchen added 2 commits November 3, 2026 14:39

lawrencecchen temporarily deployed to ci-routing-agent November 3, 2026 14:47 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to sentry-dsyms November 3, 2026 14:58 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts November 3, 2026 14:58 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts November 3, 2026 15:01 — with GitHub Actions Inactive

Merge feat-cmux-next (3a43646) into rewrite/r-durability-s1-journal-c…

89293fb

…rate Merge-main commit by scripts/merge-main.sh. Merged by scripts/merge-main.sh: origin/feat-cmux-next at 3a43646. Merge-main-previous-head: 0027391 Merge-main-base: 3a43646

lawrencecchen temporarily deployed to ci-routing-agent November 3, 2026 15:23 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts November 3, 2026 15:33 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to sentry-dsyms November 3, 2026 15:33 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts November 3, 2026 15:38 — with GitHub Actions Inactive

github-actions Bot temporarily deployed to ci-routing-agent November 3, 2026 15:39 Inactive

lawrencecchen added 2 commits November 3, 2026 17:55

cmux-journal: a sync a signal interrupts is made again (macOS)

a936848

fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r).

lawrencecchen temporarily deployed to artifacts November 4, 2026 04:15 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to ci-routing-agent November 4, 2026 04:17 — with GitHub Actions Inactive

lawrencecchen added 3 commits November 4, 2026 04:51

cmux-journal: sort the quarantine list with sort_by_key (clippy unnec…

9c47594

…essary_sort_by) Fleet d9f22bed at b589044: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`).

lawrencecchen temporarily deployed to ci-routing-agent November 4, 2026 05:40 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to ci-routing-agent November 4, 2026 05:42 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to sentry-dsyms November 4, 2026 05:52 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts November 4, 2026 05:52 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts November 4, 2026 05:55 — with GitHub Actions Inactive

lawrencecchen mentioned this pull request Nov 4, 2026

acpmux: the event log through one append-only journal, behind store.journal (R-DURABILITY S2) #34901

Merged

lawrencecchen added 2 commits November 4, 2026 06:02

lawrencecchen temporarily deployed to ci-routing-agent November 4, 2026 06:22 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to artifacts November 4, 2026 06:24 — with GitHub Actions Inactive

lawrencecchen temporarily deployed to ci-routing-agent November 4, 2026 06:27 — with GitHub Actions Inactive

lawrencecchen added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Nov 4, 2026

lawrencecchen merged commit da881c3 into feat-cmux-next Nov 4, 2026
267 of 271 checks passed

lawrencecchen deleted the rewrite/r-durability-s1-journal-crate branch November 4, 2026 07:06

lawrencecchen mentioned this pull request Nov 4, 2026

Pane motion: during a close the kept resize handle stays centred on a visible gap; the last column and the last row close without showing the window background (FRAME OWNER S3d) #35140

Merged

Conversation

lawrencecchen commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Review fixes (independent review FIX at 89293fb, comment 6610468202)

Re-review fixes (FIX at 3498975, comment 6620113993)

Third-review fixes (FIX at cb893c6, comment 6621681018)

Fourth-review fixes (FIX at 215c8f0, comment 6622740919)

Fifth-review fixes (FIX at 9c47594, comment 6623794646)

Testing

Observed / Not run

Changelog

Proof

Uh oh!

cursor Bot commented Nov 3, 2026

Bugbot needs on-demand usage enabled

Uh oh!

github-actions Bot commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blacksmith-sh Bot commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Failures

Uh oh!

lawrencecchen commented Nov 3, 2026

Uh oh!

cursor Bot commented Nov 3, 2026

Bugbot needs on-demand usage enabled

Uh oh!

lawrencecchen commented Nov 3, 2026

Uh oh!

lawrencecchen commented Nov 3, 2026

Uh oh!

lawrencecchen commented Nov 3, 2026

Uh oh!

lawrencecchen commented Nov 3, 2026

Findings

F1 (major): a crash during quarantine gives the quarantined LSNs to new records

F2 (major): a damaged header in the last segment gives its LSNs to new records

F3 (major): a panic on the writer thread stops durability, and nobody is told

F4 (major): the failed-sync rule does not survive a restart during the backoff

F5 (minor): the crash simulation cannot find F1 or F4

F6 (minor): recovery holds one open file per segment, and segments are never retired

F7 (nit): two doc sentences contradict the code

What I checked and found correct

Evidence

Uh oh!

lawrencecchen commented Nov 4, 2026

Findings

G8 (major impact, rare; the G7 class): a recovery copy is numbered above the quarantine, but its first LSN is below the quarantined LSNs

N3 (nit): a dropped loss takes its moved names with it

N4 (nit): DAMAGE is written after the renames

What I checked and found correct

Evidence

Uh oh!

lawrencecchen commented Nov 4, 2026

Findings

G9 (major impact, rare; the G7/G8 class): the floor from a damaged record in the last segment is lost once a copy follows it

What I checked and found correct

Evidence

Uh oh!

lawrencecchen commented Nov 4, 2026

Findings

What I checked

Evidence

Not verified

Uh oh!

lawrencecchen commented Nov 4, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

lawrencecchen commented Nov 3, 2026 •

edited

Loading

Review fixes (independent review FIX at `89293fb`, comment 6610468202)

Re-review fixes (FIX at `3498975`, comment 6620113993)

Third-review fixes (FIX at `cb893c6`, comment 6621681018)

Fourth-review fixes (FIX at `215c8f0`, comment 6622740919)

Fifth-review fixes (FIX at `9c47594`, comment 6623794646)

github-actions Bot commented Nov 3, 2026 •

edited

Loading

blacksmith-sh Bot commented Nov 3, 2026 •

edited

Loading

N3 (nit): a dropped loss takes its `moved` names with it