Repository navigation
acpmux: replies and events wait until what they show is durable; the daemon stops for a restart when the journal cannot go on (R-DURABILITY S2c) - #35400
Merged
Conversation
…nd ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner).
…clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red.
… segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee.
Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c8: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5.
fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r).
…1 review regressions (red) The S1 review (FIX at 89293fb) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open
…eral segments (red) The reviewer's F2 scenario has two segments: S1 holds 1..10, S2 (the last) holds 11..20, and S2's header is damaged. The test now rotates segments (300-byte segments, 20 acked records) and damages the last one's header, so the report must say lost_through 20 and the next append must take 21. On the code before the fix the moved segment's LSNs read as 0, so the next append reuses one.
…S crash) (red) On the code before the S1 fix, the crash-sim step was green (fleet 184129d8 at 263da86): 6 of 7 runs passed, and the mixed run failed only seed 140 in the full run and passed when it ran alone. The random faults reached the order the review found (F4) in 1 of 800 runs. So these tests were no red/green proof. The order: the sync of a write that spans pages fails and the disk gets none of them (Linux keeps them clean in the cache, so a later sync reports success); the process is killed in the writer's backoff; the next open reads the failed segment from the cache, replays its records and goes on; an OS crash or a power loss then takes those pages, and records an open already replayed are gone. - sim.rs: `fail_next_sync(pages, kind)` fails the next sync that has at least that many dirty pages, and the disk gets none of them. `crash_after_failure(n)` makes the n-th operation after the next injected failure the crash point. - crash_sim_tests.rs: a new run forces the order on every seed (a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record): epoch 0 aims the fault, the kill is a process crash at the writer's retry, the open after it does not crash, epoch 1 ends in an OS crash, and a third open checks that the records the second open replayed are all there. In the random runs, one epoch in four aims the fault, with the crash 1 to 3 operations after the failed sync and a crash of any kind after it. - A sim test checks the aimed fault itself (the_sim_fails_the_aimed_sync_and_crashes_in_the_backoff). Expected on this commit (old journal code): the forced run fails every seed; the mixed run fails most runs. The fix commit follows.
…y, LSN floor, writer end) Review FIX at 89293fb (PR #33919 comment 6610468202). The regression tests are the two commits before this one. - Sync markers (format.rs): an entry with flags 1 and no payload, whose LSN field says "every record of this segment before me is synced". The writer appends one after each successful sync that covered records (after the durable listeners, so no reply waits for it), and syncs it fully at a seal and at a rotation. - F4, recovery (step 4 and 6): the last segment is trusted only up to its last marker. Its records after the marker may sit in pages a failed sync dropped (a later sync on another descriptor reports success without them), so recovery writes them again into a new segment: header, records and a marker, fully synced under `<name>.seg.new`, renamed into place, folder synced. The journal always goes on in a segment this open made, never in one an earlier process wrote. A restart in the writer's backoff after a failed sync is then safe. - F1, quarantine: every LSN a quarantined segment holds counts in the LSN floor at every open, and a quarantine that no new segment followed (a crash in the middle of an earlier recovery) is reported as damage again. The highest LSN of every moved segment is read before the first rename. - F2, highest LSN: a segment whose header is damaged is read from its first record (each record's checksum covers its LSN), then the bytes after the first bad entry are scanned for records and markers that still check out. The same scan runs after a cut in the last segment: a marker there is media damage (reported), records there are a torn write (`torn_lsns`); their LSNs are never used again. - F3, writer end: a guard closes the journal when the writer thread ends, also by a panic: appends fail with Closed, waits end with WriterEnded, the listeners are dropped (so a channel a held reply waits on closes), and stats.failing says why. The crate does not abort the process; the daemon owns that policy. - F6: recovery closes each segment after its header is read and opens one again only to read it. (Retirement of compacted segments is an S2 API, before the first consumer.) - F5: recovery syncs the parent folder after it makes the journal folder. - F7: format.rs and lib.rs now say what the code does: a failed write is cut back and retried in the same segment; the cut removes only bytes never synced and never acknowledged; nothing shortens a segment past a synced record. Docs: journal.rs says when open's replayed records are durable (on Ok) and what a stopped writer does.
… (review G5) sim.rs had 924 lines and 63 functions, over the 60-function limit for a new Rust file (the god-file check warned "not in baseline: split it"). The machine model (each file's page cache and disk, the ordered log of names, and what an OS crash or a power loss leaves) moves to sim/machine.rs (297 lines, 19 functions); the process view and the Storage implementation stay in sim.rs (639 lines, 30 functions). No behavior change: the items move as they are, made visible to the sim module.
…ash sim (red) Re-review FIX at 3498975 (PR #33919 comment 6620113993). Tests and sim only; the fix follows. - sim: the pages a failed sync dropped stay in the cache until an eviction; `evict_dropped_pages_after_read(true)` makes each read evict the dropped pages it read, so a second read of them sees the disk's bytes. - G1 recovery_copies_the_records_it_checked_when_the_cache_drops_them: a sync fails and drops record 2's pages, a kill in the backoff, the next open reads record 2 from the cache (replayed) and the pages are evicted after that read; then an OS crash. Record 2 and the record 3 acked after the restart must survive. On this code recovery's second, unchecked read stages zeros under a vouching marker, and the last open keeps only record 1 and reports damage. - G2: the F1 regression asserts that every open after the damage reports the loss (lost_through 40). On this code a crash at the open's last step (the folder sync after the new segment's rename) loses the report for good. - G3 a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system): a journal in a symlinked state folder, and a journal folder that is a symlink. On this code the parent folder's sync opens with O_NOFOLLOW (ELOOP). - G4 an_open_after_a_finished_quarantine_reads_no_quarantined_segment: on this code every open reads every quarantined segment in full.
Re-review FIX at 3498975 (PR #33919 comment 6620113993); the regression tests are the commit before this one (red: fleet de35f6d0). - G1: recovery writes again exactly the bytes its checked read returned. The last segment's entries after its last marker are kept from the checked read (header and payload) and staged from memory; the second, unchecked read of the file is gone. A page that a failed sync dropped can be evicted between two reads, and the second read then returned the disk's zeros, which the copy kept under a vouching marker. - G2: the damage an open finds is written to `DAMAGE` (whole, fully synced, renamed into place, folder synced) before the new segment that hides it from later opens. Every open reports the recorded damage first, until the daemon calls the new `Journal::clear_damage`, which moves the record into `quarantine/` (kept, never deleted). - G3: a folder sync follows a symlink in its path (no O_NOFOLLOW): a state folder, or the journal folder itself, may be a symlink. The sync is for the folder that holds the entries. - G4: the quarantined segments are read for their LSNs only while no live segment is numbered above them (an unfinished quarantine); once one is, its first LSN already carries their floor. - G6: `checked_div` in the scan of the bytes after a cut (the crate's one tracked arithmetic hit). - G5 (the sim split) is the commit before the regressions.
Third review FIX at cb893c6 (PR #33919 comment 6621681018). Tests only. - G7 A a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn: after a finished quarantine, one byte of the empty new segment's header changes. On this code the next open skips the quarantine scan (the listing counts the damaged segment) and the next append reuses a quarantined LSN. - G7 B a_damaged_newest_segment_after_a_quarantine_reports_its_records: the newest segment holds acked 41..43, and its header and its first record are damaged. On this code the moved segment is scanned with a base far below 41, so the loss is not reported through 43 and the next append is not 44. - N1: the F1 regression also asserts that an open reports each loss once (no reported LSN range lies inside another). On this code a crash after the quarantine's renames makes the next open report the same loss twice.
…y.rs Third review FIX at cb893c6 (PR #33919 comment 6621681018); the regression tests are the commit before this one. - G7: the quarantine scan is skipped only when a segment whose header checked out is numbered above every quarantined segment (the folder listing also counted a newest segment with a damaged header, which carries no floor). The scans start from the running floor: `quarantined_floor` and `quarantine` pass the highest LSN counted so far as the base of the next file's scan, so a quarantined file with no readable header and no readable first record is scanned for the LSNs right above what is counted. - The "unfinished quarantine" report is made only when the quarantine holds LSNs nothing else counted (an old quarantine behind a segment that a crash tore at its creation holds none, and a crash is not damage). Its reason now also names a damaged newest header. - N1: a loss whose LSN range lies inside a recorded one is the same loss found again after a crash: it is neither recorded nor reported twice. - N2: recovery.rs's docs name `DAMAGE`; two doc lines over 100 columns are wrapped (lib.rs, sim.rs). - recovery.rs was at 987 of its 1000 lines: the damage record moves to recovery/damage.rs, the entry reader and the LSN scans to recovery/scan.rs (moved as they are).
…e (N1) The red run of the N1 regression (fleet 27d011ca, op 64 of the F1 test) shows the case the previous commit did not cover: a crash after the quarantine's renames and before DAMAGE is written; the next open finds both the damaged synced record of the last segment (7..14) and the unfinished quarantine (7..40), and reported both. Now every loss inside another (recorded or found by this open) is dropped, the larger range kept; DAMAGE is rewritten only when its set of losses changed.
Fourth review FIX at 215c8f0 (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report.
… the moves (G8, N3, N4) Fourth review FIX at 215c8f0 (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format.
…essary_sort_by) Fleet d9f22bed at b589044: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`).
…S2's projector) R-DURABILITY S2 (acpmux event logs through the journal) projects each chat's log line into its ndjson file once the journal has WRITTEN the batch that holds it, so readers see stream lines at once. The listener runs on the writer thread after the write and before the sync, once per batch that moved the written LSN: a batch written again into a new segment after a failed sync is not heard twice. When the writer stops, these listeners are dropped too. Test: heard while the batch's sync still runs, once. Port of 145faa6 (branch rewrite/r-durability-s2-acpmux-logs) onto the S1 review fixes.
…paction point covers) R-DURABILITY S2: a daemon's projector compacts the journal into its own files (acpmux: the chat ndjson logs, then a synced checkpoint). Without a delete the segment count only grows (review finding F6). `retire_through(compacted)` deletes the segments that hold no record after the caller's compaction point: a segment goes when the next segment's "previous last LSN" is at or before it, so the newest segment never goes. The point is capped at the durable LSN. The deletes and a folder sync run on the caller's thread (the projector, never a request path). A delete that fails leaves its segment live for a later call. Why a crash during a retirement is safe: the caller reopens from a point at or after `compacted`, and recovery reads nothing from a segment the point covers (it skips it on its next segment's header), so any subset of the deletes may reach the disk. A point older than the retired segments is reported as Damage::Missing. Quarantined segments are never deleted. - Storage::remove_file (FsStorage: std::fs::remove_file; SimStorage: one more change in the ordered log of names, durable at the folder sync). - The live segment list: recovery's, then each segment the writer makes. - JournalStats::segments_retired. - Tests: only covered segments go, and a second call deletes none; an OS crash after it keeps every record after the point and an older point is reported; a crash at every operation of a retirement (process, OS crash, power loss) keeps every record after the point and never reuses an LSN; the newest segment stays; a point in the middle of what this writer wrote. This adds a delete of journal data, so its PR is risk:high.
store.rs (2139 lines, 120 fns) and config.rs (1909 lines, 97 fns) are over
the god-file limit, and the journal store (R-DURABILITY S2) must not grow
them. MemoryStore moves to store/memory.rs and StoreMode, StoreConfig and its
defaults to config/store_config.rs; both stay exported where they were
(crate::store::MemoryStore, crate::config::{StoreConfig, StoreMode}). No
behavior change: the code moves as it is.
R-DURABILITY S2 (DESIGN.md 3.3b). Today no event-log append is synced: a line
a client was told about is on the disk only after the next flush (shutdown),
so an OS crash or a power loss takes back every line since (E12). This commit
adds only the switch, a seam and the tests; the journal store follows.
- `store.journal` (StoreConfig::journal, default false).
- `Store::wait_durable(timeout)`: tests and tools wait until every appended
record is durable. The default returns at once (the Local and Memory stores
never sync an append), which is today's behavior.
- `store::open_with_journal` (tests): the store with the journal on a given
`cmux_journal::Storage`. Here it ignores the journal and opens LocalStore.
- acpmux depends on cmux-journal (dev: with the `sim` feature).
- store/journal_store_tests.rs, red on this commit:
- a_line_the_store_says_is_durable_survives_an_os_crash (lines 2..20 are
lost: nothing synced them);
- a_deleted_chat_stays_deleted_after_a_crash and
a_line_the_log_refused_is_never_written_later pass here, and must keep
passing with the journal store (no replay may bring them back);
- a_kill_keeps_every_line_once passes here and must keep passing (a replay
after a kill must write no line twice).
…DURABILITY S2)
DESIGN.md 3.3b. With `store.journal` on, each event-log append is one record
in `<home>/journal/` (cmux-journal: group commit, one sync per batch; a power
loss takes back at most the last second, always a suffix), then the same line
goes to the chat's ndjson as before. Readers, segments and file formats do not
change. Default off.
- store/journal_store.rs (JournalStore over LocalStore):
- append: under one lock, the journal record (a memory copy, no I/O), then
the log line; a line the log refuses gets a `cancel` record. Agent stream
lines (`in` session/update, claude.*, codex.*) are Class::Stream, every
other line Class::Ack.
- checkpoint (`<home>/journal-projected.json`): every 30 s while the journal
grew, past 64 MiB, at flush (shutdown) and before a retention commit. It
syncs each log written since (LocalStore::sync_log, with no lock held
while the disk works) and the sessions folder, writes the LSN (temp file,
sync, rename, folder sync, every error checked), then retires the journal
before it. A log close that failed its sync freezes the checkpoint until a
restart (LocalStore::sync_failed).
- recovery: the records after the checkpoint; for each chat, the log is cut
right after its last readable line below the first journaled seq (later
segments removed, the boundary one shortened, their kind-index copies
dropped), and the journaled lines are written again in order. `forget`
deletes a chat again; `cancel` keeps a refused line out.
- delete: `forget` record, folder removal, then a bounded wait until the
record is durable.
- seal: the logs refuse writes and the journal syncs fully.
- LocalStore: `append` is serialize + `append_line`; `sync_log`; a failed sync
at a log's close is noted (`sync_failed`).
- store::open opens the JournalStore when `store.journal` is on.
Not yet (S2c): replies and events do not wait for durability; the daemon does
not stop when the journal closes; the writer thread keeps the process's
Utility I/O policy.
…its delete (red) - a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay: a chat is deleted (its forget record), made again under the same id (a transfer back: session.json and new lines), then a kill -9. On this code the replay deletes the folder again and its session.json is gone. - a_line_after_the_seal_is_refused_and_never_written_later: coverage of the seal (it passes here; the fix also closes a race the test cannot force).
…ps a chat made again - seal takes the appends' lock: an append lands in both the journal and the logs, or in neither. Before, an append that raced the seal could leave a journal record whose line the sealed logs refused (its cancel then failed on the closed journal), and the next start would write that line. - append and delete check the logs' seal first, under the same lock, and do not touch the journal after it (no error per append at shutdown). - replay deletes a forgotten chat again only when no line follows its forget record: otherwise the chat was made again under the same id, and its folder holds that chat's own files.
Fifth review FIX at 9c47594 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4.
Fifth review FIX at 9c47594 (PR #33919 comment 6623794646); the regression tests are the commit before this one. When an open gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which a copy then hides from every later open), the only record of that floor was the header of the empty segment it makes; a damaged header lost it, and DAMAGE is cleared by `clear_damage` (torn records are not damage at all). Now such an open makes the empty file `FLOOR.<lsn>` and syncs its folder before any new segment, and every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine's names and this open's moves say their floors already: no file is made for them.
Both sides added tests after the same test in cmux_journal_tests.rs; both are kept (the G9 tests, then the S2 retirement section).
Review FIX at ab0e953 (PR #34901 comment 6624439824). Each test fails on this commit and passes after the fix that follows it: - 1a lines_written_with_the_switch_off_survive_turning_it_on_again - 1b a_kill_while_the_journal_syncs_keeps_the_lines_the_log_got - 1c a_line_flush_session_made_durable_survives_losing_its_record - 2 a_folder_put_back_after_a_delete_is_not_deleted_by_the_replay - 3 a_kill_in_a_delete_leaves_the_chat_whole_or_gone - 4 a_delete_does_not_wait_for_the_journal_sync - 5 a_failed_log_sync_freezes_the_checkpoint, a_checkpoint_that_keeps_failing_asks_the_daemon_to_stop - 6 one_chat_that_cannot_be_repaired_does_not_stop_the_open - 7 the_journal_storage_refuses_a_lost_home - the repair keeps a log's lines after a hole, also through a crash in it: an_os_crash_hole_is_repaired_and_the_lines_after_it_kept, a_crash_inside_a_repair_loses_no_line - nits: an_unreadable_checkpoint_fails_the_open, the_open_waits_for_the_previous_daemon_to_let_go_of_the_journal, a_crashed_checkpoint_temporary_is_removed_at_open, a_crash_right_after_a_checkpoint_reports_no_damage - the catch-up with the switch off: switching_off_after_a_crash_keeps_every_acknowledged_line - cmux-journal: check_append_says_what_append_would_do So they compile here, this commit adds the seams they use, with no behavior: `Store::stop_reason` (default None), `Journal::check_append` (always Ok), `FencedStorage` (never asks its fence), the open report and a test stop point after a repair's cut.
…eld only an unused import: clippy red, review of #35400 F0)
- a_read_that_shows_a_line_covers_its_record (F1): a read sees a log line between its log write and its record (the appends' lock held there), and the reply built from it goes out with no wait for that record. - a_journal_that_syncs_nothing_asks_the_daemon_to_stop (F2): with a journal that makes nothing durable, every reply waits without a limit and nothing asks the daemon to stop. Test seams only: PAUSE_AFTER_LOG_WRITE, TEST_STALL_MS.
…load's replay sends a line before its record
…hows; a stalled journal stops the daemon F1: the append writes a line to the log, then appends its record and sets the acknowledged LSN, all under the appends' lock; reads take no lock. A read could show a line before its record existed, and the reply built from it went out at once. Now every journal-store read of the logs ends with a fence on the appends' lock (`Inner::fenced`): a line the read saw has its record appended and its LSN acknowledged when the read returns, so the reply's gate covers it. The one read that sends as it reads (`Hub::scan_events`, session/load's replay) stops at the chat's last_seq read first. F2: a journal writer that keeps failing (it retries without end) held every reply on every connection with no limit. A connection's gate now waits at most 10 s (`STALL_LIMIT`); past it, `Durability::stalled` asks the daemon to stop for a restart, as a failed log sync does. Tests: the 3 of 3916f84 and c553a19.
…r, checkpointer, per-connection gate; review of #35400)
Conflict: daemon.rs's module list. S2 brought feat's `term_sender` (#35147: a SIGTERM stop names its sender); S2c re-exports `EXIT_REPAIR` and `exit_status`. Both kept. The select block merged cleanly: feat's SIGTERM line and S2c's `store_stop` branch (which marks the restart itself).
…ad43f5 S2 landed as a squash, so its files conflict with S2c's own history of them. S2c holds S2's final head (3273b2a) plus its own changes, so for the files only S2 changed on feat (README.md, config/store_config.rs, daemon/stop_causes.rs, store.rs, store/journal_store.rs, store/journal_store/checkpoint.rs, store/journal_store_tests.rs, store/log_sync.rs) S2c's side is taken whole. Three other feat commits touched two of the conflicting files, and their changes are kept: #35405 (config.rs: private_state_create_new), #35291 (daemon.rs: `pub(crate) mod term_sender`, `term_sender::settled().await`) and #35359 (daemon.rs: `spawn_startup_respawns`). config.rs re-exports S2c's store_journal_enabled.
…under way kept its delay and hung the test's end)
lawrencecchen
temporarily deployed
to
ci-routing-agent
November 4, 2026 15:38 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
Shipped in:
|
github-actions
Bot
added
risk:normal
|
Found 1 test failure on Blacksmith runners: Failure
|
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light-3x.png)
lawrencecchen
temporarily deployed
to
sentry-dsyms
November 4, 2026 15:50 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
artifacts
November 4, 2026 15:50 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
artifacts
November 4, 2026 16:18 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
Author
|
Review (r-durability-s2c-reviewer): SHIP at eb24c40 Delta review of the repair of my FIX at Verdict: SHIP. F0, F1 and F2 are closed. Each repair test is red on the code before its fix and green at the head. The merges keep S2c's restart path and feat's changes. Two notes below (P3, follow-ups). They do not hold the merge. F0, F1, F2: closed
Notes (P3, follow-ups; they do not block)N1: the
|
lawrencecchen
added
the
ready-to-land
lawrencecchen
added a commit
that referenced
this pull request
Nov 4, 2026
…35483) * cmux-journal: one append-only journal per daemon, with group commit and ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner). * cmux-journal: type a test range; drop an unread segment field (fleet clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71e3f9 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red. * cmux-journal: a failed F_FULLFSYNC is an error, not a plain fsync; no segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee. * cmux-journal: the past-the-end test reads back the storage it wrote to Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c87b8d: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5c53e. * cmux-journal: a sync a signal interrupts is made again (macOS) fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r). * cmux-journal: crash sim of write-back errors and crash in recovery; S1 review regressions (red) The S1 review (FIX at 89293fb15f39) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open * cmux-journal: the F2 regression damages the header of the last of several segments (red) The reviewer's F2 scenario has two segments: S1 holds 1..10, S2 (the last) holds 11..20, and S2's header is damaged. The test now rotates segments (300-byte segments, 20 acked records) and damages the last one's header, so the report must say lost_through 20 and the next append must take 21. On the code before the fix the moved segment's LSNs read as 0, so the next append reuses one. * cmux-journal: the crash sim forces the F4 order (failed sync, kill, OS crash) (red) On the code before the S1 fix, the crash-sim step was green (fleet 184129d8 at 263da86f223): 6 of 7 runs passed, and the mixed run failed only seed 140 in the full run and passed when it ran alone. The random faults reached the order the review found (F4) in 1 of 800 runs. So these tests were no red/green proof. The order: the sync of a write that spans pages fails and the disk gets none of them (Linux keeps them clean in the cache, so a later sync reports success); the process is killed in the writer's backoff; the next open reads the failed segment from the cache, replays its records and goes on; an OS crash or a power loss then takes those pages, and records an open already replayed are gone. - sim.rs: `fail_next_sync(pages, kind)` fails the next sync that has at least that many dirty pages, and the disk gets none of them. `crash_after_failure(n)` makes the n-th operation after the next injected failure the crash point. - crash_sim_tests.rs: a new run forces the order on every seed (a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record): epoch 0 aims the fault, the kill is a process crash at the writer's retry, the open after it does not crash, epoch 1 ends in an OS crash, and a third open checks that the records the second open replayed are all there. In the random runs, one epoch in four aims the fault, with the crash 1 to 3 operations after the failed sync and a crash of any kind after it. - A sim test checks the aimed fault itself (the_sim_fails_the_aimed_sync_and_crashes_in_the_backoff). Expected on this commit (old journal code): the forced run fails every seed; the mixed run fails most runs. The fix commit follows. * cmux-journal: fix the S1 review findings F1-F7 (markers, recovery copy, LSN floor, writer end) Review FIX at 89293fb15f39 (PR #33919 comment 6610468202). The regression tests are the two commits before this one. - Sync markers (format.rs): an entry with flags 1 and no payload, whose LSN field says "every record of this segment before me is synced". The writer appends one after each successful sync that covered records (after the durable listeners, so no reply waits for it), and syncs it fully at a seal and at a rotation. - F4, recovery (step 4 and 6): the last segment is trusted only up to its last marker. Its records after the marker may sit in pages a failed sync dropped (a later sync on another descriptor reports success without them), so recovery writes them again into a new segment: header, records and a marker, fully synced under `<name>.seg.new`, renamed into place, folder synced. The journal always goes on in a segment this open made, never in one an earlier process wrote. A restart in the writer's backoff after a failed sync is then safe. - F1, quarantine: every LSN a quarantined segment holds counts in the LSN floor at every open, and a quarantine that no new segment followed (a crash in the middle of an earlier recovery) is reported as damage again. The highest LSN of every moved segment is read before the first rename. - F2, highest LSN: a segment whose header is damaged is read from its first record (each record's checksum covers its LSN), then the bytes after the first bad entry are scanned for records and markers that still check out. The same scan runs after a cut in the last segment: a marker there is media damage (reported), records there are a torn write (`torn_lsns`); their LSNs are never used again. - F3, writer end: a guard closes the journal when the writer thread ends, also by a panic: appends fail with Closed, waits end with WriterEnded, the listeners are dropped (so a channel a held reply waits on closes), and stats.failing says why. The crate does not abort the process; the daemon owns that policy. - F6: recovery closes each segment after its header is read and opens one again only to read it. (Retirement of compacted segments is an S2 API, before the first consumer.) - F5: recovery syncs the parent folder after it makes the journal folder. - F7: format.rs and lib.rs now say what the code does: a failed write is cut back and retried in the same segment; the cut removes only bytes never synced and never acknowledged; nothing shortens a segment past a synced record. Docs: journal.rs says when open's replayed records are durable (on Ok) and what a stopped writer does. * cmux-journal: split the crash sim's machine model into sim/machine.rs (review G5) sim.rs had 924 lines and 63 functions, over the 60-function limit for a new Rust file (the god-file check warned "not in baseline: split it"). The machine model (each file's page cache and disk, the ordered log of names, and what an OS crash or a power loss leaves) moves to sim/machine.rs (297 lines, 19 functions); the process view and the Storage implementation stay in sim.rs (639 lines, 30 functions). No behavior change: the items move as they are, made visible to the sim module. * cmux-journal: re-review regressions G1-G4 and page eviction in the crash sim (red) Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993). Tests and sim only; the fix follows. - sim: the pages a failed sync dropped stay in the cache until an eviction; `evict_dropped_pages_after_read(true)` makes each read evict the dropped pages it read, so a second read of them sees the disk's bytes. - G1 recovery_copies_the_records_it_checked_when_the_cache_drops_them: a sync fails and drops record 2's pages, a kill in the backoff, the next open reads record 2 from the cache (replayed) and the pages are evicted after that read; then an OS crash. Record 2 and the record 3 acked after the restart must survive. On this code recovery's second, unchecked read stages zeros under a vouching marker, and the last open keeps only record 1 and reports damage. - G2: the F1 regression asserts that every open after the damage reports the loss (lost_through 40). On this code a crash at the open's last step (the folder sync after the new segment's rename) loses the report for good. - G3 a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system): a journal in a symlinked state folder, and a journal folder that is a symlink. On this code the parent folder's sync opens with O_NOFOLLOW (ELOOP). - G4 an_open_after_a_finished_quarantine_reads_no_quarantined_segment: on this code every open reads every quarantined segment in full. * cmux-journal: fix the re-review findings G1-G4 and G6 Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993); the regression tests are the commit before this one (red: fleet de35f6d0). - G1: recovery writes again exactly the bytes its checked read returned. The last segment's entries after its last marker are kept from the checked read (header and payload) and staged from memory; the second, unchecked read of the file is gone. A page that a failed sync dropped can be evicted between two reads, and the second read then returned the disk's zeros, which the copy kept under a vouching marker. - G2: the damage an open finds is written to `DAMAGE` (whole, fully synced, renamed into place, folder synced) before the new segment that hides it from later opens. Every open reports the recorded damage first, until the daemon calls the new `Journal::clear_damage`, which moves the record into `quarantine/` (kept, never deleted). - G3: a folder sync follows a symlink in its path (no O_NOFOLLOW): a state folder, or the journal folder itself, may be a symlink. The sync is for the folder that holds the entries. - G4: the quarantined segments are read for their LSNs only while no live segment is numbered above them (an unfinished quarantine); once one is, its first LSN already carries their floor. - G6: `checked_div` in the scan of the bytes after a cut (the crate's one tracked arithmetic hit). - G5 (the sim split) is the commit before the regressions. * cmux-journal: third-review regressions G7 (A, B) and N1 (red) Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018). Tests only. - G7 A a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn: after a finished quarantine, one byte of the empty new segment's header changes. On this code the next open skips the quarantine scan (the listing counts the damaged segment) and the next append reuses a quarantined LSN. - G7 B a_damaged_newest_segment_after_a_quarantine_reports_its_records: the newest segment holds acked 41..43, and its header and its first record are damaged. On this code the moved segment is scanned with a base far below 41, so the loss is not reported through 43 and the next append is not 44. - N1: the F1 regression also asserts that an open reports each loss once (no reported LSN range lies inside another). On this code a crash after the quarantine's renames makes the next open report the same loss twice. * cmux-journal: fix the third-review findings G7, N1, N2; split recovery.rs Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018); the regression tests are the commit before this one. - G7: the quarantine scan is skipped only when a segment whose header checked out is numbered above every quarantined segment (the folder listing also counted a newest segment with a damaged header, which carries no floor). The scans start from the running floor: `quarantined_floor` and `quarantine` pass the highest LSN counted so far as the base of the next file's scan, so a quarantined file with no readable header and no readable first record is scanned for the LSNs right above what is counted. - The "unfinished quarantine" report is made only when the quarantine holds LSNs nothing else counted (an old quarantine behind a segment that a crash tore at its creation holds none, and a crash is not damage). Its reason now also names a damaged newest header. - N1: a loss whose LSN range lies inside a recorded one is the same loss found again after a crash: it is neither recorded nor reported twice. - N2: recovery.rs's docs name `DAMAGE`; two doc lines over 100 columns are wrapped (lib.rs, sim.rs). - recovery.rs was at 987 of its 1000 lines: the damage record moves to recovery/damage.rs, the entry reader and the LSN scans to recovery/scan.rs (moved as they are). * cmux-journal: report each loss once, also when one open finds it twice (N1) The red run of the N1 regression (fleet 27d011ca, op 64 of the F1 test) shows the case the previous commit did not cover: a crash after the quarantine's renames and before DAMAGE is written; the next open finds both the damaged synced record of the last segment (7..14) and the unfinished quarantine (7..40), and reported both. Now every loss inside another (recorded or found by this open) is dropped, the larger range kept; DAMAGE is rewritten only when its set of losses changed. * cmux-journal: fourth-review regressions G8, N3, N4 (red) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report. * cmux-journal: the quarantine's floor in its file names; DAMAGE before the moves (G8, N3, N4) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format. * cmux-journal: sort the quarantine list with sort_by_key (clippy unnecessary_sort_by) Fleet d9f22bed at b589044b852: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`). * cmux-journal: Journal::on_written, a listener per written batch (for S2's projector) R-DURABILITY S2 (acpmux event logs through the journal) projects each chat's log line into its ndjson file once the journal has WRITTEN the batch that holds it, so readers see stream lines at once. The listener runs on the writer thread after the write and before the sync, once per batch that moved the written LSN: a batch written again into a new segment after a failed sync is not heard twice. When the writer stops, these listeners are dropped too. Test: heard while the batch's sync still runs, once. Port of 145faa6faa8 (branch rewrite/r-durability-s2-acpmux-logs) onto the S1 review fixes. * cmux-journal: Journal::retire_through, the one delete (segments a compaction point covers) R-DURABILITY S2: a daemon's projector compacts the journal into its own files (acpmux: the chat ndjson logs, then a synced checkpoint). Without a delete the segment count only grows (review finding F6). `retire_through(compacted)` deletes the segments that hold no record after the caller's compaction point: a segment goes when the next segment's "previous last LSN" is at or before it, so the newest segment never goes. The point is capped at the durable LSN. The deletes and a folder sync run on the caller's thread (the projector, never a request path). A delete that fails leaves its segment live for a later call. Why a crash during a retirement is safe: the caller reopens from a point at or after `compacted`, and recovery reads nothing from a segment the point covers (it skips it on its next segment's header), so any subset of the deletes may reach the disk. A point older than the retired segments is reported as Damage::Missing. Quarantined segments are never deleted. - Storage::remove_file (FsStorage: std::fs::remove_file; SimStorage: one more change in the ordered log of names, durable at the folder sync). - The live segment list: recovery's, then each segment the writer makes. - JournalStats::segments_retired. - Tests: only covered segments go, and a second call deletes none; an OS crash after it keeps every record after the point and an older point is reported; a crash at every operation of a retirement (process, OS crash, power loss) keeps every record after the point and never reuses an LSN; the newest segment stays; a point in the middle of what this writer wrote. This adds a delete of journal data, so its PR is risk:high. * acpmux: move MemoryStore and the store settings into their own files store.rs (2139 lines, 120 fns) and config.rs (1909 lines, 97 fns) are over the god-file limit, and the journal store (R-DURABILITY S2) must not grow them. MemoryStore moves to store/memory.rs and StoreMode, StoreConfig and its defaults to config/store_config.rs; both stay exported where they were (crate::store::MemoryStore, crate::config::{StoreConfig, StoreMode}). No behavior change: the code moves as it is. * acpmux: the event log's journal switch and its crash tests (red) R-DURABILITY S2 (DESIGN.md 3.3b). Today no event-log append is synced: a line a client was told about is on the disk only after the next flush (shutdown), so an OS crash or a power loss takes back every line since (E12). This commit adds only the switch, a seam and the tests; the journal store follows. - `store.journal` (StoreConfig::journal, default false). - `Store::wait_durable(timeout)`: tests and tools wait until every appended record is durable. The default returns at once (the Local and Memory stores never sync an append), which is today's behavior. - `store::open_with_journal` (tests): the store with the journal on a given `cmux_journal::Storage`. Here it ignores the journal and opens LocalStore. - acpmux depends on cmux-journal (dev: with the `sim` feature). - store/journal_store_tests.rs, red on this commit: - a_line_the_store_says_is_durable_survives_an_os_crash (lines 2..20 are lost: nothing synced them); - a_deleted_chat_stays_deleted_after_a_crash and a_line_the_log_refused_is_never_written_later pass here, and must keep passing with the journal store (no replay may bring them back); - a_kill_keeps_every_line_once passes here and must keep passing (a replay after a kill must write no line twice). * acpmux: the event log through one journal, behind `store.journal` (R-DURABILITY S2) DESIGN.md 3.3b. With `store.journal` on, each event-log append is one record in `<home>/journal/` (cmux-journal: group commit, one sync per batch; a power loss takes back at most the last second, always a suffix), then the same line goes to the chat's ndjson as before. Readers, segments and file formats do not change. Default off. - store/journal_store.rs (JournalStore over LocalStore): - append: under one lock, the journal record (a memory copy, no I/O), then the log line; a line the log refuses gets a `cancel` record. Agent stream lines (`in` session/update, claude.*, codex.*) are Class::Stream, every other line Class::Ack. - checkpoint (`<home>/journal-projected.json`): every 30 s while the journal grew, past 64 MiB, at flush (shutdown) and before a retention commit. It syncs each log written since (LocalStore::sync_log, with no lock held while the disk works) and the sessions folder, writes the LSN (temp file, sync, rename, folder sync, every error checked), then retires the journal before it. A log close that failed its sync freezes the checkpoint until a restart (LocalStore::sync_failed). - recovery: the records after the checkpoint; for each chat, the log is cut right after its last readable line below the first journaled seq (later segments removed, the boundary one shortened, their kind-index copies dropped), and the journaled lines are written again in order. `forget` deletes a chat again; `cancel` keeps a refused line out. - delete: `forget` record, folder removal, then a bounded wait until the record is durable. - seal: the logs refuse writes and the journal syncs fully. - LocalStore: `append` is serialize + `append_line`; `sync_log`; a failed sync at a log's close is noted (`sync_failed`). - store::open opens the JournalStore when `store.journal` is on. Not yet (S2c): replies and events do not wait for durability; the daemon does not stop when the journal closes; the writer thread keeps the process's Utility I/O policy. * acpmux: journal store tests for the seal and a chat made again after its delete (red) - a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay: a chat is deleted (its forget record), made again under the same id (a transfer back: session.json and new lines), then a kill -9. On this code the replay deletes the folder again and its session.json is gone. - a_line_after_the_seal_is_refused_and_never_written_later: coverage of the seal (it passes here; the fix also closes a race the test cannot force). * acpmux: journal store: the seal under the appends' lock; a replay keeps a chat made again - seal takes the appends' lock: an append lands in both the journal and the logs, or in neither. Before, an append that raced the seal could leave a journal record whose line the sealed logs refused (its cancel then failed on the closed journal), and the next start would write that line. - append and delete check the logs' seal first, under the same lock, and do not touch the journal after it (no error per append at shutdown). - replay deletes a forgotten chat again only when no line follows its forget record: otherwise the chat was made again under the same id, and its folder holds that chat's own files. * cmux-journal: fifth-review regressions G9 (red) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4. * cmux-journal: name a skipped LSN floor in a FLOOR.<lsn> file (G9) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646); the regression tests are the commit before this one. When an open gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which a copy then hides from every later open), the only record of that floor was the header of the empty segment it makes; a damaged header lost it, and DAMAGE is cleared by `clear_damage` (torn records are not damage at all). Now such an open makes the empty file `FLOOR.<lsn>` and syncs its folder before any new segment, and every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine's names and this open's moves say their floors already: no file is made for them. * acpmux: journal store regressions from the S2 review (red) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824). Each test fails on this commit and passes after the fix that follows it: - 1a lines_written_with_the_switch_off_survive_turning_it_on_again - 1b a_kill_while_the_journal_syncs_keeps_the_lines_the_log_got - 1c a_line_flush_session_made_durable_survives_losing_its_record - 2 a_folder_put_back_after_a_delete_is_not_deleted_by_the_replay - 3 a_kill_in_a_delete_leaves_the_chat_whole_or_gone - 4 a_delete_does_not_wait_for_the_journal_sync - 5 a_failed_log_sync_freezes_the_checkpoint, a_checkpoint_that_keeps_failing_asks_the_daemon_to_stop - 6 one_chat_that_cannot_be_repaired_does_not_stop_the_open - 7 the_journal_storage_refuses_a_lost_home - the repair keeps a log's lines after a hole, also through a crash in it: an_os_crash_hole_is_repaired_and_the_lines_after_it_kept, a_crash_inside_a_repair_loses_no_line - nits: an_unreadable_checkpoint_fails_the_open, the_open_waits_for_the_previous_daemon_to_let_go_of_the_journal, a_crashed_checkpoint_temporary_is_removed_at_open, a_crash_right_after_a_checkpoint_reports_no_damage - the catch-up with the switch off: switching_off_after_a_crash_keeps_every_acknowledged_line - cmux-journal: check_append_says_what_append_would_do So they compile here, this commit adds the seams they use, with no behavior: `Store::stop_reason` (default None), `Journal::check_append` (always Ok), `FencedStorage` (never asks its fence), the open report and a test stop point after a repair's cut. * acpmux: the checkpoint freeze test fails a log sync, not a folder open (red) The freeze guards against a failed fsync (a later fsync of the same file may report success for pages the kernel dropped), not against a folder the store cannot open: a chmod 000 folder is an open error, which the fix retries instead of freezing. The test now fails the log sync itself through a test seam, `FAIL_NEXT_LOG_SYNC`. Red on the code before the fix, like the other review regressions (fleet d8e7037d, 1c22175e, 63604b26 at e030f4fe402: 17 of 17 new acpmux tests and the crate's check_append test fail). * acpmux: the journal store repairs logs, never rebuilds them (S2 review) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824); the regression tests are the two commits before this one. DESIGN.md 3.3c. - 1a/1b/1c: recovery no longer cuts a chat's log at its first journaled line and writes back only the journal's lines. It merges the log's lines and the journal's by seq: a line in both must be equal, a line only the log has stays, a line only the journal has goes in its place. Only bytes that do not read (a torn tail, a hole of zeros) are dropped. When the merge must change the log before its end, the lines only the log holds from there are journaled and made durable first, so a crash inside the repair loses no line. A log that moved on without the journal (a line in both that differs, lines out of order) is left as it is, and the journal's lines go to journal-unapplied/. - Append writes the log first, then the record, under one lock, after `Journal::check_append` (new in cmux-journal: the append's refusals, asked before any write). So a record always has its log line, and the `cancel` record is gone. - 2/3/4: the replay never deletes a folder and never makes a missing one again (a forget only drops the chat's earlier journaled lines). Delete appends its forget under the lock and removes the folder off it, with no wait for the record. - 5: a failed fsync in a checkpoint freezes it (a folder that does not open is retried); a frozen checkpoint returns an error, and the store asks the daemon to stop for a restart (`Store::stop_reason`, wired in daemon.rs), whose replay writes the logs again. Ten failed checkpoints in a row ask the same, so the journal past the checkpoint stays bounded. - 6: an error in one chat keeps its journaled lines in journal-unapplied/ and the open goes on. - 7: the journal's storage refuses writes under a lost home (`FencedStorage`, `config::refuse_lost_home`). - With store.journal off, a journal left by a run with it on is replayed, checkpointed and retired first (`catch_up`). - Nits: the checkpoint is never past the journal's durable end (no `Ahead` after a normal crash); damage is cleared after the next checkpoint; a kind-index copy goes before its segment is cut; an unreadable checkpoint fails the open; the open waits up to 5 s for the journal lock; crashed checkpoint temporaries are removed; `on_written` leaves cmux-journal (no user); README lists store.journal. * acpmux: split the journal store under its size budget (a move) journal_store.rs grew to 1085 lines and 71 functions with the review fix (budget 1000 and 60), and the fix grew store.rs and daemon.rs, which are over budget on the base. A pure move, no behavior change: - store/journal_store/{checkpoint,fenced,repair}.rs: the checkpoint and its thread, `FencedStorage`, the replay and the repair. - store/log_sync.rs: `LocalStore::sync_log`, `SyncFailed` and the test seam `FAIL_NEXT_LOG_SYNC`, out of store.rs (2139 lines on the base, 2113 now). - daemon/stop_causes.rs: `AbortOnDrop`, `next_signal` and `store_asks_to_stop`, out of daemon.rs (2052 on the base, 2046 now). check-no-godfiles.sh --base 51cfdee876b on every changed file: no file over budget grew, and no new file is over budget. * acpmux: the unrepairable-chat test waits for line 1 to be durable before its checkpoint At 03fccdef69c (fleet 32df3c72 on aws-m4pro-9, 144d4ccc on Linux) the test kept chat-b's lines [1, 2, 3] aside, not [2, 3]: it flushed right after appending line 1, and the checkpoint is now never past the journal's durable end (the review's Ahead nit), so line 1 was still past it. The fix's behavior is right; the test now waits until line 1 is durable, so its checkpoint covers it. On the code before the fix the test still fails (the open stops at chat-b's link). * acpmux: replies and events wait for durability, tests (red, S2c) DESIGN.md 3.3d. With the event journal on and every journal sync taking 400 ms (`SimStorage::set_sync_delay`): - an_acked_change_is_answered_once_its_record_is_durable: a rename's reply must come no sooner than one sync after its `renamed` record. Red here: the reply goes at once. - agent_output_is_not_held_behind_another_chats_acked_change: agent output to a connection attached to its chat goes at once while another chat's acknowledged change waits for its sync. Green here (nothing waits); it guards the fix against holding agent output behind changes the connection does not hear. * acpmux: replies and events wait until what they show is durable (S2c) DESIGN.md 3.3d; the tests are the commit before this one. With `store.journal` on, each connection gets an outbound gate (server/durable_gate.rs) between it and its transport: - Every line a connection queues raises the gate's `needs` to the last acknowledged LSN (`Store::durability().acked`, set in the journal store's append before the hub publishes anything of the record), except the lines of a `Stream` record's event (agent output, DESIGN C4): the fan-out delivers those in a thread-local stream scope. - The gate's task takes every queued line at once, reads `needs`, waits (async) until the journal's durable LSN reaches it, then sends the batch in order: one wait per batch, as the journal's group commit. - The durable LSN is a `tokio::sync::watch` fed by `Journal::on_durable`. The listener owns a guard: a writer that ends while the store is not being dropped (a panic on its thread) asks the daemon to stop (`Store::stop_reason`, decision 9.8), and the watch closes, so each gate drops what it holds and ends, which closes its connection. - Without the journal, `Store::durability()` is None: no gate, and `Conn::raise_gate` does nothing. Moves, so server/mod.rs and store.rs stay under their size on the base: the outbound queue helpers to server/outbound.rs, `open_with_journal` to the journal store, and `streamed` to store/durability.rs. * acpmux: the journal lock wait's deadline without an unchecked Instant addition `Instant + Duration` can panic on overflow (clippy::arithmetic_side_effects, tracked by the panic ratchet). The wait now has no deadline only in the case that cannot happen (an Instant 5 s ahead overflowing), where it stops waiting at once. * acpmux: a failed log sync stops the real daemon for a restart, test (red) Lead rule 2026-11-04 (3): a daemon-level test of the store's "stop for a restart" before S2c merges, red first. With store.journal on and every log sync failing (a debug build's ACPMUX_TEST_FAIL_LOG_SYNC; the checkpoint every 200 ms through ACPMUX_TEST_JOURNAL_CHECKPOINT_MS), a rename of a chat on disk makes its log dirty, the checkpoint's sync fails and freezes it, and the store asks the daemon to stop. The test asserts: the WebSocket client gets the restart close (1012), the socket client's pending `_acpmux/wait` ends with the connection (no hang), and the daemon exits with status 74. Red here: the daemon stops and sends 1012, but exits 0 (no exit status says that it stopped to be repaired). This commit adds only the two debug-build seams (release builds have none) and the test. * acpmux: a daemon its store stopped for a restart exits with status 74 The test is the commit before this one. `daemon::run` returned Ok and the process exited 0 for every stop, so a supervisor or a person could not tell a stop to be repaired from a normal one; with systemd `Restart=on-failure` it would not even start again. Now the store's stop (`store_asks_to_stop`: a failed log sync froze the event journal's checkpoint, or the journal's writer ended) marks itself, sets the restart handoff (WebSocket clients get 1012, as before), and the process exits with `daemon::EXIT_REPAIR` (74, EX_IOERR). The units prevent only 75, so launchd KeepAlive and systemd Restart=always start it again; the app does not read the daemon's exit status. README: store.journal names the status. * acpmux: the restart test's request in flight goes through the acpmux client Lead rule (3): the client sees a typed error, not a hang. The socket client is now the acpmux client library (acpmux::client::Client, as the CLI uses it), and the test asserts what the CLI makes of the request that the stop ends: `daemon_closed`, retryable (cli::errors::classify). The first red (fleet 41f2d037, hq run 41804640229, at 80c5fb2b7f8) failed only on the exit status (0, wanted 74), with 1012 and the end of the connection passing. * acpmux: the daemon's disk tier with the journal on, tests (red) Lead rule 2026-11-04 (2): a reply the person waits for never waits on a throttled disk sync. The fleet A/B (STATUS 11554) found that at IOPOL_UTILITY an acknowledged journal append waits about 16 ms p50 under another process's disk stress (0.7 to 1.1 ms at the default tier). - the_daemon_with_the_journal_keeps_the_default_disk_tier: with store.journal on, the daemon process keeps the default tier; without it, utility as before (each case in a child of the test binary). - a_background_thread_yields_its_disk_work: a dedicated background thread (the checkpointer) works at utility; a thread beside it does not. The API they use is stubbed here (`qos::enter_daemon` ignores the journal; `qos::background_thread` does nothing), so both fail on macOS. * acpmux: the restart test borrows no &str twice (clippy needless_borrows_for_generic_args) Found by fleet 7f8afd9d (hq run: the S2c server step at 62b4f68ef8b); the step's other clippy red is the base red in link_push_bench.rs. * acpmux: with the journal on, the daemon keeps the default disk tier (S2c) The tests are 994f74119fb. Lead rule 2026-11-04: a reply the person waits for never waits on a throttled disk sync. Measured on two owned Macs (STATUS 11554): one acknowledged append to durable at IOPOL_UTILITY under another app's disk stress took 16.3-16.7 ms p50 and 37-45 ms p95, against 0.7-1.1 ms p50 and 1.5-1.8 ms p95 at the default tier. - qos::enter_daemon(journal): with store.journal on, the daemon process keeps the default disk tier; without it, utility as before. The switch is read alone from config.json (config::store_journal_enabled), before any thread starts, with no harness discovery. - qos::background_thread: the journal store's checkpointer lowers its own disk I/O to utility for its whole life (a thread that does nothing else: a thread-scope policy drops it out of QoS for good). * acpmux: an agent chunk is a Stream record, test (red) The hub's tap logs an ACP session/update under its sessionUpdate kind (hub/tap.rs: `kind = su.to_owned()`), so a record's kind is `agent_message_chunk`, `tool_call`, ... and never `session/update`; the message's method is. The journal store's `streamed` checked the kind, so every agent chunk was journaled as Ack: a sync per batch at the token rate, and in S2c each token's event waited for that sync (the S2c I/O A/B: STRESS token to client 2.2 ms -> 20.3 ms p50 with the journal on). Red here: the chunk reads back as Ack. * acpmux: an agent chunk is a Stream record (the record's message names session/update) The test is the commit before this one. `streamed` now reads the logged message's method: the hub's tap logs a session/update under its sessionUpdate kind (hub/tap.rs), so the kind never said it. Agent output is again Stream (synced with the next acknowledged batch or a second later), as DESIGN.md 3.3b says. * acpmux: review 2 of #34901, tests (red): a crash model that follows real log syncs The logs' OS crash model now follows the syncs that ran (store/sync_model.rs, tests only): a segment keeps its length at its last sync that counts; a failed sync loses what was written since the one before, and a later sync does not bring it back; a cut ends a loss it removes. Every log segment sync goes through log_sync::sync_segment (same calls as before), where the model and FAIL_NEXT_LOG_SYNC hook in. Red tests, one per finding: - 1a an_os_crash_after_a_restart_from_a_kill_keeps_every_acked_line: the restart's first checkpoint retires records whose log lines no sync made durable; expected [1..5], the head gives [1]. - 1b an_os_crash_after_a_restart_from_a_failed_sync_keeps_every_acked_line: the same after a frozen checkpoint and a restart; expected [1..5]. - 2 a_hole_that_starts_inside_a_line_is_repaired (the reviewer's test): a torn line is read as line 6 of a log that moved on; expected [1..9], the head gives [1, 2, 3, 4, 5, 9]. - 3 a_failed_sync_in_flush_session_freezes_the_checkpoint: a failed sync in flush_session does not freeze the checkpoint. * acpmux: review 2 of #34901: a restart writes the replayed lines again; a torn line is not a line; flush_session freezes Finding 1 (high): the repair now writes every merged line again from the repair start, even when the log already has them all. A killed daemon's log writes may be in the page cache only, and after a failed sync the kernel may have dropped pages that still read back and that a later sync reports as written; the first checkpoint after the open then syncs bytes this process wrote before it retires their records. The lines only the log has from the repair start are journaled and made durable first, as before a cut. The work is bounded by the checkpoint interval. Finding 2: a piece of the log is a line only when it parses as one JSON value (piece_seq). A hole that starts inside a line keeps that line's {"seq":N, head and runs into a later line's tail; it was read as a line of a log that moved on, and the chat's journaled lines went aside. Now its bytes go and the lines are written again. Finding 3: a failed sync in flush_session freezes the checkpoint (its file sync is a SyncFailed now), as a failed sync in a checkpoint or at a log's close does. Nits: cut_at syncs events/ after its removals (and the kind-index folder after its copies go), so a removed segment cannot come back beside the lines written again; the README says that with the switch on an event line over 64 MiB is refused and its log does not get it. Tests: the four red tests of d2c63860ece. * acpmux: chat metadata through the event journal, tests (red, R-DURABILITY S3a) DESIGN.md 3.3e. Red on S2c: - last_seq_never_passes_the_log_after_an_os_crash (E13): session.json is synced at each save while the agent's lines wait for the journal's next sync, so after an OS crash the metadata names lines the log lost. - a_chat_made_after_the_checkpoint_survives_an_os_crash_that_loses_its_folder: nothing syncs the sessions folder when a chat's folder is made, and the replay never makes a folder again, so an acknowledged new chat is lost. - a_metadata_change_is_written_to_session_json_by_the_checkpoint: today each change writes and syncs session.json on the request path. Guards (green before and after): a chat made before the checkpoint and deleted is not made again; the replay never writes an older image over session.json. * acpmux: chat metadata through the event journal (R-DURABILITY S3a) DESIGN.md 3.3e, behind `store.journal` (default off). A metadata change is a `meta` record (kind 4: chat id, a `created` flag, the whole image) instead of a session.json write with two syncs on the request path; the reply waits for the record through S2c's gate, one group commit for a burst. - Overlay: each chat's newest image until a checkpoint writes session.json; load, load_startup and list read it first (list under the projection lock, so a checkpoint cannot drop an entry between its two reads). - Checkpoint: waits on its own thread until every record through its LSN is durable, then writes each image it read (session.json carries the image's LSN as `journalLsn`), then the logs and the sessions folder, as before. An image so names only lines the journal holds (E13). A chat deleted or made again meanwhile is passed (its overlay entry is gone or new). - Replay: per chat after its last `forget`, the newest image, written when newer than the file's `journalLsn`. A missing folder is made again only for a chat a replayed `created` record made; a chat made before the checkpoint whose folder is gone keeps its image in journal-unapplied/. - flush_session also waits until the chat's records are durable. - Delete drops the overlay entry with its `forget`, and removes the folder under the folders lock a checkpoint's write also takes. Tests: the 5 of 0505e9bee04 (3 red there). * acpmux: S3a, tests (red): folders handed out of the store Code outside the store reads and moves chat folders directly (transfer_in puts the live folder aside and installs a transferred chat; session_dir hands the folder to the transfer, the session files view and others). With the metadata in the overlay until a checkpoint: - a_folder_replaced_outside_the_store_keeps_its_own_metadata: a read, the checkpoint and the replay must not put the old folder's image in the new one. - session_dir_hands_out_a_folder_whose_metadata_is_written. - flush_session_writes_the_chats_metadata (the trait says log and metadata). - the_checkpoint_never_makes_a_folder_removed_outside_the_store_again. Red on 71bc09a1e63 (S3a's first implementation). * acpmux: S3a red tests: a replay after a folder swap; flush_session's chat has a log (as its callers' chats do) * acpmux: S3a: each metadata image is bound to its chat folder; session_dir and flush_session write it first Code outside the store reads and moves chat folders directly (a transfer puts the live folder aside and installs a transferred chat under the same id). So a `meta` record now carries its folder's identity (file id and birth time): - reads use an overlay image only while the folder at the path is its folder; - the checkpoint writes an image only into its own folder, never makes a missing one, and keeps the image of a removed or replaced folder in journal-unapplied/; - the replay writes an image only into its own folder (or a folder a replayed `created` record made), else keeps it aside. `session_dir` (the transfer, the session files view) and `flush_session` (the trait: "log and metadata to stable storage") write the chat's image first, once its record is durable. The list reads the overlay before the files (a checkpoint writes a file before it drops the entry), so it needs no lock. Tests: the 5 of 0280ac37f5e and 0e617489731. * acpmux: the qos journal tests module is macOS-only (its Linux build held only an unused import: clippy red, review of #35400 F0) * acpmux: review of #35400 F1 and F2, tests (red) - a_read_that_shows_a_line_covers_its_record (F1): a read sees a log line between its log write and its record (the appends' lock held there), and the reply built from it goes out with no wait for that record. - a_journal_that_syncs_nothing_asks_the_daemon_to_stop (F2): with a journal that makes nothing durable, every reply waits without a limit and nothing asks the daemon to stop. Test seams only: PAUSE_AFTER_LOG_WRITE, TEST_STALL_MS. * acpmux: review of #35400 F1, the streaming read, test (red): session/load's replay sends a line before its record * acpmux: the test seam PAUSE_AFTER_LOG_WRITE is reachable from the server tests * acpmux: review of #35400: a read covers the records of the lines it shows; a stalled journal stops the daemon F1: the append writes a line to the log, then appends its record and sets the acknowledged LSN, all under the appends' lock; reads take no lock. A read could show a line before its record existed, and the reply built from it went out at once. Now every journal-store read of the logs ends with a fence on the appends' lock (`Inner::fenced`): a line the read saw has its record appended and its LSN acknowledged when the read returns, so the reply's gate covers it. The one read that sends as it reads (`Hub::scan_events`, session/load's replay) stops at the chat's last_seq read first. F2: a journal writer that keeps failing (it retries without end) held every reply on every connection with no limit. A connection's gate now waits at most 10 s (`STALL_LIMIT`); past it, `Durability::stalled` asks the daemon to stop for a restart, as a failed log sync does. Tests: the 3 of 3916f84dd68 and c553a19fd8b. * threading-model: the acpmux event journal's threads and bounds (writer, checkpointer, per-connection gate; review of #35400) * acpmux: the stall test's journal syncs take 3 s, not an hour (a sync under way kept its delay and hung the test's end) * acpmux: S3a review (#35483) red tests: switch-off metadata and deletes, a late settle, a failed session.json folder sync, a purge after the checkpoint F1: metadata written with the switch off survives turning it on again; a chat deleted with the switch off after a failed catch-up is not made again. F2: a late settle never writes an older image over a checkpointed one (the reviewer's test and hold). F3: a failed folder sync after a metadata write freezes the checkpoint. F4: a chat made after the checkpoint and deleted is not made again after a kill. Test seams only; the code is the S3a head merged with feat. * acpmux: S3a review (#35483) red tests: a new chat is acknowledged only once its folder is durable; session_path never waits The root of F4 and of F1's delete case: the replay makes a chat's folder again, so a chat removed by any path comes back. The fix will never make a folder again, which needs an acknowledged chat's folder to survive an OS crash: the test models what each folder's last sync kept (sync_model) and asserts the gate's durability passes a new chat only after its sessions entry is synced. P3 1 of the review: a caller that needs only the path never waits for the journal (Store::session_path, a stub with session_dir's behavior here). * acpmux: S3a review (#35483) fixes: the replay never makes a folder again; switch-off changes are kept; no late or unsynced metadata write F1: each meta record carries the stamp (file id, modification time, length) of the session.json its save saw. The replay writes an image over a file the journal did not write only when one of the chat's records saw that file; otherwise the file moved on without the journal (a run with store.journal off whose catch-up did not replay, or an older build) and the image goes to journal-unapplied/, as S2 does for a log. F1 (deletes) and F4 at the root: the replay never makes a chat's folder again. A save that made a folder notes its record's LSN; the new acpmux-journal-folders thread (the daemon's disk tier) syncs sessions/ at once, and the durable LSN the S2c gate waits for stays below that record until then (journal_store/new_folders.rs). So an acknowledged chat's folder survives an OS crash, and a folder that is gone was removed: a purge whose forget a kill took back, a run with the switch off, the transfer recovery. The created flag and the replay's folder remake are gone. F2: write_through does not write a file that holds the same or a newer journalLsn. F3: session.json is written with its own syncs (file, then folder); a failed sync is SyncFailed and freezes the checkpoint; the replay and the kept-aside images return their errors. P3 1: Store::session_path gives the path with no lock, wait or write; an agent's start, the fixed-id check, the temporary-folder sweep, the host probe and the transfer recovery's root use it. The replaced test (a chat made after the checkpoint survives an OS crash that loses its folder) asserted a remake that cannot be right any more; the new test asserts the guarantee that makes the loss impossible, then the crash. Red (7 of 17 journal_meta_tests, each for its reason; the other 10 pass): macOS 5bc434a9 and 624d5df4, Linux a2207ff0 and 638a44ef. * acpmux: S3a delta review (N1) red tests: a change to a chat whose session.json another journal wrote reaches it A folder moved in from another home (build adoption), checkpointed and replayed, and a journal made again from nothing: the other journal's journalLsn must not hide a change made here. The first test is the reviewer's (fleet 4d0d98bc and 071ad7c4 at e944a7c21de). * acpmux: S3a delta review (N1): journalLsn is bound to its journal's id; a file another journal wrote follows the stamp rule session.json now carries journalId (16 hex digits, a string) beside journalLsn. The checkpoint's "held" check (F2) and the replay's rule compare an LSN only for a file with this journal's id: a file from another journal (a chat folder build adoption moved in from another home, or a home whose journal was made again) counts as one this journal did not write, so the stamp rule decides, as for a run with the switch off. cmux-journal: Journal::journal_id (the id already in every segment header; kept across opens and retirements), with a unit test. Red at 7c8cb957745: fleet c29b659d (Linux) and 47570494 (macOS) running; the reviewer's red at e944a7c21de: 4d0d98bc (macOS) and 071ad7c4 (Linux).
lawrencecchen
added a commit
that referenced
this pull request
Nov 4, 2026
…a hole's last zero splits, an idle checkpointer sleeps (#35562) * cmux-journal: one append-only journal per daemon, with group commit and ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner). * cmux-journal: type a test range; drop an unread segment field (fleet clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71e3f9 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red. * cmux-journal: a failed F_FULLFSYNC is an error, not a plain fsync; no segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee. * cmux-journal: the past-the-end test reads back the storage it wrote to Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c87b8d: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5c53e. * cmux-journal: a sync a signal interrupts is made again (macOS) fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r). * cmux-journal: crash sim of write-back errors and crash in recovery; S1 review regressions (red) The S1 review (FIX at 89293fb15f39) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open * cmux-journal: the F2 regression damages the header of the last of several segments (red) The reviewer's F2 scenario has two segments: S1 holds 1..10, S2 (the last) holds 11..20, and S2's header is damaged. The test now rotates segments (300-byte segments, 20 acked records) and damages the last one's header, so the report must say lost_through 20 and the next append must take 21. On the code before the fix the moved segment's LSNs read as 0, so the next append reuses one. * cmux-journal: the crash sim forces the F4 order (failed sync, kill, OS crash) (red) On the code before the S1 fix, the crash-sim step was green (fleet 184129d8 at 263da86f223): 6 of 7 runs passed, and the mixed run failed only seed 140 in the full run and passed when it ran alone. The random faults reached the order the review found (F4) in 1 of 800 runs. So these tests were no red/green proof. The order: the sync of a write that spans pages fails and the disk gets none of them (Linux keeps them clean in the cache, so a later sync reports success); the process is killed in the writer's backoff; the next open reads the failed segment from the cache, replays its records and goes on; an OS crash or a power loss then takes those pages, and records an open already replayed are gone. - sim.rs: `fail_next_sync(pages, kind)` fails the next sync that has at least that many dirty pages, and the disk gets none of them. `crash_after_failure(n)` makes the n-th operation after the next injected failure the crash point. - crash_sim_tests.rs: a new run forces the order on every seed (a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record): epoch 0 aims the fault, the kill is a process crash at the writer's retry, the open after it does not crash, epoch 1 ends in an OS crash, and a third open checks that the records the second open replayed are all there. In the random runs, one epoch in four aims the fault, with the crash 1 to 3 operations after the failed sync and a crash of any kind after it. - A sim test checks the aimed fault itself (the_sim_fails_the_aimed_sync_and_crashes_in_the_backoff). Expected on this commit (old journal code): the forced run fails every seed; the mixed run fails most runs. The fix commit follows. * cmux-journal: fix the S1 review findings F1-F7 (markers, recovery copy, LSN floor, writer end) Review FIX at 89293fb15f39 (PR #33919 comment 6610468202). The regression tests are the two commits before this one. - Sync markers (format.rs): an entry with flags 1 and no payload, whose LSN field says "every record of this segment before me is synced". The writer appends one after each successful sync that covered records (after the durable listeners, so no reply waits for it), and syncs it fully at a seal and at a rotation. - F4, recovery (step 4 and 6): the last segment is trusted only up to its last marker. Its records after the marker may sit in pages a failed sync dropped (a later sync on another descriptor reports success without them), so recovery writes them again into a new segment: header, records and a marker, fully synced under `<name>.seg.new`, renamed into place, folder synced. The journal always goes on in a segment this open made, never in one an earlier process wrote. A restart in the writer's backoff after a failed sync is then safe. - F1, quarantine: every LSN a quarantined segment holds counts in the LSN floor at every open, and a quarantine that no new segment followed (a crash in the middle of an earlier recovery) is reported as damage again. The highest LSN of every moved segment is read before the first rename. - F2, highest LSN: a segment whose header is damaged is read from its first record (each record's checksum covers its LSN), then the bytes after the first bad entry are scanned for records and markers that still check out. The same scan runs after a cut in the last segment: a marker there is media damage (reported), records there are a torn write (`torn_lsns`); their LSNs are never used again. - F3, writer end: a guard closes the journal when the writer thread ends, also by a panic: appends fail with Closed, waits end with WriterEnded, the listeners are dropped (so a channel a held reply waits on closes), and stats.failing says why. The crate does not abort the process; the daemon owns that policy. - F6: recovery closes each segment after its header is read and opens one again only to read it. (Retirement of compacted segments is an S2 API, before the first consumer.) - F5: recovery syncs the parent folder after it makes the journal folder. - F7: format.rs and lib.rs now say what the code does: a failed write is cut back and retried in the same segment; the cut removes only bytes never synced and never acknowledged; nothing shortens a segment past a synced record. Docs: journal.rs says when open's replayed records are durable (on Ok) and what a stopped writer does. * cmux-journal: split the crash sim's machine model into sim/machine.rs (review G5) sim.rs had 924 lines and 63 functions, over the 60-function limit for a new Rust file (the god-file check warned "not in baseline: split it"). The machine model (each file's page cache and disk, the ordered log of names, and what an OS crash or a power loss leaves) moves to sim/machine.rs (297 lines, 19 functions); the process view and the Storage implementation stay in sim.rs (639 lines, 30 functions). No behavior change: the items move as they are, made visible to the sim module. * cmux-journal: re-review regressions G1-G4 and page eviction in the crash sim (red) Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993). Tests and sim only; the fix follows. - sim: the pages a failed sync dropped stay in the cache until an eviction; `evict_dropped_pages_after_read(true)` makes each read evict the dropped pages it read, so a second read of them sees the disk's bytes. - G1 recovery_copies_the_records_it_checked_when_the_cache_drops_them: a sync fails and drops record 2's pages, a kill in the backoff, the next open reads record 2 from the cache (replayed) and the pages are evicted after that read; then an OS crash. Record 2 and the record 3 acked after the restart must survive. On this code recovery's second, unchecked read stages zeros under a vouching marker, and the last open keeps only record 1 and reports damage. - G2: the F1 regression asserts that every open after the damage reports the loss (lost_through 40). On this code a crash at the open's last step (the folder sync after the new segment's rename) loses the report for good. - G3 a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system): a journal in a symlinked state folder, and a journal folder that is a symlink. On this code the parent folder's sync opens with O_NOFOLLOW (ELOOP). - G4 an_open_after_a_finished_quarantine_reads_no_quarantined_segment: on this code every open reads every quarantined segment in full. * cmux-journal: fix the re-review findings G1-G4 and G6 Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993); the regression tests are the commit before this one (red: fleet de35f6d0). - G1: recovery writes again exactly the bytes its checked read returned. The last segment's entries after its last marker are kept from the checked read (header and payload) and staged from memory; the second, unchecked read of the file is gone. A page that a failed sync dropped can be evicted between two reads, and the second read then returned the disk's zeros, which the copy kept under a vouching marker. - G2: the damage an open finds is written to `DAMAGE` (whole, fully synced, renamed into place, folder synced) before the new segment that hides it from later opens. Every open reports the recorded damage first, until the daemon calls the new `Journal::clear_damage`, which moves the record into `quarantine/` (kept, never deleted). - G3: a folder sync follows a symlink in its path (no O_NOFOLLOW): a state folder, or the journal folder itself, may be a symlink. The sync is for the folder that holds the entries. - G4: the quarantined segments are read for their LSNs only while no live segment is numbered above them (an unfinished quarantine); once one is, its first LSN already carries their floor. - G6: `checked_div` in the scan of the bytes after a cut (the crate's one tracked arithmetic hit). - G5 (the sim split) is the commit before the regressions. * cmux-journal: third-review regressions G7 (A, B) and N1 (red) Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018). Tests only. - G7 A a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn: after a finished quarantine, one byte of the empty new segment's header changes. On this code the next open skips the quarantine scan (the listing counts the damaged segment) and the next append reuses a quarantined LSN. - G7 B a_damaged_newest_segment_after_a_quarantine_reports_its_records: the newest segment holds acked 41..43, and its header and its first record are damaged. On this code the moved segment is scanned with a base far below 41, so the loss is not reported through 43 and the next append is not 44. - N1: the F1 regression also asserts that an open reports each loss once (no reported LSN range lies inside another). On this code a crash after the quarantine's renames makes the next open report the same loss twice. * cmux-journal: fix the third-review findings G7, N1, N2; split recovery.rs Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018); the regression tests are the commit before this one. - G7: the quarantine scan is skipped only when a segment whose header checked out is numbered above every quarantined segment (the folder listing also counted a newest segment with a damaged header, which carries no floor). The scans start from the running floor: `quarantined_floor` and `quarantine` pass the highest LSN counted so far as the base of the next file's scan, so a quarantined file with no readable header and no readable first record is scanned for the LSNs right above what is counted. - The "unfinished quarantine" report is made only when the quarantine holds LSNs nothing else counted (an old quarantine behind a segment that a crash tore at its creation holds none, and a crash is not damage). Its reason now also names a damaged newest header. - N1: a loss whose LSN range lies inside a recorded one is the same loss found again after a crash: it is neither recorded nor reported twice. - N2: recovery.rs's docs name `DAMAGE`; two doc lines over 100 columns are wrapped (lib.rs, sim.rs). - recovery.rs was at 987 of its 1000 lines: the damage record moves to recovery/damage.rs, the entry reader and the LSN scans to recovery/scan.rs (moved as they are). * cmux-journal: report each loss once, also when one open finds it twice (N1) The red run of the N1 regression (fleet 27d011ca, op 64 of the F1 test) shows the case the previous commit did not cover: a crash after the quarantine's renames and before DAMAGE is written; the next open finds both the damaged synced record of the last segment (7..14) and the unfinished quarantine (7..40), and reported both. Now every loss inside another (recorded or found by this open) is dropped, the larger range kept; DAMAGE is rewritten only when its set of losses changed. * cmux-journal: fourth-review regressions G8, N3, N4 (red) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report. * cmux-journal: the quarantine's floor in its file names; DAMAGE before the moves (G8, N3, N4) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format. * cmux-journal: sort the quarantine list with sort_by_key (clippy unnecessary_sort_by) Fleet d9f22bed at b589044b852: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`). * cmux-journal: Journal::on_written, a listener per written batch (for S2's projector) R-DURABILITY S2 (acpmux event logs through the journal) projects each chat's log line into its ndjson file once the journal has WRITTEN the batch that holds it, so readers see stream lines at once. The listener runs on the writer thread after the write and before the sync, once per batch that moved the written LSN: a batch written again into a new segment after a failed sync is not heard twice. When the writer stops, these listeners are dropped too. Test: heard while the batch's sync still runs, once. Port of 145faa6faa8 (branch rewrite/r-durability-s2-acpmux-logs) onto the S1 review fixes. * cmux-journal: Journal::retire_through, the one delete (segments a compaction point covers) R-DURABILITY S2: a daemon's projector compacts the journal into its own files (acpmux: the chat ndjson logs, then a synced checkpoint). Without a delete the segment count only grows (review finding F6). `retire_through(compacted)` deletes the segments that hold no record after the caller's compaction point: a segment goes when the next segment's "previous last LSN" is at or before it, so the newest segment never goes. The point is capped at the durable LSN. The deletes and a folder sync run on the caller's thread (the projector, never a request path). A delete that fails leaves its segment live for a later call. Why a crash during a retirement is safe: the caller reopens from a point at or after `compacted`, and recovery reads nothing from a segment the point covers (it skips it on its next segment's header), so any subset of the deletes may reach the disk. A point older than the retired segments is reported as Damage::Missing. Quarantined segments are never deleted. - Storage::remove_file (FsStorage: std::fs::remove_file; SimStorage: one more change in the ordered log of names, durable at the folder sync). - The live segment list: recovery's, then each segment the writer makes. - JournalStats::segments_retired. - Tests: only covered segments go, and a second call deletes none; an OS crash after it keeps every record after the point and an older point is reported; a crash at every operation of a retirement (process, OS crash, power loss) keeps every record after the point and never reuses an LSN; the newest segment stays; a point in the middle of what this writer wrote. This adds a delete of journal data, so its PR is risk:high. * acpmux: move MemoryStore and the store settings into their own files store.rs (2139 lines, 120 fns) and config.rs (1909 lines, 97 fns) are over the god-file limit, and the journal store (R-DURABILITY S2) must not grow them. MemoryStore moves to store/memory.rs and StoreMode, StoreConfig and its defaults to config/store_config.rs; both stay exported where they were (crate::store::MemoryStore, crate::config::{StoreConfig, StoreMode}). No behavior change: the code moves as it is. * acpmux: the event log's journal switch and its crash tests (red) R-DURABILITY S2 (DESIGN.md 3.3b). Today no event-log append is synced: a line a client was told about is on the disk only after the next flush (shutdown), so an OS crash or a power loss takes back every line since (E12). This commit adds only the switch, a seam and the tests; the journal store follows. - `store.journal` (StoreConfig::journal, default false). - `Store::wait_durable(timeout)`: tests and tools wait until every appended record is durable. The default returns at once (the Local and Memory stores never sync an append), which is today's behavior. - `store::open_with_journal` (tests): the store with the journal on a given `cmux_journal::Storage`. Here it ignores the journal and opens LocalStore. - acpmux depends on cmux-journal (dev: with the `sim` feature). - store/journal_store_tests.rs, red on this commit: - a_line_the_store_says_is_durable_survives_an_os_crash (lines 2..20 are lost: nothing synced them); - a_deleted_chat_stays_deleted_after_a_crash and a_line_the_log_refused_is_never_written_later pass here, and must keep passing with the journal store (no replay may bring them back); - a_kill_keeps_every_line_once passes here and must keep passing (a replay after a kill must write no line twice). * acpmux: the event log through one journal, behind `store.journal` (R-DURABILITY S2) DESIGN.md 3.3b. With `store.journal` on, each event-log append is one record in `<home>/journal/` (cmux-journal: group commit, one sync per batch; a power loss takes back at most the last second, always a suffix), then the same line goes to the chat's ndjson as before. Readers, segments and file formats do not change. Default off. - store/journal_store.rs (JournalStore over LocalStore): - append: under one lock, the journal record (a memory copy, no I/O), then the log line; a line the log refuses gets a `cancel` record. Agent stream lines (`in` session/update, claude.*, codex.*) are Class::Stream, every other line Class::Ack. - checkpoint (`<home>/journal-projected.json`): every 30 s while the journal grew, past 64 MiB, at flush (shutdown) and before a retention commit. It syncs each log written since (LocalStore::sync_log, with no lock held while the disk works) and the sessions folder, writes the LSN (temp file, sync, rename, folder sync, every error checked), then retires the journal before it. A log close that failed its sync freezes the checkpoint until a restart (LocalStore::sync_failed). - recovery: the records after the checkpoint; for each chat, the log is cut right after its last readable line below the first journaled seq (later segments removed, the boundary one shortened, their kind-index copies dropped), and the journaled lines are written again in order. `forget` deletes a chat again; `cancel` keeps a refused line out. - delete: `forget` record, folder removal, then a bounded wait until the record is durable. - seal: the logs refuse writes and the journal syncs fully. - LocalStore: `append` is serialize + `append_line`; `sync_log`; a failed sync at a log's close is noted (`sync_failed`). - store::open opens the JournalStore when `store.journal` is on. Not yet (S2c): replies and events do not wait for durability; the daemon does not stop when the journal closes; the writer thread keeps the process's Utility I/O policy. * acpmux: journal store tests for the seal and a chat made again after its delete (red) - a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay: a chat is deleted (its forget record), made again under the same id (a transfer back: session.json and new lines), then a kill -9. On this code the replay deletes the folder again and its session.json is gone. - a_line_after_the_seal_is_refused_and_never_written_later: coverage of the seal (it passes here; the fix also closes a race the test cannot force). * acpmux: journal store: the seal under the appends' lock; a replay keeps a chat made again - seal takes the appends' lock: an append lands in both the journal and the logs, or in neither. Before, an append that raced the seal could leave a journal record whose line the sealed logs refused (its cancel then failed on the closed journal), and the next start would write that line. - append and delete check the logs' seal first, under the same lock, and do not touch the journal after it (no error per append at shutdown). - replay deletes a forgotten chat again only when no line follows its forget record: otherwise the chat was made again under the same id, and its folder holds that chat's own files. * cmux-journal: fifth-review regressions G9 (red) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4. * cmux-journal: name a skipped LSN floor in a FLOOR.<lsn> file (G9) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646); the regression tests are the commit before this one. When an open gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which a copy then hides from every later open), the only record of that floor was the header of the empty segment it makes; a damaged header lost it, and DAMAGE is cleared by `clear_damage` (torn records are not damage at all). Now such an open makes the empty file `FLOOR.<lsn>` and syncs its folder before any new segment, and every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine's names and this open's moves say their floors already: no file is made for them. * acpmux: journal store regressions from the S2 review (red) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824). Each test fails on this commit and passes after the fix that follows it: - 1a lines_written_with_the_switch_off_survive_turning_it_on_again - 1b a_kill_while_the_journal_syncs_keeps_the_lines_the_log_got - 1c a_line_flush_session_made_durable_survives_losing_its_record - 2 a_folder_put_back_after_a_delete_is_not_deleted_by_the_replay - 3 a_kill_in_a_delete_leaves_the_chat_whole_or_gone - 4 a_delete_does_not_wait_for_the_journal_sync - 5 a_failed_log_sync_freezes_the_checkpoint, a_checkpoint_that_keeps_failing_asks_the_daemon_to_stop - 6 one_chat_that_cannot_be_repaired_does_not_stop_the_open - 7 the_journal_storage_refuses_a_lost_home - the repair keeps a log's lines after a hole, also through a crash in it: an_os_crash_hole_is_repaired_and_the_lines_after_it_kept, a_crash_inside_a_repair_loses_no_line - nits: an_unreadable_checkpoint_fails_the_open, the_open_waits_for_the_previous_daemon_to_let_go_of_the_journal, a_crashed_checkpoint_temporary_is_removed_at_open, a_crash_right_after_a_checkpoint_reports_no_damage - the catch-up with the switch off: switching_off_after_a_crash_keeps_every_acknowledged_line - cmux-journal: check_append_says_what_append_would_do So they compile here, this commit adds the seams they use, with no behavior: `Store::stop_reason` (default None), `Journal::check_append` (always Ok), `FencedStorage` (never asks its fence), the open report and a test stop point after a repair's cut. * acpmux: the checkpoint freeze test fails a log sync, not a folder open (red) The freeze guards against a failed fsync (a later fsync of the same file may report success for pages the kernel dropped), not against a folder the store cannot open: a chmod 000 folder is an open error, which the fix retries instead of freezing. The test now fails the log sync itself through a test seam, `FAIL_NEXT_LOG_SYNC`. Red on the code before the fix, like the other review regressions (fleet d8e7037d, 1c22175e, 63604b26 at e030f4fe402: 17 of 17 new acpmux tests and the crate's check_append test fail). * acpmux: the journal store repairs logs, never rebuilds them (S2 review) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824); the regression tests are the two commits before this one. DESIGN.md 3.3c. - 1a/1b/1c: recovery no longer cuts a chat's log at its first journaled line and writes back only the journal's lines. It merges the log's lines and the journal's by seq: a line in both must be equal, a line only the log has stays, a line only the journal has goes in its place. Only bytes that do not read (a torn tail, a hole of zeros) are dropped. When the merge must change the log before its end, the lines only the log holds from there are journaled and made durable first, so a crash inside the repair loses no line. A log that moved on without the journal (a line in both that differs, lines out of order) is left as it is, and the journal's lines go to journal-unapplied/. - Append writes the log first, then the record, under one lock, after `Journal::check_append` (new in cmux-journal: the append's refusals, asked before any write). So a record always has its log line, and the `cancel` record is gone. - 2/3/4: the replay never deletes a folder and never makes a missing one again (a forget only drops the chat's earlier journaled lines). Delete appends its forget under the lock and removes the folder off it, with no wait for the record. - 5: a failed fsync in a checkpoint freezes it (a folder that does not open is retried); a frozen checkpoint returns an error, and the store asks the daemon to stop for a restart (`Store::stop_reason`, wired in daemon.rs), whose replay writes the logs again. Ten failed checkpoints in a row ask the same, so the journal past the checkpoint stays bounded. - 6: an error in one chat keeps its journaled lines in journal-unapplied/ and the open goes on. - 7: the journal's storage refuses writes under a lost home (`FencedStorage`, `config::refuse_lost_home`). - With store.journal off, a journal left by a run with it on is replayed, checkpointed and retired first (`catch_up`). - Nits: the checkpoint is never past the journal's durable end (no `Ahead` after a normal crash); damage is cleared after the next checkpoint; a kind-index copy goes before its segment is cut; an unreadable checkpoint fails the open; the open waits up to 5 s for the journal lock; crashed checkpoint temporaries are removed; `on_written` leaves cmux-journal (no user); README lists store.journal. * acpmux: split the journal store under its size budget (a move) journal_store.rs grew to 1085 lines and 71 functions with the review fix (budget 1000 and 60), and the fix grew store.rs and daemon.rs, which are over budget on the base. A pure move, no behavior change: - store/journal_store/{checkpoint,fenced,repair}.rs: the checkpoint and its thread, `FencedStorage`, the replay and the repair. - store/log_sync.rs: `LocalStore::sync_log`, `SyncFailed` and the test seam `FAIL_NEXT_LOG_SYNC`, out of store.rs (2139 lines on the base, 2113 now). - daemon/stop_causes.rs: `AbortOnDrop`, `next_signal` and `store_asks_to_stop`, out of daemon.rs (2052 on the base, 2046 now). check-no-godfiles.sh --base 51cfdee876b on every changed file: no file over budget grew, and no new file is over budget. * acpmux: the unrepairable-chat test waits for line 1 to be durable before its checkpoint At 03fccdef69c (fleet 32df3c72 on aws-m4pro-9, 144d4ccc on Linux) the test kept chat-b's lines [1, 2, 3] aside, not [2, 3]: it flushed right after appending line 1, and the checkpoint is now never past the journal's durable end (the review's Ahead nit), so line 1 was still past it. The fix's behavior is right; the test now waits until line 1 is durable, so its checkpoint covers it. On the code before the fix the test still fails (the open stops at chat-b's link). * acpmux: replies and events wait for durability, tests (red, S2c) DESIGN.md 3.3d. With the event journal on and every journal sync taking 400 ms (`SimStorage::set_sync_delay`): - an_acked_change_is_answered_once_its_record_is_durable: a rename's reply must come no sooner than one sync after its `renamed` record. Red here: the reply goes at once. - agent_output_is_not_held_behind_another_chats_acked_change: agent output to a connection attached to its chat goes at once while another chat's acknowledged change waits for its sync. Green here (nothing waits); it guards the fix against holding agent output behind changes the connection does not hear. * acpmux: replies and events wait until what they show is durable (S2c) DESIGN.md 3.3d; the tests are the commit before this one. With `store.journal` on, each connection gets an outbound gate (server/durable_gate.rs) between it and its transport: - Every line a connection queues raises the gate's `needs` to the last acknowledged LSN (`Store::durability().acked`, set in the journal store's append before the hub publishes anything of the record), except the lines of a `Stream` record's event (agent output, DESIGN C4): the fan-out delivers those in a thread-local stream scope. - The gate's task takes every queued line at once, reads `needs`, waits (async) until the journal's durable LSN reaches it, then sends the batch in order: one wait per batch, as the journal's group commit. - The durable LSN is a `tokio::sync::watch` fed by `Journal::on_durable`. The listener owns a guard: a writer that ends while the store is not being dropped (a panic on its thread) asks the daemon to stop (`Store::stop_reason`, decision 9.8), and the watch closes, so each gate drops what it holds and ends, which closes its connection. - Without the journal, `Store::durability()` is None: no gate, and `Conn::raise_gate` does nothing. Moves, so server/mod.rs and store.rs stay under their size on the base: the outbound queue helpers to server/outbound.rs, `open_with_journal` to the journal store, and `streamed` to store/durability.rs. * acpmux: the journal lock wait's deadline without an unchecked Instant addition `Instant + Duration` can panic on overflow (clippy::arithmetic_side_effects, tracked by the panic ratchet). The wait now has no deadline only in the case that cannot happen (an Instant 5 s ahead overflowing), where it stops waiting at once. * acpmux: a failed log sync stops the real daemon for a restart, test (red) Lead rule 2026-11-04 (3): a daemon-level test of the store's "stop for a restart" before S2c merges, red first. With store.journal on and every log sync failing (a debug build's ACPMUX_TEST_FAIL_LOG_SYNC; the checkpoint every 200 ms through ACPMUX_TEST_JOURNAL_CHECKPOINT_MS), a rename of a chat on disk makes its log dirty, the checkpoint's sync fails and freezes it, and the store asks the daemon to stop. The test asserts: the WebSocket client gets the restart close (1012), the socket client's pending `_acpmux/wait` ends with the connection (no hang), and the daemon exits with status 74. Red here: the daemon stops and sends 1012, but exits 0 (no exit status says that it stopped to be repaired). This commit adds only the two debug-build seams (release builds have none) and the test. * acpmux: a daemon its store stopped for a restart exits with status 74 The test is the commit before this one. `daemon::run` returned Ok and the process exited 0 for every stop, so a supervisor or a person could not tell a stop to be repaired from a normal one; with systemd `Restart=on-failure` it would not even start again. Now the store's stop (`store_asks_to_stop`: a failed log sync froze the event journal's checkpoint, or the journal's writer ended) marks itself, sets the restart handoff (WebSocket clients get 1012, as before), and the process exits with `daemon::EXIT_REPAIR` (74, EX_IOERR). The units prevent only 75, so launchd KeepAlive and systemd Restart=always start it again; the app does not read the daemon's exit status. README: store.journal names the status. * acpmux: the restart test's request in flight goes through the acpmux client Lead rule (3): the client sees a typed error, not a hang. The socket client is now the acpmux client library (acpmux::client::Client, as the CLI uses it), and the test asserts what the CLI makes of the request that the stop ends: `daemon_closed`, retryable (cli::errors::classify). The first red (fleet 41f2d037, hq run 41804640229, at 80c5fb2b7f8) failed only on the exit status (0, wanted 74), with 1012 and the end of the connection passing. * acpmux: the daemon's disk tier with the journal on, tests (red) Lead rule 2026-11-04 (2): a reply the person waits for never waits on a throttled disk sync. The fleet A/B (STATUS 11554) found that at IOPOL_UTILITY an acknowledged journal append waits about 16 ms p50 under another process's disk stress (0.7 to 1.1 ms at the default tier). - the_daemon_with_the_journal_keeps_the_default_disk_tier: with store.journal on, the daemon process keeps the default tier; without it, utility as before (each case in a child of the test binary). - a_background_thread_yields_its_disk_work: a dedicated background thread (the checkpointer) works at utility; a thread beside it does not. The API they use is stubbed here (`qos::enter_daemon` ignores the journal; `qos::background_thread` does nothing), so both fail on macOS. * acpmux: the restart test borrows no &str twice (clippy needless_borrows_for_generic_args) Found by fleet 7f8afd9d (hq run: the S2c server step at 62b4f68ef8b); the step's other clippy red is the base red in link_push_bench.rs. * acpmux: with the journal on, the daemon keeps the default disk tier (S2c) The tests are 994f74119fb. Lead rule 2026-11-04: a reply the person waits for never waits on a throttled disk sync. Measured on two owned Macs (STATUS 11554): one acknowledged append to durable at IOPOL_UTILITY under another app's disk stress took 16.3-16.7 ms p50 and 37-45 ms p95, against 0.7-1.1 ms p50 and 1.5-1.8 ms p95 at the default tier. - qos::enter_daemon(journal): with store.journal on, the daemon process keeps the default disk tier; without it, utility as before. The switch is read alone from config.json (config::store_journal_enabled), before any thread starts, with no harness discovery. - qos::background_thread: the journal store's checkpointer lowers its own disk I/O to utility for its whole life (a thread that does nothing else: a thread-scope policy drops it out of QoS for good). * acpmux: an agent chunk is a Stream record, test (red) The hub's tap logs an ACP session/update under its sessionUpdate kind (hub/tap.rs: `kind = su.to_owned()`), so a record's kind is `agent_message_chunk`, `tool_call`, ... and never `session/update`; the message's method is. The journal store's `streamed` checked the kind, so every agent chunk was journaled as Ack: a sync per batch at the token rate, and in S2c each token's event waited for that sync (the S2c I/O A/B: STRESS token to client 2.2 ms -> 20.3 ms p50 with the journal on). Red here: the chunk reads back as Ack. * acpmux: an agent chunk is a Stream record (the record's message names session/update) The test is the commit before this one. `streamed` now reads the logged message's method: the hub's tap logs a session/update under its sessionUpdate kind (hub/tap.rs), so the kind never said it. Agent output is again Stream (synced with the next acknowledged batch or a second later), as DESIGN.md 3.3b says. * acpmux: review 2 of #34901, tests (red): a crash model that follows real log syncs The logs' OS crash model now follows the syncs that ran (store/sync_model.rs, tests only): a segment keeps its length at its last sync that counts; a failed sync loses what was written since the one before, and a later sync does not bring it back; a cut ends a loss it removes. Every log segment sync goes through log_sync::sync_segment (same calls as before), where the model and FAIL_NEXT_LOG_SYNC hook in. Red tests, one per finding: - 1a an_os_crash_after_a_restart_from_a_kill_keeps_every_acked_line: the restart's first checkpoint retires records whose log lines no sync made durable; expected [1..5], the head gives [1]. - 1b an_os_crash_after_a_restart_from_a_failed_sync_keeps_every_acked_line: the same after a frozen checkpoint and a restart; expected [1..5]. - 2 a_hole_that_starts_inside_a_line_is_repaired (the reviewer's test): a torn line is read as line 6 of a log that moved on; expected [1..9], the head gives [1, 2, 3, 4, 5, 9]. - 3 a_failed_sync_in_flush_session_freezes_the_checkpoint: a failed sync in flush_session does not freeze the checkpoint. * acpmux: review 2 of #34901: a restart writes the replayed lines again; a torn line is not a line; flush_session freezes Finding 1 (high): the repair now writes every merged line again from the repair start, even when the log already has them all. A killed daemon's log writes may be in the page cache only, and after a failed sync the kernel may have dropped pages that still read back and that a later sync reports as written; the first checkpoint after the open then syncs bytes this process wrote before it retires their records. The lines only the log has from the repair start are journaled and made durable first, as before a cut. The work is bounded by the checkpoint interval. Finding 2: a piece of the log is a line only when it parses as one JSON value (piece_seq). A hole that starts inside a line keeps that line's {"seq":N, head and runs into a later line's tail; it was read as a line of a log that moved on, and the chat's journaled lines went aside. Now its bytes go and the lines are written again. Finding 3: a failed sync in flush_session freezes the checkpoint (its file sync is a SyncFailed now), as a failed sync in a checkpoint or at a log's close does. Nits: cut_at syncs events/ after its removals (and the kind-index folder after its copies go), so a removed segment cannot come back beside the lines written again; the README says that with the switch on an event line over 64 MiB is refused and its log does not get it. Tests: the four red tests of d2c63860ece. * acpmux: chat metadata through the event journal, tests (red, R-DURABILITY S3a) DESIGN.md 3.3e. Red on S2c: - last_seq_never_passes_the_log_after_an_os_crash (E13): session.json is synced at each save while the agent's lines wait for the journal's next sync, so after an OS crash the metadata names lines the log lost. - a_chat_made_after_the_checkpoint_survives_an_os_crash_that_loses_its_folder: nothing syncs the sessions folder when a chat's folder is made, and the replay never makes a folder again, so an acknowledged new chat is lost. - a_metadata_change_is_written_to_session_json_by_the_checkpoint: today each change writes and syncs session.json on the request path. Guards (green before and after): a chat made before the checkpoint and deleted is not made again; the replay never writes an older image over session.json. * acpmux: chat metadata through the event journal (R-DURABILITY S3a) DESIGN.md 3.3e, behind `store.journal` (default off). A metadata change is a `meta` record (kind 4: chat id, a `created` flag, the whole image) instead of a session.json write with two syncs on the request path; the reply waits for the record through S2c's gate, one group commit for a burst. - Overlay: each chat's newest image until a checkpoint writes session.json; load, load_startup and list read it first (list under the projection lock, so a checkpoint cannot drop an entry between its two reads). - Checkpoint: waits on its own thread until every record through its LSN is durable, then writes each image it read (session.json carries the image's LSN as `journalLsn`), then the logs and the sessions folder, as before. An image so names only lines the journal holds (E13). A chat deleted or made again meanwhile is passed (its overlay entry is gone or new). - Replay: per chat after its last `forget`, the newest image, written when newer than the file's `journalLsn`. A missing folder is made again only for a chat a replayed `created` record made; a chat made before the checkpoint whose folder is gone keeps its image in journal-unapplied/. - flush_session also waits until the chat's records are durable. - Delete drops the overlay entry with its `forget`, and removes the folder under the folders lock a checkpoint's write also takes. Tests: the 5 of 0505e9bee04 (3 red there). * acpmux: S3a, tests (red): folders handed out of the store Code outside the store reads and moves chat folders directly (transfer_in puts the live folder aside and installs a transferred chat; session_dir hands the folder to the transfer, the session files view and others). With the metadata in the overlay until a checkpoint: - a_folder_replaced_outside_the_store_keeps_its_own_metadata: a read, the checkpoint and the replay must not put the old folder's image in the new one. - session_dir_hands_out_a_folder_whose_metadata_is_written. - flush_session_writes_the_chats_metadata (the trait says log and metadata). - the_checkpoint_never_makes_a_folder_removed_outside_the_store_again. Red on 71bc09a1e63 (S3a's first implementation). * acpmux: S3a red tests: a replay after a folder swap; flush_session's chat has a log (as its callers' chats do) * acpmux: S3a: each metadata image is bound to its chat folder; session_dir and flush_session write it first Code outside the store reads and moves chat folders directly (a transfer puts the live folder aside and installs a transferred chat under the same id). So a `meta` record now carries its folder's identity (file id and birth time): - reads use an overlay image only while the folder at the path is its folder; - the checkpoint writes an image only into its own folder, never makes a missing one, and keeps the image of a removed or replaced folder in journal-unapplied/; - the replay writes an image only into its own folder (or a folder a replayed `created` record made), else keeps it aside. `session_dir` (the transfer, the session files view) and `flush_session` (the trait: "log and metadata to stable storage") write the chat's image first, once its record is durable. The list reads the overlay before the files (a checkpoint writes a file before it drops the entry), so it needs no lock. Tests: the 5 of 0280ac37f5e and 0e617489731. * acpmux: S2 follow-ups from review 3 of #34901, tests (red) - a_chat_whose_repair_failed_is_repaired_at_the_next_start (follow-up 1): the open's checkpoint retires the records of a chat whose repair failed, so no later start puts its lines back (they are only in journal-unapplied/). - a_hole_that_ends_at_a_line_start_keeps_that_line (follow-up 2): a torn line, zeros, then a whole line only the log holds: the piece does not parse and the whole line is dropped. - an_idle_store_does_not_wake_its_checkpointer (follow-up 4, cmux-ad m9): the checkpointer wakes every period with nothing to do. Test seams only (no behavior change): CHECKPOINTER_WAKES and TEST_CHECKPOINT_EVERY_MS. * acpmux: S2 follow-ups from review 3 of #34901: a failed repair holds its records; a hole's last zero splits; an idle checkpointer sleeps - Follow-up 1: a chat whose repair fails at the open (its log cannot be read or written now) keeps its lines in journal-unapplied/ as before, and the checkpoint now stays below its first journaled LSN (`Order::held`), so the next start repairs it again. Bounded: past 256 MiB of journal after it, the hold goes and the lines stay only in journal-unapplied/ (an error says so). - Follow-up 2: a piece splits at its last zero byte (a whole JSON line holds none), so a hole that ends at a line start, after a torn line, keeps that line (it mattered for a line only the log holds). - Follow-up 4 (cmux-ad m9): with nothing to checkpoint, the checkpointer sleeps until a change wakes it (`Inner::touched`, one atomic read per append), then checkpoints one period later; before, it woke every 30 s. Tests: the 3 of d147057bf85. * acpmux: the qos journal tests module is macOS-only (its Linux build held only an unused import: clippy red, review of #35400 F0) * acpmux: review of #35400 F1 and F2, tests (red) - a_read_that_shows_a_line_covers_its_record (F1): a read sees a log line between its log write and its record (the appends' lock held there), and the reply built from it goes out with no wait for that record. - a_journal_that_syncs_nothing_asks_the_daemon_to_stop (F2): with a journal that makes nothing durable, every reply waits without a limit and nothing asks the daemon to stop. Test seams only: PAUSE_AFTER_LOG_WRITE, TEST_STALL_MS. * acpmux: review of #35400 F1, the streaming read, test (red): session/load's replay sends a line before its record * acpmux: the test seam PAUSE_AFTER_LOG_WRITE is reachable from the server tests * acpmux: review of #35400: a read covers the records of the lines it shows; a stalled journal stops the daemon F1: the append writes a line to the log, then appends its record and sets the acknowledged LSN, all under the appends' lock; reads take no lock. A read could show a line before its record existed, and the reply built from it went out at once. Now every journal-store read of the logs ends with a fence on the appends' lock (`Inner::fenced`): a line the read saw has its record appended and its LSN acknowledged when the read returns, so the reply's gate covers it. The one read that sends as it reads (`Hub::scan_events`, session/load's replay) stops at the chat's last_seq read first. F2: a journal writer that keeps failing (it retries without end) held every reply on every connection with no limit. A connection's gate now waits at most 10 s (`STALL_LIMIT`); past it, `Durability::stalled` asks the daemon to stop for a restart, as a failed log sync does. Tests: the 3 of 3916f84dd68 and c553a19fd8b. * threading-model: the acpmux event journal's threads and bounds (writer, checkpointer, per-connection gate; review of #35400) * acpmux: the stall test's journal syncs take 3 s, not an hour (a sync under way kept its delay and hung the test's end) * acpmux: the idle checkpointer's period, test (red): a change seen by its second look must not start a checkpoint at once - a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period: the checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies when it gets the timer lock. The second look saw the change, so the checkpointer is in its period wait by then, and the notify ends that wait: a checkpoint runs at once on the checkpointer's thread. This is the cause of a_failed_log_sync_freezes_the_checkpoint failing in the full store:: run of fleet 0577ac62 (that early checkpoint took the line whose sync the test fails, on a thread where the failure is not set). The test holds both threads at the two points, so the order is exact. Test seams only (no behavior change): the checkpointer's test seams are now by the store's home (period, wake count, the two holds), so tests that run as threads of one process under cargo test do not see each other's. * acpmux: the idle checkpointer's period ends only on its own state, not on a wake meant for its idle wait The checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies once it gets the timer lock. Its change shows in the second look, so the checkpointer has gone to its period wait by then, and that notify ended the period at once: a checkpoint ran right away on the checkpointer's thread (a burst no longer shared one checkpoint), and a spurious wakeup could do the same. In the full store:: run of fleet 0577ac62 such a checkpoint took the line whose sync a_failed_log_sync_freezes_the_checkpoint fails, so that test's own flush had nothing to do and returned Ok. The period wait now ends only on the period, `wanted` or `stop` (`wait_timeout_while`): each of the checkpointer's two waits ends only on the state its waker sets before it notifies under the timer lock. No sleep, no retry. The idle wait is unchanged. Test: a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period (edca3b4823d). * acpmux: S3a review (#35483) red tests: switch-off metadata and deletes, a late settle, a failed session.json folder sync, a purge after the checkpoint F1: metadata written with the switch off survives turning it on again; a chat deleted with the switch off after a failed catch-up is not made again. F2: a late settle never writes an older image over a checkpointed one (the reviewer's test and hold). F3: a failed folder sync after a metadata write freezes the checkpoint. F4: a chat made after the checkpoint and deleted is not made again after a kill. Test seams only; the code is the S3a head merged with feat. * acpmux: S3a review (#35483) red tests: a new chat is acknowledged only once its folder is durable; session_path never waits The root of F4 and of F1's delete case: the replay makes a chat's folder again, so a chat removed by any path comes back. The fix will never make a folder again, which needs an acknowledged chat's folder to survive an OS crash: the test models what each folder's last sync kept (sync_model) and asserts the gate's durability passes a new chat only after its sessions entry is synced. P3 1 of the review: a caller that needs only the path never waits for the journal (Store::session_path, a stub with session_dir's behavior here). * acpmux: S3a review (#35483) fixes: the replay never makes a folder again; switch-off changes are kept; no late or unsynced metadata write F1: each meta record carries the stamp (file id, modification time, length) of the session.json its save saw. The replay writes an image over a file the journal did not write only when one of the chat's records saw that file; otherwise the file moved on without the journal (a run with store.journal off whose catch-up did not replay, or an older build) and the image goes to journal-unapplied/, as S2 does for a log. F1 (deletes) and F4 at the root: the replay never makes a chat's folder again. A save that made a folder notes its record's LSN; the new acpmux-journal-folders thread (the daemon's disk tier) syncs sessions/ at once, and the durable LSN the S2c gate waits for stays below that record until then (journal_store/new_folders.rs). So an acknowledged chat's folder survives an OS crash, and a folder that is gone was removed: a purge whose forget a kill took back, a run with the switch off, the transfer recovery. The created flag and the replay's folder remake are gone. F2: write_through does not write a file that holds the same or a newer journalLsn. F3: session.json is written with its own syncs (file, then folder); a failed sync is SyncFailed and freezes the checkpoint; the replay and the kept-aside images return their errors. P3 1: Store::session_path gives the path with no lock, wait or write; an agent's start, the fixed-id check, the temporary-folder sweep, the host probe and the transfer recovery's root use it. The replaced test (a chat made after the checkpoint survives an OS crash that loses its folder) asserted a remake that cannot be right any more; the new test asserts the guarantee that makes the loss impossible, then the crash. Red (7 of 17 journal_meta_tests, each for its reason; the other 10 pass): macOS 5bc434a9 and 624d5df4, Linux a2207ff0 and 638a44ef. * acpmux: review of #35562 red tests: a failed repair holds no other chat's checkpoint; a checkpoint after it writes only durable images F1 (the reviewer's test, renamed): a checkpoint after a failed repair writes only durable metadata images. F2/F3 at the root: a failed repair does not hold the checkpoint of other chats; a retention rewrite of another chat after a failed repair stays after a restart. Guards: the reviewer's two switch-off tests (green here since the S3a fix 0d246364d90: the stamp rule and no folder remake), and a chat deleted and made again after its failed repair gets none of the old chat's lines. * acpmux: review of #35562: a failed repair keeps its chat's lines in a repair file; no hold of the global checkpoint The hold (Order::held, 49d7e5cef83) kept every chat's records past the checkpoint while one chat's repair failed. That let a checkpoint write metadata images whose records were not durable (F1), made each start with the switch off replay the whole window again and undid a retention rewrite (F2), and grew the journal with each restart (F3, the 256 MiB bound counted one run only). Now a chat whose log cannot be read or written at the open keeps its lines in journal-unapplied/<id>.<lsn>.repair.ndjson (a header with the chat id, then the lines, synced). The next open journals them again, waits until they are durable, removes the file and repairs the chat with them as with any journaled line (a line in both is the same bytes, so a file journaled twice after a crash changes nothing). The checkpoint of every chat goes on as before S2's follow-ups. A delete removes its chat's repair file after the folder, so a chat made again under the same id gets none of the old lines. Red at 23870301064 (macOS a8be0684): the 3 fail for their reasons ("the checkpoint stayed at 1 below chat-a's records (the journal ends at 5)"; "the checkpoint wrote an image to session.json before its record was durable"; "the open kept a copy of a rewritten log aside"); the guards pass; 339 of 342. * acpmux: S3a delta review (N1) red tests: a change to a chat whose session.json another journal wrote reaches it A folder moved in from another home (build adoption), checkpointed and replayed, and a journal made again from nothing: the other journal's journalLsn must not hide a change made here. The first test is the reviewer's (fleet 4d0d98bc and 071ad7c4 at e944a7c21de). * acpmux: S3a delta review (N1): journalLsn is bound to its journal's id; a file another journal wrote follows the stamp rule session.json now carries journalId (16 hex digits, a string) beside journalLsn. The checkpoint's "held" check (F2) and the replay's rule compare an LSN only for a file with this journal's id: a file from another journal (a chat folder build adoption moved in from another home, or a home whose journal was made again) counts as one this journal did not write, so the stamp rule decides, as for a run with the switch off. cmux-journal: Journal::journal_id (the id already in every segment header; kept across opens and retirements), with a unit test. Red at 7c8cb957745: fleet c29b659d (Linux) and 47570494 (macOS) running; the reviewer's red at e944a7c21de: 4d0d98bc (macOS) and 071ad7c4 (Linux). * acpmux: delta review of #35562 red tests: a repair file and its folder; a torn file; a failed delete; a file larger than the journal's queue D1 (the reviewer's test, renamed): a failed repair's kept lines survive a folder that is aside at the next open; and they never go into a folder that replaced the one they were kept for. D2 (the reviewer's test, renamed): a torn repair file does not stop the chat's repair. D3: a delete that succeeds after a failed one removes the chat's repair file (test seam FAIL_NEXT_FOLDER_REMOVAL). D4: a repair file larger than max_pending_bytes is applied. * acpmux: delta review of #35562: a repair file applies only to the folder it was kept for; written whole; torn lines passed; a full queue waits; a failed delete keeps its file D1 (P2): the open took a repair file even when the chat's folder was not there (a transfer held it aside at a kill, and the hub put it back after the store's open), and the lines went nowhere. The file's header now has the folder's identity (meta.rs FolderId, the same one S3a binds metadata images to); take_repairs journals a file only when the folder at the chat's path has that identity. Otherwise the file stays: a later open applies it when that folder is back, and else it is a person's. A failed repair with no folder to bind to keeps its lines as a plain kept file, as before S2's follow-ups. D2 (P3): kept files are written whole (a temporary copy, synced, renamed, folder synced); the open removes a temporary a crash left; bytes in a repair file that are not a whole JSON line are passed. D3 (P3): delete keeps the chat's repair-file entry until the file is removed, so a delete tried again after a failed folder removal still removes it. D4 (P3): the open's repair appends wait for the writer when the journal's queue is full (append_waiting), so a repair file larger than max_pending_bytes is applied; journal_lines uses it too. Red at c0a20e09f1f: macOS fleet 54945ca4, 5 of 41 journal_store_tests fail, each for its reason; Linux fleet 393040c6 running. * acpmux: the delete test seam goes before the append pause's doc comment, not between it and its item (clippy unused_doc_comments, fleet 56f7ec6c)
lawrencecchen
added a commit
that referenced
this pull request
Nov 5, 2026
…URABILITY S3b) (#35565) * cmux-journal: one append-only journal per daemon, with group commit and ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner). * cmux-journal: type a test range; drop an unread segment field (fleet clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71e3f9 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red. * cmux-journal: a failed F_FULLFSYNC is an error, not a plain fsync; no segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee. * cmux-journal: the past-the-end test reads back the storage it wrote to Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c87b8d: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5c53e. * cmux-journal: a sync a signal interrupts is made again (macOS) fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r). * cmux-journal: crash sim of write-back errors and crash in recovery; S1 review regressions (red) The S1 review (FIX at 89293fb15f39) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open * cmux-journal: the F2 regression damages the header of the last of several segments (red) The reviewer's F2 scenario has two segments: S1 holds 1..10, S2 (the last) holds 11..20, and S2's header is damaged. The test now rotates segments (300-byte segments, 20 acked records) and damages the last one's header, so the report must say lost_through 20 and the next append must take 21. On the code before the fix the moved segment's LSNs read as 0, so the next append reuses one. * cmux-journal: the crash sim forces the F4 order (failed sync, kill, OS crash) (red) On the code before the S1 fix, the crash-sim step was green (fleet 184129d8 at 263da86f223): 6 of 7 runs passed, and the mixed run failed only seed 140 in the full run and passed when it ran alone. The random faults reached the order the review found (F4) in 1 of 800 runs. So these tests were no red/green proof. The order: the sync of a write that spans pages fails and the disk gets none of them (Linux keeps them clean in the cache, so a later sync reports success); the process is killed in the writer's backoff; the next open reads the failed segment from the cache, replays its records and goes on; an OS crash or a power loss then takes those pages, and records an open already replayed are gone. - sim.rs: `fail_next_sync(pages, kind)` fails the next sync that has at least that many dirty pages, and the disk gets none of them. `crash_after_failure(n)` makes the n-th operation after the next injected failure the crash point. - crash_sim_tests.rs: a new run forces the order on every seed (a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record): epoch 0 aims the fault, the kill is a process crash at the writer's retry, the open after it does not crash, epoch 1 ends in an OS crash, and a third open checks that the records the second open replayed are all there. In the random runs, one epoch in four aims the fault, with the crash 1 to 3 operations after the failed sync and a crash of any kind after it. - A sim test checks the aimed fault itself (the_sim_fails_the_aimed_sync_and_crashes_in_the_backoff). Expected on this commit (old journal code): the forced run fails every seed; the mixed run fails most runs. The fix commit follows. * cmux-journal: fix the S1 review findings F1-F7 (markers, recovery copy, LSN floor, writer end) Review FIX at 89293fb15f39 (PR #33919 comment 6610468202). The regression tests are the two commits before this one. - Sync markers (format.rs): an entry with flags 1 and no payload, whose LSN field says "every record of this segment before me is synced". The writer appends one after each successful sync that covered records (after the durable listeners, so no reply waits for it), and syncs it fully at a seal and at a rotation. - F4, recovery (step 4 and 6): the last segment is trusted only up to its last marker. Its records after the marker may sit in pages a failed sync dropped (a later sync on another descriptor reports success without them), so recovery writes them again into a new segment: header, records and a marker, fully synced under `<name>.seg.new`, renamed into place, folder synced. The journal always goes on in a segment this open made, never in one an earlier process wrote. A restart in the writer's backoff after a failed sync is then safe. - F1, quarantine: every LSN a quarantined segment holds counts in the LSN floor at every open, and a quarantine that no new segment followed (a crash in the middle of an earlier recovery) is reported as damage again. The highest LSN of every moved segment is read before the first rename. - F2, highest LSN: a segment whose header is damaged is read from its first record (each record's checksum covers its LSN), then the bytes after the first bad entry are scanned for records and markers that still check out. The same scan runs after a cut in the last segment: a marker there is media damage (reported), records there are a torn write (`torn_lsns`); their LSNs are never used again. - F3, writer end: a guard closes the journal when the writer thread ends, also by a panic: appends fail with Closed, waits end with WriterEnded, the listeners are dropped (so a channel a held reply waits on closes), and stats.failing says why. The crate does not abort the process; the daemon owns that policy. - F6: recovery closes each segment after its header is read and opens one again only to read it. (Retirement of compacted segments is an S2 API, before the first consumer.) - F5: recovery syncs the parent folder after it makes the journal folder. - F7: format.rs and lib.rs now say what the code does: a failed write is cut back and retried in the same segment; the cut removes only bytes never synced and never acknowledged; nothing shortens a segment past a synced record. Docs: journal.rs says when open's replayed records are durable (on Ok) and what a stopped writer does. * cmux-journal: split the crash sim's machine model into sim/machine.rs (review G5) sim.rs had 924 lines and 63 functions, over the 60-function limit for a new Rust file (the god-file check warned "not in baseline: split it"). The machine model (each file's page cache and disk, the ordered log of names, and what an OS crash or a power loss leaves) moves to sim/machine.rs (297 lines, 19 functions); the process view and the Storage implementation stay in sim.rs (639 lines, 30 functions). No behavior change: the items move as they are, made visible to the sim module. * cmux-journal: re-review regressions G1-G4 and page eviction in the crash sim (red) Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993). Tests and sim only; the fix follows. - sim: the pages a failed sync dropped stay in the cache until an eviction; `evict_dropped_pages_after_read(true)` makes each read evict the dropped pages it read, so a second read of them sees the disk's bytes. - G1 recovery_copies_the_records_it_checked_when_the_cache_drops_them: a sync fails and drops record 2's pages, a kill in the backoff, the next open reads record 2 from the cache (replayed) and the pages are evicted after that read; then an OS crash. Record 2 and the record 3 acked after the restart must survive. On this code recovery's second, unchecked read stages zeros under a vouching marker, and the last open keeps only record 1 and reports damage. - G2: the F1 regression asserts that every open after the damage reports the loss (lost_through 40). On this code a crash at the open's last step (the folder sync after the new segment's rename) loses the report for good. - G3 a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system): a journal in a symlinked state folder, and a journal folder that is a symlink. On this code the parent folder's sync opens with O_NOFOLLOW (ELOOP). - G4 an_open_after_a_finished_quarantine_reads_no_quarantined_segment: on this code every open reads every quarantined segment in full. * cmux-journal: fix the re-review findings G1-G4 and G6 Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993); the regression tests are the commit before this one (red: fleet de35f6d0). - G1: recovery writes again exactly the bytes its checked read returned. The last segment's entries after its last marker are kept from the checked read (header and payload) and staged from memory; the second, unchecked read of the file is gone. A page that a failed sync dropped can be evicted between two reads, and the second read then returned the disk's zeros, which the copy kept under a vouching marker. - G2: the damage an open finds is written to `DAMAGE` (whole, fully synced, renamed into place, folder synced) before the new segment that hides it from later opens. Every open reports the recorded damage first, until the daemon calls the new `Journal::clear_damage`, which moves the record into `quarantine/` (kept, never deleted). - G3: a folder sync follows a symlink in its path (no O_NOFOLLOW): a state folder, or the journal folder itself, may be a symlink. The sync is for the folder that holds the entries. - G4: the quarantined segments are read for their LSNs only while no live segment is numbered above them (an unfinished quarantine); once one is, its first LSN already carries their floor. - G6: `checked_div` in the scan of the bytes after a cut (the crate's one tracked arithmetic hit). - G5 (the sim split) is the commit before the regressions. * cmux-journal: third-review regressions G7 (A, B) and N1 (red) Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018). Tests only. - G7 A a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn: after a finished quarantine, one byte of the empty new segment's header changes. On this code the next open skips the quarantine scan (the listing counts the damaged segment) and the next append reuses a quarantined LSN. - G7 B a_damaged_newest_segment_after_a_quarantine_reports_its_records: the newest segment holds acked 41..43, and its header and its first record are damaged. On this code the moved segment is scanned with a base far below 41, so the loss is not reported through 43 and the next append is not 44. - N1: the F1 regression also asserts that an open reports each loss once (no reported LSN range lies inside another). On this code a crash after the quarantine's renames makes the next open report the same loss twice. * cmux-journal: fix the third-review findings G7, N1, N2; split recovery.rs Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018); the regression tests are the commit before this one. - G7: the quarantine scan is skipped only when a segment whose header checked out is numbered above every quarantined segment (the folder listing also counted a newest segment with a damaged header, which carries no floor). The scans start from the running floor: `quarantined_floor` and `quarantine` pass the highest LSN counted so far as the base of the next file's scan, so a quarantined file with no readable header and no readable first record is scanned for the LSNs right above what is counted. - The "unfinished quarantine" report is made only when the quarantine holds LSNs nothing else counted (an old quarantine behind a segment that a crash tore at its creation holds none, and a crash is not damage). Its reason now also names a damaged newest header. - N1: a loss whose LSN range lies inside a recorded one is the same loss found again after a crash: it is neither recorded nor reported twice. - N2: recovery.rs's docs name `DAMAGE`; two doc lines over 100 columns are wrapped (lib.rs, sim.rs). - recovery.rs was at 987 of its 1000 lines: the damage record moves to recovery/damage.rs, the entry reader and the LSN scans to recovery/scan.rs (moved as they are). * cmux-journal: report each loss once, also when one open finds it twice (N1) The red run of the N1 regression (fleet 27d011ca, op 64 of the F1 test) shows the case the previous commit did not cover: a crash after the quarantine's renames and before DAMAGE is written; the next open finds both the damaged synced record of the last segment (7..14) and the unfinished quarantine (7..40), and reported both. Now every loss inside another (recorded or found by this open) is dropped, the larger range kept; DAMAGE is rewritten only when its set of losses changed. * cmux-journal: fourth-review regressions G8, N3, N4 (red) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report. * cmux-journal: the quarantine's floor in its file names; DAMAGE before the moves (G8, N3, N4) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format. * cmux-journal: sort the quarantine list with sort_by_key (clippy unnecessary_sort_by) Fleet d9f22bed at b589044b852: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`). * cmux-journal: Journal::on_written, a listener per written batch (for S2's projector) R-DURABILITY S2 (acpmux event logs through the journal) projects each chat's log line into its ndjson file once the journal has WRITTEN the batch that holds it, so readers see stream lines at once. The listener runs on the writer thread after the write and before the sync, once per batch that moved the written LSN: a batch written again into a new segment after a failed sync is not heard twice. When the writer stops, these listeners are dropped too. Test: heard while the batch's sync still runs, once. Port of 145faa6faa8 (branch rewrite/r-durability-s2-acpmux-logs) onto the S1 review fixes. * cmux-journal: Journal::retire_through, the one delete (segments a compaction point covers) R-DURABILITY S2: a daemon's projector compacts the journal into its own files (acpmux: the chat ndjson logs, then a synced checkpoint). Without a delete the segment count only grows (review finding F6). `retire_through(compacted)` deletes the segments that hold no record after the caller's compaction point: a segment goes when the next segment's "previous last LSN" is at or before it, so the newest segment never goes. The point is capped at the durable LSN. The deletes and a folder sync run on the caller's thread (the projector, never a request path). A delete that fails leaves its segment live for a later call. Why a crash during a retirement is safe: the caller reopens from a point at or after `compacted`, and recovery reads nothing from a segment the point covers (it skips it on its next segment's header), so any subset of the deletes may reach the disk. A point older than the retired segments is reported as Damage::Missing. Quarantined segments are never deleted. - Storage::remove_file (FsStorage: std::fs::remove_file; SimStorage: one more change in the ordered log of names, durable at the folder sync). - The live segment list: recovery's, then each segment the writer makes. - JournalStats::segments_retired. - Tests: only covered segments go, and a second call deletes none; an OS crash after it keeps every record after the point and an older point is reported; a crash at every operation of a retirement (process, OS crash, power loss) keeps every record after the point and never reuses an LSN; the newest segment stays; a point in the middle of what this writer wrote. This adds a delete of journal data, so its PR is risk:high. * acpmux: move MemoryStore and the store settings into their own files store.rs (2139 lines, 120 fns) and config.rs (1909 lines, 97 fns) are over the god-file limit, and the journal store (R-DURABILITY S2) must not grow them. MemoryStore moves to store/memory.rs and StoreMode, StoreConfig and its defaults to config/store_config.rs; both stay exported where they were (crate::store::MemoryStore, crate::config::{StoreConfig, StoreMode}). No behavior change: the code moves as it is. * acpmux: the event log's journal switch and its crash tests (red) R-DURABILITY S2 (DESIGN.md 3.3b). Today no event-log append is synced: a line a client was told about is on the disk only after the next flush (shutdown), so an OS crash or a power loss takes back every line since (E12). This commit adds only the switch, a seam and the tests; the journal store follows. - `store.journal` (StoreConfig::journal, default false). - `Store::wait_durable(timeout)`: tests and tools wait until every appended record is durable. The default returns at once (the Local and Memory stores never sync an append), which is today's behavior. - `store::open_with_journal` (tests): the store with the journal on a given `cmux_journal::Storage`. Here it ignores the journal and opens LocalStore. - acpmux depends on cmux-journal (dev: with the `sim` feature). - store/journal_store_tests.rs, red on this commit: - a_line_the_store_says_is_durable_survives_an_os_crash (lines 2..20 are lost: nothing synced them); - a_deleted_chat_stays_deleted_after_a_crash and a_line_the_log_refused_is_never_written_later pass here, and must keep passing with the journal store (no replay may bring them back); - a_kill_keeps_every_line_once passes here and must keep passing (a replay after a kill must write no line twice). * acpmux: the event log through one journal, behind `store.journal` (R-DURABILITY S2) DESIGN.md 3.3b. With `store.journal` on, each event-log append is one record in `<home>/journal/` (cmux-journal: group commit, one sync per batch; a power loss takes back at most the last second, always a suffix), then the same line goes to the chat's ndjson as before. Readers, segments and file formats do not change. Default off. - store/journal_store.rs (JournalStore over LocalStore): - append: under one lock, the journal record (a memory copy, no I/O), then the log line; a line the log refuses gets a `cancel` record. Agent stream lines (`in` session/update, claude.*, codex.*) are Class::Stream, every other line Class::Ack. - checkpoint (`<home>/journal-projected.json`): every 30 s while the journal grew, past 64 MiB, at flush (shutdown) and before a retention commit. It syncs each log written since (LocalStore::sync_log, with no lock held while the disk works) and the sessions folder, writes the LSN (temp file, sync, rename, folder sync, every error checked), then retires the journal before it. A log close that failed its sync freezes the checkpoint until a restart (LocalStore::sync_failed). - recovery: the records after the checkpoint; for each chat, the log is cut right after its last readable line below the first journaled seq (later segments removed, the boundary one shortened, their kind-index copies dropped), and the journaled lines are written again in order. `forget` deletes a chat again; `cancel` keeps a refused line out. - delete: `forget` record, folder removal, then a bounded wait until the record is durable. - seal: the logs refuse writes and the journal syncs fully. - LocalStore: `append` is serialize + `append_line`; `sync_log`; a failed sync at a log's close is noted (`sync_failed`). - store::open opens the JournalStore when `store.journal` is on. Not yet (S2c): replies and events do not wait for durability; the daemon does not stop when the journal closes; the writer thread keeps the process's Utility I/O policy. * acpmux: journal store tests for the seal and a chat made again after its delete (red) - a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay: a chat is deleted (its forget record), made again under the same id (a transfer back: session.json and new lines), then a kill -9. On this code the replay deletes the folder again and its session.json is gone. - a_line_after_the_seal_is_refused_and_never_written_later: coverage of the seal (it passes here; the fix also closes a race the test cannot force). * acpmux: journal store: the seal under the appends' lock; a replay keeps a chat made again - seal takes the appends' lock: an append lands in both the journal and the logs, or in neither. Before, an append that raced the seal could leave a journal record whose line the sealed logs refused (its cancel then failed on the closed journal), and the next start would write that line. - append and delete check the logs' seal first, under the same lock, and do not touch the journal after it (no error per append at shutdown). - replay deletes a forgotten chat again only when no line follows its forget record: otherwise the chat was made again under the same id, and its folder holds that chat's own files. * cmux-journal: fifth-review regressions G9 (red) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4. * cmux-journal: name a skipped LSN floor in a FLOOR.<lsn> file (G9) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646); the regression tests are the commit before this one. When an open gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which a copy then hides from every later open), the only record of that floor was the header of the empty segment it makes; a damaged header lost it, and DAMAGE is cleared by `clear_damage` (torn records are not damage at all). Now such an open makes the empty file `FLOOR.<lsn>` and syncs its folder before any new segment, and every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine's names and this open's moves say their floors already: no file is made for them. * acpmux: journal store regressions from the S2 review (red) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824). Each test fails on this commit and passes after the fix that follows it: - 1a lines_written_with_the_switch_off_survive_turning_it_on_again - 1b a_kill_while_the_journal_syncs_keeps_the_lines_the_log_got - 1c a_line_flush_session_made_durable_survives_losing_its_record - 2 a_folder_put_back_after_a_delete_is_not_deleted_by_the_replay - 3 a_kill_in_a_delete_leaves_the_chat_whole_or_gone - 4 a_delete_does_not_wait_for_the_journal_sync - 5 a_failed_log_sync_freezes_the_checkpoint, a_checkpoint_that_keeps_failing_asks_the_daemon_to_stop - 6 one_chat_that_cannot_be_repaired_does_not_stop_the_open - 7 the_journal_storage_refuses_a_lost_home - the repair keeps a log's lines after a hole, also through a crash in it: an_os_crash_hole_is_repaired_and_the_lines_after_it_kept, a_crash_inside_a_repair_loses_no_line - nits: an_unreadable_checkpoint_fails_the_open, the_open_waits_for_the_previous_daemon_to_let_go_of_the_journal, a_crashed_checkpoint_temporary_is_removed_at_open, a_crash_right_after_a_checkpoint_reports_no_damage - the catch-up with the switch off: switching_off_after_a_crash_keeps_every_acknowledged_line - cmux-journal: check_append_says_what_append_would_do So they compile here, this commit adds the seams they use, with no behavior: `Store::stop_reason` (default None), `Journal::check_append` (always Ok), `FencedStorage` (never asks its fence), the open report and a test stop point after a repair's cut. * acpmux: the checkpoint freeze test fails a log sync, not a folder open (red) The freeze guards against a failed fsync (a later fsync of the same file may report success for pages the kernel dropped), not against a folder the store cannot open: a chmod 000 folder is an open error, which the fix retries instead of freezing. The test now fails the log sync itself through a test seam, `FAIL_NEXT_LOG_SYNC`. Red on the code before the fix, like the other review regressions (fleet d8e7037d, 1c22175e, 63604b26 at e030f4fe402: 17 of 17 new acpmux tests and the crate's check_append test fail). * acpmux: the journal store repairs logs, never rebuilds them (S2 review) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824); the regression tests are the two commits before this one. DESIGN.md 3.3c. - 1a/1b/1c: recovery no longer cuts a chat's log at its first journaled line and writes back only the journal's lines. It merges the log's lines and the journal's by seq: a line in both must be equal, a line only the log has stays, a line only the journal has goes in its place. Only bytes that do not read (a torn tail, a hole of zeros) are dropped. When the merge must change the log before its end, the lines only the log holds from there are journaled and made durable first, so a crash inside the repair loses no line. A log that moved on without the journal (a line in both that differs, lines out of order) is left as it is, and the journal's lines go to journal-unapplied/. - Append writes the log first, then the record, under one lock, after `Journal::check_append` (new in cmux-journal: the append's refusals, asked before any write). So a record always has its log line, and the `cancel` record is gone. - 2/3/4: the replay never deletes a folder and never makes a missing one again (a forget only drops the chat's earlier journaled lines). Delete appends its forget under the lock and removes the folder off it, with no wait for the record. - 5: a failed fsync in a checkpoint freezes it (a folder that does not open is retried); a frozen checkpoint returns an error, and the store asks the daemon to stop for a restart (`Store::stop_reason`, wired in daemon.rs), whose replay writes the logs again. Ten failed checkpoints in a row ask the same, so the journal past the checkpoint stays bounded. - 6: an error in one chat keeps its journaled lines in journal-unapplied/ and the open goes on. - 7: the journal's storage refuses writes under a lost home (`FencedStorage`, `config::refuse_lost_home`). - With store.journal off, a journal left by a run with it on is replayed, checkpointed and retired first (`catch_up`). - Nits: the checkpoint is never past the journal's durable end (no `Ahead` after a normal crash); damage is cleared after the next checkpoint; a kind-index copy goes before its segment is cut; an unreadable checkpoint fails the open; the open waits up to 5 s for the journal lock; crashed checkpoint temporaries are removed; `on_written` leaves cmux-journal (no user); README lists store.journal. * acpmux: split the journal store under its size budget (a move) journal_store.rs grew to 1085 lines and 71 functions with the review fix (budget 1000 and 60), and the fix grew store.rs and daemon.rs, which are over budget on the base. A pure move, no behavior change: - store/journal_store/{checkpoint,fenced,repair}.rs: the checkpoint and its thread, `FencedStorage`, the replay and the repair. - store/log_sync.rs: `LocalStore::sync_log`, `SyncFailed` and the test seam `FAIL_NEXT_LOG_SYNC`, out of store.rs (2139 lines on the base, 2113 now). - daemon/stop_causes.rs: `AbortOnDrop`, `next_signal` and `store_asks_to_stop`, out of daemon.rs (2052 on the base, 2046 now). check-no-godfiles.sh --base 51cfdee876b on every changed file: no file over budget grew, and no new file is over budget. * acpmux: the unrepairable-chat test waits for line 1 to be durable before its checkpoint At 03fccdef69c (fleet 32df3c72 on aws-m4pro-9, 144d4ccc on Linux) the test kept chat-b's lines [1, 2, 3] aside, not [2, 3]: it flushed right after appending line 1, and the checkpoint is now never past the journal's durable end (the review's Ahead nit), so line 1 was still past it. The fix's behavior is right; the test now waits until line 1 is durable, so its checkpoint covers it. On the code before the fix the test still fails (the open stops at chat-b's link). * acpmux: replies and events wait for durability, tests (red, S2c) DESIGN.md 3.3d. With the event journal on and every journal sync taking 400 ms (`SimStorage::set_sync_delay`): - an_acked_change_is_answered_once_its_record_is_durable: a rename's reply must come no sooner than one sync after its `renamed` record. Red here: the reply goes at once. - agent_output_is_not_held_behind_another_chats_acked_change: agent output to a connection attached to its chat goes at once while another chat's acknowledged change waits for its sync. Green here (nothing waits); it guards the fix against holding agent output behind changes the connection does not hear. * acpmux: replies and events wait until what they show is durable (S2c) DESIGN.md 3.3d; the tests are the commit before this one. With `store.journal` on, each connection gets an outbound gate (server/durable_gate.rs) between it and its transport: - Every line a connection queues raises the gate's `needs` to the last acknowledged LSN (`Store::durability().acked`, set in the journal store's append before the hub publishes anything of the record), except the lines of a `Stream` record's event (agent output, DESIGN C4): the fan-out delivers those in a thread-local stream scope. - The gate's task takes every queued line at once, reads `needs`, waits (async) until the journal's durable LSN reaches it, then sends the batch in order: one wait per batch, as the journal's group commit. - The durable LSN is a `tokio::sync::watch` fed by `Journal::on_durable`. The listener owns a guard: a writer that ends while the store is not being dropped (a panic on its thread) asks the daemon to stop (`Store::stop_reason`, decision 9.8), and the watch closes, so each gate drops what it holds and ends, which closes its connection. - Without the journal, `Store::durability()` is None: no gate, and `Conn::raise_gate` does nothing. Moves, so server/mod.rs and store.rs stay under their size on the base: the outbound queue helpers to server/outbound.rs, `open_with_journal` to the journal store, and `streamed` to store/durability.rs. * acpmux: the journal lock wait's deadline without an unchecked Instant addition `Instant + Duration` can panic on overflow (clippy::arithmetic_side_effects, tracked by the panic ratchet). The wait now has no deadline only in the case that cannot happen (an Instant 5 s ahead overflowing), where it stops waiting at once. * acpmux: a failed log sync stops the real daemon for a restart, test (red) Lead rule 2026-11-04 (3): a daemon-level test of the store's "stop for a restart" before S2c merges, red first. With store.journal on and every log sync failing (a debug build's ACPMUX_TEST_FAIL_LOG_SYNC; the checkpoint every 200 ms through ACPMUX_TEST_JOURNAL_CHECKPOINT_MS), a rename of a chat on disk makes its log dirty, the checkpoint's sync fails and freezes it, and the store asks the daemon to stop. The test asserts: the WebSocket client gets the restart close (1012), the socket client's pending `_acpmux/wait` ends with the connection (no hang), and the daemon exits with status 74. Red here: the daemon stops and sends 1012, but exits 0 (no exit status says that it stopped to be repaired). This commit adds only the two debug-build seams (release builds have none) and the test. * acpmux: a daemon its store stopped for a restart exits with status 74 The test is the commit before this one. `daemon::run` returned Ok and the process exited 0 for every stop, so a supervisor or a person could not tell a stop to be repaired from a normal one; with systemd `Restart=on-failure` it would not even start again. Now the store's stop (`store_asks_to_stop`: a failed log sync froze the event journal's checkpoint, or the journal's writer ended) marks itself, sets the restart handoff (WebSocket clients get 1012, as before), and the process exits with `daemon::EXIT_REPAIR` (74, EX_IOERR). The units prevent only 75, so launchd KeepAlive and systemd Restart=always start it again; the app does not read the daemon's exit status. README: store.journal names the status. * acpmux: the restart test's request in flight goes through the acpmux client Lead rule (3): the client sees a typed error, not a hang. The socket client is now the acpmux client library (acpmux::client::Client, as the CLI uses it), and the test asserts what the CLI makes of the request that the stop ends: `daemon_closed`, retryable (cli::errors::classify). The first red (fleet 41f2d037, hq run 41804640229, at 80c5fb2b7f8) failed only on the exit status (0, wanted 74), with 1012 and the end of the connection passing. * acpmux: the daemon's disk tier with the journal on, tests (red) Lead rule 2026-11-04 (2): a reply the person waits for never waits on a throttled disk sync. The fleet A/B (STATUS 11554) found that at IOPOL_UTILITY an acknowledged journal append waits about 16 ms p50 under another process's disk stress (0.7 to 1.1 ms at the default tier). - the_daemon_with_the_journal_keeps_the_default_disk_tier: with store.journal on, the daemon process keeps the default tier; without it, utility as before (each case in a child of the test binary). - a_background_thread_yields_its_disk_work: a dedicated background thread (the checkpointer) works at utility; a thread beside it does not. The API they use is stubbed here (`qos::enter_daemon` ignores the journal; `qos::background_thread` does nothing), so both fail on macOS. * acpmux: the restart test borrows no &str twice (clippy needless_borrows_for_generic_args) Found by fleet 7f8afd9d (hq run: the S2c server step at 62b4f68ef8b); the step's other clippy red is the base red in link_push_bench.rs. * acpmux: with the journal on, the daemon keeps the default disk tier (S2c) The tests are 994f74119fb. Lead rule 2026-11-04: a reply the person waits for never waits on a throttled disk sync. Measured on two owned Macs (STATUS 11554): one acknowledged append to durable at IOPOL_UTILITY under another app's disk stress took 16.3-16.7 ms p50 and 37-45 ms p95, against 0.7-1.1 ms p50 and 1.5-1.8 ms p95 at the default tier. - qos::enter_daemon(journal): with store.journal on, the daemon process keeps the default disk tier; without it, utility as before. The switch is read alone from config.json (config::store_journal_enabled), before any thread starts, with no harness discovery. - qos::background_thread: the journal store's checkpointer lowers its own disk I/O to utility for its whole life (a thread that does nothing else: a thread-scope policy drops it out of QoS for good). * acpmux: an agent chunk is a Stream record, test (red) The hub's tap logs an ACP session/update under its sessionUpdate kind (hub/tap.rs: `kind = su.to_owned()`), so a record's kind is `agent_message_chunk`, `tool_call`, ... and never `session/update`; the message's method is. The journal store's `streamed` checked the kind, so every agent chunk was journaled as Ack: a sync per batch at the token rate, and in S2c each token's event waited for that sync (the S2c I/O A/B: STRESS token to client 2.2 ms -> 20.3 ms p50 with the journal on). Red here: the chunk reads back as Ack. * acpmux: an agent chunk is a Stream record (the record's message names session/update) The test is the commit before this one. `streamed` now reads the logged message's method: the hub's tap logs a session/update under its sessionUpdate kind (hub/tap.rs), so the kind never said it. Agent output is again Stream (synced with the next acknowledged batch or a second later), as DESIGN.md 3.3b says. * acpmux: review 2 of #34901, tests (red): a crash model that follows real log syncs The logs' OS crash model now follows the syncs that ran (store/sync_model.rs, tests only): a segment keeps its length at its last sync that counts; a failed sync loses what was written since the one before, and a later sync does not bring it back; a cut ends a loss it removes. Every log segment sync goes through log_sync::sync_segment (same calls as before), where the model and FAIL_NEXT_LOG_SYNC hook in. Red tests, one per finding: - 1a an_os_crash_after_a_restart_from_a_kill_keeps_every_acked_line: the restart's first checkpoint retires records whose log lines no sync made durable; expected [1..5], the head gives [1]. - 1b an_os_crash_after_a_restart_from_a_failed_sync_keeps_every_acked_line: the same after a frozen checkpoint and a restart; expected [1..5]. - 2 a_hole_that_starts_inside_a_line_is_repaired (the reviewer's test): a torn line is read as line 6 of a log that moved on; expected [1..9], the head gives [1, 2, 3, 4, 5, 9]. - 3 a_failed_sync_in_flush_session_freezes_the_checkpoint: a failed sync in flush_session does not freeze the checkpoint. * acpmux: review 2 of #34901: a restart writes the replayed lines again; a torn line is not a line; flush_session freezes Finding 1 (high): the repair now writes every merged line again from the repair start, even when the log already has them all. A killed daemon's log writes may be in the page cache only, and after a failed sync the kernel may have dropped pages that still read back and that a later sync reports as written; the first checkpoint after the open then syncs bytes this process wrote before it retires their records. The lines only the log has from the repair start are journaled and made durable first, as before a cut. The work is bounded by the checkpoint interval. Finding 2: a piece of the log is a line only when it parses as one JSON value (piece_seq). A hole that starts inside a line keeps that line's {"seq":N, head and runs into a later line's tail; it was read as a line of a log that moved on, and the chat's journaled lines went aside. Now its bytes go and the lines are written again. Finding 3: a failed sync in flush_session freezes the checkpoint (its file sync is a SyncFailed now), as a failed sync in a checkpoint or at a log's close does. Nits: cut_at syncs events/ after its removals (and the kind-index folder after its copies go), so a removed segment cannot come back beside the lines written again; the README says that with the switch on an event line over 64 MiB is refused and its log does not get it. Tests: the four red tests of d2c63860ece. * acpmux: chat metadata through the event journal, tests (red, R-DURABILITY S3a) DESIGN.md 3.3e. Red on S2c: - last_seq_never_passes_the_log_after_an_os_crash (E13): session.json is synced at each save while the agent's lines wait for the journal's next sync, so after an OS crash the metadata names lines the log lost. - a_chat_made_after_the_checkpoint_survives_an_os_crash_that_loses_its_folder: nothing syncs the sessions folder when a chat's folder is made, and the replay never makes a folder again, so an acknowledged new chat is lost. - a_metadata_change_is_written_to_session_json_by_the_checkpoint: today each change writes and syncs session.json on the request path. Guards (green before and after): a chat made before the checkpoint and deleted is not made again; the replay never writes an older image over session.json. * acpmux: chat metadata through the event journal (R-DURABILITY S3a) DESIGN.md 3.3e, behind `store.journal` (default off). A metadata change is a `meta` record (kind 4: chat id, a `created` flag, the whole image) instead of a session.json write with two syncs on the request path; the reply waits for the record through S2c's gate, one group commit for a burst. - Overlay: each chat's newest image until a checkpoint writes session.json; load, load_startup and list read it first (list under the projection lock, so a checkpoint cannot drop an entry between its two reads). - Checkpoint: waits on its own thread until every record through its LSN is durable, then writes each image it read (session.json carries the image's LSN as `journalLsn`), then the logs and the sessions folder, as before. An image so names only lines the journal holds (E13). A chat deleted or made again meanwhile is passed (its overlay entry is gone or new). - Replay: per chat after its last `forget`, the newest image, written when newer than the file's `journalLsn`. A missing folder is made again only for a chat a replayed `created` record made; a chat made before the checkpoint whose folder is gone keeps its image in journal-unapplied/. - flush_session also waits until the chat's records are durable. - Delete drops the overlay entry with its `forget`, and removes the folder under the folders lock a checkpoint's write also takes. Tests: the 5 of 0505e9bee04 (3 red there). * acpmux: S3a, tests (red): folders handed out of the store Code outside the store reads and moves chat folders directly (transfer_in puts the live folder aside and installs a transferred chat; session_dir hands the folder to the transfer, the session files view and others). With the metadata in the overlay until a checkpoint: - a_folder_replaced_outside_the_store_keeps_its_own_metadata: a read, the checkpoint and the replay must not put the old folder's image in the new one. - session_dir_hands_out_a_folder_whose_metadata_is_written. - flush_session_writes_the_chats_metadata (the trait says log and metadata). - the_checkpoint_never_makes_a_folder_removed_outside_the_store_again. Red on 71bc09a1e63 (S3a's first implementation). * acpmux: S3a red tests: a replay after a folder swap; flush_session's chat has a log (as its callers' chats do) * acpmux: S3a: each metadata image is bound to its chat folder; session_dir and flush_session write it first Code outside the store reads and moves chat folders directly (a transfer puts the live folder aside and installs a transferred chat under the same id). So a `meta` record now carries its folder's identity (file id and birth time): - reads use an overlay image only while the folder at the path is its folder; - the checkpoint writes an image only into its own folder, never makes a missing one, and keeps the image of a removed or replaced folder in journal-unapplied/; - the replay writes an image only into its own folder (or a folder a replayed `created` record made), else keeps it aside. `session_dir` (the transfer, the session files view) and `flush_session` (the trait: "log and metadata to stable storage") write the chat's image first, once its record is durable. The list reads the overlay before the files (a checkpoint writes a file before it drops the entry), so it needs no lock. Tests: the 5 of 0280ac37f5e and 0e617489731. * acpmux: S2 follow-ups from review 3 of #34901, tests (red) - a_chat_whose_repair_failed_is_repaired_at_the_next_start (follow-up 1): the open's checkpoint retires the records of a chat whose repair failed, so no later start puts its lines back (they are only in journal-unapplied/). - a_hole_that_ends_at_a_line_start_keeps_that_line (follow-up 2): a torn line, zeros, then a whole line only the log holds: the piece does not parse and the whole line is dropped. - an_idle_store_does_not_wake_its_checkpointer (follow-up 4, cmux-ad m9): the checkpointer wakes every period with nothing to do. Test seams only (no behavior change): CHECKPOINTER_WAKES and TEST_CHECKPOINT_EVERY_MS. * acpmux: S2 follow-ups from review 3 of #34901: a failed repair holds its records; a hole's last zero splits; an idle checkpointer sleeps - Follow-up 1: a chat whose repair fails at the open (its log cannot be read or written now) keeps its lines in journal-unapplied/ as before, and the checkpoint now stays below its first journaled LSN (`Order::held`), so the next start repairs it again. Bounded: past 256 MiB of journal after it, the hold goes and the lines stay only in journal-unapplied/ (an error says so). - Follow-up 2: a piece splits at its last zero byte (a whole JSON line holds none), so a hole that ends at a line start, after a torn line, keeps that line (it mattered for a line only the log holds). - Follow-up 4 (cmux-ad m9): with nothing to checkpoint, the checkpointer sleeps until a change wakes it (`Inner::touched`, one atomic read per append), then checkpoints one period later; before, it woke every 30 s. Tests: the 3 of d147057bf85. * acpmux: a metadata change logs its line before its metadata, tests (red, R-DURABILITY S3b) DESIGN.md 3.3e, E6: with the event journal, records survive a crash as a prefix in append order. So a change whose line reaches the store before any metadata image that holds it is never, after a crash, in the chat's metadata without its line in the log. A recording store checks the order of the store calls: - a_tag_change_logs_its_line_before_its_metadata: red (tags saves first). - a_rules_change_logs_its_line_before_its_metadata: red (rules saves first). - a_rename_logs_its_line_before_its_metadata: green (a guard). * acpmux: the qos journal tests module is macOS-only (its Linux build held only an unused import: clippy red, review of #35400 F0) * acpmux: review of #35400 F1 and F2, tests (red) - a_read_that_shows_a_line_covers_its_record (F1): a read sees a log line between its log write and its record (the appends' lock held there), and the reply built from it goes out with no wait for that record. - a_journal_that_syncs_nothing_asks_the_daemon_to_stop (F2): with a journal that makes nothing durable, every reply waits without a limit and nothing asks the daemon to stop. Test seams only: PAUSE_AFTER_LOG_WRITE, TEST_STALL_MS. * acpmux: review of #35400 F1, the streaming read, test (red): session/load's replay sends a line before its record * acpmux: the test seam PAUSE_AFTER_LOG_WRITE is reachable from the server tests * acpmux: review of #35400: a read covers the records of the lines it shows; a stalled journal stops the daemon F1: the append writes a line to the log, then appends its record and sets the acknowledged LSN, all under the appends' lock; reads take no lock. A read could show a line before its record existed, and the reply built from it went out at once. Now every journal-store read of the logs ends with a fence on the appends' lock (`Inner::fenced`): a line the read saw has its record appended and its LSN acknowledged when the read returns, so the reply's gate covers it. The one read that sends as it reads (`Hub::scan_events`, session/load's replay) stops at the chat's last_seq read first. F2: a journal writer that keeps failing (it retries without end) held every reply on every connection with no limit. A connection's gate now waits at most 10 s (`STALL_LIMIT`); past it, `Durability::stalled` asks the daemon to stop for a restart, as a failed log sync does. Tests: the 3 of 3916f84dd68 and c553a19fd8b. * acpmux: S3b red tests: the recording store implements delete (a required Store method) * threading-model: the acpmux event journal's threads and bounds (writer, checkpointer, per-connection gate; review of #35400) * acpmux: the stall test's journal syncs take 3 s, not an hour (a sync under way kept its delay and hung the test's end) * acpmux: a tags or rules change logs its line before its metadata (R-DURABILITY S3b) DESIGN.md 3.3e (revised): with the event journal, records survive a crash as a prefix in append order, so a change whose line reaches the store before any metadata image that holds it is never, after a crash, in the chat's metadata without its line in the log (E6). `renamed`, `policy` and `mode` already did this; `tags` and `rules` saved first. Their reply still waits for the save. Tests: the 3 of 5bb3a27a2da and 7a4f02a56f3 (red at 7a4f02a56f3, fleet 88309e91: tags ["save:tagged", "line:tags"]; rules ["line:permission_chat_allowance", "save:ruled", "line:rules"]). * acpmux: the idle checkpointer's period, test (red): a change seen by its second look must not start a checkpoint at once - a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period: the checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies when it gets the timer lock. The second look saw the change, so the checkpointer is in its period wait by then, and the notify ends that wait: a checkpoint runs at once on the checkpointer's thread. This is the cause of a_failed_log_sync_freezes_the_checkpoint failing in the full store:: run of fleet 0577ac62 (that early checkpoint took the line whose sync the test fails, on a thread where the failure is not set). The test holds both threads at the two points, so the order is exact. Test seams only (no behavior change): the checkpointer's test seams are now by the store's home (period, wake count, the two holds), so tests that run as threads of one process under cargo test do not see each other's. * acpmux: the idle checkpointer's period ends only on its own state, not on a wake meant for its idle wait The checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies once it gets the timer lock. Its change shows in the second look, so the checkpointer has gone to its period wait by then, and that notify ended the period at once: a checkpoint ran right away on the checkpointer's thread (a burst no longer shared one checkpoint), and a spurious wakeup could do the same. In the full store:: run of fleet 0577ac62 such a checkpoint took the line whose sync a_failed_log_sync_freezes_the_checkpoint fails, so that test's own flush had nothing to do and returned Ok. The period wait now ends only on the period, `wanted` or `stop` (`wait_timeout_while`): each of the checkpointer's two waits ends only on the state its waker sets before it notifies under the timer lock. No sleep, no retry. The idle wait is unchanged. Test: a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period (edca3b4823d). * acpmux: S3a review (#35483) red tests: switch-off metadata and deletes, a late settle, a failed session.json folder sync, a purge after the checkpoint F1: metadata written with the switch off survives turning it on again; a chat deleted with the switch off after a failed catch-up is not made again. F2: a late settle never writes an older image over a checkpointed one (the reviewer's test and hold). F3: a failed folder sync after a metadata write freezes the checkpoint. F4: a chat made after the checkpoint and deleted is not made again after a kill. Test seams only; the code is the S3a head merged with feat. * acpmux: S3a review (#35483) red tests: a new chat is acknowledged only once its folder is durable; session_path never waits The root of F4 and of F1's delete case: the replay makes a chat's folder again, so a chat removed by any path comes back. The fix will never make a folder again, which needs an acknowledged chat's folder to survive an OS crash: the test models what each folder's last sync kept (sync_model) and asserts the gate's durability passes a new chat only after its sessions entry is synced. P3 1 of the review: a caller that needs only the path never waits for the journal (Store::session_path, a stub with session_dir's behavior here). * acpmux: S3a review (#35483) fixes: the replay never makes a folder again; switch-off changes are kept; no late or unsynced metadata write F1: each meta record carries the stamp (file id, modification time, length) of the session.json its save saw. The replay writes an image over a file the journal did not write only when one of the chat's records saw that file; otherwise the file moved on without the journal (a run with store.journal off whose catch-up did not replay, or an older build) and the image goes to journal-unapplied/, as S2 does for a log. F1 (deletes) and F4 at the root: the replay never makes a chat's folder again. A save that made a folder notes its record's LSN; the new acpmux-journal-folders thread (the daemon's disk tier) syncs sessions/ at once, and the durable LSN the S2c gate waits for stays below that record until then (journal_store/new_folders.rs). So an acknowledged chat's folder survives an OS crash, and a folder that is gone was removed: a purge whose forget a kill took back, a run with the switch off, the transfer recovery. The created flag and the replay's folder remake are gone. F2: write_through does not write a file that holds the same or a newer journalLsn. F3: session.json is written with its own syncs (file, then folder); a failed sync is SyncFailed and freezes the checkpoint; the replay and the kept-aside images return their errors. P3 1: Store::session_path gives the path with no lock, wait or write; an agent's start, the fixed-id check, the temporary-folder sweep, the host probe and the transfer recovery's root use it. The replaced test (a chat made after the checkpoint survives an OS crash that loses its folder) asserted a remake that cannot be right any more; the new test asserts the guarantee that makes the loss impossible, then the crash. Red (7 of 17 journal_meta_tests, each for its reason; the other 10 pass): macOS 5bc434a9 and 624d5df4, Linux a2207ff0 and 638a44ef. * acpmux: review of #35562 red tests: a failed repair holds no other chat's checkpoint; a checkpoint after it writes only durable images F1 (the reviewer's test, renamed): a checkpoint after a failed repair writes only durable metadata images. F2/F3 at the root: a failed repair does not hold the checkpoint of other chats; a retention rewrite of another chat after a failed repair stays after a restart. Guards: the reviewer's two switch-off tests (green here since the S3a fix 0d246364d90: the stamp rule and no folder remake), and a chat deleted and made again after its failed repair gets none of the old chat's lines. * acpmux: review of #35562: a failed repair keeps its chat's lines in a repair file; no hold of the global checkpoint The hold (Order::held, 49d7e5cef83) kept every chat's records past the checkpoint while one chat's repair failed. That let a checkpoint write metadata images whose records were not durable (F1), made each start with the switch off replay the whole window again and undid a retention rewrite (F2), and grew the journal with each restart (F3, the 256 MiB bound counted one run only). Now a chat whose log cannot be read or written at the open keeps its lines in journal-unapplied/<id>.<lsn>.repair.ndjson (a header with the chat id, then the lines, synced). The next open journals them again, waits until they are durable, removes the file and repairs the chat with them as with any journaled line (a line in both is the same bytes, so a file journaled twice after a crash changes nothing). The checkpoint of every chat goes on as before S2's follow-ups. A delete removes its chat's repair file after the folder, so a chat made again under the same id gets none of the old lines. Red at 23870301064 (macOS a8be0684): the 3 fail for their reasons ("the checkpoint stayed at 1 below chat-a's records (the journal ends at 5)"; "the checkpoint wrote an image to session.json before its record was durable"; "the open kept a copy of a rewritten log aside"); the guards pass; 339 of 342. * acpmux: S3a delta review (N1) red tests: a change to a chat whose session.json another journal wrote reaches it A folder moved in from another home (build adoption), checkpointed and replayed, and a journal made again from nothing: the other journal's journalLsn must not hide a change made here. The first test is the reviewer's (fleet 4d0d98bc and 071ad7c4 at e944a7c21de). * acpmux: S3a delta review (N1): journalLsn is bound to its journal's id; a file another journal wrote follows the stamp rule session.json now carries journalId (16 hex digits, a string) beside journalLsn. The checkpoint's "held" check (F2) and the replay's rule compare an LSN only for a file with this journal's id: a file from another journal (a chat folder build adoption moved in from another home, or a home whose journal was made again) counts as one this journal did not write, so the stamp rule decides, as for a run with the switch off. cmux-journal: Journal::journal_id (the id already in every segment header; kept across opens and retirements), with a unit test. Red at 7c8cb957745: fleet c29b659d (Linux) and 47570494 (macOS) running; the reviewer's red at e944a7c21de: 4d0d98bc (macOS) and 071ad7c4 (Linux).
lawrencecchen
added a commit
that referenced
this pull request
Nov 5, 2026
…lastSeq after a hard stop (review of #35400, N1) (#36022) * acpmux: a replay after a hard stop sends the log's tail past the saved lastSeq, test (red; review of #35400 N1) Review of #35400 (S2c), finding N1. S2c capped Hub::scan_events (session/load's streaming replay) at the chat's last_seq, so a replay never sends a line before its journal record. After a hard stop the log can be ahead of the saved lastSeq (metadata is saved less often than lines). A daemon reads a chat's log only after its ready line (load_logs) or at the chat's first append, so the first replay of a chat before then leaves out the log's newest lines, and they are never sent live either. With the journal switch off too: the cap is the default path. a_replay_after_a_hard_stop_sends_the_tail_past_the_saved_last_seq: three lines in the log, lastSeq saved at 1, a new daemon's hub built as the daemon builds it (Hub::new_deferred: logs read later). The replay must send 1, 2, 3. Red on feat: it sends only 1. This is the test of the earlier branch (rewrite/r-durability-s2c-n1, red at its scratch 51c1e9b, fleet 12c48e79), rebuilt on feat after S2c landed as one squash commit. * acpmux: session/load's replay counts a chat's log tail before it stops at lastSeq (review of #35400 N1) Hub::scan_events caps the replay at the chat's last_seq, so it never sends a line before its record. Before the cap it now counts the log's tail past the saved lastSeq (Hub::catch_up), once per chat and only when the chat is not caught up yet: no append ran on it in this daemon, so the catch-up waits only for the previous run's tail. A caught-up chat takes no lock, so a replay never waits for an append under way (a_replay_never_sends_a_line_before_its_record stays green; an earlier version that caught up on every replay broke it, fleet 7e8edbc5). Green: a_replay_after_a_hard_stop_sends_the_tail_past_the_saved_last_seq (red at 35cb32f). * acpmux: a replay during another catch-up of the chat, test (red; review of #36022, finding 1) Review of #36022 at d43d664, finding 1 (P2): catch_up_locked marks a chat caught up (caught_up.swap(true)) before it counts the tail, and the new path in scan_events reads that mark without the chat's append lock. A replay that comes while another catch-up of the chat counts (a first replay, the background log pass or the chat's first append) reads the mark, skips its catch-up and stops at the saved lastSeq, leaving out the tail. a_replay_during_another_catch_up_sends_the_tail_past_the_saved_last_seq is the reviewer's scenario (its repro 99f0c20, red on fleet 8c5881e0), with handshakes instead of two 300 ms sleeps. New test seams in hub/log_load.rs, one-shot notes a thread sends at a point: NOTE_IN_CATCH_UP (a catch-up past its check, before it counts), NOTE_BEFORE_CATCH_UP_LOCK and NOTE_BEFORE_REPLAY_SCAN. The first catch-up's read waits at the journal's read fence while another chat's append holds the appends' lock (PAUSE_AFTER_LOG_WRITE). The append is let go only after the second replay says where it stopped. The test also checks first that the chat's log is not loaded (review P3 3). Red here: the second replay reads the log having skipped its catch-up, and gets [1]. * acpmux: a chat is marked caught up only once its tail is counted (review of #36022, finding 1) catch_up_locked set caught_up with a swap before it counted the tail. Every caller holds the chat's append lock, so that only kept the count to one; but scan_events reads the mark without the lock (the N1 fix), and a replay that read it during another catch-up skipped its own and stopped at the saved lastSeq. Now catch_up_locked checks the mark and sets it after the count: the count still runs once (under the lock), and a replay that reads true knows last_seq counts the tail. A replay that reads false waits for the chat's append lock, so for the catch-up under way, then finds the mark set. The reviewer's fix (7a3f0bb). Session::caught_up's doc says so. Green: a_replay_during_another_catch_up_sends_the_tail_past_the_saved_last_seq (red at e6be8e1).
lawrencecchen
added a commit
that referenced
this pull request
Nov 5, 2026
…ion never waits for a checkpoint in flight (R-DURABILITY, follow-up 4 of review 3 of #34901) (#35866) * cmux-journal: one append-only journal per daemon, with group commit and ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner). * cmux-journal: type a test range; drop an unread segment field (fleet clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71e3f9 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red. * cmux-journal: a failed F_FULLFSYNC is an error, not a plain fsync; no segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee. * cmux-journal: the past-the-end test reads back the storage it wrote to Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c87b8d: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5c53e. * cmux-journal: a sync a signal interrupts is made again (macOS) fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r). * cmux-journal: crash sim of write-back errors and crash in recovery; S1 review regressions (red) The S1 review (FIX at 89293fb15f39) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open * cmux-journal: the F2 regression damages the header of the last of several segments (red) The reviewer's F2 scenario has two segments: S1 holds 1..10, S2 (the last) holds 11..20, and S2's header is damaged. The test now rotates segments (300-byte segments, 20 acked records) and damages the last one's header, so the report must say lost_through 20 and the next append must take 21. On the code before the fix the moved segment's LSNs read as 0, so the next append reuses one. * cmux-journal: the crash sim forces the F4 order (failed sync, kill, OS crash) (red) On the code before the S1 fix, the crash-sim step was green (fleet 184129d8 at 263da86f223): 6 of 7 runs passed, and the mixed run failed only seed 140 in the full run and passed when it ran alone. The random faults reached the order the review found (F4) in 1 of 800 runs. So these tests were no red/green proof. The order: the sync of a write that spans pages fails and the disk gets none of them (Linux keeps them clean in the cache, so a later sync reports success); the process is killed in the writer's backoff; the next open reads the failed segment from the cache, replays its records and goes on; an OS crash or a power loss then takes those pages, and records an open already replayed are gone. - sim.rs: `fail_next_sync(pages, kind)` fails the next sync that has at least that many dirty pages, and the disk gets none of them. `crash_after_failure(n)` makes the n-th operation after the next injected failure the crash point. - crash_sim_tests.rs: a new run forces the order on every seed (a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record): epoch 0 aims the fault, the kill is a process crash at the writer's retry, the open after it does not crash, epoch 1 ends in an OS crash, and a third open checks that the records the second open replayed are all there. In the random runs, one epoch in four aims the fault, with the crash 1 to 3 operations after the failed sync and a crash of any kind after it. - A sim test checks the aimed fault itself (the_sim_fails_the_aimed_sync_and_crashes_in_the_backoff). Expected on this commit (old journal code): the forced run fails every seed; the mixed run fails most runs. The fix commit follows. * cmux-journal: fix the S1 review findings F1-F7 (markers, recovery copy, LSN floor, writer end) Review FIX at 89293fb15f39 (PR #33919 comment 6610468202). The regression tests are the two commits before this one. - Sync markers (format.rs): an entry with flags 1 and no payload, whose LSN field says "every record of this segment before me is synced". The writer appends one after each successful sync that covered records (after the durable listeners, so no reply waits for it), and syncs it fully at a seal and at a rotation. - F4, recovery (step 4 and 6): the last segment is trusted only up to its last marker. Its records after the marker may sit in pages a failed sync dropped (a later sync on another descriptor reports success without them), so recovery writes them again into a new segment: header, records and a marker, fully synced under `<name>.seg.new`, renamed into place, folder synced. The journal always goes on in a segment this open made, never in one an earlier process wrote. A restart in the writer's backoff after a failed sync is then safe. - F1, quarantine: every LSN a quarantined segment holds counts in the LSN floor at every open, and a quarantine that no new segment followed (a crash in the middle of an earlier recovery) is reported as damage again. The highest LSN of every moved segment is read before the first rename. - F2, highest LSN: a segment whose header is damaged is read from its first record (each record's checksum covers its LSN), then the bytes after the first bad entry are scanned for records and markers that still check out. The same scan runs after a cut in the last segment: a marker there is media damage (reported), records there are a torn write (`torn_lsns`); their LSNs are never used again. - F3, writer end: a guard closes the journal when the writer thread ends, also by a panic: appends fail with Closed, waits end with WriterEnded, the listeners are dropped (so a channel a held reply waits on closes), and stats.failing says why. The crate does not abort the process; the daemon owns that policy. - F6: recovery closes each segment after its header is read and opens one again only to read it. (Retirement of compacted segments is an S2 API, before the first consumer.) - F5: recovery syncs the parent folder after it makes the journal folder. - F7: format.rs and lib.rs now say what the code does: a failed write is cut back and retried in the same segment; the cut removes only bytes never synced and never acknowledged; nothing shortens a segment past a synced record. Docs: journal.rs says when open's replayed records are durable (on Ok) and what a stopped writer does. * cmux-journal: split the crash sim's machine model into sim/machine.rs (review G5) sim.rs had 924 lines and 63 functions, over the 60-function limit for a new Rust file (the god-file check warned "not in baseline: split it"). The machine model (each file's page cache and disk, the ordered log of names, and what an OS crash or a power loss leaves) moves to sim/machine.rs (297 lines, 19 functions); the process view and the Storage implementation stay in sim.rs (639 lines, 30 functions). No behavior change: the items move as they are, made visible to the sim module. * cmux-journal: re-review regressions G1-G4 and page eviction in the crash sim (red) Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993). Tests and sim only; the fix follows. - sim: the pages a failed sync dropped stay in the cache until an eviction; `evict_dropped_pages_after_read(true)` makes each read evict the dropped pages it read, so a second read of them sees the disk's bytes. - G1 recovery_copies_the_records_it_checked_when_the_cache_drops_them: a sync fails and drops record 2's pages, a kill in the backoff, the next open reads record 2 from the cache (replayed) and the pages are evicted after that read; then an OS crash. Record 2 and the record 3 acked after the restart must survive. On this code recovery's second, unchecked read stages zeros under a vouching marker, and the last open keeps only record 1 and reports damage. - G2: the F1 regression asserts that every open after the damage reports the loss (lost_through 40). On this code a crash at the open's last step (the folder sync after the new segment's rename) loses the report for good. - G3 a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system): a journal in a symlinked state folder, and a journal folder that is a symlink. On this code the parent folder's sync opens with O_NOFOLLOW (ELOOP). - G4 an_open_after_a_finished_quarantine_reads_no_quarantined_segment: on this code every open reads every quarantined segment in full. * cmux-journal: fix the re-review findings G1-G4 and G6 Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993); the regression tests are the commit before this one (red: fleet de35f6d0). - G1: recovery writes again exactly the bytes its checked read returned. The last segment's entries after its last marker are kept from the checked read (header and payload) and staged from memory; the second, unchecked read of the file is gone. A page that a failed sync dropped can be evicted between two reads, and the second read then returned the disk's zeros, which the copy kept under a vouching marker. - G2: the damage an open finds is written to `DAMAGE` (whole, fully synced, renamed into place, folder synced) before the new segment that hides it from later opens. Every open reports the recorded damage first, until the daemon calls the new `Journal::clear_damage`, which moves the record into `quarantine/` (kept, never deleted). - G3: a folder sync follows a symlink in its path (no O_NOFOLLOW): a state folder, or the journal folder itself, may be a symlink. The sync is for the folder that holds the entries. - G4: the quarantined segments are read for their LSNs only while no live segment is numbered above them (an unfinished quarantine); once one is, its first LSN already carries their floor. - G6: `checked_div` in the scan of the bytes after a cut (the crate's one tracked arithmetic hit). - G5 (the sim split) is the commit before the regressions. * cmux-journal: third-review regressions G7 (A, B) and N1 (red) Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018). Tests only. - G7 A a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn: after a finished quarantine, one byte of the empty new segment's header changes. On this code the next open skips the quarantine scan (the listing counts the damaged segment) and the next append reuses a quarantined LSN. - G7 B a_damaged_newest_segment_after_a_quarantine_reports_its_records: the newest segment holds acked 41..43, and its header and its first record are damaged. On this code the moved segment is scanned with a base far below 41, so the loss is not reported through 43 and the next append is not 44. - N1: the F1 regression also asserts that an open reports each loss once (no reported LSN range lies inside another). On this code a crash after the quarantine's renames makes the next open report the same loss twice. * cmux-journal: fix the third-review findings G7, N1, N2; split recovery.rs Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018); the regression tests are the commit before this one. - G7: the quarantine scan is skipped only when a segment whose header checked out is numbered above every quarantined segment (the folder listing also counted a newest segment with a damaged header, which carries no floor). The scans start from the running floor: `quarantined_floor` and `quarantine` pass the highest LSN counted so far as the base of the next file's scan, so a quarantined file with no readable header and no readable first record is scanned for the LSNs right above what is counted. - The "unfinished quarantine" report is made only when the quarantine holds LSNs nothing else counted (an old quarantine behind a segment that a crash tore at its creation holds none, and a crash is not damage). Its reason now also names a damaged newest header. - N1: a loss whose LSN range lies inside a recorded one is the same loss found again after a crash: it is neither recorded nor reported twice. - N2: recovery.rs's docs name `DAMAGE`; two doc lines over 100 columns are wrapped (lib.rs, sim.rs). - recovery.rs was at 987 of its 1000 lines: the damage record moves to recovery/damage.rs, the entry reader and the LSN scans to recovery/scan.rs (moved as they are). * cmux-journal: report each loss once, also when one open finds it twice (N1) The red run of the N1 regression (fleet 27d011ca, op 64 of the F1 test) shows the case the previous commit did not cover: a crash after the quarantine's renames and before DAMAGE is written; the next open finds both the damaged synced record of the last segment (7..14) and the unfinished quarantine (7..40), and reported both. Now every loss inside another (recorded or found by this open) is dropped, the larger range kept; DAMAGE is rewritten only when its set of losses changed. * cmux-journal: fourth-review regressions G8, N3, N4 (red) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report. * cmux-journal: the quarantine's floor in its file names; DAMAGE before the moves (G8, N3, N4) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format. * cmux-journal: sort the quarantine list with sort_by_key (clippy unnecessary_sort_by) Fleet d9f22bed at b589044b852: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`). * cmux-journal: Journal::on_written, a listener per written batch (for S2's projector) R-DURABILITY S2 (acpmux event logs through the journal) projects each chat's log line into its ndjson file once the journal has WRITTEN the batch that holds it, so readers see stream lines at once. The listener runs on the writer thread after the write and before the sync, once per batch that moved the written LSN: a batch written again into a new segment after a failed sync is not heard twice. When the writer stops, these listeners are dropped too. Test: heard while the batch's sync still runs, once. Port of 145faa6faa8 (branch rewrite/r-durability-s2-acpmux-logs) onto the S1 review fixes. * cmux-journal: Journal::retire_through, the one delete (segments a compaction point covers) R-DURABILITY S2: a daemon's projector compacts the journal into its own files (acpmux: the chat ndjson logs, then a synced checkpoint). Without a delete the segment count only grows (review finding F6). `retire_through(compacted)` deletes the segments that hold no record after the caller's compaction point: a segment goes when the next segment's "previous last LSN" is at or before it, so the newest segment never goes. The point is capped at the durable LSN. The deletes and a folder sync run on the caller's thread (the projector, never a request path). A delete that fails leaves its segment live for a later call. Why a crash during a retirement is safe: the caller reopens from a point at or after `compacted`, and recovery reads nothing from a segment the point covers (it skips it on its next segment's header), so any subset of the deletes may reach the disk. A point older than the retired segments is reported as Damage::Missing. Quarantined segments are never deleted. - Storage::remove_file (FsStorage: std::fs::remove_file; SimStorage: one more change in the ordered log of names, durable at the folder sync). - The live segment list: recovery's, then each segment the writer makes. - JournalStats::segments_retired. - Tests: only covered segments go, and a second call deletes none; an OS crash after it keeps every record after the point and an older point is reported; a crash at every operation of a retirement (process, OS crash, power loss) keeps every record after the point and never reuses an LSN; the newest segment stays; a point in the middle of what this writer wrote. This adds a delete of journal data, so its PR is risk:high. * acpmux: move MemoryStore and the store settings into their own files store.rs (2139 lines, 120 fns) and config.rs (1909 lines, 97 fns) are over the god-file limit, and the journal store (R-DURABILITY S2) must not grow them. MemoryStore moves to store/memory.rs and StoreMode, StoreConfig and its defaults to config/store_config.rs; both stay exported where they were (crate::store::MemoryStore, crate::config::{StoreConfig, StoreMode}). No behavior change: the code moves as it is. * acpmux: the event log's journal switch and its crash tests (red) R-DURABILITY S2 (DESIGN.md 3.3b). Today no event-log append is synced: a line a client was told about is on the disk only after the next flush (shutdown), so an OS crash or a power loss takes back every line since (E12). This commit adds only the switch, a seam and the tests; the journal store follows. - `store.journal` (StoreConfig::journal, default false). - `Store::wait_durable(timeout)`: tests and tools wait until every appended record is durable. The default returns at once (the Local and Memory stores never sync an append), which is today's behavior. - `store::open_with_journal` (tests): the store with the journal on a given `cmux_journal::Storage`. Here it ignores the journal and opens LocalStore. - acpmux depends on cmux-journal (dev: with the `sim` feature). - store/journal_store_tests.rs, red on this commit: - a_line_the_store_says_is_durable_survives_an_os_crash (lines 2..20 are lost: nothing synced them); - a_deleted_chat_stays_deleted_after_a_crash and a_line_the_log_refused_is_never_written_later pass here, and must keep passing with the journal store (no replay may bring them back); - a_kill_keeps_every_line_once passes here and must keep passing (a replay after a kill must write no line twice). * acpmux: the event log through one journal, behind `store.journal` (R-DURABILITY S2) DESIGN.md 3.3b. With `store.journal` on, each event-log append is one record in `<home>/journal/` (cmux-journal: group commit, one sync per batch; a power loss takes back at most the last second, always a suffix), then the same line goes to the chat's ndjson as before. Readers, segments and file formats do not change. Default off. - store/journal_store.rs (JournalStore over LocalStore): - append: under one lock, the journal record (a memory copy, no I/O), then the log line; a line the log refuses gets a `cancel` record. Agent stream lines (`in` session/update, claude.*, codex.*) are Class::Stream, every other line Class::Ack. - checkpoint (`<home>/journal-projected.json`): every 30 s while the journal grew, past 64 MiB, at flush (shutdown) and before a retention commit. It syncs each log written since (LocalStore::sync_log, with no lock held while the disk works) and the sessions folder, writes the LSN (temp file, sync, rename, folder sync, every error checked), then retires the journal before it. A log close that failed its sync freezes the checkpoint until a restart (LocalStore::sync_failed). - recovery: the records after the checkpoint; for each chat, the log is cut right after its last readable line below the first journaled seq (later segments removed, the boundary one shortened, their kind-index copies dropped), and the journaled lines are written again in order. `forget` deletes a chat again; `cancel` keeps a refused line out. - delete: `forget` record, folder removal, then a bounded wait until the record is durable. - seal: the logs refuse writes and the journal syncs fully. - LocalStore: `append` is serialize + `append_line`; `sync_log`; a failed sync at a log's close is noted (`sync_failed`). - store::open opens the JournalStore when `store.journal` is on. Not yet (S2c): replies and events do not wait for durability; the daemon does not stop when the journal closes; the writer thread keeps the process's Utility I/O policy. * acpmux: journal store tests for the seal and a chat made again after its delete (red) - a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay: a chat is deleted (its forget record), made again under the same id (a transfer back: session.json and new lines), then a kill -9. On this code the replay deletes the folder again and its session.json is gone. - a_line_after_the_seal_is_refused_and_never_written_later: coverage of the seal (it passes here; the fix also closes a race the test cannot force). * acpmux: journal store: the seal under the appends' lock; a replay keeps a chat made again - seal takes the appends' lock: an append lands in both the journal and the logs, or in neither. Before, an append that raced the seal could leave a journal record whose line the sealed logs refused (its cancel then failed on the closed journal), and the next start would write that line. - append and delete check the logs' seal first, under the same lock, and do not touch the journal after it (no error per append at shutdown). - replay deletes a forgotten chat again only when no line follows its forget record: otherwise the chat was made again under the same id, and its folder holds that chat's own files. * cmux-journal: fifth-review regressions G9 (red) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4. * cmux-journal: name a skipped LSN floor in a FLOOR.<lsn> file (G9) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646); the regression tests are the commit before this one. When an open gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which a copy then hides from every later open), the only record of that floor was the header of the empty segment it makes; a damaged header lost it, and DAMAGE is cleared by `clear_damage` (torn records are not damage at all). Now such an open makes the empty file `FLOOR.<lsn>` and syncs its folder before any new segment, and every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine's names and this open's moves say their floors already: no file is made for them. * acpmux: journal store regressions from the S2 review (red) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824). Each test fails on this commit and passes after the fix that follows it: - 1a lines_written_with_the_switch_off_survive_turning_it_on_again - 1b a_kill_while_the_journal_syncs_keeps_the_lines_the_log_got - 1c a_line_flush_session_made_durable_survives_losing_its_record - 2 a_folder_put_back_after_a_delete_is_not_deleted_by_the_replay - 3 a_kill_in_a_delete_leaves_the_chat_whole_or_gone - 4 a_delete_does_not_wait_for_the_journal_sync - 5 a_failed_log_sync_freezes_the_checkpoint, a_checkpoint_that_keeps_failing_asks_the_daemon_to_stop - 6 one_chat_that_cannot_be_repaired_does_not_stop_the_open - 7 the_journal_storage_refuses_a_lost_home - the repair keeps a log's lines after a hole, also through a crash in it: an_os_crash_hole_is_repaired_and_the_lines_after_it_kept, a_crash_inside_a_repair_loses_no_line - nits: an_unreadable_checkpoint_fails_the_open, the_open_waits_for_the_previous_daemon_to_let_go_of_the_journal, a_crashed_checkpoint_temporary_is_removed_at_open, a_crash_right_after_a_checkpoint_reports_no_damage - the catch-up with the switch off: switching_off_after_a_crash_keeps_every_acknowledged_line - cmux-journal: check_append_says_what_append_would_do So they compile here, this commit adds the seams they use, with no behavior: `Store::stop_reason` (default None), `Journal::check_append` (always Ok), `FencedStorage` (never asks its fence), the open report and a test stop point after a repair's cut. * acpmux: the checkpoint freeze test fails a log sync, not a folder open (red) The freeze guards against a failed fsync (a later fsync of the same file may report success for pages the kernel dropped), not against a folder the store cannot open: a chmod 000 folder is an open error, which the fix retries instead of freezing. The test now fails the log sync itself through a test seam, `FAIL_NEXT_LOG_SYNC`. Red on the code before the fix, like the other review regressions (fleet d8e7037d, 1c22175e, 63604b26 at e030f4fe402: 17 of 17 new acpmux tests and the crate's check_append test fail). * acpmux: the journal store repairs logs, never rebuilds them (S2 review) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824); the regression tests are the two commits before this one. DESIGN.md 3.3c. - 1a/1b/1c: recovery no longer cuts a chat's log at its first journaled line and writes back only the journal's lines. It merges the log's lines and the journal's by seq: a line in both must be equal, a line only the log has stays, a line only the journal has goes in its place. Only bytes that do not read (a torn tail, a hole of zeros) are dropped. When the merge must change the log before its end, the lines only the log holds from there are journaled and made durable first, so a crash inside the repair loses no line. A log that moved on without the journal (a line in both that differs, lines out of order) is left as it is, and the journal's lines go to journal-unapplied/. - Append writes the log first, then the record, under one lock, after `Journal::check_append` (new in cmux-journal: the append's refusals, asked before any write). So a record always has its log line, and the `cancel` record is gone. - 2/3/4: the replay never deletes a folder and never makes a missing one again (a forget only drops the chat's earlier journaled lines). Delete appends its forget under the lock and removes the folder off it, with no wait for the record. - 5: a failed fsync in a checkpoint freezes it (a folder that does not open is retried); a frozen checkpoint returns an error, and the store asks the daemon to stop for a restart (`Store::stop_reason`, wired in daemon.rs), whose replay writes the logs again. Ten failed checkpoints in a row ask the same, so the journal past the checkpoint stays bounded. - 6: an error in one chat keeps its journaled lines in journal-unapplied/ and the open goes on. - 7: the journal's storage refuses writes under a lost home (`FencedStorage`, `config::refuse_lost_home`). - With store.journal off, a journal left by a run with it on is replayed, checkpointed and retired first (`catch_up`). - Nits: the checkpoint is never past the journal's durable end (no `Ahead` after a normal crash); damage is cleared after the next checkpoint; a kind-index copy goes before its segment is cut; an unreadable checkpoint fails the open; the open waits up to 5 s for the journal lock; crashed checkpoint temporaries are removed; `on_written` leaves cmux-journal (no user); README lists store.journal. * acpmux: split the journal store under its size budget (a move) journal_store.rs grew to 1085 lines and 71 functions with the review fix (budget 1000 and 60), and the fix grew store.rs and daemon.rs, which are over budget on the base. A pure move, no behavior change: - store/journal_store/{checkpoint,fenced,repair}.rs: the checkpoint and its thread, `FencedStorage`, the replay and the repair. - store/log_sync.rs: `LocalStore::sync_log`, `SyncFailed` and the test seam `FAIL_NEXT_LOG_SYNC`, out of store.rs (2139 lines on the base, 2113 now). - daemon/stop_causes.rs: `AbortOnDrop`, `next_signal` and `store_asks_to_stop`, out of daemon.rs (2052 on the base, 2046 now). check-no-godfiles.sh --base 51cfdee876b on every changed file: no file over budget grew, and no new file is over budget. * acpmux: the unrepairable-chat test waits for line 1 to be durable before its checkpoint At 03fccdef69c (fleet 32df3c72 on aws-m4pro-9, 144d4ccc on Linux) the test kept chat-b's lines [1, 2, 3] aside, not [2, 3]: it flushed right after appending line 1, and the checkpoint is now never past the journal's durable end (the review's Ahead nit), so line 1 was still past it. The fix's behavior is right; the test now waits until line 1 is durable, so its checkpoint covers it. On the code before the fix the test still fails (the open stops at chat-b's link). * acpmux: replies and events wait for durability, tests (red, S2c) DESIGN.md 3.3d. With the event journal on and every journal sync taking 400 ms (`SimStorage::set_sync_delay`): - an_acked_change_is_answered_once_its_record_is_durable: a rename's reply must come no sooner than one sync after its `renamed` record. Red here: the reply goes at once. - agent_output_is_not_held_behind_another_chats_acked_change: agent output to a connection attached to its chat goes at once while another chat's acknowledged change waits for its sync. Green here (nothing waits); it guards the fix against holding agent output behind changes the connection does not hear. * acpmux: replies and events wait until what they show is durable (S2c) DESIGN.md 3.3d; the tests are the commit before this one. With `store.journal` on, each connection gets an outbound gate (server/durable_gate.rs) between it and its transport: - Every line a connection queues raises the gate's `needs` to the last acknowledged LSN (`Store::durability().acked`, set in the journal store's append before the hub publishes anything of the record), except the lines of a `Stream` record's event (agent output, DESIGN C4): the fan-out delivers those in a thread-local stream scope. - The gate's task takes every queued line at once, reads `needs`, waits (async) until the journal's durable LSN reaches it, then sends the batch in order: one wait per batch, as the journal's group commit. - The durable LSN is a `tokio::sync::watch` fed by `Journal::on_durable`. The listener owns a guard: a writer that ends while the store is not being dropped (a panic on its thread) asks the daemon to stop (`Store::stop_reason`, decision 9.8), and the watch closes, so each gate drops what it holds and ends, which closes its connection. - Without the journal, `Store::durability()` is None: no gate, and `Conn::raise_gate` does nothing. Moves, so server/mod.rs and store.rs stay under their size on the base: the outbound queue helpers to server/outbound.rs, `open_with_journal` to the journal store, and `streamed` to store/durability.rs. * acpmux: the journal lock wait's deadline without an unchecked Instant addition `Instant + Duration` can panic on overflow (clippy::arithmetic_side_effects, tracked by the panic ratchet). The wait now has no deadline only in the case that cannot happen (an Instant 5 s ahead overflowing), where it stops waiting at once. * acpmux: a failed log sync stops the real daemon for a restart, test (red) Lead rule 2026-11-04 (3): a daemon-level test of the store's "stop for a restart" before S2c merges, red first. With store.journal on and every log sync failing (a debug build's ACPMUX_TEST_FAIL_LOG_SYNC; the checkpoint every 200 ms through ACPMUX_TEST_JOURNAL_CHECKPOINT_MS), a rename of a chat on disk makes its log dirty, the checkpoint's sync fails and freezes it, and the store asks the daemon to stop. The test asserts: the WebSocket client gets the restart close (1012), the socket client's pending `_acpmux/wait` ends with the connection (no hang), and the daemon exits with status 74. Red here: the daemon stops and sends 1012, but exits 0 (no exit status says that it stopped to be repaired). This commit adds only the two debug-build seams (release builds have none) and the test. * acpmux: a daemon its store stopped for a restart exits with status 74 The test is the commit before this one. `daemon::run` returned Ok and the process exited 0 for every stop, so a supervisor or a person could not tell a stop to be repaired from a normal one; with systemd `Restart=on-failure` it would not even start again. Now the store's stop (`store_asks_to_stop`: a failed log sync froze the event journal's checkpoint, or the journal's writer ended) marks itself, sets the restart handoff (WebSocket clients get 1012, as before), and the process exits with `daemon::EXIT_REPAIR` (74, EX_IOERR). The units prevent only 75, so launchd KeepAlive and systemd Restart=always start it again; the app does not read the daemon's exit status. README: store.journal names the status. * acpmux: the restart test's request in flight goes through the acpmux client Lead rule (3): the client sees a typed error, not a hang. The socket client is now the acpmux client library (acpmux::client::Client, as the CLI uses it), and the test asserts what the CLI makes of the request that the stop ends: `daemon_closed`, retryable (cli::errors::classify). The first red (fleet 41f2d037, hq run 41804640229, at 80c5fb2b7f8) failed only on the exit status (0, wanted 74), with 1012 and the end of the connection passing. * acpmux: the daemon's disk tier with the journal on, tests (red) Lead rule 2026-11-04 (2): a reply the person waits for never waits on a throttled disk sync. The fleet A/B (STATUS 11554) found that at IOPOL_UTILITY an acknowledged journal append waits about 16 ms p50 under another process's disk stress (0.7 to 1.1 ms at the default tier). - the_daemon_with_the_journal_keeps_the_default_disk_tier: with store.journal on, the daemon process keeps the default tier; without it, utility as before (each case in a child of the test binary). - a_background_thread_yields_its_disk_work: a dedicated background thread (the checkpointer) works at utility; a thread beside it does not. The API they use is stubbed here (`qos::enter_daemon` ignores the journal; `qos::background_thread` does nothing), so both fail on macOS. * acpmux: the restart test borrows no &str twice (clippy needless_borrows_for_generic_args) Found by fleet 7f8afd9d (hq run: the S2c server step at 62b4f68ef8b); the step's other clippy red is the base red in link_push_bench.rs. * acpmux: with the journal on, the daemon keeps the default disk tier (S2c) The tests are 994f74119fb. Lead rule 2026-11-04: a reply the person waits for never waits on a throttled disk sync. Measured on two owned Macs (STATUS 11554): one acknowledged append to durable at IOPOL_UTILITY under another app's disk stress took 16.3-16.7 ms p50 and 37-45 ms p95, against 0.7-1.1 ms p50 and 1.5-1.8 ms p95 at the default tier. - qos::enter_daemon(journal): with store.journal on, the daemon process keeps the default disk tier; without it, utility as before. The switch is read alone from config.json (config::store_journal_enabled), before any thread starts, with no harness discovery. - qos::background_thread: the journal store's checkpointer lowers its own disk I/O to utility for its whole life (a thread that does nothing else: a thread-scope policy drops it out of QoS for good). * acpmux: an agent chunk is a Stream record, test (red) The hub's tap logs an ACP session/update under its sessionUpdate kind (hub/tap.rs: `kind = su.to_owned()`), so a record's kind is `agent_message_chunk`, `tool_call`, ... and never `session/update`; the message's method is. The journal store's `streamed` checked the kind, so every agent chunk was journaled as Ack: a sync per batch at the token rate, and in S2c each token's event waited for that sync (the S2c I/O A/B: STRESS token to client 2.2 ms -> 20.3 ms p50 with the journal on). Red here: the chunk reads back as Ack. * acpmux: an agent chunk is a Stream record (the record's message names session/update) The test is the commit before this one. `streamed` now reads the logged message's method: the hub's tap logs a session/update under its sessionUpdate kind (hub/tap.rs), so the kind never said it. Agent output is again Stream (synced with the next acknowledged batch or a second later), as DESIGN.md 3.3b says. * acpmux: review 2 of #34901, tests (red): a crash model that follows real log syncs The logs' OS crash model now follows the syncs that ran (store/sync_model.rs, tests only): a segment keeps its length at its last sync that counts; a failed sync loses what was written since the one before, and a later sync does not bring it back; a cut ends a loss it removes. Every log segment sync goes through log_sync::sync_segment (same calls as before), where the model and FAIL_NEXT_LOG_SYNC hook in. Red tests, one per finding: - 1a an_os_crash_after_a_restart_from_a_kill_keeps_every_acked_line: the restart's first checkpoint retires records whose log lines no sync made durable; expected [1..5], the head gives [1]. - 1b an_os_crash_after_a_restart_from_a_failed_sync_keeps_every_acked_line: the same after a frozen checkpoint and a restart; expected [1..5]. - 2 a_hole_that_starts_inside_a_line_is_repaired (the reviewer's test): a torn line is read as line 6 of a log that moved on; expected [1..9], the head gives [1, 2, 3, 4, 5, 9]. - 3 a_failed_sync_in_flush_session_freezes_the_checkpoint: a failed sync in flush_session does not freeze the checkpoint. * acpmux: review 2 of #34901: a restart writes the replayed lines again; a torn line is not a line; flush_session freezes Finding 1 (high): the repair now writes every merged line again from the repair start, even when the log already has them all. A killed daemon's log writes may be in the page cache only, and after a failed sync the kernel may have dropped pages that still read back and that a later sync reports as written; the first checkpoint after the open then syncs bytes this process wrote before it retires their records. The lines only the log has from the repair start are journaled and made durable first, as before a cut. The work is bounded by the checkpoint interval. Finding 2: a piece of the log is a line only when it parses as one JSON value (piece_seq). A hole that starts inside a line keeps that line's {"seq":N, head and runs into a later line's tail; it was read as a line of a log that moved on, and the chat's journaled lines went aside. Now its bytes go and the lines are written again. Finding 3: a failed sync in flush_session freezes the checkpoint (its file sync is a SyncFailed now), as a failed sync in a checkpoint or at a log's close does. Nits: cut_at syncs events/ after its removals (and the kind-index folder after its copies go), so a removed segment cannot come back beside the lines written again; the README says that with the switch on an event line over 64 MiB is refused and its log does not get it. Tests: the four red tests of d2c63860ece. * acpmux: chat metadata through the event journal, tests (red, R-DURABILITY S3a) DESIGN.md 3.3e. Red on S2c: - last_seq_never_passes_the_log_after_an_os_crash (E13): session.json is synced at each save while the agent's lines wait for the journal's next sync, so after an OS crash the metadata names lines the log lost. - a_chat_made_after_the_checkpoint_survives_an_os_crash_that_loses_its_folder: nothing syncs the sessions folder when a chat's folder is made, and the replay never makes a folder again, so an acknowledged new chat is lost. - a_metadata_change_is_written_to_session_json_by_the_checkpoint: today each change writes and syncs session.json on the request path. Guards (green before and after): a chat made before the checkpoint and deleted is not made again; the replay never writes an older image over session.json. * acpmux: chat metadata through the event journal (R-DURABILITY S3a) DESIGN.md 3.3e, behind `store.journal` (default off). A metadata change is a `meta` record (kind 4: chat id, a `created` flag, the whole image) instead of a session.json write with two syncs on the request path; the reply waits for the record through S2c's gate, one group commit for a burst. - Overlay: each chat's newest image until a checkpoint writes session.json; load, load_startup and list read it first (list under the projection lock, so a checkpoint cannot drop an entry between its two reads). - Checkpoint: waits on its own thread until every record through its LSN is durable, then writes each image it read (session.json carries the image's LSN as `journalLsn`), then the logs and the sessions folder, as before. An image so names only lines the journal holds (E13). A chat deleted or made again meanwhile is passed (its overlay entry is gone or new). - Replay: per chat after its last `forget`, the newest image, written when newer than the file's `journalLsn`. A missing folder is made again only for a chat a replayed `created` record made; a chat made before the checkpoint whose folder is gone keeps its image in journal-unapplied/. - flush_session also waits until the chat's records are durable. - Delete drops the overlay entry with its `forget`, and removes the folder under the folders lock a checkpoint's write also takes. Tests: the 5 of 0505e9bee04 (3 red there). * acpmux: S3a, tests (red): folders handed out of the store Code outside the store reads and moves chat folders directly (transfer_in puts the live folder aside and installs a transferred chat; session_dir hands the folder to the transfer, the session files view and others). With the metadata in the overlay until a checkpoint: - a_folder_replaced_outside_the_store_keeps_its_own_metadata: a read, the checkpoint and the replay must not put the old folder's image in the new one. - session_dir_hands_out_a_folder_whose_metadata_is_written. - flush_session_writes_the_chats_metadata (the trait says log and metadata). - the_checkpoint_never_makes_a_folder_removed_outside_the_store_again. Red on 71bc09a1e63 (S3a's first implementation). * acpmux: S3a red tests: a replay after a folder swap; flush_session's chat has a log (as its callers' chats do) * acpmux: S3a: each metadata image is bound to its chat folder; session_dir and flush_session write it first Code outside the store reads and moves chat folders directly (a transfer puts the live folder aside and installs a transferred chat under the same id). So a `meta` record now carries its folder's identity (file id and birth time): - reads use an overlay image only while the folder at the path is its folder; - the checkpoint writes an image only into its own folder, never makes a missing one, and keeps the image of a removed or replaced folder in journal-unapplied/; - the replay writes an image only into its own folder (or a folder a replayed `created` record made), else keeps it aside. `session_dir` (the transfer, the session files view) and `flush_session` (the trait: "log and metadata to stable storage") write the chat's image first, once its record is durable. The list reads the overlay before the files (a checkpoint writes a file before it drops the entry), so it needs no lock. Tests: the 5 of 0280ac37f5e and 0e617489731. * acpmux: S2 follow-ups from review 3 of #34901, tests (red) - a_chat_whose_repair_failed_is_repaired_at_the_next_start (follow-up 1): the open's checkpoint retires the records of a chat whose repair failed, so no later start puts its lines back (they are only in journal-unapplied/). - a_hole_that_ends_at_a_line_start_keeps_that_line (follow-up 2): a torn line, zeros, then a whole line only the log holds: the piece does not parse and the whole line is dropped. - an_idle_store_does_not_wake_its_checkpointer (follow-up 4, cmux-ad m9): the checkpointer wakes every period with nothing to do. Test seams only (no behavior change): CHECKPOINTER_WAKES and TEST_CHECKPOINT_EVERY_MS. * acpmux: S2 follow-ups from review 3 of #34901: a failed repair holds its records; a hole's last zero splits; an idle checkpointer sleeps - Follow-up 1: a chat whose repair fails at the open (its log cannot be read or written now) keeps its lines in journal-unapplied/ as before, and the checkpoint now stays below its first journaled LSN (`Order::held`), so the next start repairs it again. Bounded: past 256 MiB of journal after it, the hold goes and the lines stay only in journal-unapplied/ (an error says so). - Follow-up 2: a piece splits at its last zero byte (a whole JSON line holds none), so a hole that ends at a line start, after a torn line, keeps that line (it mattered for a line only the log holds). - Follow-up 4 (cmux-ad m9): with nothing to checkpoint, the checkpointer sleeps until a change wakes it (`Inner::touched`, one atomic read per append), then checkpoints one period later; before, it woke every 30 s. Tests: the 3 of d147057bf85. * acpmux: a metadata change logs its line before its metadata, tests (red, R-DURABILITY S3b) DESIGN.md 3.3e, E6: with the event journal, records survive a crash as a prefix in append order. So a change whose line reaches the store before any metadata image that holds it is never, after a crash, in the chat's metadata without its line in the log. A recording store checks the order of the store calls: - a_tag_change_logs_its_line_before_its_metadata: red (tags saves first). - a_rules_change_logs_its_line_before_its_metadata: red (rules saves first). - a_rename_logs_its_line_before_its_metadata: green (a guard). * acpmux: the qos journal tests module is macOS-only (its Linux build held only an unused import: clippy red, review of #35400 F0) * acpmux: review of #35400 F1 and F2, tests (red) - a_read_that_shows_a_line_covers_its_record (F1): a read sees a log line between its log write and its record (the appends' lock held there), and the reply built from it goes out with no wait for that record. - a_journal_that_syncs_nothing_asks_the_daemon_to_stop (F2): with a journal that makes nothing durable, every reply waits without a limit and nothing asks the daemon to stop. Test seams only: PAUSE_AFTER_LOG_WRITE, TEST_STALL_MS. * acpmux: review of #35400 F1, the streaming read, test (red): session/load's replay sends a line before its record * acpmux: the test seam PAUSE_AFTER_LOG_WRITE is reachable from the server tests * acpmux: review of #35400: a read covers the records of the lines it shows; a stalled journal stops the daemon F1: the append writes a line to the log, then appends its record and sets the acknowledged LSN, all under the appends' lock; reads take no lock. A read could show a line before its record existed, and the reply built from it went out at once. Now every journal-store read of the logs ends with a fence on the appends' lock (`Inner::fenced`): a line the read saw has its record appended and its LSN acknowledged when the read returns, so the reply's gate covers it. The one read that sends as it reads (`Hub::scan_events`, session/load's replay) stops at the chat's last_seq read first. F2: a journal writer that keeps failing (it retries without end) held every reply on every connection with no limit. A connection's gate now waits at most 10 s (`STALL_LIMIT`); past it, `Durability::stalled` asks the daemon to stop for a restart, as a failed log sync does. Tests: the 3 of 3916f84dd68 and c553a19fd8b. * acpmux: S3b red tests: the recording store implements delete (a required Store method) * threading-model: the acpmux event journal's threads and bounds (writer, checkpointer, per-connection gate; review of #35400) * acpmux: the stall test's journal syncs take 3 s, not an hour (a sync under way kept its delay and hung the test's end) * acpmux: a tags or rules change logs its line before its metadata (R-DURABILITY S3b) DESIGN.md 3.3e (revised): with the event journal, records survive a crash as a prefix in append order, so a change whose line reaches the store before any metadata image that holds it is never, after a crash, in the chat's metadata without its line in the log (E6). `renamed`, `policy` and `mode` already did this; `tags` and `rules` saved first. Their reply still waits for the save. Tests: the 3 of 5bb3a27a2da and 7a4f02a56f3 (red at 7a4f02a56f3, fleet 88309e91: tags ["save:tagged", "line:tags"]; rules ["line:permission_chat_allowance", "save:ruled", "line:rules"]). * acpmux: an open that replays many chats flushes the drive a bounded number of times, test (red) Measured (fleet ead05e82, owned Mac aws-m4pro-5): the event journal store's open after a kill takes 29 ms per replayed chat on macOS (8.8 s for 300 chats, past the 8 s start budget; Linux 0.6 ms per chat, 681725fb). Each replayed chat costs about 7 F_FULLFSYNC (every sync_all/sync_data on macOS flushes the drive's cache): the rewrite's truncate and folder syncs, session.json's write_atomic, the checkpoint's log and folder syncs. Test seams only (no behavior change): FULL_SYNCS counts the store's full syncs (`log_sync::sync_full`, `sync_segment`, the folder syncs, session.json). * acpmux: the journal store's checkpoint and recovery sync in write order and flush the drive once Measured (fleet ead05e82, owned Mac aws-m4pro-5): an open after a kill took 29 ms per replayed chat on macOS (8.8 s for 300 chats, past the 8 s start budget), because every sync was an F_FULLFSYNC (std's sync_all and sync_data on macOS), about 7 per chat. Now the checkpoint's log and folder syncs, the repair's cut, and session.json's write sync with F_BARRIERFSYNC (durable against a crash of any process and a kernel panic, in write order; Linux fdatasync as before), and the checkpoint file's own syncs stay full: its F_FULLFSYNC flushes the drive's whole cache once per checkpoint, before any journal record is retired, so everything synced before it is durable against a power loss too (DESIGN.md 3.4: the checkpoint and recovery still end fully durable). cmux-journal: `sync_file(file, SyncLevel)` is public (the crate's tested F_BARRIERFSYNC/F_FULLFSYNC call with its ENOTSUP fallback), so acpmux uses the same sync. Test: an_open_that_replays_many_chats_flushes_the_drive_a_bounded_number_of_times (red at b2240f1b050). * acpmux: the open after a kill syncs nothing per replayed chat, test (red) At 15cfa1421c4 (ordered syncs) an open after a kill still took 6.1 s for 300 replayed chats on an owned Mac (fleet 59175d6b): about 8 ordered syncs per chat at about 2 ms each under load, plus a full flush per log evicted past 64 open logs. The test counts every sync the store makes during the open (seam ALL_SYNCS, tests only) and wants a constant, not one per chat; then the checkpoint (flush) makes every line durable through an OS crash. * acpmux: the open-syncs test counts the opening thread's syncs only (the checkpointer's thread is not the open's cost) * acpmux: the open after a kill syncs nothing per replayed chat; a freeze leaves a durable marker At 15cfa1421c4 an open after a kill still took 6.1 s for 300 replayed chats on an owned Mac (59175d6b), and it synced 283 times for 40 chats (90d87e10): every replayed chat's log was cut and written again (review 2 of #34901, finding 1, the forced rewrite) and its session.json written, then the first checkpoint ran before the open returned. Now: - The repair leaves a log that already holds every merged line, in order and untorn, as it is, and records its segments; the next checkpoint syncs them (ordered) before it retires their records, so finding 1a stays closed. - A freeze writes a durable marker (`journal-frozen`): the next open writes every replayed chat's lines again (finding 1b: the kernel may have dropped pages that still read back); the first checkpoint after it removes the marker, unless a chat's repair failed (the hold keeps it for the next start). - Replayed metadata images go to the overlay; the checkpoint writes session.json. - The open's first checkpoint runs on the checkpointer's thread at once, after the open returns (the daemon binds its socket after the open). Tests: an_open_after_a_kill_syncs_nothing_per_chat_and_the_checkpoint_makes_it_durable (red at 6def18f92c5), a_freeze_marker_makes_the_next_open_write_the_lines_again_then_goes; the restart-then-OS-crash tests of review 2 now flush after the reopen (the open's checkpoint is no longer inside it). * acpmux: the idle checkpointer's period, test (red): a change seen by its second look must not start a checkpoint at once - a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period: the checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies when it gets the timer lock. The second look saw the change, so the checkpointer is in its period wait by then, and the notify ends that wait: a checkpoint runs at once on the checkpointer's thread. This is the cause of a_failed_log_sync_freezes_the_checkpoint failing in the full store:: run of fleet 0577ac62 (that early checkpoint took the line whose sync the test fails, on a thread where the failure is not set). The test holds both threads at the two points, so the order is exact. Test seams only (no behavior change): the checkpointer's test seams are now by the store's home (period, wake count, the two holds), so tests that run as threads of one process under cargo test do not see each other's. * acpmux: the idle checkpointer's period ends only on its own state, not on a wake meant for its idle wait The checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies once it gets the timer lock. Its change shows in the second look, so the checkpointer has gone to its period wait by then, and that notify ended the period at once: a checkpoint ran right away on the checkpointer's thread (a burst no longer shared one checkpoint), and a spurious wakeup could do the same. In the full store:: run of fleet 0577ac62 such a checkpoint took the line whose sync a_failed_log_sync_freezes_the_checkpoint fails, so that test's own flush had nothing to do and returned Ok. The period wait now ends only on the period, `wanted` or `stop` (`wait_timeout_while`): each of the checkpointer's two waits ends only on the state its waker sets before it notifies under the timer lock. No sleep, no retry. The idle wait is unchanged. Test: a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period (edca3b4823d). * acpmux: S3a review (#35483) red tests: switch-off metadata and deletes, a late settle, a failed session.json folder sync, a purge after the checkpoint F1: metadata written with the switch off survives turning it on again; a chat deleted with the switch off after a failed catch-up is not made again. F2: a late settle never writes an older image over a checkpointed one (the reviewer's test and hold). F3: a failed folder sync after a metadata write freezes the checkpoint. F4: a chat made after the checkpoint and deleted is not made again after a kill. Test seams only; the code is the S3a head merged with feat. * acpmux: S3a review (#35483) red tests: a new chat is acknowledged only once its folder is durable; session_path never waits The root of F4 and of F1's delete case: the replay makes a chat's folder again, so a chat removed by any path comes back. The fix will never make a folder again, which needs an acknowledged chat's folder to survive an OS crash: the test models what each folder's last sync kept (sync_model) and asserts the gate's durability passes a new chat only after its sessions entry is synced. P3 1 of the review: a caller that needs only the path never waits for the journal (Store::session_path, a stub with session_dir's behavior here). * acpmux: S3a review (#35483) fixes: the replay never makes a folder again; switch-off changes are kept; no late or unsynced metadata write F1: each meta record carries the stamp (file id, modification time, length) of the session.json its save saw. The replay writes an image over a file the journal did not write only when one of the chat's records saw that file; otherwise the file moved on without the journal (a run with store.journal off whose catch-up did not replay, or an older build) and the image goes to journal-unapplied/, as S2 does for a log. F1 (deletes) and F4 at the root: the replay never makes a chat's folder again. A save that made a folder notes its record's LSN; the new acpmux-journal-folders thread (the daemon's disk tier) syncs sessions/ at once, and the durable LSN the S2c gate waits for stays below that record until then (journal_store/new_folders.rs). So an acknowledged chat's folder survives an OS crash, and a folder that is gone was removed: a purge whose forget a kill took back, a run with the switch off, the transfer recovery. The created flag and the replay's folder remake are gone. F2: write_through does not write a file that holds the same or a newer journalLsn. F3: session.json is written with its own syncs (file, then folder); a failed sync is SyncFailed and freezes the checkpoint; the replay and the kept-aside images return their errors. P3 1: Store::session_path gives the path with no lock, wait or write; an agent's start, the fixed-id check, the temporary-folder sweep, the host probe and the transfer recovery's root use it. The replaced test (a chat made after the checkpoint survives an OS crash that loses its folder) asserted a remake that cannot be right any more; the new test asserts the guarantee that makes the loss impossible, then the crash. Red (7 of 17 journal_meta_tests, each for its reason; the other 10 pass): macOS 5bc434a9 and 624d5df4, Linux a2207ff0 and 638a44ef. * acpmux: review of #35562 red tests: a failed repair holds no other chat's checkpoint; a checkpoint after it writes only durable images F1 (the reviewer's test, renamed): a checkpoint after a failed repair writes only durable metadata images. F2/F3 at the root: a failed repair does not hold the checkpoint of other chats; a retention rewrite of another chat after a failed repair stays after a restart. Guards: the reviewer's two switch-off tests (green here since the S3a fix 0d2…
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stage S2c of the R-DURABILITY rewrite (design:
handoff/rewrites/R-DURABILITY/DESIGN.md3.3d). It follows #34901 (the acpmux event logs through one journal,store.journal, default off), which landed on feat-cmux-next as6ad43f5d1bb; the base is now feat-cmux-next.With
store.journalon, a client is no longer told about a change before the change is durable:server/durable_gate.rs) between it and its transport. Every line it queues raises the gate to the store's last acknowledged LSN (Store::durability()), except the lines of agent output (aStreamrecord's event, delivered in a thread-local stream scope: DESIGN C4). The gate's task takes every queued line at once, waits (async, no thread blocked) until the journal's durable LSN reaches what they may show, then sends them in order: one wait per batch, as the journal's group commit is one sync per batch. A connection that heard nothing acknowledged never waits.Store::stop_reason); the daemon makes it a restart handoff (WebSocket clients get 1012) and exits with status 74 (EX_IOERR), which launchdKeepAliveand systemdRestart=alwaysstart again (the units prevent only 75). The next start's replay repairs the logs. DESIGN decision 9.8 (lead: keep it; nopanic = "abort").Store::durability()is None: no gate, and a send costs oneOptioncheck more.I/O policy (lead rule 2026-11-04: a reply the person waits for never waits on a throttled disk sync). acpmux ran the whole daemon at
IOPOL_UTILITY(qos.rs), and acknowledged replies now wait on the journal writer's sync. A thread's disk tier is the most throttled of its process's, its own and its QoS class's, so the writer cannot be raised alone. Measured on owned Macs (STATUS 11554; macOS 26.7 and 27.0.1): one acknowledged append to durable under another app's disk stress took 16.3-16.7 ms p50 and 37-45 ms p95 at utility, against 0.7-1.1 ms p50 and 1.5-1.8 ms p95 at the default tier. With the daemon (the real one, 16 chats streaming to two attached clients, plus another process's disk stress; scratch testtests/perf_ops.rs, never lands): after the Stream class fix (S2'sce8dcba5458), stresstoken_to_client(an agent's token to the attached client) p50/p95/max: journal on at the default tier 4.3/20.5/51 ms (fleet 2950e66f, aws-m4pro-4); journal off at the default tier 0.7/7.7/12.8 ms (b72720b0, aws-m4pro-9); journal off at utility, as feat today, 2.2/20.5/223 ms (c7b3649b, aws-m4pro-3). Acknowledged replies under stress,renameanddraft_setp50/p95: journal on 21.2/25.8 and 15.0/19.5 ms; today 21.1/22.3 and 20.4/22.7 ms. Three hosts, one run each: a same-host repeat is a follow-up. Before the Stream fix, agent chunks wereAckrecords and the journal-on token p50 was 20.3 ms. So withstore.journalon, the daemon keeps the default tier (qos::enter_daemon, which reads only that switch before any thread starts), and its background thread lowers itself: the checkpointer (qos::background_thread, a thread-scope utility tier for a thread that does nothing else). Without the journal nothing changes.What changes
store/durability.rs(new):Durability { acked, durable },streamed(moved from the journal store).store/journal_store.rs: the acknowledged LSN (an atomic set in the append, before the hub publishes anything of the record), the durable watch fed byJournal::on_durable, and a guard owned by that listener that asks the daemon to stop when the writer ends while the store is not being dropped.server/durable_gate.rs(new): the gate, the stream scope andinstall;server/mod.rs:Conn::gate, raised insend_replying,send_serialized,send_request_in_orderandsend_line;server/fanout.rs: the stream scope around aStreamrecord's delivery.daemon/stop_causes.rs:store_asks_to_stopmarks the stop and sets the restart handoff;daemon::exit_status,EXIT_REPAIR;cli/entry.rsexits with it.ACPMUX_TEST_FAIL_LOG_SYNC(every log sync fails),ACPMUX_TEST_JOURNAL_CHECKPOINT_MS(the checkpoint period).server/mod.rsdoes not grow (it is over its size budget on the base): the outbound queue helpers toserver/outbound.rs.Review fixes (independent review FIX at d9938eb, comment 6633213210)
unused import: super::*atqos_journal_tests.rs:6(a macOS-only module)cfg(all(test, target_os = "macos"))Inner::fenced), so a line the read saw has its record appended and its LSN acknowledged when the read returns;Hub::scan_events, the one read that sends as it reads (session/load's replay), stops at the chat'slast_seqread firsta_read_that_shows_a_line_covers_its_record(red: "a read showed line 2 before its record was appended (acked 1, then 2)", fleet 2eeae197);a_replay_never_sends_a_line_before_its_record(red: "the replay sent line 2 before its record: [1, 2]", fleet d0c6b406)STALL_LIMIT); past it,Durability::stalledasks the daemon to stop for a restarta_journal_that_syncs_nothing_asks_the_daemon_to_stop(red: fleet e0ccde6e)threading-model.mdnot updatedThe P3 items F3-F8 are follow-ups (Known gaps).
Known gaps
renamed) can be seen by another connection's reply in that window. The reply then waits only for what was acknowledged before it, not for that record. S3 does not close it (DESIGN.md 3.3d, decided 2026-11-04): it needs every metadata write site to append before the change is visible; the window is microseconds on one thread, and a crash must follow within one group commit.available_commands_update, built at delivery time (it can show a later acknowledged change unraised); F4 a tailnet device can get a close without 1012 when the writer ends; F5qos::enter_daemon(true)sets nothing, so a daemon a utility-tier process starts stays at utility, and with the journal on the daemon's children run at the default tier; F6 retention holds a chat's append lock while it waits for the utility-tier checkpointer; F7 acknowledgements to the agent host still go out before durability (E12, stage S4); F8 a failed open logs a false "writer ended" error; no test covers the "writer ended" path.Testing
Red on the code before each fix, green after (fleet steps,
scripts/ci/cmux-tui-rust-check.sh):server::durable_gate_tests(a connection throughserve_connectionon a journal store whose simulated sync takes 400 ms):an_acked_change_is_answered_once_its_record_is_durable: red at2abbb1a0a4a(fleet 783893c4, hq run 41797031252): the rename was answered after 1.1 ms, before its record could be durable; also alone.agent_output_is_not_held_behind_another_chats_acked_change: green there and after (it guards the fix against holding agent output).tests/daemon_restart_close.rsa_failed_log_sync_stops_the_daemon_for_a_restart(lead rule: the real daemon, a failed fsync through the hook, the restart exit and a typed client error, not a hang): red at80c5fb2b7f8(fleet 41f2d037, hq run 41804640229): the daemon exited 0 (wanted 74), while the 1012 close and the end of the connection already passed; the stricter version (the request in flight throughacpmux::client::Client, the CLI's typed and retryabledaemon_closed) is red at31d276386b1(fleet b6ef221a, hq run 41806243685): the 1012 close and the typed, retryabledaemon_closedpassed, and the daemon exited 0 (wanted 74).Green at the head
eb24c40e1dfe23615f170567d9c46dddb7f71371(the review fixes; feat-cmux-next merged after #34901 landed:9cf1425a7d5merges3dcd5db58a3with S2c's side for the S2 files,3969cb049b1merges35a1a909a11cleanly):3969cb049b1, the same code but the stall test):all store::: clippy pass, 318 of 318 tests. Its fmt step is red only on 7 acpmux files this PR does not change (agent_host/host.rs,claude_stdio/inbound.rs,cli/peer_status.rs,hub/admission.rs,subsession_mcp.rs,tests/host_lifetime.rs,tests/peer_status.rs): a base red on feat (from acpmux: messages and hints name the renamed machine commands #33676 and direct pushes; a base-red PR fixes it).eb24c40e1df): red on the code before the F2 fix (fleet 92cff3a1: "a reply waited past the stall limit and the store did not ask to stop", in 3 s, no hang).eb24c40e1dfon owned Macs:server::1576 of 1576, the three new gate tests included (fleet 51b2adb4, aws-m4pro-7); the restart testa_failed_log_sync_stops_the_daemon_for_a_restart(89fde7f4);daemon833 of 834 (0033a5b0), the one failure being the base redacpmux_daemon_idle_has_no_periodic_wakeups(it also fails on the merge base alone, fleet 6c027917);sigterm16 of 17 (10e278d9):daemon_binds_before_login_env_reports_ready_and_stops_on_sigtermsaid "ready after 2.10 s" against its 2 s limit on a loaded host; it passed alone at this head (7a9a3e38) and on featd17e9736d91(5d0e2ed2), and it failed on feat alone on another loaded host (the acpmux: the event log through one append-only journal, behind store.journal (R-DURABILITY S2) #34901 merge review, 6.85 s): a timing flake, not this PR.Observed / Not run
type_complexityatcmux-tui/crates/cmux-tui/tests/link_push_bench.rs:59(from Cloud copy: compressed fs.write bytes (fs-write-deflate-v1) and fewer round trips per file #34538);acpmux::daemon_lifecycle a_daemon_whose_home_is_removed_stops(fails on featd422b94b60dalone, owner BASE RED ACPMUX);server::fs_wire::tests::a_deflate_write_stream_over_its_decoded_size_leaves_no_fileandacpmux_daemon_idle_has_no_periodic_wakeups(above).store.journalon (the switch is off by default).Changelog
none
Proof
No capture: a daemon storage change behind a default-off setting, with no UI.