Repository navigation
acpmux: the event log through one append-only journal, behind store.journal (R-DURABILITY S2) - #34901
Merged
Merged
Conversation
…nd ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner).
…clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red.
… segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee.
Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c8: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5.
fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r).
…1 review regressions (red) The S1 review (FIX at 89293fb) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open
…eral segments (red) The reviewer's F2 scenario has two segments: S1 holds 1..10, S2 (the last) holds 11..20, and S2's header is damaged. The test now rotates segments (300-byte segments, 20 acked records) and damages the last one's header, so the report must say lost_through 20 and the next append must take 21. On the code before the fix the moved segment's LSNs read as 0, so the next append reuses one.
…S crash) (red) On the code before the S1 fix, the crash-sim step was green (fleet 184129d8 at 263da86): 6 of 7 runs passed, and the mixed run failed only seed 140 in the full run and passed when it ran alone. The random faults reached the order the review found (F4) in 1 of 800 runs. So these tests were no red/green proof. The order: the sync of a write that spans pages fails and the disk gets none of them (Linux keeps them clean in the cache, so a later sync reports success); the process is killed in the writer's backoff; the next open reads the failed segment from the cache, replays its records and goes on; an OS crash or a power loss then takes those pages, and records an open already replayed are gone. - sim.rs: `fail_next_sync(pages, kind)` fails the next sync that has at least that many dirty pages, and the disk gets none of them. `crash_after_failure(n)` makes the n-th operation after the next injected failure the crash point. - crash_sim_tests.rs: a new run forces the order on every seed (a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record): epoch 0 aims the fault, the kill is a process crash at the writer's retry, the open after it does not crash, epoch 1 ends in an OS crash, and a third open checks that the records the second open replayed are all there. In the random runs, one epoch in four aims the fault, with the crash 1 to 3 operations after the failed sync and a crash of any kind after it. - A sim test checks the aimed fault itself (the_sim_fails_the_aimed_sync_and_crashes_in_the_backoff). Expected on this commit (old journal code): the forced run fails every seed; the mixed run fails most runs. The fix commit follows.
…y, LSN floor, writer end) Review FIX at 89293fb (PR #33919 comment 6610468202). The regression tests are the two commits before this one. - Sync markers (format.rs): an entry with flags 1 and no payload, whose LSN field says "every record of this segment before me is synced". The writer appends one after each successful sync that covered records (after the durable listeners, so no reply waits for it), and syncs it fully at a seal and at a rotation. - F4, recovery (step 4 and 6): the last segment is trusted only up to its last marker. Its records after the marker may sit in pages a failed sync dropped (a later sync on another descriptor reports success without them), so recovery writes them again into a new segment: header, records and a marker, fully synced under `<name>.seg.new`, renamed into place, folder synced. The journal always goes on in a segment this open made, never in one an earlier process wrote. A restart in the writer's backoff after a failed sync is then safe. - F1, quarantine: every LSN a quarantined segment holds counts in the LSN floor at every open, and a quarantine that no new segment followed (a crash in the middle of an earlier recovery) is reported as damage again. The highest LSN of every moved segment is read before the first rename. - F2, highest LSN: a segment whose header is damaged is read from its first record (each record's checksum covers its LSN), then the bytes after the first bad entry are scanned for records and markers that still check out. The same scan runs after a cut in the last segment: a marker there is media damage (reported), records there are a torn write (`torn_lsns`); their LSNs are never used again. - F3, writer end: a guard closes the journal when the writer thread ends, also by a panic: appends fail with Closed, waits end with WriterEnded, the listeners are dropped (so a channel a held reply waits on closes), and stats.failing says why. The crate does not abort the process; the daemon owns that policy. - F6: recovery closes each segment after its header is read and opens one again only to read it. (Retirement of compacted segments is an S2 API, before the first consumer.) - F5: recovery syncs the parent folder after it makes the journal folder. - F7: format.rs and lib.rs now say what the code does: a failed write is cut back and retried in the same segment; the cut removes only bytes never synced and never acknowledged; nothing shortens a segment past a synced record. Docs: journal.rs says when open's replayed records are durable (on Ok) and what a stopped writer does.
… (review G5) sim.rs had 924 lines and 63 functions, over the 60-function limit for a new Rust file (the god-file check warned "not in baseline: split it"). The machine model (each file's page cache and disk, the ordered log of names, and what an OS crash or a power loss leaves) moves to sim/machine.rs (297 lines, 19 functions); the process view and the Storage implementation stay in sim.rs (639 lines, 30 functions). No behavior change: the items move as they are, made visible to the sim module.
…ash sim (red) Re-review FIX at 3498975 (PR #33919 comment 6620113993). Tests and sim only; the fix follows. - sim: the pages a failed sync dropped stay in the cache until an eviction; `evict_dropped_pages_after_read(true)` makes each read evict the dropped pages it read, so a second read of them sees the disk's bytes. - G1 recovery_copies_the_records_it_checked_when_the_cache_drops_them: a sync fails and drops record 2's pages, a kill in the backoff, the next open reads record 2 from the cache (replayed) and the pages are evicted after that read; then an OS crash. Record 2 and the record 3 acked after the restart must survive. On this code recovery's second, unchecked read stages zeros under a vouching marker, and the last open keeps only record 1 and reports damage. - G2: the F1 regression asserts that every open after the damage reports the loss (lost_through 40). On this code a crash at the open's last step (the folder sync after the new segment's rename) loses the report for good. - G3 a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system): a journal in a symlinked state folder, and a journal folder that is a symlink. On this code the parent folder's sync opens with O_NOFOLLOW (ELOOP). - G4 an_open_after_a_finished_quarantine_reads_no_quarantined_segment: on this code every open reads every quarantined segment in full.
Re-review FIX at 3498975 (PR #33919 comment 6620113993); the regression tests are the commit before this one (red: fleet de35f6d0). - G1: recovery writes again exactly the bytes its checked read returned. The last segment's entries after its last marker are kept from the checked read (header and payload) and staged from memory; the second, unchecked read of the file is gone. A page that a failed sync dropped can be evicted between two reads, and the second read then returned the disk's zeros, which the copy kept under a vouching marker. - G2: the damage an open finds is written to `DAMAGE` (whole, fully synced, renamed into place, folder synced) before the new segment that hides it from later opens. Every open reports the recorded damage first, until the daemon calls the new `Journal::clear_damage`, which moves the record into `quarantine/` (kept, never deleted). - G3: a folder sync follows a symlink in its path (no O_NOFOLLOW): a state folder, or the journal folder itself, may be a symlink. The sync is for the folder that holds the entries. - G4: the quarantined segments are read for their LSNs only while no live segment is numbered above them (an unfinished quarantine); once one is, its first LSN already carries their floor. - G6: `checked_div` in the scan of the bytes after a cut (the crate's one tracked arithmetic hit). - G5 (the sim split) is the commit before the regressions.
Third review FIX at cb893c6 (PR #33919 comment 6621681018). Tests only. - G7 A a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn: after a finished quarantine, one byte of the empty new segment's header changes. On this code the next open skips the quarantine scan (the listing counts the damaged segment) and the next append reuses a quarantined LSN. - G7 B a_damaged_newest_segment_after_a_quarantine_reports_its_records: the newest segment holds acked 41..43, and its header and its first record are damaged. On this code the moved segment is scanned with a base far below 41, so the loss is not reported through 43 and the next append is not 44. - N1: the F1 regression also asserts that an open reports each loss once (no reported LSN range lies inside another). On this code a crash after the quarantine's renames makes the next open report the same loss twice.
…y.rs Third review FIX at cb893c6 (PR #33919 comment 6621681018); the regression tests are the commit before this one. - G7: the quarantine scan is skipped only when a segment whose header checked out is numbered above every quarantined segment (the folder listing also counted a newest segment with a damaged header, which carries no floor). The scans start from the running floor: `quarantined_floor` and `quarantine` pass the highest LSN counted so far as the base of the next file's scan, so a quarantined file with no readable header and no readable first record is scanned for the LSNs right above what is counted. - The "unfinished quarantine" report is made only when the quarantine holds LSNs nothing else counted (an old quarantine behind a segment that a crash tore at its creation holds none, and a crash is not damage). Its reason now also names a damaged newest header. - N1: a loss whose LSN range lies inside a recorded one is the same loss found again after a crash: it is neither recorded nor reported twice. - N2: recovery.rs's docs name `DAMAGE`; two doc lines over 100 columns are wrapped (lib.rs, sim.rs). - recovery.rs was at 987 of its 1000 lines: the damage record moves to recovery/damage.rs, the entry reader and the LSN scans to recovery/scan.rs (moved as they are).
…e (N1) The red run of the N1 regression (fleet 27d011ca, op 64 of the F1 test) shows the case the previous commit did not cover: a crash after the quarantine's renames and before DAMAGE is written; the next open finds both the damaged synced record of the last segment (7..14) and the unfinished quarantine (7..40), and reported both. Now every loss inside another (recorded or found by this open) is dropped, the larger range kept; DAMAGE is rewritten only when its set of losses changed.
Fourth review FIX at 215c8f0 (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report.
… the moves (G8, N3, N4) Fourth review FIX at 215c8f0 (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format.
…essary_sort_by) Fleet d9f22bed at b589044: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`).
…S2's projector) R-DURABILITY S2 (acpmux event logs through the journal) projects each chat's log line into its ndjson file once the journal has WRITTEN the batch that holds it, so readers see stream lines at once. The listener runs on the writer thread after the write and before the sync, once per batch that moved the written LSN: a batch written again into a new segment after a failed sync is not heard twice. When the writer stops, these listeners are dropped too. Test: heard while the batch's sync still runs, once. Port of 145faa6 (branch rewrite/r-durability-s2-acpmux-logs) onto the S1 review fixes.
…paction point covers) R-DURABILITY S2: a daemon's projector compacts the journal into its own files (acpmux: the chat ndjson logs, then a synced checkpoint). Without a delete the segment count only grows (review finding F6). `retire_through(compacted)` deletes the segments that hold no record after the caller's compaction point: a segment goes when the next segment's "previous last LSN" is at or before it, so the newest segment never goes. The point is capped at the durable LSN. The deletes and a folder sync run on the caller's thread (the projector, never a request path). A delete that fails leaves its segment live for a later call. Why a crash during a retirement is safe: the caller reopens from a point at or after `compacted`, and recovery reads nothing from a segment the point covers (it skips it on its next segment's header), so any subset of the deletes may reach the disk. A point older than the retired segments is reported as Damage::Missing. Quarantined segments are never deleted. - Storage::remove_file (FsStorage: std::fs::remove_file; SimStorage: one more change in the ordered log of names, durable at the folder sync). - The live segment list: recovery's, then each segment the writer makes. - JournalStats::segments_retired. - Tests: only covered segments go, and a second call deletes none; an OS crash after it keeps every record after the point and an older point is reported; a crash at every operation of a retirement (process, OS crash, power loss) keeps every record after the point and never reuses an LSN; the newest segment stays; a point in the middle of what this writer wrote. This adds a delete of journal data, so its PR is risk:high.
store.rs (2139 lines, 120 fns) and config.rs (1909 lines, 97 fns) are over
the god-file limit, and the journal store (R-DURABILITY S2) must not grow
them. MemoryStore moves to store/memory.rs and StoreMode, StoreConfig and its
defaults to config/store_config.rs; both stay exported where they were
(crate::store::MemoryStore, crate::config::{StoreConfig, StoreMode}). No
behavior change: the code moves as it is.
R-DURABILITY S2 (DESIGN.md 3.3b). Today no event-log append is synced: a line
a client was told about is on the disk only after the next flush (shutdown),
so an OS crash or a power loss takes back every line since (E12). This commit
adds only the switch, a seam and the tests; the journal store follows.
- `store.journal` (StoreConfig::journal, default false).
- `Store::wait_durable(timeout)`: tests and tools wait until every appended
record is durable. The default returns at once (the Local and Memory stores
never sync an append), which is today's behavior.
- `store::open_with_journal` (tests): the store with the journal on a given
`cmux_journal::Storage`. Here it ignores the journal and opens LocalStore.
- acpmux depends on cmux-journal (dev: with the `sim` feature).
- store/journal_store_tests.rs, red on this commit:
- a_line_the_store_says_is_durable_survives_an_os_crash (lines 2..20 are
lost: nothing synced them);
- a_deleted_chat_stays_deleted_after_a_crash and
a_line_the_log_refused_is_never_written_later pass here, and must keep
passing with the journal store (no replay may bring them back);
- a_kill_keeps_every_line_once passes here and must keep passing (a replay
after a kill must write no line twice).
…DURABILITY S2)
DESIGN.md 3.3b. With `store.journal` on, each event-log append is one record
in `<home>/journal/` (cmux-journal: group commit, one sync per batch; a power
loss takes back at most the last second, always a suffix), then the same line
goes to the chat's ndjson as before. Readers, segments and file formats do not
change. Default off.
- store/journal_store.rs (JournalStore over LocalStore):
- append: under one lock, the journal record (a memory copy, no I/O), then
the log line; a line the log refuses gets a `cancel` record. Agent stream
lines (`in` session/update, claude.*, codex.*) are Class::Stream, every
other line Class::Ack.
- checkpoint (`<home>/journal-projected.json`): every 30 s while the journal
grew, past 64 MiB, at flush (shutdown) and before a retention commit. It
syncs each log written since (LocalStore::sync_log, with no lock held
while the disk works) and the sessions folder, writes the LSN (temp file,
sync, rename, folder sync, every error checked), then retires the journal
before it. A log close that failed its sync freezes the checkpoint until a
restart (LocalStore::sync_failed).
- recovery: the records after the checkpoint; for each chat, the log is cut
right after its last readable line below the first journaled seq (later
segments removed, the boundary one shortened, their kind-index copies
dropped), and the journaled lines are written again in order. `forget`
deletes a chat again; `cancel` keeps a refused line out.
- delete: `forget` record, folder removal, then a bounded wait until the
record is durable.
- seal: the logs refuse writes and the journal syncs fully.
- LocalStore: `append` is serialize + `append_line`; `sync_log`; a failed sync
at a log's close is noted (`sync_failed`).
- store::open opens the JournalStore when `store.journal` is on.
Not yet (S2c): replies and events do not wait for durability; the daemon does
not stop when the journal closes; the writer thread keeps the process's
Utility I/O policy.
…its delete (red) - a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay: a chat is deleted (its forget record), made again under the same id (a transfer back: session.json and new lines), then a kill -9. On this code the replay deletes the folder again and its session.json is gone. - a_line_after_the_seal_is_refused_and_never_written_later: coverage of the seal (it passes here; the fix also closes a race the test cannot force).
…ps a chat made again - seal takes the appends' lock: an append lands in both the journal and the logs, or in neither. Before, an append that raced the seal could leave a journal record whose line the sealed logs refused (its cancel then failed on the closed journal), and the next start would write that line. - append and delete check the logs' seal first, under the same lock, and do not touch the journal after it (no error per append at shutdown). - replay deletes a forgotten chat again only when no line follows its forget record: otherwise the chat was made again under the same id, and its folder holds that chat's own files.
Fifth review FIX at 9c47594 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4.
lawrencecchen
temporarily deployed
to
artifacts
November 4, 2026 06:04 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
sentry-dsyms
November 4, 2026 06:04 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
…eal log syncs The logs' OS crash model now follows the syncs that ran (store/sync_model.rs, tests only): a segment keeps its length at its last sync that counts; a failed sync loses what was written since the one before, and a later sync does not bring it back; a cut ends a loss it removes. Every log segment sync goes through log_sync::sync_segment (same calls as before), where the model and FAIL_NEXT_LOG_SYNC hook in. Red tests, one per finding: - 1a an_os_crash_after_a_restart_from_a_kill_keeps_every_acked_line: the restart's first checkpoint retires records whose log lines no sync made durable; expected [1..5], the head gives [1]. - 1b an_os_crash_after_a_restart_from_a_failed_sync_keeps_every_acked_line: the same after a frozen checkpoint and a restart; expected [1..5]. - 2 a_hole_that_starts_inside_a_line_is_repaired (the reviewer's test): a torn line is read as line 6 of a log that moved on; expected [1..9], the head gives [1, 2, 3, 4, 5, 9]. - 3 a_failed_sync_in_flush_session_freezes_the_checkpoint: a failed sync in flush_session does not freeze the checkpoint.
…; a torn line is not a line; flush_session freezes
Finding 1 (high): the repair now writes every merged line again from the repair
start, even when the log already has them all. A killed daemon's log writes may
be in the page cache only, and after a failed sync the kernel may have dropped
pages that still read back and that a later sync reports as written; the first
checkpoint after the open then syncs bytes this process wrote before it retires
their records. The lines only the log has from the repair start are journaled
and made durable first, as before a cut. The work is bounded by the checkpoint
interval.
Finding 2: a piece of the log is a line only when it parses as one JSON value
(piece_seq). A hole that starts inside a line keeps that line's {"seq":N, head
and runs into a later line's tail; it was read as a line of a log that moved
on, and the chat's journaled lines went aside. Now its bytes go and the lines
are written again.
Finding 3: a failed sync in flush_session freezes the checkpoint (its file
sync is a SyncFailed now), as a failed sync in a checkpoint or at a log's
close does.
Nits: cut_at syncs events/ after its removals (and the kind-index folder after
its copies go), so a removed segment cannot come back beside the lines written
again; the README says that with the switch on an event line over 64 MiB is
refused and its log does not get it.
Tests: the four red tests of d2c6386.
lawrencecchen
temporarily deployed
to
ci-routing-agent
November 4, 2026 12:28 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
artifacts
November 4, 2026 12:40 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
sentry-dsyms
November 4, 2026 12:40 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
artifacts
November 4, 2026 12:44 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
Author
|
Review (r-durability-s2-reviewer, subagent, correctness first): SHIP at 84f9c26 Scope: dcfecad..84f9c26 (4 commits, no merge), the repair for my FIX at dcfecad (comment 6629184957). I read every changed file in full first, then the PR body and DESIGN.md 3.3c. Each of the three findings is closed at its root, and so are the nits. The forced rewrite keeps both properties you asked about: no whole line and no folder is deleted, and a crash at any step loses nothing. The follow-ups below do not block. They need an I/O error during recovery or a rare crash shape, or they are cost items, and the switch is off by default. The findings of review 2, checked
The forced rewrite: no whole line or folder deleted, safe at every crash point
Follow-ups (not blocking)
Also checked
Evidence
|
Conflict: daemon.rs's module list. feat added `term_sender` (#35147: a SIGTERM stop names its sender); S2 moved the stop helpers to `daemon/stop_causes.rs` and added the store's stop branch. Both kept; the select block merged cleanly (feat's SIGTERM line, S2's `store_stop` branch).
lawrencecchen
temporarily deployed
to
ci-routing-agent
November 4, 2026 13:14 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
artifacts
November 4, 2026 13:24 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
sentry-dsyms
November 4, 2026 13:24 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
lawrencecchen
temporarily deployed
to
artifacts
November 4, 2026 13:31 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
Author
|
Review (r-durability-s2-merge-reviewer): SHIP at 3273b2a Scope: only the two feat-cmux-next merges after the SHIP at 84f9c26 (comment 6630992897). I did not review the PR's own code again. Read-only review. I ran no cargo locally; the tests below ran on the fleet. Merge 1: 3b245c1 (84f9c26 + feat b43ce2b, merge base d422b94)
Merge 2: 3273b2a (3b245c1 + feat d3d5a62)
FindingsNone against the merges. Evidence at 3273b2a (I read each log)
Base findings (not this PR's red)
Not run
|
Contributor
Author
|
Non-green checks at the head 3273b2a, classified (none is in a file this PR changes):
The build jobs (aarch64/x86_64 Apple and Linux musl), |
lawrencecchen
added
the
ready-to-land
Contributor
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Shipped in:
|
lawrencecchen
added a commit
that referenced
this pull request
Nov 4, 2026
…daemon stops for a restart when the journal cannot go on (R-DURABILITY S2c) (#35400) * cmux-journal: one append-only journal per daemon, with group commit and ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner). * cmux-journal: type a test range; drop an unread segment field (fleet clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red. * cmux-journal: a failed F_FULLFSYNC is an error, not a plain fsync; no segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee. * cmux-journal: the past-the-end test reads back the storage it wrote to Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c8: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5. * cmux-journal: a sync a signal interrupts is made again (macOS) fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r). * cmux-journal: crash sim of write-back errors and crash in recovery; S1 review regressions (red) The S1 review (FIX at 89293fb) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open * cmux-journal: the F2 regression damages the header of the last of several segments (red) The reviewer's F2 scenario has two segments: S1 holds 1..10, S2 (the last) holds 11..20, and S2's header is damaged. The test now rotates segments (300-byte segments, 20 acked records) and damages the last one's header, so the report must say lost_through 20 and the next append must take 21. On the code before the fix the moved segment's LSNs read as 0, so the next append reuses one. * cmux-journal: the crash sim forces the F4 order (failed sync, kill, OS crash) (red) On the code before the S1 fix, the crash-sim step was green (fleet 184129d8 at 263da86): 6 of 7 runs passed, and the mixed run failed only seed 140 in the full run and passed when it ran alone. The random faults reached the order the review found (F4) in 1 of 800 runs. So these tests were no red/green proof. The order: the sync of a write that spans pages fails and the disk gets none of them (Linux keeps them clean in the cache, so a later sync reports success); the process is killed in the writer's backoff; the next open reads the failed segment from the cache, replays its records and goes on; an OS crash or a power loss then takes those pages, and records an open already replayed are gone. - sim.rs: `fail_next_sync(pages, kind)` fails the next sync that has at least that many dirty pages, and the disk gets none of them. `crash_after_failure(n)` makes the n-th operation after the next injected failure the crash point. - crash_sim_tests.rs: a new run forces the order on every seed (a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record): epoch 0 aims the fault, the kill is a process crash at the writer's retry, the open after it does not crash, epoch 1 ends in an OS crash, and a third open checks that the records the second open replayed are all there. In the random runs, one epoch in four aims the fault, with the crash 1 to 3 operations after the failed sync and a crash of any kind after it. - A sim test checks the aimed fault itself (the_sim_fails_the_aimed_sync_and_crashes_in_the_backoff). Expected on this commit (old journal code): the forced run fails every seed; the mixed run fails most runs. The fix commit follows. * cmux-journal: fix the S1 review findings F1-F7 (markers, recovery copy, LSN floor, writer end) Review FIX at 89293fb (PR #33919 comment 6610468202). The regression tests are the two commits before this one. - Sync markers (format.rs): an entry with flags 1 and no payload, whose LSN field says "every record of this segment before me is synced". The writer appends one after each successful sync that covered records (after the durable listeners, so no reply waits for it), and syncs it fully at a seal and at a rotation. - F4, recovery (step 4 and 6): the last segment is trusted only up to its last marker. Its records after the marker may sit in pages a failed sync dropped (a later sync on another descriptor reports success without them), so recovery writes them again into a new segment: header, records and a marker, fully synced under `<name>.seg.new`, renamed into place, folder synced. The journal always goes on in a segment this open made, never in one an earlier process wrote. A restart in the writer's backoff after a failed sync is then safe. - F1, quarantine: every LSN a quarantined segment holds counts in the LSN floor at every open, and a quarantine that no new segment followed (a crash in the middle of an earlier recovery) is reported as damage again. The highest LSN of every moved segment is read before the first rename. - F2, highest LSN: a segment whose header is damaged is read from its first record (each record's checksum covers its LSN), then the bytes after the first bad entry are scanned for records and markers that still check out. The same scan runs after a cut in the last segment: a marker there is media damage (reported), records there are a torn write (`torn_lsns`); their LSNs are never used again. - F3, writer end: a guard closes the journal when the writer thread ends, also by a panic: appends fail with Closed, waits end with WriterEnded, the listeners are dropped (so a channel a held reply waits on closes), and stats.failing says why. The crate does not abort the process; the daemon owns that policy. - F6: recovery closes each segment after its header is read and opens one again only to read it. (Retirement of compacted segments is an S2 API, before the first consumer.) - F5: recovery syncs the parent folder after it makes the journal folder. - F7: format.rs and lib.rs now say what the code does: a failed write is cut back and retried in the same segment; the cut removes only bytes never synced and never acknowledged; nothing shortens a segment past a synced record. Docs: journal.rs says when open's replayed records are durable (on Ok) and what a stopped writer does. * cmux-journal: split the crash sim's machine model into sim/machine.rs (review G5) sim.rs had 924 lines and 63 functions, over the 60-function limit for a new Rust file (the god-file check warned "not in baseline: split it"). The machine model (each file's page cache and disk, the ordered log of names, and what an OS crash or a power loss leaves) moves to sim/machine.rs (297 lines, 19 functions); the process view and the Storage implementation stay in sim.rs (639 lines, 30 functions). No behavior change: the items move as they are, made visible to the sim module. * cmux-journal: re-review regressions G1-G4 and page eviction in the crash sim (red) Re-review FIX at 3498975 (PR #33919 comment 6620113993). Tests and sim only; the fix follows. - sim: the pages a failed sync dropped stay in the cache until an eviction; `evict_dropped_pages_after_read(true)` makes each read evict the dropped pages it read, so a second read of them sees the disk's bytes. - G1 recovery_copies_the_records_it_checked_when_the_cache_drops_them: a sync fails and drops record 2's pages, a kill in the backoff, the next open reads record 2 from the cache (replayed) and the pages are evicted after that read; then an OS crash. Record 2 and the record 3 acked after the restart must survive. On this code recovery's second, unchecked read stages zeros under a vouching marker, and the last open keeps only record 1 and reports damage. - G2: the F1 regression asserts that every open after the damage reports the loss (lost_through 40). On this code a crash at the open's last step (the folder sync after the new segment's rename) loses the report for good. - G3 a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system): a journal in a symlinked state folder, and a journal folder that is a symlink. On this code the parent folder's sync opens with O_NOFOLLOW (ELOOP). - G4 an_open_after_a_finished_quarantine_reads_no_quarantined_segment: on this code every open reads every quarantined segment in full. * cmux-journal: fix the re-review findings G1-G4 and G6 Re-review FIX at 3498975 (PR #33919 comment 6620113993); the regression tests are the commit before this one (red: fleet de35f6d0). - G1: recovery writes again exactly the bytes its checked read returned. The last segment's entries after its last marker are kept from the checked read (header and payload) and staged from memory; the second, unchecked read of the file is gone. A page that a failed sync dropped can be evicted between two reads, and the second read then returned the disk's zeros, which the copy kept under a vouching marker. - G2: the damage an open finds is written to `DAMAGE` (whole, fully synced, renamed into place, folder synced) before the new segment that hides it from later opens. Every open reports the recorded damage first, until the daemon calls the new `Journal::clear_damage`, which moves the record into `quarantine/` (kept, never deleted). - G3: a folder sync follows a symlink in its path (no O_NOFOLLOW): a state folder, or the journal folder itself, may be a symlink. The sync is for the folder that holds the entries. - G4: the quarantined segments are read for their LSNs only while no live segment is numbered above them (an unfinished quarantine); once one is, its first LSN already carries their floor. - G6: `checked_div` in the scan of the bytes after a cut (the crate's one tracked arithmetic hit). - G5 (the sim split) is the commit before the regressions. * cmux-journal: third-review regressions G7 (A, B) and N1 (red) Third review FIX at cb893c6 (PR #33919 comment 6621681018). Tests only. - G7 A a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn: after a finished quarantine, one byte of the empty new segment's header changes. On this code the next open skips the quarantine scan (the listing counts the damaged segment) and the next append reuses a quarantined LSN. - G7 B a_damaged_newest_segment_after_a_quarantine_reports_its_records: the newest segment holds acked 41..43, and its header and its first record are damaged. On this code the moved segment is scanned with a base far below 41, so the loss is not reported through 43 and the next append is not 44. - N1: the F1 regression also asserts that an open reports each loss once (no reported LSN range lies inside another). On this code a crash after the quarantine's renames makes the next open report the same loss twice. * cmux-journal: fix the third-review findings G7, N1, N2; split recovery.rs Third review FIX at cb893c6 (PR #33919 comment 6621681018); the regression tests are the commit before this one. - G7: the quarantine scan is skipped only when a segment whose header checked out is numbered above every quarantined segment (the folder listing also counted a newest segment with a damaged header, which carries no floor). The scans start from the running floor: `quarantined_floor` and `quarantine` pass the highest LSN counted so far as the base of the next file's scan, so a quarantined file with no readable header and no readable first record is scanned for the LSNs right above what is counted. - The "unfinished quarantine" report is made only when the quarantine holds LSNs nothing else counted (an old quarantine behind a segment that a crash tore at its creation holds none, and a crash is not damage). Its reason now also names a damaged newest header. - N1: a loss whose LSN range lies inside a recorded one is the same loss found again after a crash: it is neither recorded nor reported twice. - N2: recovery.rs's docs name `DAMAGE`; two doc lines over 100 columns are wrapped (lib.rs, sim.rs). - recovery.rs was at 987 of its 1000 lines: the damage record moves to recovery/damage.rs, the entry reader and the LSN scans to recovery/scan.rs (moved as they are). * cmux-journal: report each loss once, also when one open finds it twice (N1) The red run of the N1 regression (fleet 27d011ca, op 64 of the F1 test) shows the case the previous commit did not cover: a crash after the quarantine's renames and before DAMAGE is written; the next open finds both the damaged synced record of the last segment (7..14) and the unfinished quarantine (7..40), and reported both. Now every loss inside another (recorded or found by this open) is dropped, the larger range kept; DAMAGE is rewritten only when its set of losses changed. * cmux-journal: fourth-review regressions G8, N3, N4 (red) Fourth review FIX at 215c8f0 (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report. * cmux-journal: the quarantine's floor in its file names; DAMAGE before the moves (G8, N3, N4) Fourth review FIX at 215c8f0 (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format. * cmux-journal: sort the quarantine list with sort_by_key (clippy unnecessary_sort_by) Fleet d9f22bed at b589044: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`). * cmux-journal: Journal::on_written, a listener per written batch (for S2's projector) R-DURABILITY S2 (acpmux event logs through the journal) projects each chat's log line into its ndjson file once the journal has WRITTEN the batch that holds it, so readers see stream lines at once. The listener runs on the writer thread after the write and before the sync, once per batch that moved the written LSN: a batch written again into a new segment after a failed sync is not heard twice. When the writer stops, these listeners are dropped too. Test: heard while the batch's sync still runs, once. Port of 145faa6 (branch rewrite/r-durability-s2-acpmux-logs) onto the S1 review fixes. * cmux-journal: Journal::retire_through, the one delete (segments a compaction point covers) R-DURABILITY S2: a daemon's projector compacts the journal into its own files (acpmux: the chat ndjson logs, then a synced checkpoint). Without a delete the segment count only grows (review finding F6). `retire_through(compacted)` deletes the segments that hold no record after the caller's compaction point: a segment goes when the next segment's "previous last LSN" is at or before it, so the newest segment never goes. The point is capped at the durable LSN. The deletes and a folder sync run on the caller's thread (the projector, never a request path). A delete that fails leaves its segment live for a later call. Why a crash during a retirement is safe: the caller reopens from a point at or after `compacted`, and recovery reads nothing from a segment the point covers (it skips it on its next segment's header), so any subset of the deletes may reach the disk. A point older than the retired segments is reported as Damage::Missing. Quarantined segments are never deleted. - Storage::remove_file (FsStorage: std::fs::remove_file; SimStorage: one more change in the ordered log of names, durable at the folder sync). - The live segment list: recovery's, then each segment the writer makes. - JournalStats::segments_retired. - Tests: only covered segments go, and a second call deletes none; an OS crash after it keeps every record after the point and an older point is reported; a crash at every operation of a retirement (process, OS crash, power loss) keeps every record after the point and never reuses an LSN; the newest segment stays; a point in the middle of what this writer wrote. This adds a delete of journal data, so its PR is risk:high. * acpmux: move MemoryStore and the store settings into their own files store.rs (2139 lines, 120 fns) and config.rs (1909 lines, 97 fns) are over the god-file limit, and the journal store (R-DURABILITY S2) must not grow them. MemoryStore moves to store/memory.rs and StoreMode, StoreConfig and its defaults to config/store_config.rs; both stay exported where they were (crate::store::MemoryStore, crate::config::{StoreConfig, StoreMode}). No behavior change: the code moves as it is. * acpmux: the event log's journal switch and its crash tests (red) R-DURABILITY S2 (DESIGN.md 3.3b). Today no event-log append is synced: a line a client was told about is on the disk only after the next flush (shutdown), so an OS crash or a power loss takes back every line since (E12). This commit adds only the switch, a seam and the tests; the journal store follows. - `store.journal` (StoreConfig::journal, default false). - `Store::wait_durable(timeout)`: tests and tools wait until every appended record is durable. The default returns at once (the Local and Memory stores never sync an append), which is today's behavior. - `store::open_with_journal` (tests): the store with the journal on a given `cmux_journal::Storage`. Here it ignores the journal and opens LocalStore. - acpmux depends on cmux-journal (dev: with the `sim` feature). - store/journal_store_tests.rs, red on this commit: - a_line_the_store_says_is_durable_survives_an_os_crash (lines 2..20 are lost: nothing synced them); - a_deleted_chat_stays_deleted_after_a_crash and a_line_the_log_refused_is_never_written_later pass here, and must keep passing with the journal store (no replay may bring them back); - a_kill_keeps_every_line_once passes here and must keep passing (a replay after a kill must write no line twice). * acpmux: the event log through one journal, behind `store.journal` (R-DURABILITY S2) DESIGN.md 3.3b. With `store.journal` on, each event-log append is one record in `<home>/journal/` (cmux-journal: group commit, one sync per batch; a power loss takes back at most the last second, always a suffix), then the same line goes to the chat's ndjson as before. Readers, segments and file formats do not change. Default off. - store/journal_store.rs (JournalStore over LocalStore): - append: under one lock, the journal record (a memory copy, no I/O), then the log line; a line the log refuses gets a `cancel` record. Agent stream lines (`in` session/update, claude.*, codex.*) are Class::Stream, every other line Class::Ack. - checkpoint (`<home>/journal-projected.json`): every 30 s while the journal grew, past 64 MiB, at flush (shutdown) and before a retention commit. It syncs each log written since (LocalStore::sync_log, with no lock held while the disk works) and the sessions folder, writes the LSN (temp file, sync, rename, folder sync, every error checked), then retires the journal before it. A log close that failed its sync freezes the checkpoint until a restart (LocalStore::sync_failed). - recovery: the records after the checkpoint; for each chat, the log is cut right after its last readable line below the first journaled seq (later segments removed, the boundary one shortened, their kind-index copies dropped), and the journaled lines are written again in order. `forget` deletes a chat again; `cancel` keeps a refused line out. - delete: `forget` record, folder removal, then a bounded wait until the record is durable. - seal: the logs refuse writes and the journal syncs fully. - LocalStore: `append` is serialize + `append_line`; `sync_log`; a failed sync at a log's close is noted (`sync_failed`). - store::open opens the JournalStore when `store.journal` is on. Not yet (S2c): replies and events do not wait for durability; the daemon does not stop when the journal closes; the writer thread keeps the process's Utility I/O policy. * acpmux: journal store tests for the seal and a chat made again after its delete (red) - a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay: a chat is deleted (its forget record), made again under the same id (a transfer back: session.json and new lines), then a kill -9. On this code the replay deletes the folder again and its session.json is gone. - a_line_after_the_seal_is_refused_and_never_written_later: coverage of the seal (it passes here; the fix also closes a race the test cannot force). * acpmux: journal store: the seal under the appends' lock; a replay keeps a chat made again - seal takes the appends' lock: an append lands in both the journal and the logs, or in neither. Before, an append that raced the seal could leave a journal record whose line the sealed logs refused (its cancel then failed on the closed journal), and the next start would write that line. - append and delete check the logs' seal first, under the same lock, and do not touch the journal after it (no error per append at shutdown). - replay deletes a forgotten chat again only when no line follows its forget record: otherwise the chat was made again under the same id, and its folder holds that chat's own files. * cmux-journal: fifth-review regressions G9 (red) Fifth review FIX at 9c47594 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4. * cmux-journal: name a skipped LSN floor in a FLOOR.<lsn> file (G9) Fifth review FIX at 9c47594 (PR #33919 comment 6623794646); the regression tests are the commit before this one. When an open gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which a copy then hides from every later open), the only record of that floor was the header of the empty segment it makes; a damaged header lost it, and DAMAGE is cleared by `clear_damage` (torn records are not damage at all). Now such an open makes the empty file `FLOOR.<lsn>` and syncs its folder before any new segment, and every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine's names and this open's moves say their floors already: no file is made for them. * acpmux: journal store regressions from the S2 review (red) Review FIX at ab0e953 (PR #34901 comment 6624439824). Each test fails on this commit and passes after the fix that follows it: - 1a lines_written_with_the_switch_off_survive_turning_it_on_again - 1b a_kill_while_the_journal_syncs_keeps_the_lines_the_log_got - 1c a_line_flush_session_made_durable_survives_losing_its_record - 2 a_folder_put_back_after_a_delete_is_not_deleted_by_the_replay - 3 a_kill_in_a_delete_leaves_the_chat_whole_or_gone - 4 a_delete_does_not_wait_for_the_journal_sync - 5 a_failed_log_sync_freezes_the_checkpoint, a_checkpoint_that_keeps_failing_asks_the_daemon_to_stop - 6 one_chat_that_cannot_be_repaired_does_not_stop_the_open - 7 the_journal_storage_refuses_a_lost_home - the repair keeps a log's lines after a hole, also through a crash in it: an_os_crash_hole_is_repaired_and_the_lines_after_it_kept, a_crash_inside_a_repair_loses_no_line - nits: an_unreadable_checkpoint_fails_the_open, the_open_waits_for_the_previous_daemon_to_let_go_of_the_journal, a_crashed_checkpoint_temporary_is_removed_at_open, a_crash_right_after_a_checkpoint_reports_no_damage - the catch-up with the switch off: switching_off_after_a_crash_keeps_every_acknowledged_line - cmux-journal: check_append_says_what_append_would_do So they compile here, this commit adds the seams they use, with no behavior: `Store::stop_reason` (default None), `Journal::check_append` (always Ok), `FencedStorage` (never asks its fence), the open report and a test stop point after a repair's cut. * acpmux: the checkpoint freeze test fails a log sync, not a folder open (red) The freeze guards against a failed fsync (a later fsync of the same file may report success for pages the kernel dropped), not against a folder the store cannot open: a chmod 000 folder is an open error, which the fix retries instead of freezing. The test now fails the log sync itself through a test seam, `FAIL_NEXT_LOG_SYNC`. Red on the code before the fix, like the other review regressions (fleet d8e7037d, 1c22175e, 63604b26 at e030f4f: 17 of 17 new acpmux tests and the crate's check_append test fail). * acpmux: the journal store repairs logs, never rebuilds them (S2 review) Review FIX at ab0e953 (PR #34901 comment 6624439824); the regression tests are the two commits before this one. DESIGN.md 3.3c. - 1a/1b/1c: recovery no longer cuts a chat's log at its first journaled line and writes back only the journal's lines. It merges the log's lines and the journal's by seq: a line in both must be equal, a line only the log has stays, a line only the journal has goes in its place. Only bytes that do not read (a torn tail, a hole of zeros) are dropped. When the merge must change the log before its end, the lines only the log holds from there are journaled and made durable first, so a crash inside the repair loses no line. A log that moved on without the journal (a line in both that differs, lines out of order) is left as it is, and the journal's lines go to journal-unapplied/. - Append writes the log first, then the record, under one lock, after `Journal::check_append` (new in cmux-journal: the append's refusals, asked before any write). So a record always has its log line, and the `cancel` record is gone. - 2/3/4: the replay never deletes a folder and never makes a missing one again (a forget only drops the chat's earlier journaled lines). Delete appends its forget under the lock and removes the folder off it, with no wait for the record. - 5: a failed fsync in a checkpoint freezes it (a folder that does not open is retried); a frozen checkpoint returns an error, and the store asks the daemon to stop for a restart (`Store::stop_reason`, wired in daemon.rs), whose replay writes the logs again. Ten failed checkpoints in a row ask the same, so the journal past the checkpoint stays bounded. - 6: an error in one chat keeps its journaled lines in journal-unapplied/ and the open goes on. - 7: the journal's storage refuses writes under a lost home (`FencedStorage`, `config::refuse_lost_home`). - With store.journal off, a journal left by a run with it on is replayed, checkpointed and retired first (`catch_up`). - Nits: the checkpoint is never past the journal's durable end (no `Ahead` after a normal crash); damage is cleared after the next checkpoint; a kind-index copy goes before its segment is cut; an unreadable checkpoint fails the open; the open waits up to 5 s for the journal lock; crashed checkpoint temporaries are removed; `on_written` leaves cmux-journal (no user); README lists store.journal. * acpmux: split the journal store under its size budget (a move) journal_store.rs grew to 1085 lines and 71 functions with the review fix (budget 1000 and 60), and the fix grew store.rs and daemon.rs, which are over budget on the base. A pure move, no behavior change: - store/journal_store/{checkpoint,fenced,repair}.rs: the checkpoint and its thread, `FencedStorage`, the replay and the repair. - store/log_sync.rs: `LocalStore::sync_log`, `SyncFailed` and the test seam `FAIL_NEXT_LOG_SYNC`, out of store.rs (2139 lines on the base, 2113 now). - daemon/stop_causes.rs: `AbortOnDrop`, `next_signal` and `store_asks_to_stop`, out of daemon.rs (2052 on the base, 2046 now). check-no-godfiles.sh --base 51cfdee on every changed file: no file over budget grew, and no new file is over budget. * acpmux: the unrepairable-chat test waits for line 1 to be durable before its checkpoint At 03fccde (fleet 32df3c72 on aws-m4pro-9, 144d4ccc on Linux) the test kept chat-b's lines [1, 2, 3] aside, not [2, 3]: it flushed right after appending line 1, and the checkpoint is now never past the journal's durable end (the review's Ahead nit), so line 1 was still past it. The fix's behavior is right; the test now waits until line 1 is durable, so its checkpoint covers it. On the code before the fix the test still fails (the open stops at chat-b's link). * acpmux: replies and events wait for durability, tests (red, S2c) DESIGN.md 3.3d. With the event journal on and every journal sync taking 400 ms (`SimStorage::set_sync_delay`): - an_acked_change_is_answered_once_its_record_is_durable: a rename's reply must come no sooner than one sync after its `renamed` record. Red here: the reply goes at once. - agent_output_is_not_held_behind_another_chats_acked_change: agent output to a connection attached to its chat goes at once while another chat's acknowledged change waits for its sync. Green here (nothing waits); it guards the fix against holding agent output behind changes the connection does not hear. * acpmux: replies and events wait until what they show is durable (S2c) DESIGN.md 3.3d; the tests are the commit before this one. With `store.journal` on, each connection gets an outbound gate (server/durable_gate.rs) between it and its transport: - Every line a connection queues raises the gate's `needs` to the last acknowledged LSN (`Store::durability().acked`, set in the journal store's append before the hub publishes anything of the record), except the lines of a `Stream` record's event (agent output, DESIGN C4): the fan-out delivers those in a thread-local stream scope. - The gate's task takes every queued line at once, reads `needs`, waits (async) until the journal's durable LSN reaches it, then sends the batch in order: one wait per batch, as the journal's group commit. - The durable LSN is a `tokio::sync::watch` fed by `Journal::on_durable`. The listener owns a guard: a writer that ends while the store is not being dropped (a panic on its thread) asks the daemon to stop (`Store::stop_reason`, decision 9.8), and the watch closes, so each gate drops what it holds and ends, which closes its connection. - Without the journal, `Store::durability()` is None: no gate, and `Conn::raise_gate` does nothing. Moves, so server/mod.rs and store.rs stay under their size on the base: the outbound queue helpers to server/outbound.rs, `open_with_journal` to the journal store, and `streamed` to store/durability.rs. * acpmux: the journal lock wait's deadline without an unchecked Instant addition `Instant + Duration` can panic on overflow (clippy::arithmetic_side_effects, tracked by the panic ratchet). The wait now has no deadline only in the case that cannot happen (an Instant 5 s ahead overflowing), where it stops waiting at once. * acpmux: a failed log sync stops the real daemon for a restart, test (red) Lead rule 2026-11-04 (3): a daemon-level test of the store's "stop for a restart" before S2c merges, red first. With store.journal on and every log sync failing (a debug build's ACPMUX_TEST_FAIL_LOG_SYNC; the checkpoint every 200 ms through ACPMUX_TEST_JOURNAL_CHECKPOINT_MS), a rename of a chat on disk makes its log dirty, the checkpoint's sync fails and freezes it, and the store asks the daemon to stop. The test asserts: the WebSocket client gets the restart close (1012), the socket client's pending `_acpmux/wait` ends with the connection (no hang), and the daemon exits with status 74. Red here: the daemon stops and sends 1012, but exits 0 (no exit status says that it stopped to be repaired). This commit adds only the two debug-build seams (release builds have none) and the test. * acpmux: a daemon its store stopped for a restart exits with status 74 The test is the commit before this one. `daemon::run` returned Ok and the process exited 0 for every stop, so a supervisor or a person could not tell a stop to be repaired from a normal one; with systemd `Restart=on-failure` it would not even start again. Now the store's stop (`store_asks_to_stop`: a failed log sync froze the event journal's checkpoint, or the journal's writer ended) marks itself, sets the restart handoff (WebSocket clients get 1012, as before), and the process exits with `daemon::EXIT_REPAIR` (74, EX_IOERR). The units prevent only 75, so launchd KeepAlive and systemd Restart=always start it again; the app does not read the daemon's exit status. README: store.journal names the status. * acpmux: the restart test's request in flight goes through the acpmux client Lead rule (3): the client sees a typed error, not a hang. The socket client is now the acpmux client library (acpmux::client::Client, as the CLI uses it), and the test asserts what the CLI makes of the request that the stop ends: `daemon_closed`, retryable (cli::errors::classify). The first red (fleet 41f2d037, hq run 41804640229, at 80c5fb2) failed only on the exit status (0, wanted 74), with 1012 and the end of the connection passing. * acpmux: the daemon's disk tier with the journal on, tests (red) Lead rule 2026-11-04 (2): a reply the person waits for never waits on a throttled disk sync. The fleet A/B (STATUS 11554) found that at IOPOL_UTILITY an acknowledged journal append waits about 16 ms p50 under another process's disk stress (0.7 to 1.1 ms at the default tier). - the_daemon_with_the_journal_keeps_the_default_disk_tier: with store.journal on, the daemon process keeps the default tier; without it, utility as before (each case in a child of the test binary). - a_background_thread_yields_its_disk_work: a dedicated background thread (the checkpointer) works at utility; a thread beside it does not. The API they use is stubbed here (`qos::enter_daemon` ignores the journal; `qos::background_thread` does nothing), so both fail on macOS. * acpmux: the restart test borrows no &str twice (clippy needless_borrows_for_generic_args) Found by fleet 7f8afd9d (hq run: the S2c server step at 62b4f68); the step's other clippy red is the base red in link_push_bench.rs. * acpmux: with the journal on, the daemon keeps the default disk tier (S2c) The tests are 994f741. Lead rule 2026-11-04: a reply the person waits for never waits on a throttled disk sync. Measured on two owned Macs (STATUS 11554): one acknowledged append to durable at IOPOL_UTILITY under another app's disk stress took 16.3-16.7 ms p50 and 37-45 ms p95, against 0.7-1.1 ms p50 and 1.5-1.8 ms p95 at the default tier. - qos::enter_daemon(journal): with store.journal on, the daemon process keeps the default disk tier; without it, utility as before. The switch is read alone from config.json (config::store_journal_enabled), before any thread starts, with no harness discovery. - qos::background_thread: the journal store's checkpointer lowers its own disk I/O to utility for its whole life (a thread that does nothing else: a thread-scope policy drops it out of QoS for good). * acpmux: an agent chunk is a Stream record, test (red) The hub's tap logs an ACP session/update under its sessionUpdate kind (hub/tap.rs: `kind = su.to_owned()`), so a record's kind is `agent_message_chunk`, `tool_call`, ... and never `session/update`; the message's method is. The journal store's `streamed` checked the kind, so every agent chunk was journaled as Ack: a sync per batch at the token rate, and in S2c each token's event waited for that sync (the S2c I/O A/B: STRESS token to client 2.2 ms -> 20.3 ms p50 with the journal on). Red here: the chunk reads back as Ack. * acpmux: an agent chunk is a Stream record (the record's message names session/update) The test is the commit before this one. `streamed` now reads the logged message's method: the hub's tap logs a session/update under its sessionUpdate kind (hub/tap.rs), so the kind never said it. Agent output is again Stream (synced with the next acknowledged batch or a second later), as DESIGN.md 3.3b says. * acpmux: review 2 of #34901, tests (red): a crash model that follows real log syncs The logs' OS crash model now follows the syncs that ran (store/sync_model.rs, tests only): a segment keeps its length at its last sync that counts; a failed sync loses what was written since the one before, and a later sync does not bring it back; a cut ends a loss it removes. Every log segment sync goes through log_sync::sync_segment (same calls as before), where the model and FAIL_NEXT_LOG_SYNC hook in. Red tests, one per finding: - 1a an_os_crash_after_a_restart_from_a_kill_keeps_every_acked_line: the restart's first checkpoint retires records whose log lines no sync made durable; expected [1..5], the head gives [1]. - 1b an_os_crash_after_a_restart_from_a_failed_sync_keeps_every_acked_line: the same after a frozen checkpoint and a restart; expected [1..5]. - 2 a_hole_that_starts_inside_a_line_is_repaired (the reviewer's test): a torn line is read as line 6 of a log that moved on; expected [1..9], the head gives [1, 2, 3, 4, 5, 9]. - 3 a_failed_sync_in_flush_session_freezes_the_checkpoint: a failed sync in flush_session does not freeze the checkpoint. * acpmux: review 2 of #34901: a restart writes the replayed lines again; a torn line is not a line; flush_session freezes Finding 1 (high): the repair now writes every merged line again from the repair start, even when the log already has them all. A killed daemon's log writes may be in the page cache only, and after a failed sync the kernel may have dropped pages that still read back and that a later sync reports as written; the first checkpoint after the open then syncs bytes this process wrote before it retires their records. The lines only the log has from the repair start are journaled and made durable first, as before a cut. The work is bounded by the checkpoint interval. Finding 2: a piece of the log is a line only when it parses as one JSON value (piece_seq). A hole that starts inside a line keeps that line's {"seq":N, head and runs into a later line's tail; it was read as a line of a log that moved on, and the chat's journaled lines went aside. Now its bytes go and the lines are written again. Finding 3: a failed sync in flush_session freezes the checkpoint (its file sync is a SyncFailed now), as a failed sync in a checkpoint or at a log's close does. Nits: cut_at syncs events/ after its removals (and the kind-index folder after its copies go), so a removed segment cannot come back beside the lines written again; the README says that with the switch on an event line over 64 MiB is refused and its log does not get it. Tests: the four red tests of d2c6386. * acpmux: the qos journal tests module is macOS-only (its Linux build held only an unused import: clippy red, review of #35400 F0) * acpmux: review of #35400 F1 and F2, tests (red) - a_read_that_shows_a_line_covers_its_record (F1): a read sees a log line between its log write and its record (the appends' lock held there), and the reply built from it goes out with no wait for that record. - a_journal_that_syncs_nothing_asks_the_daemon_to_stop (F2): with a journal that makes nothing durable, every reply waits without a limit and nothing asks the daemon to stop. Test seams only: PAUSE_AFTER_LOG_WRITE, TEST_STALL_MS. * acpmux: review of #35400 F1, the streaming read, test (red): session/load's replay sends a line before its record * acpmux: the test seam PAUSE_AFTER_LOG_WRITE is reachable from the server tests * acpmux: review of #35400: a read covers the records of the lines it shows; a stalled journal stops the daemon F1: the append writes a line to the log, then appends its record and sets the acknowledged LSN, all under the appends' lock; reads take no lock. A read could show a line before its record existed, and the reply built from it went out at once. Now every journal-store read of the logs ends with a fence on the appends' lock (`Inner::fenced`): a line the read saw has its record appended and its LSN acknowledged when the read returns, so the reply's gate covers it. The one read that sends as it reads (`Hub::scan_events`, session/load's replay) stops at the chat's last_seq read first. F2: a journal writer that keeps failing (it retries without end) held every reply on every connection with no limit. A connection's gate now waits at most 10 s (`STALL_LIMIT`); past it, `Durability::stalled` asks the daemon to stop for a restart, as a failed log sync does. Tests: the 3 of 3916f84 and c553a19. * threading-model: the acpmux event journal's threads and bounds (writer, checkpointer, per-connection gate; review of #35400) * acpmux: the stall test's journal syncs take 3 s, not an hour (a sync under way kept its delay and hung the test's end)
lawrencecchen
added a commit
that referenced
this pull request
Nov 4, 2026
…35483) * cmux-journal: one append-only journal per daemon, with group commit and ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner). * cmux-journal: type a test range; drop an unread segment field (fleet clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71e3f9 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red. * cmux-journal: a failed F_FULLFSYNC is an error, not a plain fsync; no segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee. * cmux-journal: the past-the-end test reads back the storage it wrote to Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c87b8d: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5c53e. * cmux-journal: a sync a signal interrupts is made again (macOS) fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r). * cmux-journal: crash sim of write-back errors and crash in recovery; S1 review regressions (red) The S1 review (FIX at 89293fb15f39) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open * cmux-journal: the F2 regression damages the header of the last of several segments (red) The reviewer's F2 scenario has two segments: S1 holds 1..10, S2 (the last) holds 11..20, and S2's header is damaged. The test now rotates segments (300-byte segments, 20 acked records) and damages the last one's header, so the report must say lost_through 20 and the next append must take 21. On the code before the fix the moved segment's LSNs read as 0, so the next append reuses one. * cmux-journal: the crash sim forces the F4 order (failed sync, kill, OS crash) (red) On the code before the S1 fix, the crash-sim step was green (fleet 184129d8 at 263da86f223): 6 of 7 runs passed, and the mixed run failed only seed 140 in the full run and passed when it ran alone. The random faults reached the order the review found (F4) in 1 of 800 runs. So these tests were no red/green proof. The order: the sync of a write that spans pages fails and the disk gets none of them (Linux keeps them clean in the cache, so a later sync reports success); the process is killed in the writer's backoff; the next open reads the failed segment from the cache, replays its records and goes on; an OS crash or a power loss then takes those pages, and records an open already replayed are gone. - sim.rs: `fail_next_sync(pages, kind)` fails the next sync that has at least that many dirty pages, and the disk gets none of them. `crash_after_failure(n)` makes the n-th operation after the next injected failure the crash point. - crash_sim_tests.rs: a new run forces the order on every seed (a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record): epoch 0 aims the fault, the kill is a process crash at the writer's retry, the open after it does not crash, epoch 1 ends in an OS crash, and a third open checks that the records the second open replayed are all there. In the random runs, one epoch in four aims the fault, with the crash 1 to 3 operations after the failed sync and a crash of any kind after it. - A sim test checks the aimed fault itself (the_sim_fails_the_aimed_sync_and_crashes_in_the_backoff). Expected on this commit (old journal code): the forced run fails every seed; the mixed run fails most runs. The fix commit follows. * cmux-journal: fix the S1 review findings F1-F7 (markers, recovery copy, LSN floor, writer end) Review FIX at 89293fb15f39 (PR #33919 comment 6610468202). The regression tests are the two commits before this one. - Sync markers (format.rs): an entry with flags 1 and no payload, whose LSN field says "every record of this segment before me is synced". The writer appends one after each successful sync that covered records (after the durable listeners, so no reply waits for it), and syncs it fully at a seal and at a rotation. - F4, recovery (step 4 and 6): the last segment is trusted only up to its last marker. Its records after the marker may sit in pages a failed sync dropped (a later sync on another descriptor reports success without them), so recovery writes them again into a new segment: header, records and a marker, fully synced under `<name>.seg.new`, renamed into place, folder synced. The journal always goes on in a segment this open made, never in one an earlier process wrote. A restart in the writer's backoff after a failed sync is then safe. - F1, quarantine: every LSN a quarantined segment holds counts in the LSN floor at every open, and a quarantine that no new segment followed (a crash in the middle of an earlier recovery) is reported as damage again. The highest LSN of every moved segment is read before the first rename. - F2, highest LSN: a segment whose header is damaged is read from its first record (each record's checksum covers its LSN), then the bytes after the first bad entry are scanned for records and markers that still check out. The same scan runs after a cut in the last segment: a marker there is media damage (reported), records there are a torn write (`torn_lsns`); their LSNs are never used again. - F3, writer end: a guard closes the journal when the writer thread ends, also by a panic: appends fail with Closed, waits end with WriterEnded, the listeners are dropped (so a channel a held reply waits on closes), and stats.failing says why. The crate does not abort the process; the daemon owns that policy. - F6: recovery closes each segment after its header is read and opens one again only to read it. (Retirement of compacted segments is an S2 API, before the first consumer.) - F5: recovery syncs the parent folder after it makes the journal folder. - F7: format.rs and lib.rs now say what the code does: a failed write is cut back and retried in the same segment; the cut removes only bytes never synced and never acknowledged; nothing shortens a segment past a synced record. Docs: journal.rs says when open's replayed records are durable (on Ok) and what a stopped writer does. * cmux-journal: split the crash sim's machine model into sim/machine.rs (review G5) sim.rs had 924 lines and 63 functions, over the 60-function limit for a new Rust file (the god-file check warned "not in baseline: split it"). The machine model (each file's page cache and disk, the ordered log of names, and what an OS crash or a power loss leaves) moves to sim/machine.rs (297 lines, 19 functions); the process view and the Storage implementation stay in sim.rs (639 lines, 30 functions). No behavior change: the items move as they are, made visible to the sim module. * cmux-journal: re-review regressions G1-G4 and page eviction in the crash sim (red) Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993). Tests and sim only; the fix follows. - sim: the pages a failed sync dropped stay in the cache until an eviction; `evict_dropped_pages_after_read(true)` makes each read evict the dropped pages it read, so a second read of them sees the disk's bytes. - G1 recovery_copies_the_records_it_checked_when_the_cache_drops_them: a sync fails and drops record 2's pages, a kill in the backoff, the next open reads record 2 from the cache (replayed) and the pages are evicted after that read; then an OS crash. Record 2 and the record 3 acked after the restart must survive. On this code recovery's second, unchecked read stages zeros under a vouching marker, and the last open keeps only record 1 and reports damage. - G2: the F1 regression asserts that every open after the damage reports the loss (lost_through 40). On this code a crash at the open's last step (the folder sync after the new segment's rename) loses the report for good. - G3 a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system): a journal in a symlinked state folder, and a journal folder that is a symlink. On this code the parent folder's sync opens with O_NOFOLLOW (ELOOP). - G4 an_open_after_a_finished_quarantine_reads_no_quarantined_segment: on this code every open reads every quarantined segment in full. * cmux-journal: fix the re-review findings G1-G4 and G6 Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993); the regression tests are the commit before this one (red: fleet de35f6d0). - G1: recovery writes again exactly the bytes its checked read returned. The last segment's entries after its last marker are kept from the checked read (header and payload) and staged from memory; the second, unchecked read of the file is gone. A page that a failed sync dropped can be evicted between two reads, and the second read then returned the disk's zeros, which the copy kept under a vouching marker. - G2: the damage an open finds is written to `DAMAGE` (whole, fully synced, renamed into place, folder synced) before the new segment that hides it from later opens. Every open reports the recorded damage first, until the daemon calls the new `Journal::clear_damage`, which moves the record into `quarantine/` (kept, never deleted). - G3: a folder sync follows a symlink in its path (no O_NOFOLLOW): a state folder, or the journal folder itself, may be a symlink. The sync is for the folder that holds the entries. - G4: the quarantined segments are read for their LSNs only while no live segment is numbered above them (an unfinished quarantine); once one is, its first LSN already carries their floor. - G6: `checked_div` in the scan of the bytes after a cut (the crate's one tracked arithmetic hit). - G5 (the sim split) is the commit before the regressions. * cmux-journal: third-review regressions G7 (A, B) and N1 (red) Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018). Tests only. - G7 A a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn: after a finished quarantine, one byte of the empty new segment's header changes. On this code the next open skips the quarantine scan (the listing counts the damaged segment) and the next append reuses a quarantined LSN. - G7 B a_damaged_newest_segment_after_a_quarantine_reports_its_records: the newest segment holds acked 41..43, and its header and its first record are damaged. On this code the moved segment is scanned with a base far below 41, so the loss is not reported through 43 and the next append is not 44. - N1: the F1 regression also asserts that an open reports each loss once (no reported LSN range lies inside another). On this code a crash after the quarantine's renames makes the next open report the same loss twice. * cmux-journal: fix the third-review findings G7, N1, N2; split recovery.rs Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018); the regression tests are the commit before this one. - G7: the quarantine scan is skipped only when a segment whose header checked out is numbered above every quarantined segment (the folder listing also counted a newest segment with a damaged header, which carries no floor). The scans start from the running floor: `quarantined_floor` and `quarantine` pass the highest LSN counted so far as the base of the next file's scan, so a quarantined file with no readable header and no readable first record is scanned for the LSNs right above what is counted. - The "unfinished quarantine" report is made only when the quarantine holds LSNs nothing else counted (an old quarantine behind a segment that a crash tore at its creation holds none, and a crash is not damage). Its reason now also names a damaged newest header. - N1: a loss whose LSN range lies inside a recorded one is the same loss found again after a crash: it is neither recorded nor reported twice. - N2: recovery.rs's docs name `DAMAGE`; two doc lines over 100 columns are wrapped (lib.rs, sim.rs). - recovery.rs was at 987 of its 1000 lines: the damage record moves to recovery/damage.rs, the entry reader and the LSN scans to recovery/scan.rs (moved as they are). * cmux-journal: report each loss once, also when one open finds it twice (N1) The red run of the N1 regression (fleet 27d011ca, op 64 of the F1 test) shows the case the previous commit did not cover: a crash after the quarantine's renames and before DAMAGE is written; the next open finds both the damaged synced record of the last segment (7..14) and the unfinished quarantine (7..40), and reported both. Now every loss inside another (recorded or found by this open) is dropped, the larger range kept; DAMAGE is rewritten only when its set of losses changed. * cmux-journal: fourth-review regressions G8, N3, N4 (red) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report. * cmux-journal: the quarantine's floor in its file names; DAMAGE before the moves (G8, N3, N4) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format. * cmux-journal: sort the quarantine list with sort_by_key (clippy unnecessary_sort_by) Fleet d9f22bed at b589044b852: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`). * cmux-journal: Journal::on_written, a listener per written batch (for S2's projector) R-DURABILITY S2 (acpmux event logs through the journal) projects each chat's log line into its ndjson file once the journal has WRITTEN the batch that holds it, so readers see stream lines at once. The listener runs on the writer thread after the write and before the sync, once per batch that moved the written LSN: a batch written again into a new segment after a failed sync is not heard twice. When the writer stops, these listeners are dropped too. Test: heard while the batch's sync still runs, once. Port of 145faa6faa8 (branch rewrite/r-durability-s2-acpmux-logs) onto the S1 review fixes. * cmux-journal: Journal::retire_through, the one delete (segments a compaction point covers) R-DURABILITY S2: a daemon's projector compacts the journal into its own files (acpmux: the chat ndjson logs, then a synced checkpoint). Without a delete the segment count only grows (review finding F6). `retire_through(compacted)` deletes the segments that hold no record after the caller's compaction point: a segment goes when the next segment's "previous last LSN" is at or before it, so the newest segment never goes. The point is capped at the durable LSN. The deletes and a folder sync run on the caller's thread (the projector, never a request path). A delete that fails leaves its segment live for a later call. Why a crash during a retirement is safe: the caller reopens from a point at or after `compacted`, and recovery reads nothing from a segment the point covers (it skips it on its next segment's header), so any subset of the deletes may reach the disk. A point older than the retired segments is reported as Damage::Missing. Quarantined segments are never deleted. - Storage::remove_file (FsStorage: std::fs::remove_file; SimStorage: one more change in the ordered log of names, durable at the folder sync). - The live segment list: recovery's, then each segment the writer makes. - JournalStats::segments_retired. - Tests: only covered segments go, and a second call deletes none; an OS crash after it keeps every record after the point and an older point is reported; a crash at every operation of a retirement (process, OS crash, power loss) keeps every record after the point and never reuses an LSN; the newest segment stays; a point in the middle of what this writer wrote. This adds a delete of journal data, so its PR is risk:high. * acpmux: move MemoryStore and the store settings into their own files store.rs (2139 lines, 120 fns) and config.rs (1909 lines, 97 fns) are over the god-file limit, and the journal store (R-DURABILITY S2) must not grow them. MemoryStore moves to store/memory.rs and StoreMode, StoreConfig and its defaults to config/store_config.rs; both stay exported where they were (crate::store::MemoryStore, crate::config::{StoreConfig, StoreMode}). No behavior change: the code moves as it is. * acpmux: the event log's journal switch and its crash tests (red) R-DURABILITY S2 (DESIGN.md 3.3b). Today no event-log append is synced: a line a client was told about is on the disk only after the next flush (shutdown), so an OS crash or a power loss takes back every line since (E12). This commit adds only the switch, a seam and the tests; the journal store follows. - `store.journal` (StoreConfig::journal, default false). - `Store::wait_durable(timeout)`: tests and tools wait until every appended record is durable. The default returns at once (the Local and Memory stores never sync an append), which is today's behavior. - `store::open_with_journal` (tests): the store with the journal on a given `cmux_journal::Storage`. Here it ignores the journal and opens LocalStore. - acpmux depends on cmux-journal (dev: with the `sim` feature). - store/journal_store_tests.rs, red on this commit: - a_line_the_store_says_is_durable_survives_an_os_crash (lines 2..20 are lost: nothing synced them); - a_deleted_chat_stays_deleted_after_a_crash and a_line_the_log_refused_is_never_written_later pass here, and must keep passing with the journal store (no replay may bring them back); - a_kill_keeps_every_line_once passes here and must keep passing (a replay after a kill must write no line twice). * acpmux: the event log through one journal, behind `store.journal` (R-DURABILITY S2) DESIGN.md 3.3b. With `store.journal` on, each event-log append is one record in `<home>/journal/` (cmux-journal: group commit, one sync per batch; a power loss takes back at most the last second, always a suffix), then the same line goes to the chat's ndjson as before. Readers, segments and file formats do not change. Default off. - store/journal_store.rs (JournalStore over LocalStore): - append: under one lock, the journal record (a memory copy, no I/O), then the log line; a line the log refuses gets a `cancel` record. Agent stream lines (`in` session/update, claude.*, codex.*) are Class::Stream, every other line Class::Ack. - checkpoint (`<home>/journal-projected.json`): every 30 s while the journal grew, past 64 MiB, at flush (shutdown) and before a retention commit. It syncs each log written since (LocalStore::sync_log, with no lock held while the disk works) and the sessions folder, writes the LSN (temp file, sync, rename, folder sync, every error checked), then retires the journal before it. A log close that failed its sync freezes the checkpoint until a restart (LocalStore::sync_failed). - recovery: the records after the checkpoint; for each chat, the log is cut right after its last readable line below the first journaled seq (later segments removed, the boundary one shortened, their kind-index copies dropped), and the journaled lines are written again in order. `forget` deletes a chat again; `cancel` keeps a refused line out. - delete: `forget` record, folder removal, then a bounded wait until the record is durable. - seal: the logs refuse writes and the journal syncs fully. - LocalStore: `append` is serialize + `append_line`; `sync_log`; a failed sync at a log's close is noted (`sync_failed`). - store::open opens the JournalStore when `store.journal` is on. Not yet (S2c): replies and events do not wait for durability; the daemon does not stop when the journal closes; the writer thread keeps the process's Utility I/O policy. * acpmux: journal store tests for the seal and a chat made again after its delete (red) - a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay: a chat is deleted (its forget record), made again under the same id (a transfer back: session.json and new lines), then a kill -9. On this code the replay deletes the folder again and its session.json is gone. - a_line_after_the_seal_is_refused_and_never_written_later: coverage of the seal (it passes here; the fix also closes a race the test cannot force). * acpmux: journal store: the seal under the appends' lock; a replay keeps a chat made again - seal takes the appends' lock: an append lands in both the journal and the logs, or in neither. Before, an append that raced the seal could leave a journal record whose line the sealed logs refused (its cancel then failed on the closed journal), and the next start would write that line. - append and delete check the logs' seal first, under the same lock, and do not touch the journal after it (no error per append at shutdown). - replay deletes a forgotten chat again only when no line follows its forget record: otherwise the chat was made again under the same id, and its folder holds that chat's own files. * cmux-journal: fifth-review regressions G9 (red) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4. * cmux-journal: name a skipped LSN floor in a FLOOR.<lsn> file (G9) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646); the regression tests are the commit before this one. When an open gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which a copy then hides from every later open), the only record of that floor was the header of the empty segment it makes; a damaged header lost it, and DAMAGE is cleared by `clear_damage` (torn records are not damage at all). Now such an open makes the empty file `FLOOR.<lsn>` and syncs its folder before any new segment, and every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine's names and this open's moves say their floors already: no file is made for them. * acpmux: journal store regressions from the S2 review (red) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824). Each test fails on this commit and passes after the fix that follows it: - 1a lines_written_with_the_switch_off_survive_turning_it_on_again - 1b a_kill_while_the_journal_syncs_keeps_the_lines_the_log_got - 1c a_line_flush_session_made_durable_survives_losing_its_record - 2 a_folder_put_back_after_a_delete_is_not_deleted_by_the_replay - 3 a_kill_in_a_delete_leaves_the_chat_whole_or_gone - 4 a_delete_does_not_wait_for_the_journal_sync - 5 a_failed_log_sync_freezes_the_checkpoint, a_checkpoint_that_keeps_failing_asks_the_daemon_to_stop - 6 one_chat_that_cannot_be_repaired_does_not_stop_the_open - 7 the_journal_storage_refuses_a_lost_home - the repair keeps a log's lines after a hole, also through a crash in it: an_os_crash_hole_is_repaired_and_the_lines_after_it_kept, a_crash_inside_a_repair_loses_no_line - nits: an_unreadable_checkpoint_fails_the_open, the_open_waits_for_the_previous_daemon_to_let_go_of_the_journal, a_crashed_checkpoint_temporary_is_removed_at_open, a_crash_right_after_a_checkpoint_reports_no_damage - the catch-up with the switch off: switching_off_after_a_crash_keeps_every_acknowledged_line - cmux-journal: check_append_says_what_append_would_do So they compile here, this commit adds the seams they use, with no behavior: `Store::stop_reason` (default None), `Journal::check_append` (always Ok), `FencedStorage` (never asks its fence), the open report and a test stop point after a repair's cut. * acpmux: the checkpoint freeze test fails a log sync, not a folder open (red) The freeze guards against a failed fsync (a later fsync of the same file may report success for pages the kernel dropped), not against a folder the store cannot open: a chmod 000 folder is an open error, which the fix retries instead of freezing. The test now fails the log sync itself through a test seam, `FAIL_NEXT_LOG_SYNC`. Red on the code before the fix, like the other review regressions (fleet d8e7037d, 1c22175e, 63604b26 at e030f4fe402: 17 of 17 new acpmux tests and the crate's check_append test fail). * acpmux: the journal store repairs logs, never rebuilds them (S2 review) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824); the regression tests are the two commits before this one. DESIGN.md 3.3c. - 1a/1b/1c: recovery no longer cuts a chat's log at its first journaled line and writes back only the journal's lines. It merges the log's lines and the journal's by seq: a line in both must be equal, a line only the log has stays, a line only the journal has goes in its place. Only bytes that do not read (a torn tail, a hole of zeros) are dropped. When the merge must change the log before its end, the lines only the log holds from there are journaled and made durable first, so a crash inside the repair loses no line. A log that moved on without the journal (a line in both that differs, lines out of order) is left as it is, and the journal's lines go to journal-unapplied/. - Append writes the log first, then the record, under one lock, after `Journal::check_append` (new in cmux-journal: the append's refusals, asked before any write). So a record always has its log line, and the `cancel` record is gone. - 2/3/4: the replay never deletes a folder and never makes a missing one again (a forget only drops the chat's earlier journaled lines). Delete appends its forget under the lock and removes the folder off it, with no wait for the record. - 5: a failed fsync in a checkpoint freezes it (a folder that does not open is retried); a frozen checkpoint returns an error, and the store asks the daemon to stop for a restart (`Store::stop_reason`, wired in daemon.rs), whose replay writes the logs again. Ten failed checkpoints in a row ask the same, so the journal past the checkpoint stays bounded. - 6: an error in one chat keeps its journaled lines in journal-unapplied/ and the open goes on. - 7: the journal's storage refuses writes under a lost home (`FencedStorage`, `config::refuse_lost_home`). - With store.journal off, a journal left by a run with it on is replayed, checkpointed and retired first (`catch_up`). - Nits: the checkpoint is never past the journal's durable end (no `Ahead` after a normal crash); damage is cleared after the next checkpoint; a kind-index copy goes before its segment is cut; an unreadable checkpoint fails the open; the open waits up to 5 s for the journal lock; crashed checkpoint temporaries are removed; `on_written` leaves cmux-journal (no user); README lists store.journal. * acpmux: split the journal store under its size budget (a move) journal_store.rs grew to 1085 lines and 71 functions with the review fix (budget 1000 and 60), and the fix grew store.rs and daemon.rs, which are over budget on the base. A pure move, no behavior change: - store/journal_store/{checkpoint,fenced,repair}.rs: the checkpoint and its thread, `FencedStorage`, the replay and the repair. - store/log_sync.rs: `LocalStore::sync_log`, `SyncFailed` and the test seam `FAIL_NEXT_LOG_SYNC`, out of store.rs (2139 lines on the base, 2113 now). - daemon/stop_causes.rs: `AbortOnDrop`, `next_signal` and `store_asks_to_stop`, out of daemon.rs (2052 on the base, 2046 now). check-no-godfiles.sh --base 51cfdee876b on every changed file: no file over budget grew, and no new file is over budget. * acpmux: the unrepairable-chat test waits for line 1 to be durable before its checkpoint At 03fccdef69c (fleet 32df3c72 on aws-m4pro-9, 144d4ccc on Linux) the test kept chat-b's lines [1, 2, 3] aside, not [2, 3]: it flushed right after appending line 1, and the checkpoint is now never past the journal's durable end (the review's Ahead nit), so line 1 was still past it. The fix's behavior is right; the test now waits until line 1 is durable, so its checkpoint covers it. On the code before the fix the test still fails (the open stops at chat-b's link). * acpmux: replies and events wait for durability, tests (red, S2c) DESIGN.md 3.3d. With the event journal on and every journal sync taking 400 ms (`SimStorage::set_sync_delay`): - an_acked_change_is_answered_once_its_record_is_durable: a rename's reply must come no sooner than one sync after its `renamed` record. Red here: the reply goes at once. - agent_output_is_not_held_behind_another_chats_acked_change: agent output to a connection attached to its chat goes at once while another chat's acknowledged change waits for its sync. Green here (nothing waits); it guards the fix against holding agent output behind changes the connection does not hear. * acpmux: replies and events wait until what they show is durable (S2c) DESIGN.md 3.3d; the tests are the commit before this one. With `store.journal` on, each connection gets an outbound gate (server/durable_gate.rs) between it and its transport: - Every line a connection queues raises the gate's `needs` to the last acknowledged LSN (`Store::durability().acked`, set in the journal store's append before the hub publishes anything of the record), except the lines of a `Stream` record's event (agent output, DESIGN C4): the fan-out delivers those in a thread-local stream scope. - The gate's task takes every queued line at once, reads `needs`, waits (async) until the journal's durable LSN reaches it, then sends the batch in order: one wait per batch, as the journal's group commit. - The durable LSN is a `tokio::sync::watch` fed by `Journal::on_durable`. The listener owns a guard: a writer that ends while the store is not being dropped (a panic on its thread) asks the daemon to stop (`Store::stop_reason`, decision 9.8), and the watch closes, so each gate drops what it holds and ends, which closes its connection. - Without the journal, `Store::durability()` is None: no gate, and `Conn::raise_gate` does nothing. Moves, so server/mod.rs and store.rs stay under their size on the base: the outbound queue helpers to server/outbound.rs, `open_with_journal` to the journal store, and `streamed` to store/durability.rs. * acpmux: the journal lock wait's deadline without an unchecked Instant addition `Instant + Duration` can panic on overflow (clippy::arithmetic_side_effects, tracked by the panic ratchet). The wait now has no deadline only in the case that cannot happen (an Instant 5 s ahead overflowing), where it stops waiting at once. * acpmux: a failed log sync stops the real daemon for a restart, test (red) Lead rule 2026-11-04 (3): a daemon-level test of the store's "stop for a restart" before S2c merges, red first. With store.journal on and every log sync failing (a debug build's ACPMUX_TEST_FAIL_LOG_SYNC; the checkpoint every 200 ms through ACPMUX_TEST_JOURNAL_CHECKPOINT_MS), a rename of a chat on disk makes its log dirty, the checkpoint's sync fails and freezes it, and the store asks the daemon to stop. The test asserts: the WebSocket client gets the restart close (1012), the socket client's pending `_acpmux/wait` ends with the connection (no hang), and the daemon exits with status 74. Red here: the daemon stops and sends 1012, but exits 0 (no exit status says that it stopped to be repaired). This commit adds only the two debug-build seams (release builds have none) and the test. * acpmux: a daemon its store stopped for a restart exits with status 74 The test is the commit before this one. `daemon::run` returned Ok and the process exited 0 for every stop, so a supervisor or a person could not tell a stop to be repaired from a normal one; with systemd `Restart=on-failure` it would not even start again. Now the store's stop (`store_asks_to_stop`: a failed log sync froze the event journal's checkpoint, or the journal's writer ended) marks itself, sets the restart handoff (WebSocket clients get 1012, as before), and the process exits with `daemon::EXIT_REPAIR` (74, EX_IOERR). The units prevent only 75, so launchd KeepAlive and systemd Restart=always start it again; the app does not read the daemon's exit status. README: store.journal names the status. * acpmux: the restart test's request in flight goes through the acpmux client Lead rule (3): the client sees a typed error, not a hang. The socket client is now the acpmux client library (acpmux::client::Client, as the CLI uses it), and the test asserts what the CLI makes of the request that the stop ends: `daemon_closed`, retryable (cli::errors::classify). The first red (fleet 41f2d037, hq run 41804640229, at 80c5fb2b7f8) failed only on the exit status (0, wanted 74), with 1012 and the end of the connection passing. * acpmux: the daemon's disk tier with the journal on, tests (red) Lead rule 2026-11-04 (2): a reply the person waits for never waits on a throttled disk sync. The fleet A/B (STATUS 11554) found that at IOPOL_UTILITY an acknowledged journal append waits about 16 ms p50 under another process's disk stress (0.7 to 1.1 ms at the default tier). - the_daemon_with_the_journal_keeps_the_default_disk_tier: with store.journal on, the daemon process keeps the default tier; without it, utility as before (each case in a child of the test binary). - a_background_thread_yields_its_disk_work: a dedicated background thread (the checkpointer) works at utility; a thread beside it does not. The API they use is stubbed here (`qos::enter_daemon` ignores the journal; `qos::background_thread` does nothing), so both fail on macOS. * acpmux: the restart test borrows no &str twice (clippy needless_borrows_for_generic_args) Found by fleet 7f8afd9d (hq run: the S2c server step at 62b4f68ef8b); the step's other clippy red is the base red in link_push_bench.rs. * acpmux: with the journal on, the daemon keeps the default disk tier (S2c) The tests are 994f74119fb. Lead rule 2026-11-04: a reply the person waits for never waits on a throttled disk sync. Measured on two owned Macs (STATUS 11554): one acknowledged append to durable at IOPOL_UTILITY under another app's disk stress took 16.3-16.7 ms p50 and 37-45 ms p95, against 0.7-1.1 ms p50 and 1.5-1.8 ms p95 at the default tier. - qos::enter_daemon(journal): with store.journal on, the daemon process keeps the default disk tier; without it, utility as before. The switch is read alone from config.json (config::store_journal_enabled), before any thread starts, with no harness discovery. - qos::background_thread: the journal store's checkpointer lowers its own disk I/O to utility for its whole life (a thread that does nothing else: a thread-scope policy drops it out of QoS for good). * acpmux: an agent chunk is a Stream record, test (red) The hub's tap logs an ACP session/update under its sessionUpdate kind (hub/tap.rs: `kind = su.to_owned()`), so a record's kind is `agent_message_chunk`, `tool_call`, ... and never `session/update`; the message's method is. The journal store's `streamed` checked the kind, so every agent chunk was journaled as Ack: a sync per batch at the token rate, and in S2c each token's event waited for that sync (the S2c I/O A/B: STRESS token to client 2.2 ms -> 20.3 ms p50 with the journal on). Red here: the chunk reads back as Ack. * acpmux: an agent chunk is a Stream record (the record's message names session/update) The test is the commit before this one. `streamed` now reads the logged message's method: the hub's tap logs a session/update under its sessionUpdate kind (hub/tap.rs), so the kind never said it. Agent output is again Stream (synced with the next acknowledged batch or a second later), as DESIGN.md 3.3b says. * acpmux: review 2 of #34901, tests (red): a crash model that follows real log syncs The logs' OS crash model now follows the syncs that ran (store/sync_model.rs, tests only): a segment keeps its length at its last sync that counts; a failed sync loses what was written since the one before, and a later sync does not bring it back; a cut ends a loss it removes. Every log segment sync goes through log_sync::sync_segment (same calls as before), where the model and FAIL_NEXT_LOG_SYNC hook in. Red tests, one per finding: - 1a an_os_crash_after_a_restart_from_a_kill_keeps_every_acked_line: the restart's first checkpoint retires records whose log lines no sync made durable; expected [1..5], the head gives [1]. - 1b an_os_crash_after_a_restart_from_a_failed_sync_keeps_every_acked_line: the same after a frozen checkpoint and a restart; expected [1..5]. - 2 a_hole_that_starts_inside_a_line_is_repaired (the reviewer's test): a torn line is read as line 6 of a log that moved on; expected [1..9], the head gives [1, 2, 3, 4, 5, 9]. - 3 a_failed_sync_in_flush_session_freezes_the_checkpoint: a failed sync in flush_session does not freeze the checkpoint. * acpmux: review 2 of #34901: a restart writes the replayed lines again; a torn line is not a line; flush_session freezes Finding 1 (high): the repair now writes every merged line again from the repair start, even when the log already has them all. A killed daemon's log writes may be in the page cache only, and after a failed sync the kernel may have dropped pages that still read back and that a later sync reports as written; the first checkpoint after the open then syncs bytes this process wrote before it retires their records. The lines only the log has from the repair start are journaled and made durable first, as before a cut. The work is bounded by the checkpoint interval. Finding 2: a piece of the log is a line only when it parses as one JSON value (piece_seq). A hole that starts inside a line keeps that line's {"seq":N, head and runs into a later line's tail; it was read as a line of a log that moved on, and the chat's journaled lines went aside. Now its bytes go and the lines are written again. Finding 3: a failed sync in flush_session freezes the checkpoint (its file sync is a SyncFailed now), as a failed sync in a checkpoint or at a log's close does. Nits: cut_at syncs events/ after its removals (and the kind-index folder after its copies go), so a removed segment cannot come back beside the lines written again; the README says that with the switch on an event line over 64 MiB is refused and its log does not get it. Tests: the four red tests of d2c63860ece. * acpmux: chat metadata through the event journal, tests (red, R-DURABILITY S3a) DESIGN.md 3.3e. Red on S2c: - last_seq_never_passes_the_log_after_an_os_crash (E13): session.json is synced at each save while the agent's lines wait for the journal's next sync, so after an OS crash the metadata names lines the log lost. - a_chat_made_after_the_checkpoint_survives_an_os_crash_that_loses_its_folder: nothing syncs the sessions folder when a chat's folder is made, and the replay never makes a folder again, so an acknowledged new chat is lost. - a_metadata_change_is_written_to_session_json_by_the_checkpoint: today each change writes and syncs session.json on the request path. Guards (green before and after): a chat made before the checkpoint and deleted is not made again; the replay never writes an older image over session.json. * acpmux: chat metadata through the event journal (R-DURABILITY S3a) DESIGN.md 3.3e, behind `store.journal` (default off). A metadata change is a `meta` record (kind 4: chat id, a `created` flag, the whole image) instead of a session.json write with two syncs on the request path; the reply waits for the record through S2c's gate, one group commit for a burst. - Overlay: each chat's newest image until a checkpoint writes session.json; load, load_startup and list read it first (list under the projection lock, so a checkpoint cannot drop an entry between its two reads). - Checkpoint: waits on its own thread until every record through its LSN is durable, then writes each image it read (session.json carries the image's LSN as `journalLsn`), then the logs and the sessions folder, as before. An image so names only lines the journal holds (E13). A chat deleted or made again meanwhile is passed (its overlay entry is gone or new). - Replay: per chat after its last `forget`, the newest image, written when newer than the file's `journalLsn`. A missing folder is made again only for a chat a replayed `created` record made; a chat made before the checkpoint whose folder is gone keeps its image in journal-unapplied/. - flush_session also waits until the chat's records are durable. - Delete drops the overlay entry with its `forget`, and removes the folder under the folders lock a checkpoint's write also takes. Tests: the 5 of 0505e9bee04 (3 red there). * acpmux: S3a, tests (red): folders handed out of the store Code outside the store reads and moves chat folders directly (transfer_in puts the live folder aside and installs a transferred chat; session_dir hands the folder to the transfer, the session files view and others). With the metadata in the overlay until a checkpoint: - a_folder_replaced_outside_the_store_keeps_its_own_metadata: a read, the checkpoint and the replay must not put the old folder's image in the new one. - session_dir_hands_out_a_folder_whose_metadata_is_written. - flush_session_writes_the_chats_metadata (the trait says log and metadata). - the_checkpoint_never_makes_a_folder_removed_outside_the_store_again. Red on 71bc09a1e63 (S3a's first implementation). * acpmux: S3a red tests: a replay after a folder swap; flush_session's chat has a log (as its callers' chats do) * acpmux: S3a: each metadata image is bound to its chat folder; session_dir and flush_session write it first Code outside the store reads and moves chat folders directly (a transfer puts the live folder aside and installs a transferred chat under the same id). So a `meta` record now carries its folder's identity (file id and birth time): - reads use an overlay image only while the folder at the path is its folder; - the checkpoint writes an image only into its own folder, never makes a missing one, and keeps the image of a removed or replaced folder in journal-unapplied/; - the replay writes an image only into its own folder (or a folder a replayed `created` record made), else keeps it aside. `session_dir` (the transfer, the session files view) and `flush_session` (the trait: "log and metadata to stable storage") write the chat's image first, once its record is durable. The list reads the overlay before the files (a checkpoint writes a file before it drops the entry), so it needs no lock. Tests: the 5 of 0280ac37f5e and 0e617489731. * acpmux: the qos journal tests module is macOS-only (its Linux build held only an unused import: clippy red, review of #35400 F0) * acpmux: review of #35400 F1 and F2, tests (red) - a_read_that_shows_a_line_covers_its_record (F1): a read sees a log line between its log write and its record (the appends' lock held there), and the reply built from it goes out with no wait for that record. - a_journal_that_syncs_nothing_asks_the_daemon_to_stop (F2): with a journal that makes nothing durable, every reply waits without a limit and nothing asks the daemon to stop. Test seams only: PAUSE_AFTER_LOG_WRITE, TEST_STALL_MS. * acpmux: review of #35400 F1, the streaming read, test (red): session/load's replay sends a line before its record * acpmux: the test seam PAUSE_AFTER_LOG_WRITE is reachable from the server tests * acpmux: review of #35400: a read covers the records of the lines it shows; a stalled journal stops the daemon F1: the append writes a line to the log, then appends its record and sets the acknowledged LSN, all under the appends' lock; reads take no lock. A read could show a line before its record existed, and the reply built from it went out at once. Now every journal-store read of the logs ends with a fence on the appends' lock (`Inner::fenced`): a line the read saw has its record appended and its LSN acknowledged when the read returns, so the reply's gate covers it. The one read that sends as it reads (`Hub::scan_events`, session/load's replay) stops at the chat's last_seq read first. F2: a journal writer that keeps failing (it retries without end) held every reply on every connection with no limit. A connection's gate now waits at most 10 s (`STALL_LIMIT`); past it, `Durability::stalled` asks the daemon to stop for a restart, as a failed log sync does. Tests: the 3 of 3916f84dd68 and c553a19fd8b. * threading-model: the acpmux event journal's threads and bounds (writer, checkpointer, per-connection gate; review of #35400) * acpmux: the stall test's journal syncs take 3 s, not an hour (a sync under way kept its delay and hung the test's end) * acpmux: S3a review (#35483) red tests: switch-off metadata and deletes, a late settle, a failed session.json folder sync, a purge after the checkpoint F1: metadata written with the switch off survives turning it on again; a chat deleted with the switch off after a failed catch-up is not made again. F2: a late settle never writes an older image over a checkpointed one (the reviewer's test and hold). F3: a failed folder sync after a metadata write freezes the checkpoint. F4: a chat made after the checkpoint and deleted is not made again after a kill. Test seams only; the code is the S3a head merged with feat. * acpmux: S3a review (#35483) red tests: a new chat is acknowledged only once its folder is durable; session_path never waits The root of F4 and of F1's delete case: the replay makes a chat's folder again, so a chat removed by any path comes back. The fix will never make a folder again, which needs an acknowledged chat's folder to survive an OS crash: the test models what each folder's last sync kept (sync_model) and asserts the gate's durability passes a new chat only after its sessions entry is synced. P3 1 of the review: a caller that needs only the path never waits for the journal (Store::session_path, a stub with session_dir's behavior here). * acpmux: S3a review (#35483) fixes: the replay never makes a folder again; switch-off changes are kept; no late or unsynced metadata write F1: each meta record carries the stamp (file id, modification time, length) of the session.json its save saw. The replay writes an image over a file the journal did not write only when one of the chat's records saw that file; otherwise the file moved on without the journal (a run with store.journal off whose catch-up did not replay, or an older build) and the image goes to journal-unapplied/, as S2 does for a log. F1 (deletes) and F4 at the root: the replay never makes a chat's folder again. A save that made a folder notes its record's LSN; the new acpmux-journal-folders thread (the daemon's disk tier) syncs sessions/ at once, and the durable LSN the S2c gate waits for stays below that record until then (journal_store/new_folders.rs). So an acknowledged chat's folder survives an OS crash, and a folder that is gone was removed: a purge whose forget a kill took back, a run with the switch off, the transfer recovery. The created flag and the replay's folder remake are gone. F2: write_through does not write a file that holds the same or a newer journalLsn. F3: session.json is written with its own syncs (file, then folder); a failed sync is SyncFailed and freezes the checkpoint; the replay and the kept-aside images return their errors. P3 1: Store::session_path gives the path with no lock, wait or write; an agent's start, the fixed-id check, the temporary-folder sweep, the host probe and the transfer recovery's root use it. The replaced test (a chat made after the checkpoint survives an OS crash that loses its folder) asserted a remake that cannot be right any more; the new test asserts the guarantee that makes the loss impossible, then the crash. Red (7 of 17 journal_meta_tests, each for its reason; the other 10 pass): macOS 5bc434a9 and 624d5df4, Linux a2207ff0 and 638a44ef. * acpmux: S3a delta review (N1) red tests: a change to a chat whose session.json another journal wrote reaches it A folder moved in from another home (build adoption), checkpointed and replayed, and a journal made again from nothing: the other journal's journalLsn must not hide a change made here. The first test is the reviewer's (fleet 4d0d98bc and 071ad7c4 at e944a7c21de). * acpmux: S3a delta review (N1): journalLsn is bound to its journal's id; a file another journal wrote follows the stamp rule session.json now carries journalId (16 hex digits, a string) beside journalLsn. The checkpoint's "held" check (F2) and the replay's rule compare an LSN only for a file with this journal's id: a file from another journal (a chat folder build adoption moved in from another home, or a home whose journal was made again) counts as one this journal did not write, so the stamp rule decides, as for a run with the switch off. cmux-journal: Journal::journal_id (the id already in every segment header; kept across opens and retirements), with a unit test. Red at 7c8cb957745: fleet c29b659d (Linux) and 47570494 (macOS) running; the reviewer's red at e944a7c21de: 4d0d98bc (macOS) and 071ad7c4 (Linux).
lawrencecchen
added a commit
that referenced
this pull request
Nov 4, 2026
…a hole's last zero splits, an idle checkpointer sleeps (#35562) * cmux-journal: one append-only journal per daemon, with group commit and ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner). * cmux-journal: type a test range; drop an unread segment field (fleet clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71e3f9 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red. * cmux-journal: a failed F_FULLFSYNC is an error, not a plain fsync; no segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee. * cmux-journal: the past-the-end test reads back the storage it wrote to Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c87b8d: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5c53e. * cmux-journal: a sync a signal interrupts is made again (macOS) fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r). * cmux-journal: crash sim of write-back errors and crash in recovery; S1 review regressions (red) The S1 review (FIX at 89293fb15f39) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open * cmux-journal: the F2 regression damages the header of the last of several segments (red) The reviewer's F2 scenario has two segments: S1 holds 1..10, S2 (the last) holds 11..20, and S2's header is damaged. The test now rotates segments (300-byte segments, 20 acked records) and damages the last one's header, so the report must say lost_through 20 and the next append must take 21. On the code before the fix the moved segment's LSNs read as 0, so the next append reuses one. * cmux-journal: the crash sim forces the F4 order (failed sync, kill, OS crash) (red) On the code before the S1 fix, the crash-sim step was green (fleet 184129d8 at 263da86f223): 6 of 7 runs passed, and the mixed run failed only seed 140 in the full run and passed when it ran alone. The random faults reached the order the review found (F4) in 1 of 800 runs. So these tests were no red/green proof. The order: the sync of a write that spans pages fails and the disk gets none of them (Linux keeps them clean in the cache, so a later sync reports success); the process is killed in the writer's backoff; the next open reads the failed segment from the cache, replays its records and goes on; an OS crash or a power loss then takes those pages, and records an open already replayed are gone. - sim.rs: `fail_next_sync(pages, kind)` fails the next sync that has at least that many dirty pages, and the disk gets none of them. `crash_after_failure(n)` makes the n-th operation after the next injected failure the crash point. - crash_sim_tests.rs: a new run forces the order on every seed (a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record): epoch 0 aims the fault, the kill is a process crash at the writer's retry, the open after it does not crash, epoch 1 ends in an OS crash, and a third open checks that the records the second open replayed are all there. In the random runs, one epoch in four aims the fault, with the crash 1 to 3 operations after the failed sync and a crash of any kind after it. - A sim test checks the aimed fault itself (the_sim_fails_the_aimed_sync_and_crashes_in_the_backoff). Expected on this commit (old journal code): the forced run fails every seed; the mixed run fails most runs. The fix commit follows. * cmux-journal: fix the S1 review findings F1-F7 (markers, recovery copy, LSN floor, writer end) Review FIX at 89293fb15f39 (PR #33919 comment 6610468202). The regression tests are the two commits before this one. - Sync markers (format.rs): an entry with flags 1 and no payload, whose LSN field says "every record of this segment before me is synced". The writer appends one after each successful sync that covered records (after the durable listeners, so no reply waits for it), and syncs it fully at a seal and at a rotation. - F4, recovery (step 4 and 6): the last segment is trusted only up to its last marker. Its records after the marker may sit in pages a failed sync dropped (a later sync on another descriptor reports success without them), so recovery writes them again into a new segment: header, records and a marker, fully synced under `<name>.seg.new`, renamed into place, folder synced. The journal always goes on in a segment this open made, never in one an earlier process wrote. A restart in the writer's backoff after a failed sync is then safe. - F1, quarantine: every LSN a quarantined segment holds counts in the LSN floor at every open, and a quarantine that no new segment followed (a crash in the middle of an earlier recovery) is reported as damage again. The highest LSN of every moved segment is read before the first rename. - F2, highest LSN: a segment whose header is damaged is read from its first record (each record's checksum covers its LSN), then the bytes after the first bad entry are scanned for records and markers that still check out. The same scan runs after a cut in the last segment: a marker there is media damage (reported), records there are a torn write (`torn_lsns`); their LSNs are never used again. - F3, writer end: a guard closes the journal when the writer thread ends, also by a panic: appends fail with Closed, waits end with WriterEnded, the listeners are dropped (so a channel a held reply waits on closes), and stats.failing says why. The crate does not abort the process; the daemon owns that policy. - F6: recovery closes each segment after its header is read and opens one again only to read it. (Retirement of compacted segments is an S2 API, before the first consumer.) - F5: recovery syncs the parent folder after it makes the journal folder. - F7: format.rs and lib.rs now say what the code does: a failed write is cut back and retried in the same segment; the cut removes only bytes never synced and never acknowledged; nothing shortens a segment past a synced record. Docs: journal.rs says when open's replayed records are durable (on Ok) and what a stopped writer does. * cmux-journal: split the crash sim's machine model into sim/machine.rs (review G5) sim.rs had 924 lines and 63 functions, over the 60-function limit for a new Rust file (the god-file check warned "not in baseline: split it"). The machine model (each file's page cache and disk, the ordered log of names, and what an OS crash or a power loss leaves) moves to sim/machine.rs (297 lines, 19 functions); the process view and the Storage implementation stay in sim.rs (639 lines, 30 functions). No behavior change: the items move as they are, made visible to the sim module. * cmux-journal: re-review regressions G1-G4 and page eviction in the crash sim (red) Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993). Tests and sim only; the fix follows. - sim: the pages a failed sync dropped stay in the cache until an eviction; `evict_dropped_pages_after_read(true)` makes each read evict the dropped pages it read, so a second read of them sees the disk's bytes. - G1 recovery_copies_the_records_it_checked_when_the_cache_drops_them: a sync fails and drops record 2's pages, a kill in the backoff, the next open reads record 2 from the cache (replayed) and the pages are evicted after that read; then an OS crash. Record 2 and the record 3 acked after the restart must survive. On this code recovery's second, unchecked read stages zeros under a vouching marker, and the last open keeps only record 1 and reports damage. - G2: the F1 regression asserts that every open after the damage reports the loss (lost_through 40). On this code a crash at the open's last step (the folder sync after the new segment's rename) loses the report for good. - G3 a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system): a journal in a symlinked state folder, and a journal folder that is a symlink. On this code the parent folder's sync opens with O_NOFOLLOW (ELOOP). - G4 an_open_after_a_finished_quarantine_reads_no_quarantined_segment: on this code every open reads every quarantined segment in full. * cmux-journal: fix the re-review findings G1-G4 and G6 Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993); the regression tests are the commit before this one (red: fleet de35f6d0). - G1: recovery writes again exactly the bytes its checked read returned. The last segment's entries after its last marker are kept from the checked read (header and payload) and staged from memory; the second, unchecked read of the file is gone. A page that a failed sync dropped can be evicted between two reads, and the second read then returned the disk's zeros, which the copy kept under a vouching marker. - G2: the damage an open finds is written to `DAMAGE` (whole, fully synced, renamed into place, folder synced) before the new segment that hides it from later opens. Every open reports the recorded damage first, until the daemon calls the new `Journal::clear_damage`, which moves the record into `quarantine/` (kept, never deleted). - G3: a folder sync follows a symlink in its path (no O_NOFOLLOW): a state folder, or the journal folder itself, may be a symlink. The sync is for the folder that holds the entries. - G4: the quarantined segments are read for their LSNs only while no live segment is numbered above them (an unfinished quarantine); once one is, its first LSN already carries their floor. - G6: `checked_div` in the scan of the bytes after a cut (the crate's one tracked arithmetic hit). - G5 (the sim split) is the commit before the regressions. * cmux-journal: third-review regressions G7 (A, B) and N1 (red) Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018). Tests only. - G7 A a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn: after a finished quarantine, one byte of the empty new segment's header changes. On this code the next open skips the quarantine scan (the listing counts the damaged segment) and the next append reuses a quarantined LSN. - G7 B a_damaged_newest_segment_after_a_quarantine_reports_its_records: the newest segment holds acked 41..43, and its header and its first record are damaged. On this code the moved segment is scanned with a base far below 41, so the loss is not reported through 43 and the next append is not 44. - N1: the F1 regression also asserts that an open reports each loss once (no reported LSN range lies inside another). On this code a crash after the quarantine's renames makes the next open report the same loss twice. * cmux-journal: fix the third-review findings G7, N1, N2; split recovery.rs Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018); the regression tests are the commit before this one. - G7: the quarantine scan is skipped only when a segment whose header checked out is numbered above every quarantined segment (the folder listing also counted a newest segment with a damaged header, which carries no floor). The scans start from the running floor: `quarantined_floor` and `quarantine` pass the highest LSN counted so far as the base of the next file's scan, so a quarantined file with no readable header and no readable first record is scanned for the LSNs right above what is counted. - The "unfinished quarantine" report is made only when the quarantine holds LSNs nothing else counted (an old quarantine behind a segment that a crash tore at its creation holds none, and a crash is not damage). Its reason now also names a damaged newest header. - N1: a loss whose LSN range lies inside a recorded one is the same loss found again after a crash: it is neither recorded nor reported twice. - N2: recovery.rs's docs name `DAMAGE`; two doc lines over 100 columns are wrapped (lib.rs, sim.rs). - recovery.rs was at 987 of its 1000 lines: the damage record moves to recovery/damage.rs, the entry reader and the LSN scans to recovery/scan.rs (moved as they are). * cmux-journal: report each loss once, also when one open finds it twice (N1) The red run of the N1 regression (fleet 27d011ca, op 64 of the F1 test) shows the case the previous commit did not cover: a crash after the quarantine's renames and before DAMAGE is written; the next open finds both the damaged synced record of the last segment (7..14) and the unfinished quarantine (7..40), and reported both. Now every loss inside another (recorded or found by this open) is dropped, the larger range kept; DAMAGE is rewritten only when its set of losses changed. * cmux-journal: fourth-review regressions G8, N3, N4 (red) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report. * cmux-journal: the quarantine's floor in its file names; DAMAGE before the moves (G8, N3, N4) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format. * cmux-journal: sort the quarantine list with sort_by_key (clippy unnecessary_sort_by) Fleet d9f22bed at b589044b852: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`). * cmux-journal: Journal::on_written, a listener per written batch (for S2's projector) R-DURABILITY S2 (acpmux event logs through the journal) projects each chat's log line into its ndjson file once the journal has WRITTEN the batch that holds it, so readers see stream lines at once. The listener runs on the writer thread after the write and before the sync, once per batch that moved the written LSN: a batch written again into a new segment after a failed sync is not heard twice. When the writer stops, these listeners are dropped too. Test: heard while the batch's sync still runs, once. Port of 145faa6faa8 (branch rewrite/r-durability-s2-acpmux-logs) onto the S1 review fixes. * cmux-journal: Journal::retire_through, the one delete (segments a compaction point covers) R-DURABILITY S2: a daemon's projector compacts the journal into its own files (acpmux: the chat ndjson logs, then a synced checkpoint). Without a delete the segment count only grows (review finding F6). `retire_through(compacted)` deletes the segments that hold no record after the caller's compaction point: a segment goes when the next segment's "previous last LSN" is at or before it, so the newest segment never goes. The point is capped at the durable LSN. The deletes and a folder sync run on the caller's thread (the projector, never a request path). A delete that fails leaves its segment live for a later call. Why a crash during a retirement is safe: the caller reopens from a point at or after `compacted`, and recovery reads nothing from a segment the point covers (it skips it on its next segment's header), so any subset of the deletes may reach the disk. A point older than the retired segments is reported as Damage::Missing. Quarantined segments are never deleted. - Storage::remove_file (FsStorage: std::fs::remove_file; SimStorage: one more change in the ordered log of names, durable at the folder sync). - The live segment list: recovery's, then each segment the writer makes. - JournalStats::segments_retired. - Tests: only covered segments go, and a second call deletes none; an OS crash after it keeps every record after the point and an older point is reported; a crash at every operation of a retirement (process, OS crash, power loss) keeps every record after the point and never reuses an LSN; the newest segment stays; a point in the middle of what this writer wrote. This adds a delete of journal data, so its PR is risk:high. * acpmux: move MemoryStore and the store settings into their own files store.rs (2139 lines, 120 fns) and config.rs (1909 lines, 97 fns) are over the god-file limit, and the journal store (R-DURABILITY S2) must not grow them. MemoryStore moves to store/memory.rs and StoreMode, StoreConfig and its defaults to config/store_config.rs; both stay exported where they were (crate::store::MemoryStore, crate::config::{StoreConfig, StoreMode}). No behavior change: the code moves as it is. * acpmux: the event log's journal switch and its crash tests (red) R-DURABILITY S2 (DESIGN.md 3.3b). Today no event-log append is synced: a line a client was told about is on the disk only after the next flush (shutdown), so an OS crash or a power loss takes back every line since (E12). This commit adds only the switch, a seam and the tests; the journal store follows. - `store.journal` (StoreConfig::journal, default false). - `Store::wait_durable(timeout)`: tests and tools wait until every appended record is durable. The default returns at once (the Local and Memory stores never sync an append), which is today's behavior. - `store::open_with_journal` (tests): the store with the journal on a given `cmux_journal::Storage`. Here it ignores the journal and opens LocalStore. - acpmux depends on cmux-journal (dev: with the `sim` feature). - store/journal_store_tests.rs, red on this commit: - a_line_the_store_says_is_durable_survives_an_os_crash (lines 2..20 are lost: nothing synced them); - a_deleted_chat_stays_deleted_after_a_crash and a_line_the_log_refused_is_never_written_later pass here, and must keep passing with the journal store (no replay may bring them back); - a_kill_keeps_every_line_once passes here and must keep passing (a replay after a kill must write no line twice). * acpmux: the event log through one journal, behind `store.journal` (R-DURABILITY S2) DESIGN.md 3.3b. With `store.journal` on, each event-log append is one record in `<home>/journal/` (cmux-journal: group commit, one sync per batch; a power loss takes back at most the last second, always a suffix), then the same line goes to the chat's ndjson as before. Readers, segments and file formats do not change. Default off. - store/journal_store.rs (JournalStore over LocalStore): - append: under one lock, the journal record (a memory copy, no I/O), then the log line; a line the log refuses gets a `cancel` record. Agent stream lines (`in` session/update, claude.*, codex.*) are Class::Stream, every other line Class::Ack. - checkpoint (`<home>/journal-projected.json`): every 30 s while the journal grew, past 64 MiB, at flush (shutdown) and before a retention commit. It syncs each log written since (LocalStore::sync_log, with no lock held while the disk works) and the sessions folder, writes the LSN (temp file, sync, rename, folder sync, every error checked), then retires the journal before it. A log close that failed its sync freezes the checkpoint until a restart (LocalStore::sync_failed). - recovery: the records after the checkpoint; for each chat, the log is cut right after its last readable line below the first journaled seq (later segments removed, the boundary one shortened, their kind-index copies dropped), and the journaled lines are written again in order. `forget` deletes a chat again; `cancel` keeps a refused line out. - delete: `forget` record, folder removal, then a bounded wait until the record is durable. - seal: the logs refuse writes and the journal syncs fully. - LocalStore: `append` is serialize + `append_line`; `sync_log`; a failed sync at a log's close is noted (`sync_failed`). - store::open opens the JournalStore when `store.journal` is on. Not yet (S2c): replies and events do not wait for durability; the daemon does not stop when the journal closes; the writer thread keeps the process's Utility I/O policy. * acpmux: journal store tests for the seal and a chat made again after its delete (red) - a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay: a chat is deleted (its forget record), made again under the same id (a transfer back: session.json and new lines), then a kill -9. On this code the replay deletes the folder again and its session.json is gone. - a_line_after_the_seal_is_refused_and_never_written_later: coverage of the seal (it passes here; the fix also closes a race the test cannot force). * acpmux: journal store: the seal under the appends' lock; a replay keeps a chat made again - seal takes the appends' lock: an append lands in both the journal and the logs, or in neither. Before, an append that raced the seal could leave a journal record whose line the sealed logs refused (its cancel then failed on the closed journal), and the next start would write that line. - append and delete check the logs' seal first, under the same lock, and do not touch the journal after it (no error per append at shutdown). - replay deletes a forgotten chat again only when no line follows its forget record: otherwise the chat was made again under the same id, and its folder holds that chat's own files. * cmux-journal: fifth-review regressions G9 (red) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4. * cmux-journal: name a skipped LSN floor in a FLOOR.<lsn> file (G9) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646); the regression tests are the commit before this one. When an open gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which a copy then hides from every later open), the only record of that floor was the header of the empty segment it makes; a damaged header lost it, and DAMAGE is cleared by `clear_damage` (torn records are not damage at all). Now such an open makes the empty file `FLOOR.<lsn>` and syncs its folder before any new segment, and every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine's names and this open's moves say their floors already: no file is made for them. * acpmux: journal store regressions from the S2 review (red) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824). Each test fails on this commit and passes after the fix that follows it: - 1a lines_written_with_the_switch_off_survive_turning_it_on_again - 1b a_kill_while_the_journal_syncs_keeps_the_lines_the_log_got - 1c a_line_flush_session_made_durable_survives_losing_its_record - 2 a_folder_put_back_after_a_delete_is_not_deleted_by_the_replay - 3 a_kill_in_a_delete_leaves_the_chat_whole_or_gone - 4 a_delete_does_not_wait_for_the_journal_sync - 5 a_failed_log_sync_freezes_the_checkpoint, a_checkpoint_that_keeps_failing_asks_the_daemon_to_stop - 6 one_chat_that_cannot_be_repaired_does_not_stop_the_open - 7 the_journal_storage_refuses_a_lost_home - the repair keeps a log's lines after a hole, also through a crash in it: an_os_crash_hole_is_repaired_and_the_lines_after_it_kept, a_crash_inside_a_repair_loses_no_line - nits: an_unreadable_checkpoint_fails_the_open, the_open_waits_for_the_previous_daemon_to_let_go_of_the_journal, a_crashed_checkpoint_temporary_is_removed_at_open, a_crash_right_after_a_checkpoint_reports_no_damage - the catch-up with the switch off: switching_off_after_a_crash_keeps_every_acknowledged_line - cmux-journal: check_append_says_what_append_would_do So they compile here, this commit adds the seams they use, with no behavior: `Store::stop_reason` (default None), `Journal::check_append` (always Ok), `FencedStorage` (never asks its fence), the open report and a test stop point after a repair's cut. * acpmux: the checkpoint freeze test fails a log sync, not a folder open (red) The freeze guards against a failed fsync (a later fsync of the same file may report success for pages the kernel dropped), not against a folder the store cannot open: a chmod 000 folder is an open error, which the fix retries instead of freezing. The test now fails the log sync itself through a test seam, `FAIL_NEXT_LOG_SYNC`. Red on the code before the fix, like the other review regressions (fleet d8e7037d, 1c22175e, 63604b26 at e030f4fe402: 17 of 17 new acpmux tests and the crate's check_append test fail). * acpmux: the journal store repairs logs, never rebuilds them (S2 review) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824); the regression tests are the two commits before this one. DESIGN.md 3.3c. - 1a/1b/1c: recovery no longer cuts a chat's log at its first journaled line and writes back only the journal's lines. It merges the log's lines and the journal's by seq: a line in both must be equal, a line only the log has stays, a line only the journal has goes in its place. Only bytes that do not read (a torn tail, a hole of zeros) are dropped. When the merge must change the log before its end, the lines only the log holds from there are journaled and made durable first, so a crash inside the repair loses no line. A log that moved on without the journal (a line in both that differs, lines out of order) is left as it is, and the journal's lines go to journal-unapplied/. - Append writes the log first, then the record, under one lock, after `Journal::check_append` (new in cmux-journal: the append's refusals, asked before any write). So a record always has its log line, and the `cancel` record is gone. - 2/3/4: the replay never deletes a folder and never makes a missing one again (a forget only drops the chat's earlier journaled lines). Delete appends its forget under the lock and removes the folder off it, with no wait for the record. - 5: a failed fsync in a checkpoint freezes it (a folder that does not open is retried); a frozen checkpoint returns an error, and the store asks the daemon to stop for a restart (`Store::stop_reason`, wired in daemon.rs), whose replay writes the logs again. Ten failed checkpoints in a row ask the same, so the journal past the checkpoint stays bounded. - 6: an error in one chat keeps its journaled lines in journal-unapplied/ and the open goes on. - 7: the journal's storage refuses writes under a lost home (`FencedStorage`, `config::refuse_lost_home`). - With store.journal off, a journal left by a run with it on is replayed, checkpointed and retired first (`catch_up`). - Nits: the checkpoint is never past the journal's durable end (no `Ahead` after a normal crash); damage is cleared after the next checkpoint; a kind-index copy goes before its segment is cut; an unreadable checkpoint fails the open; the open waits up to 5 s for the journal lock; crashed checkpoint temporaries are removed; `on_written` leaves cmux-journal (no user); README lists store.journal. * acpmux: split the journal store under its size budget (a move) journal_store.rs grew to 1085 lines and 71 functions with the review fix (budget 1000 and 60), and the fix grew store.rs and daemon.rs, which are over budget on the base. A pure move, no behavior change: - store/journal_store/{checkpoint,fenced,repair}.rs: the checkpoint and its thread, `FencedStorage`, the replay and the repair. - store/log_sync.rs: `LocalStore::sync_log`, `SyncFailed` and the test seam `FAIL_NEXT_LOG_SYNC`, out of store.rs (2139 lines on the base, 2113 now). - daemon/stop_causes.rs: `AbortOnDrop`, `next_signal` and `store_asks_to_stop`, out of daemon.rs (2052 on the base, 2046 now). check-no-godfiles.sh --base 51cfdee876b on every changed file: no file over budget grew, and no new file is over budget. * acpmux: the unrepairable-chat test waits for line 1 to be durable before its checkpoint At 03fccdef69c (fleet 32df3c72 on aws-m4pro-9, 144d4ccc on Linux) the test kept chat-b's lines [1, 2, 3] aside, not [2, 3]: it flushed right after appending line 1, and the checkpoint is now never past the journal's durable end (the review's Ahead nit), so line 1 was still past it. The fix's behavior is right; the test now waits until line 1 is durable, so its checkpoint covers it. On the code before the fix the test still fails (the open stops at chat-b's link). * acpmux: replies and events wait for durability, tests (red, S2c) DESIGN.md 3.3d. With the event journal on and every journal sync taking 400 ms (`SimStorage::set_sync_delay`): - an_acked_change_is_answered_once_its_record_is_durable: a rename's reply must come no sooner than one sync after its `renamed` record. Red here: the reply goes at once. - agent_output_is_not_held_behind_another_chats_acked_change: agent output to a connection attached to its chat goes at once while another chat's acknowledged change waits for its sync. Green here (nothing waits); it guards the fix against holding agent output behind changes the connection does not hear. * acpmux: replies and events wait until what they show is durable (S2c) DESIGN.md 3.3d; the tests are the commit before this one. With `store.journal` on, each connection gets an outbound gate (server/durable_gate.rs) between it and its transport: - Every line a connection queues raises the gate's `needs` to the last acknowledged LSN (`Store::durability().acked`, set in the journal store's append before the hub publishes anything of the record), except the lines of a `Stream` record's event (agent output, DESIGN C4): the fan-out delivers those in a thread-local stream scope. - The gate's task takes every queued line at once, reads `needs`, waits (async) until the journal's durable LSN reaches it, then sends the batch in order: one wait per batch, as the journal's group commit. - The durable LSN is a `tokio::sync::watch` fed by `Journal::on_durable`. The listener owns a guard: a writer that ends while the store is not being dropped (a panic on its thread) asks the daemon to stop (`Store::stop_reason`, decision 9.8), and the watch closes, so each gate drops what it holds and ends, which closes its connection. - Without the journal, `Store::durability()` is None: no gate, and `Conn::raise_gate` does nothing. Moves, so server/mod.rs and store.rs stay under their size on the base: the outbound queue helpers to server/outbound.rs, `open_with_journal` to the journal store, and `streamed` to store/durability.rs. * acpmux: the journal lock wait's deadline without an unchecked Instant addition `Instant + Duration` can panic on overflow (clippy::arithmetic_side_effects, tracked by the panic ratchet). The wait now has no deadline only in the case that cannot happen (an Instant 5 s ahead overflowing), where it stops waiting at once. * acpmux: a failed log sync stops the real daemon for a restart, test (red) Lead rule 2026-11-04 (3): a daemon-level test of the store's "stop for a restart" before S2c merges, red first. With store.journal on and every log sync failing (a debug build's ACPMUX_TEST_FAIL_LOG_SYNC; the checkpoint every 200 ms through ACPMUX_TEST_JOURNAL_CHECKPOINT_MS), a rename of a chat on disk makes its log dirty, the checkpoint's sync fails and freezes it, and the store asks the daemon to stop. The test asserts: the WebSocket client gets the restart close (1012), the socket client's pending `_acpmux/wait` ends with the connection (no hang), and the daemon exits with status 74. Red here: the daemon stops and sends 1012, but exits 0 (no exit status says that it stopped to be repaired). This commit adds only the two debug-build seams (release builds have none) and the test. * acpmux: a daemon its store stopped for a restart exits with status 74 The test is the commit before this one. `daemon::run` returned Ok and the process exited 0 for every stop, so a supervisor or a person could not tell a stop to be repaired from a normal one; with systemd `Restart=on-failure` it would not even start again. Now the store's stop (`store_asks_to_stop`: a failed log sync froze the event journal's checkpoint, or the journal's writer ended) marks itself, sets the restart handoff (WebSocket clients get 1012, as before), and the process exits with `daemon::EXIT_REPAIR` (74, EX_IOERR). The units prevent only 75, so launchd KeepAlive and systemd Restart=always start it again; the app does not read the daemon's exit status. README: store.journal names the status. * acpmux: the restart test's request in flight goes through the acpmux client Lead rule (3): the client sees a typed error, not a hang. The socket client is now the acpmux client library (acpmux::client::Client, as the CLI uses it), and the test asserts what the CLI makes of the request that the stop ends: `daemon_closed`, retryable (cli::errors::classify). The first red (fleet 41f2d037, hq run 41804640229, at 80c5fb2b7f8) failed only on the exit status (0, wanted 74), with 1012 and the end of the connection passing. * acpmux: the daemon's disk tier with the journal on, tests (red) Lead rule 2026-11-04 (2): a reply the person waits for never waits on a throttled disk sync. The fleet A/B (STATUS 11554) found that at IOPOL_UTILITY an acknowledged journal append waits about 16 ms p50 under another process's disk stress (0.7 to 1.1 ms at the default tier). - the_daemon_with_the_journal_keeps_the_default_disk_tier: with store.journal on, the daemon process keeps the default tier; without it, utility as before (each case in a child of the test binary). - a_background_thread_yields_its_disk_work: a dedicated background thread (the checkpointer) works at utility; a thread beside it does not. The API they use is stubbed here (`qos::enter_daemon` ignores the journal; `qos::background_thread` does nothing), so both fail on macOS. * acpmux: the restart test borrows no &str twice (clippy needless_borrows_for_generic_args) Found by fleet 7f8afd9d (hq run: the S2c server step at 62b4f68ef8b); the step's other clippy red is the base red in link_push_bench.rs. * acpmux: with the journal on, the daemon keeps the default disk tier (S2c) The tests are 994f74119fb. Lead rule 2026-11-04: a reply the person waits for never waits on a throttled disk sync. Measured on two owned Macs (STATUS 11554): one acknowledged append to durable at IOPOL_UTILITY under another app's disk stress took 16.3-16.7 ms p50 and 37-45 ms p95, against 0.7-1.1 ms p50 and 1.5-1.8 ms p95 at the default tier. - qos::enter_daemon(journal): with store.journal on, the daemon process keeps the default disk tier; without it, utility as before. The switch is read alone from config.json (config::store_journal_enabled), before any thread starts, with no harness discovery. - qos::background_thread: the journal store's checkpointer lowers its own disk I/O to utility for its whole life (a thread that does nothing else: a thread-scope policy drops it out of QoS for good). * acpmux: an agent chunk is a Stream record, test (red) The hub's tap logs an ACP session/update under its sessionUpdate kind (hub/tap.rs: `kind = su.to_owned()`), so a record's kind is `agent_message_chunk`, `tool_call`, ... and never `session/update`; the message's method is. The journal store's `streamed` checked the kind, so every agent chunk was journaled as Ack: a sync per batch at the token rate, and in S2c each token's event waited for that sync (the S2c I/O A/B: STRESS token to client 2.2 ms -> 20.3 ms p50 with the journal on). Red here: the chunk reads back as Ack. * acpmux: an agent chunk is a Stream record (the record's message names session/update) The test is the commit before this one. `streamed` now reads the logged message's method: the hub's tap logs a session/update under its sessionUpdate kind (hub/tap.rs), so the kind never said it. Agent output is again Stream (synced with the next acknowledged batch or a second later), as DESIGN.md 3.3b says. * acpmux: review 2 of #34901, tests (red): a crash model that follows real log syncs The logs' OS crash model now follows the syncs that ran (store/sync_model.rs, tests only): a segment keeps its length at its last sync that counts; a failed sync loses what was written since the one before, and a later sync does not bring it back; a cut ends a loss it removes. Every log segment sync goes through log_sync::sync_segment (same calls as before), where the model and FAIL_NEXT_LOG_SYNC hook in. Red tests, one per finding: - 1a an_os_crash_after_a_restart_from_a_kill_keeps_every_acked_line: the restart's first checkpoint retires records whose log lines no sync made durable; expected [1..5], the head gives [1]. - 1b an_os_crash_after_a_restart_from_a_failed_sync_keeps_every_acked_line: the same after a frozen checkpoint and a restart; expected [1..5]. - 2 a_hole_that_starts_inside_a_line_is_repaired (the reviewer's test): a torn line is read as line 6 of a log that moved on; expected [1..9], the head gives [1, 2, 3, 4, 5, 9]. - 3 a_failed_sync_in_flush_session_freezes_the_checkpoint: a failed sync in flush_session does not freeze the checkpoint. * acpmux: review 2 of #34901: a restart writes the replayed lines again; a torn line is not a line; flush_session freezes Finding 1 (high): the repair now writes every merged line again from the repair start, even when the log already has them all. A killed daemon's log writes may be in the page cache only, and after a failed sync the kernel may have dropped pages that still read back and that a later sync reports as written; the first checkpoint after the open then syncs bytes this process wrote before it retires their records. The lines only the log has from the repair start are journaled and made durable first, as before a cut. The work is bounded by the checkpoint interval. Finding 2: a piece of the log is a line only when it parses as one JSON value (piece_seq). A hole that starts inside a line keeps that line's {"seq":N, head and runs into a later line's tail; it was read as a line of a log that moved on, and the chat's journaled lines went aside. Now its bytes go and the lines are written again. Finding 3: a failed sync in flush_session freezes the checkpoint (its file sync is a SyncFailed now), as a failed sync in a checkpoint or at a log's close does. Nits: cut_at syncs events/ after its removals (and the kind-index folder after its copies go), so a removed segment cannot come back beside the lines written again; the README says that with the switch on an event line over 64 MiB is refused and its log does not get it. Tests: the four red tests of d2c63860ece. * acpmux: chat metadata through the event journal, tests (red, R-DURABILITY S3a) DESIGN.md 3.3e. Red on S2c: - last_seq_never_passes_the_log_after_an_os_crash (E13): session.json is synced at each save while the agent's lines wait for the journal's next sync, so after an OS crash the metadata names lines the log lost. - a_chat_made_after_the_checkpoint_survives_an_os_crash_that_loses_its_folder: nothing syncs the sessions folder when a chat's folder is made, and the replay never makes a folder again, so an acknowledged new chat is lost. - a_metadata_change_is_written_to_session_json_by_the_checkpoint: today each change writes and syncs session.json on the request path. Guards (green before and after): a chat made before the checkpoint and deleted is not made again; the replay never writes an older image over session.json. * acpmux: chat metadata through the event journal (R-DURABILITY S3a) DESIGN.md 3.3e, behind `store.journal` (default off). A metadata change is a `meta` record (kind 4: chat id, a `created` flag, the whole image) instead of a session.json write with two syncs on the request path; the reply waits for the record through S2c's gate, one group commit for a burst. - Overlay: each chat's newest image until a checkpoint writes session.json; load, load_startup and list read it first (list under the projection lock, so a checkpoint cannot drop an entry between its two reads). - Checkpoint: waits on its own thread until every record through its LSN is durable, then writes each image it read (session.json carries the image's LSN as `journalLsn`), then the logs and the sessions folder, as before. An image so names only lines the journal holds (E13). A chat deleted or made again meanwhile is passed (its overlay entry is gone or new). - Replay: per chat after its last `forget`, the newest image, written when newer than the file's `journalLsn`. A missing folder is made again only for a chat a replayed `created` record made; a chat made before the checkpoint whose folder is gone keeps its image in journal-unapplied/. - flush_session also waits until the chat's records are durable. - Delete drops the overlay entry with its `forget`, and removes the folder under the folders lock a checkpoint's write also takes. Tests: the 5 of 0505e9bee04 (3 red there). * acpmux: S3a, tests (red): folders handed out of the store Code outside the store reads and moves chat folders directly (transfer_in puts the live folder aside and installs a transferred chat; session_dir hands the folder to the transfer, the session files view and others). With the metadata in the overlay until a checkpoint: - a_folder_replaced_outside_the_store_keeps_its_own_metadata: a read, the checkpoint and the replay must not put the old folder's image in the new one. - session_dir_hands_out_a_folder_whose_metadata_is_written. - flush_session_writes_the_chats_metadata (the trait says log and metadata). - the_checkpoint_never_makes_a_folder_removed_outside_the_store_again. Red on 71bc09a1e63 (S3a's first implementation). * acpmux: S3a red tests: a replay after a folder swap; flush_session's chat has a log (as its callers' chats do) * acpmux: S3a: each metadata image is bound to its chat folder; session_dir and flush_session write it first Code outside the store reads and moves chat folders directly (a transfer puts the live folder aside and installs a transferred chat under the same id). So a `meta` record now carries its folder's identity (file id and birth time): - reads use an overlay image only while the folder at the path is its folder; - the checkpoint writes an image only into its own folder, never makes a missing one, and keeps the image of a removed or replaced folder in journal-unapplied/; - the replay writes an image only into its own folder (or a folder a replayed `created` record made), else keeps it aside. `session_dir` (the transfer, the session files view) and `flush_session` (the trait: "log and metadata to stable storage") write the chat's image first, once its record is durable. The list reads the overlay before the files (a checkpoint writes a file before it drops the entry), so it needs no lock. Tests: the 5 of 0280ac37f5e and 0e617489731. * acpmux: S2 follow-ups from review 3 of #34901, tests (red) - a_chat_whose_repair_failed_is_repaired_at_the_next_start (follow-up 1): the open's checkpoint retires the records of a chat whose repair failed, so no later start puts its lines back (they are only in journal-unapplied/). - a_hole_that_ends_at_a_line_start_keeps_that_line (follow-up 2): a torn line, zeros, then a whole line only the log holds: the piece does not parse and the whole line is dropped. - an_idle_store_does_not_wake_its_checkpointer (follow-up 4, cmux-ad m9): the checkpointer wakes every period with nothing to do. Test seams only (no behavior change): CHECKPOINTER_WAKES and TEST_CHECKPOINT_EVERY_MS. * acpmux: S2 follow-ups from review 3 of #34901: a failed repair holds its records; a hole's last zero splits; an idle checkpointer sleeps - Follow-up 1: a chat whose repair fails at the open (its log cannot be read or written now) keeps its lines in journal-unapplied/ as before, and the checkpoint now stays below its first journaled LSN (`Order::held`), so the next start repairs it again. Bounded: past 256 MiB of journal after it, the hold goes and the lines stay only in journal-unapplied/ (an error says so). - Follow-up 2: a piece splits at its last zero byte (a whole JSON line holds none), so a hole that ends at a line start, after a torn line, keeps that line (it mattered for a line only the log holds). - Follow-up 4 (cmux-ad m9): with nothing to checkpoint, the checkpointer sleeps until a change wakes it (`Inner::touched`, one atomic read per append), then checkpoints one period later; before, it woke every 30 s. Tests: the 3 of d147057bf85. * acpmux: the qos journal tests module is macOS-only (its Linux build held only an unused import: clippy red, review of #35400 F0) * acpmux: review of #35400 F1 and F2, tests (red) - a_read_that_shows_a_line_covers_its_record (F1): a read sees a log line between its log write and its record (the appends' lock held there), and the reply built from it goes out with no wait for that record. - a_journal_that_syncs_nothing_asks_the_daemon_to_stop (F2): with a journal that makes nothing durable, every reply waits without a limit and nothing asks the daemon to stop. Test seams only: PAUSE_AFTER_LOG_WRITE, TEST_STALL_MS. * acpmux: review of #35400 F1, the streaming read, test (red): session/load's replay sends a line before its record * acpmux: the test seam PAUSE_AFTER_LOG_WRITE is reachable from the server tests * acpmux: review of #35400: a read covers the records of the lines it shows; a stalled journal stops the daemon F1: the append writes a line to the log, then appends its record and sets the acknowledged LSN, all under the appends' lock; reads take no lock. A read could show a line before its record existed, and the reply built from it went out at once. Now every journal-store read of the logs ends with a fence on the appends' lock (`Inner::fenced`): a line the read saw has its record appended and its LSN acknowledged when the read returns, so the reply's gate covers it. The one read that sends as it reads (`Hub::scan_events`, session/load's replay) stops at the chat's last_seq read first. F2: a journal writer that keeps failing (it retries without end) held every reply on every connection with no limit. A connection's gate now waits at most 10 s (`STALL_LIMIT`); past it, `Durability::stalled` asks the daemon to stop for a restart, as a failed log sync does. Tests: the 3 of 3916f84dd68 and c553a19fd8b. * threading-model: the acpmux event journal's threads and bounds (writer, checkpointer, per-connection gate; review of #35400) * acpmux: the stall test's journal syncs take 3 s, not an hour (a sync under way kept its delay and hung the test's end) * acpmux: the idle checkpointer's period, test (red): a change seen by its second look must not start a checkpoint at once - a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period: the checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies when it gets the timer lock. The second look saw the change, so the checkpointer is in its period wait by then, and the notify ends that wait: a checkpoint runs at once on the checkpointer's thread. This is the cause of a_failed_log_sync_freezes_the_checkpoint failing in the full store:: run of fleet 0577ac62 (that early checkpoint took the line whose sync the test fails, on a thread where the failure is not set). The test holds both threads at the two points, so the order is exact. Test seams only (no behavior change): the checkpointer's test seams are now by the store's home (period, wake count, the two holds), so tests that run as threads of one process under cargo test do not see each other's. * acpmux: the idle checkpointer's period ends only on its own state, not on a wake meant for its idle wait The checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies once it gets the timer lock. Its change shows in the second look, so the checkpointer has gone to its period wait by then, and that notify ended the period at once: a checkpoint ran right away on the checkpointer's thread (a burst no longer shared one checkpoint), and a spurious wakeup could do the same. In the full store:: run of fleet 0577ac62 such a checkpoint took the line whose sync a_failed_log_sync_freezes_the_checkpoint fails, so that test's own flush had nothing to do and returned Ok. The period wait now ends only on the period, `wanted` or `stop` (`wait_timeout_while`): each of the checkpointer's two waits ends only on the state its waker sets before it notifies under the timer lock. No sleep, no retry. The idle wait is unchanged. Test: a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period (edca3b4823d). * acpmux: S3a review (#35483) red tests: switch-off metadata and deletes, a late settle, a failed session.json folder sync, a purge after the checkpoint F1: metadata written with the switch off survives turning it on again; a chat deleted with the switch off after a failed catch-up is not made again. F2: a late settle never writes an older image over a checkpointed one (the reviewer's test and hold). F3: a failed folder sync after a metadata write freezes the checkpoint. F4: a chat made after the checkpoint and deleted is not made again after a kill. Test seams only; the code is the S3a head merged with feat. * acpmux: S3a review (#35483) red tests: a new chat is acknowledged only once its folder is durable; session_path never waits The root of F4 and of F1's delete case: the replay makes a chat's folder again, so a chat removed by any path comes back. The fix will never make a folder again, which needs an acknowledged chat's folder to survive an OS crash: the test models what each folder's last sync kept (sync_model) and asserts the gate's durability passes a new chat only after its sessions entry is synced. P3 1 of the review: a caller that needs only the path never waits for the journal (Store::session_path, a stub with session_dir's behavior here). * acpmux: S3a review (#35483) fixes: the replay never makes a folder again; switch-off changes are kept; no late or unsynced metadata write F1: each meta record carries the stamp (file id, modification time, length) of the session.json its save saw. The replay writes an image over a file the journal did not write only when one of the chat's records saw that file; otherwise the file moved on without the journal (a run with store.journal off whose catch-up did not replay, or an older build) and the image goes to journal-unapplied/, as S2 does for a log. F1 (deletes) and F4 at the root: the replay never makes a chat's folder again. A save that made a folder notes its record's LSN; the new acpmux-journal-folders thread (the daemon's disk tier) syncs sessions/ at once, and the durable LSN the S2c gate waits for stays below that record until then (journal_store/new_folders.rs). So an acknowledged chat's folder survives an OS crash, and a folder that is gone was removed: a purge whose forget a kill took back, a run with the switch off, the transfer recovery. The created flag and the replay's folder remake are gone. F2: write_through does not write a file that holds the same or a newer journalLsn. F3: session.json is written with its own syncs (file, then folder); a failed sync is SyncFailed and freezes the checkpoint; the replay and the kept-aside images return their errors. P3 1: Store::session_path gives the path with no lock, wait or write; an agent's start, the fixed-id check, the temporary-folder sweep, the host probe and the transfer recovery's root use it. The replaced test (a chat made after the checkpoint survives an OS crash that loses its folder) asserted a remake that cannot be right any more; the new test asserts the guarantee that makes the loss impossible, then the crash. Red (7 of 17 journal_meta_tests, each for its reason; the other 10 pass): macOS 5bc434a9 and 624d5df4, Linux a2207ff0 and 638a44ef. * acpmux: review of #35562 red tests: a failed repair holds no other chat's checkpoint; a checkpoint after it writes only durable images F1 (the reviewer's test, renamed): a checkpoint after a failed repair writes only durable metadata images. F2/F3 at the root: a failed repair does not hold the checkpoint of other chats; a retention rewrite of another chat after a failed repair stays after a restart. Guards: the reviewer's two switch-off tests (green here since the S3a fix 0d246364d90: the stamp rule and no folder remake), and a chat deleted and made again after its failed repair gets none of the old chat's lines. * acpmux: review of #35562: a failed repair keeps its chat's lines in a repair file; no hold of the global checkpoint The hold (Order::held, 49d7e5cef83) kept every chat's records past the checkpoint while one chat's repair failed. That let a checkpoint write metadata images whose records were not durable (F1), made each start with the switch off replay the whole window again and undid a retention rewrite (F2), and grew the journal with each restart (F3, the 256 MiB bound counted one run only). Now a chat whose log cannot be read or written at the open keeps its lines in journal-unapplied/<id>.<lsn>.repair.ndjson (a header with the chat id, then the lines, synced). The next open journals them again, waits until they are durable, removes the file and repairs the chat with them as with any journaled line (a line in both is the same bytes, so a file journaled twice after a crash changes nothing). The checkpoint of every chat goes on as before S2's follow-ups. A delete removes its chat's repair file after the folder, so a chat made again under the same id gets none of the old lines. Red at 23870301064 (macOS a8be0684): the 3 fail for their reasons ("the checkpoint stayed at 1 below chat-a's records (the journal ends at 5)"; "the checkpoint wrote an image to session.json before its record was durable"; "the open kept a copy of a rewritten log aside"); the guards pass; 339 of 342. * acpmux: S3a delta review (N1) red tests: a change to a chat whose session.json another journal wrote reaches it A folder moved in from another home (build adoption), checkpointed and replayed, and a journal made again from nothing: the other journal's journalLsn must not hide a change made here. The first test is the reviewer's (fleet 4d0d98bc and 071ad7c4 at e944a7c21de). * acpmux: S3a delta review (N1): journalLsn is bound to its journal's id; a file another journal wrote follows the stamp rule session.json now carries journalId (16 hex digits, a string) beside journalLsn. The checkpoint's "held" check (F2) and the replay's rule compare an LSN only for a file with this journal's id: a file from another journal (a chat folder build adoption moved in from another home, or a home whose journal was made again) counts as one this journal did not write, so the stamp rule decides, as for a run with the switch off. cmux-journal: Journal::journal_id (the id already in every segment header; kept across opens and retirements), with a unit test. Red at 7c8cb957745: fleet c29b659d (Linux) and 47570494 (macOS) running; the reviewer's red at e944a7c21de: 4d0d98bc (macOS) and 071ad7c4 (Linux). * acpmux: delta review of #35562 red tests: a repair file and its folder; a torn file; a failed delete; a file larger than the journal's queue D1 (the reviewer's test, renamed): a failed repair's kept lines survive a folder that is aside at the next open; and they never go into a folder that replaced the one they were kept for. D2 (the reviewer's test, renamed): a torn repair file does not stop the chat's repair. D3: a delete that succeeds after a failed one removes the chat's repair file (test seam FAIL_NEXT_FOLDER_REMOVAL). D4: a repair file larger than max_pending_bytes is applied. * acpmux: delta review of #35562: a repair file applies only to the folder it was kept for; written whole; torn lines passed; a full queue waits; a failed delete keeps its file D1 (P2): the open took a repair file even when the chat's folder was not there (a transfer held it aside at a kill, and the hub put it back after the store's open), and the lines went nowhere. The file's header now has the folder's identity (meta.rs FolderId, the same one S3a binds metadata images to); take_repairs journals a file only when the folder at the chat's path has that identity. Otherwise the file stays: a later open applies it when that folder is back, and else it is a person's. A failed repair with no folder to bind to keeps its lines as a plain kept file, as before S2's follow-ups. D2 (P3): kept files are written whole (a temporary copy, synced, renamed, folder synced); the open removes a temporary a crash left; bytes in a repair file that are not a whole JSON line are passed. D3 (P3): delete keeps the chat's repair-file entry until the file is removed, so a delete tried again after a failed folder removal still removes it. D4 (P3): the open's repair appends wait for the writer when the journal's queue is full (append_waiting), so a repair file larger than max_pending_bytes is applied; journal_lines uses it too. Red at c0a20e09f1f: macOS fleet 54945ca4, 5 of 41 journal_store_tests fail, each for its reason; Linux fleet 393040c6 running. * acpmux: the delete test seam goes before the append pause's doc comment, not between it and its item (clippy unused_doc_comments, fleet 56f7ec6c)
lawrencecchen
added a commit
that referenced
this pull request
Nov 5, 2026
…URABILITY S3b) (#35565) * cmux-journal: one append-only journal per daemon, with group commit and ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner). * cmux-journal: type a test range; drop an unread segment field (fleet clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71e3f9 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red. * cmux-journal: a failed F_FULLFSYNC is an error, not a plain fsync; no segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee. * cmux-journal: the past-the-end test reads back the storage it wrote to Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c87b8d: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5c53e. * cmux-journal: a sync a signal interrupts is made again (macOS) fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r). * cmux-journal: crash sim of write-back errors and crash in recovery; S1 review regressions (red) The S1 review (FIX at 89293fb15f39) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open * cmux-journal: the F2 regression damages the header of the last of several segments (red) The reviewer's F2 scenario has two segments: S1 holds 1..10, S2 (the last) holds 11..20, and S2's header is damaged. The test now rotates segments (300-byte segments, 20 acked records) and damages the last one's header, so the report must say lost_through 20 and the next append must take 21. On the code before the fix the moved segment's LSNs read as 0, so the next append reuses one. * cmux-journal: the crash sim forces the F4 order (failed sync, kill, OS crash) (red) On the code before the S1 fix, the crash-sim step was green (fleet 184129d8 at 263da86f223): 6 of 7 runs passed, and the mixed run failed only seed 140 in the full run and passed when it ran alone. The random faults reached the order the review found (F4) in 1 of 800 runs. So these tests were no red/green proof. The order: the sync of a write that spans pages fails and the disk gets none of them (Linux keeps them clean in the cache, so a later sync reports success); the process is killed in the writer's backoff; the next open reads the failed segment from the cache, replays its records and goes on; an OS crash or a power loss then takes those pages, and records an open already replayed are gone. - sim.rs: `fail_next_sync(pages, kind)` fails the next sync that has at least that many dirty pages, and the disk gets none of them. `crash_after_failure(n)` makes the n-th operation after the next injected failure the crash point. - crash_sim_tests.rs: a new run forces the order on every seed (a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record): epoch 0 aims the fault, the kill is a process crash at the writer's retry, the open after it does not crash, epoch 1 ends in an OS crash, and a third open checks that the records the second open replayed are all there. In the random runs, one epoch in four aims the fault, with the crash 1 to 3 operations after the failed sync and a crash of any kind after it. - A sim test checks the aimed fault itself (the_sim_fails_the_aimed_sync_and_crashes_in_the_backoff). Expected on this commit (old journal code): the forced run fails every seed; the mixed run fails most runs. The fix commit follows. * cmux-journal: fix the S1 review findings F1-F7 (markers, recovery copy, LSN floor, writer end) Review FIX at 89293fb15f39 (PR #33919 comment 6610468202). The regression tests are the two commits before this one. - Sync markers (format.rs): an entry with flags 1 and no payload, whose LSN field says "every record of this segment before me is synced". The writer appends one after each successful sync that covered records (after the durable listeners, so no reply waits for it), and syncs it fully at a seal and at a rotation. - F4, recovery (step 4 and 6): the last segment is trusted only up to its last marker. Its records after the marker may sit in pages a failed sync dropped (a later sync on another descriptor reports success without them), so recovery writes them again into a new segment: header, records and a marker, fully synced under `<name>.seg.new`, renamed into place, folder synced. The journal always goes on in a segment this open made, never in one an earlier process wrote. A restart in the writer's backoff after a failed sync is then safe. - F1, quarantine: every LSN a quarantined segment holds counts in the LSN floor at every open, and a quarantine that no new segment followed (a crash in the middle of an earlier recovery) is reported as damage again. The highest LSN of every moved segment is read before the first rename. - F2, highest LSN: a segment whose header is damaged is read from its first record (each record's checksum covers its LSN), then the bytes after the first bad entry are scanned for records and markers that still check out. The same scan runs after a cut in the last segment: a marker there is media damage (reported), records there are a torn write (`torn_lsns`); their LSNs are never used again. - F3, writer end: a guard closes the journal when the writer thread ends, also by a panic: appends fail with Closed, waits end with WriterEnded, the listeners are dropped (so a channel a held reply waits on closes), and stats.failing says why. The crate does not abort the process; the daemon owns that policy. - F6: recovery closes each segment after its header is read and opens one again only to read it. (Retirement of compacted segments is an S2 API, before the first consumer.) - F5: recovery syncs the parent folder after it makes the journal folder. - F7: format.rs and lib.rs now say what the code does: a failed write is cut back and retried in the same segment; the cut removes only bytes never synced and never acknowledged; nothing shortens a segment past a synced record. Docs: journal.rs says when open's replayed records are durable (on Ok) and what a stopped writer does. * cmux-journal: split the crash sim's machine model into sim/machine.rs (review G5) sim.rs had 924 lines and 63 functions, over the 60-function limit for a new Rust file (the god-file check warned "not in baseline: split it"). The machine model (each file's page cache and disk, the ordered log of names, and what an OS crash or a power loss leaves) moves to sim/machine.rs (297 lines, 19 functions); the process view and the Storage implementation stay in sim.rs (639 lines, 30 functions). No behavior change: the items move as they are, made visible to the sim module. * cmux-journal: re-review regressions G1-G4 and page eviction in the crash sim (red) Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993). Tests and sim only; the fix follows. - sim: the pages a failed sync dropped stay in the cache until an eviction; `evict_dropped_pages_after_read(true)` makes each read evict the dropped pages it read, so a second read of them sees the disk's bytes. - G1 recovery_copies_the_records_it_checked_when_the_cache_drops_them: a sync fails and drops record 2's pages, a kill in the backoff, the next open reads record 2 from the cache (replayed) and the pages are evicted after that read; then an OS crash. Record 2 and the record 3 acked after the restart must survive. On this code recovery's second, unchecked read stages zeros under a vouching marker, and the last open keeps only record 1 and reports damage. - G2: the F1 regression asserts that every open after the damage reports the loss (lost_through 40). On this code a crash at the open's last step (the folder sync after the new segment's rename) loses the report for good. - G3 a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system): a journal in a symlinked state folder, and a journal folder that is a symlink. On this code the parent folder's sync opens with O_NOFOLLOW (ELOOP). - G4 an_open_after_a_finished_quarantine_reads_no_quarantined_segment: on this code every open reads every quarantined segment in full. * cmux-journal: fix the re-review findings G1-G4 and G6 Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993); the regression tests are the commit before this one (red: fleet de35f6d0). - G1: recovery writes again exactly the bytes its checked read returned. The last segment's entries after its last marker are kept from the checked read (header and payload) and staged from memory; the second, unchecked read of the file is gone. A page that a failed sync dropped can be evicted between two reads, and the second read then returned the disk's zeros, which the copy kept under a vouching marker. - G2: the damage an open finds is written to `DAMAGE` (whole, fully synced, renamed into place, folder synced) before the new segment that hides it from later opens. Every open reports the recorded damage first, until the daemon calls the new `Journal::clear_damage`, which moves the record into `quarantine/` (kept, never deleted). - G3: a folder sync follows a symlink in its path (no O_NOFOLLOW): a state folder, or the journal folder itself, may be a symlink. The sync is for the folder that holds the entries. - G4: the quarantined segments are read for their LSNs only while no live segment is numbered above them (an unfinished quarantine); once one is, its first LSN already carries their floor. - G6: `checked_div` in the scan of the bytes after a cut (the crate's one tracked arithmetic hit). - G5 (the sim split) is the commit before the regressions. * cmux-journal: third-review regressions G7 (A, B) and N1 (red) Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018). Tests only. - G7 A a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn: after a finished quarantine, one byte of the empty new segment's header changes. On this code the next open skips the quarantine scan (the listing counts the damaged segment) and the next append reuses a quarantined LSN. - G7 B a_damaged_newest_segment_after_a_quarantine_reports_its_records: the newest segment holds acked 41..43, and its header and its first record are damaged. On this code the moved segment is scanned with a base far below 41, so the loss is not reported through 43 and the next append is not 44. - N1: the F1 regression also asserts that an open reports each loss once (no reported LSN range lies inside another). On this code a crash after the quarantine's renames makes the next open report the same loss twice. * cmux-journal: fix the third-review findings G7, N1, N2; split recovery.rs Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018); the regression tests are the commit before this one. - G7: the quarantine scan is skipped only when a segment whose header checked out is numbered above every quarantined segment (the folder listing also counted a newest segment with a damaged header, which carries no floor). The scans start from the running floor: `quarantined_floor` and `quarantine` pass the highest LSN counted so far as the base of the next file's scan, so a quarantined file with no readable header and no readable first record is scanned for the LSNs right above what is counted. - The "unfinished quarantine" report is made only when the quarantine holds LSNs nothing else counted (an old quarantine behind a segment that a crash tore at its creation holds none, and a crash is not damage). Its reason now also names a damaged newest header. - N1: a loss whose LSN range lies inside a recorded one is the same loss found again after a crash: it is neither recorded nor reported twice. - N2: recovery.rs's docs name `DAMAGE`; two doc lines over 100 columns are wrapped (lib.rs, sim.rs). - recovery.rs was at 987 of its 1000 lines: the damage record moves to recovery/damage.rs, the entry reader and the LSN scans to recovery/scan.rs (moved as they are). * cmux-journal: report each loss once, also when one open finds it twice (N1) The red run of the N1 regression (fleet 27d011ca, op 64 of the F1 test) shows the case the previous commit did not cover: a crash after the quarantine's renames and before DAMAGE is written; the next open finds both the damaged synced record of the last segment (7..14) and the unfinished quarantine (7..40), and reported both. Now every loss inside another (recorded or found by this open) is dropped, the larger range kept; DAMAGE is rewritten only when its set of losses changed. * cmux-journal: fourth-review regressions G8, N3, N4 (red) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report. * cmux-journal: the quarantine's floor in its file names; DAMAGE before the moves (G8, N3, N4) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format. * cmux-journal: sort the quarantine list with sort_by_key (clippy unnecessary_sort_by) Fleet d9f22bed at b589044b852: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`). * cmux-journal: Journal::on_written, a listener per written batch (for S2's projector) R-DURABILITY S2 (acpmux event logs through the journal) projects each chat's log line into its ndjson file once the journal has WRITTEN the batch that holds it, so readers see stream lines at once. The listener runs on the writer thread after the write and before the sync, once per batch that moved the written LSN: a batch written again into a new segment after a failed sync is not heard twice. When the writer stops, these listeners are dropped too. Test: heard while the batch's sync still runs, once. Port of 145faa6faa8 (branch rewrite/r-durability-s2-acpmux-logs) onto the S1 review fixes. * cmux-journal: Journal::retire_through, the one delete (segments a compaction point covers) R-DURABILITY S2: a daemon's projector compacts the journal into its own files (acpmux: the chat ndjson logs, then a synced checkpoint). Without a delete the segment count only grows (review finding F6). `retire_through(compacted)` deletes the segments that hold no record after the caller's compaction point: a segment goes when the next segment's "previous last LSN" is at or before it, so the newest segment never goes. The point is capped at the durable LSN. The deletes and a folder sync run on the caller's thread (the projector, never a request path). A delete that fails leaves its segment live for a later call. Why a crash during a retirement is safe: the caller reopens from a point at or after `compacted`, and recovery reads nothing from a segment the point covers (it skips it on its next segment's header), so any subset of the deletes may reach the disk. A point older than the retired segments is reported as Damage::Missing. Quarantined segments are never deleted. - Storage::remove_file (FsStorage: std::fs::remove_file; SimStorage: one more change in the ordered log of names, durable at the folder sync). - The live segment list: recovery's, then each segment the writer makes. - JournalStats::segments_retired. - Tests: only covered segments go, and a second call deletes none; an OS crash after it keeps every record after the point and an older point is reported; a crash at every operation of a retirement (process, OS crash, power loss) keeps every record after the point and never reuses an LSN; the newest segment stays; a point in the middle of what this writer wrote. This adds a delete of journal data, so its PR is risk:high. * acpmux: move MemoryStore and the store settings into their own files store.rs (2139 lines, 120 fns) and config.rs (1909 lines, 97 fns) are over the god-file limit, and the journal store (R-DURABILITY S2) must not grow them. MemoryStore moves to store/memory.rs and StoreMode, StoreConfig and its defaults to config/store_config.rs; both stay exported where they were (crate::store::MemoryStore, crate::config::{StoreConfig, StoreMode}). No behavior change: the code moves as it is. * acpmux: the event log's journal switch and its crash tests (red) R-DURABILITY S2 (DESIGN.md 3.3b). Today no event-log append is synced: a line a client was told about is on the disk only after the next flush (shutdown), so an OS crash or a power loss takes back every line since (E12). This commit adds only the switch, a seam and the tests; the journal store follows. - `store.journal` (StoreConfig::journal, default false). - `Store::wait_durable(timeout)`: tests and tools wait until every appended record is durable. The default returns at once (the Local and Memory stores never sync an append), which is today's behavior. - `store::open_with_journal` (tests): the store with the journal on a given `cmux_journal::Storage`. Here it ignores the journal and opens LocalStore. - acpmux depends on cmux-journal (dev: with the `sim` feature). - store/journal_store_tests.rs, red on this commit: - a_line_the_store_says_is_durable_survives_an_os_crash (lines 2..20 are lost: nothing synced them); - a_deleted_chat_stays_deleted_after_a_crash and a_line_the_log_refused_is_never_written_later pass here, and must keep passing with the journal store (no replay may bring them back); - a_kill_keeps_every_line_once passes here and must keep passing (a replay after a kill must write no line twice). * acpmux: the event log through one journal, behind `store.journal` (R-DURABILITY S2) DESIGN.md 3.3b. With `store.journal` on, each event-log append is one record in `<home>/journal/` (cmux-journal: group commit, one sync per batch; a power loss takes back at most the last second, always a suffix), then the same line goes to the chat's ndjson as before. Readers, segments and file formats do not change. Default off. - store/journal_store.rs (JournalStore over LocalStore): - append: under one lock, the journal record (a memory copy, no I/O), then the log line; a line the log refuses gets a `cancel` record. Agent stream lines (`in` session/update, claude.*, codex.*) are Class::Stream, every other line Class::Ack. - checkpoint (`<home>/journal-projected.json`): every 30 s while the journal grew, past 64 MiB, at flush (shutdown) and before a retention commit. It syncs each log written since (LocalStore::sync_log, with no lock held while the disk works) and the sessions folder, writes the LSN (temp file, sync, rename, folder sync, every error checked), then retires the journal before it. A log close that failed its sync freezes the checkpoint until a restart (LocalStore::sync_failed). - recovery: the records after the checkpoint; for each chat, the log is cut right after its last readable line below the first journaled seq (later segments removed, the boundary one shortened, their kind-index copies dropped), and the journaled lines are written again in order. `forget` deletes a chat again; `cancel` keeps a refused line out. - delete: `forget` record, folder removal, then a bounded wait until the record is durable. - seal: the logs refuse writes and the journal syncs fully. - LocalStore: `append` is serialize + `append_line`; `sync_log`; a failed sync at a log's close is noted (`sync_failed`). - store::open opens the JournalStore when `store.journal` is on. Not yet (S2c): replies and events do not wait for durability; the daemon does not stop when the journal closes; the writer thread keeps the process's Utility I/O policy. * acpmux: journal store tests for the seal and a chat made again after its delete (red) - a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay: a chat is deleted (its forget record), made again under the same id (a transfer back: session.json and new lines), then a kill -9. On this code the replay deletes the folder again and its session.json is gone. - a_line_after_the_seal_is_refused_and_never_written_later: coverage of the seal (it passes here; the fix also closes a race the test cannot force). * acpmux: journal store: the seal under the appends' lock; a replay keeps a chat made again - seal takes the appends' lock: an append lands in both the journal and the logs, or in neither. Before, an append that raced the seal could leave a journal record whose line the sealed logs refused (its cancel then failed on the closed journal), and the next start would write that line. - append and delete check the logs' seal first, under the same lock, and do not touch the journal after it (no error per append at shutdown). - replay deletes a forgotten chat again only when no line follows its forget record: otherwise the chat was made again under the same id, and its folder holds that chat's own files. * cmux-journal: fifth-review regressions G9 (red) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4. * cmux-journal: name a skipped LSN floor in a FLOOR.<lsn> file (G9) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646); the regression tests are the commit before this one. When an open gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which a copy then hides from every later open), the only record of that floor was the header of the empty segment it makes; a damaged header lost it, and DAMAGE is cleared by `clear_damage` (torn records are not damage at all). Now such an open makes the empty file `FLOOR.<lsn>` and syncs its folder before any new segment, and every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine's names and this open's moves say their floors already: no file is made for them. * acpmux: journal store regressions from the S2 review (red) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824). Each test fails on this commit and passes after the fix that follows it: - 1a lines_written_with_the_switch_off_survive_turning_it_on_again - 1b a_kill_while_the_journal_syncs_keeps_the_lines_the_log_got - 1c a_line_flush_session_made_durable_survives_losing_its_record - 2 a_folder_put_back_after_a_delete_is_not_deleted_by_the_replay - 3 a_kill_in_a_delete_leaves_the_chat_whole_or_gone - 4 a_delete_does_not_wait_for_the_journal_sync - 5 a_failed_log_sync_freezes_the_checkpoint, a_checkpoint_that_keeps_failing_asks_the_daemon_to_stop - 6 one_chat_that_cannot_be_repaired_does_not_stop_the_open - 7 the_journal_storage_refuses_a_lost_home - the repair keeps a log's lines after a hole, also through a crash in it: an_os_crash_hole_is_repaired_and_the_lines_after_it_kept, a_crash_inside_a_repair_loses_no_line - nits: an_unreadable_checkpoint_fails_the_open, the_open_waits_for_the_previous_daemon_to_let_go_of_the_journal, a_crashed_checkpoint_temporary_is_removed_at_open, a_crash_right_after_a_checkpoint_reports_no_damage - the catch-up with the switch off: switching_off_after_a_crash_keeps_every_acknowledged_line - cmux-journal: check_append_says_what_append_would_do So they compile here, this commit adds the seams they use, with no behavior: `Store::stop_reason` (default None), `Journal::check_append` (always Ok), `FencedStorage` (never asks its fence), the open report and a test stop point after a repair's cut. * acpmux: the checkpoint freeze test fails a log sync, not a folder open (red) The freeze guards against a failed fsync (a later fsync of the same file may report success for pages the kernel dropped), not against a folder the store cannot open: a chmod 000 folder is an open error, which the fix retries instead of freezing. The test now fails the log sync itself through a test seam, `FAIL_NEXT_LOG_SYNC`. Red on the code before the fix, like the other review regressions (fleet d8e7037d, 1c22175e, 63604b26 at e030f4fe402: 17 of 17 new acpmux tests and the crate's check_append test fail). * acpmux: the journal store repairs logs, never rebuilds them (S2 review) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824); the regression tests are the two commits before this one. DESIGN.md 3.3c. - 1a/1b/1c: recovery no longer cuts a chat's log at its first journaled line and writes back only the journal's lines. It merges the log's lines and the journal's by seq: a line in both must be equal, a line only the log has stays, a line only the journal has goes in its place. Only bytes that do not read (a torn tail, a hole of zeros) are dropped. When the merge must change the log before its end, the lines only the log holds from there are journaled and made durable first, so a crash inside the repair loses no line. A log that moved on without the journal (a line in both that differs, lines out of order) is left as it is, and the journal's lines go to journal-unapplied/. - Append writes the log first, then the record, under one lock, after `Journal::check_append` (new in cmux-journal: the append's refusals, asked before any write). So a record always has its log line, and the `cancel` record is gone. - 2/3/4: the replay never deletes a folder and never makes a missing one again (a forget only drops the chat's earlier journaled lines). Delete appends its forget under the lock and removes the folder off it, with no wait for the record. - 5: a failed fsync in a checkpoint freezes it (a folder that does not open is retried); a frozen checkpoint returns an error, and the store asks the daemon to stop for a restart (`Store::stop_reason`, wired in daemon.rs), whose replay writes the logs again. Ten failed checkpoints in a row ask the same, so the journal past the checkpoint stays bounded. - 6: an error in one chat keeps its journaled lines in journal-unapplied/ and the open goes on. - 7: the journal's storage refuses writes under a lost home (`FencedStorage`, `config::refuse_lost_home`). - With store.journal off, a journal left by a run with it on is replayed, checkpointed and retired first (`catch_up`). - Nits: the checkpoint is never past the journal's durable end (no `Ahead` after a normal crash); damage is cleared after the next checkpoint; a kind-index copy goes before its segment is cut; an unreadable checkpoint fails the open; the open waits up to 5 s for the journal lock; crashed checkpoint temporaries are removed; `on_written` leaves cmux-journal (no user); README lists store.journal. * acpmux: split the journal store under its size budget (a move) journal_store.rs grew to 1085 lines and 71 functions with the review fix (budget 1000 and 60), and the fix grew store.rs and daemon.rs, which are over budget on the base. A pure move, no behavior change: - store/journal_store/{checkpoint,fenced,repair}.rs: the checkpoint and its thread, `FencedStorage`, the replay and the repair. - store/log_sync.rs: `LocalStore::sync_log`, `SyncFailed` and the test seam `FAIL_NEXT_LOG_SYNC`, out of store.rs (2139 lines on the base, 2113 now). - daemon/stop_causes.rs: `AbortOnDrop`, `next_signal` and `store_asks_to_stop`, out of daemon.rs (2052 on the base, 2046 now). check-no-godfiles.sh --base 51cfdee876b on every changed file: no file over budget grew, and no new file is over budget. * acpmux: the unrepairable-chat test waits for line 1 to be durable before its checkpoint At 03fccdef69c (fleet 32df3c72 on aws-m4pro-9, 144d4ccc on Linux) the test kept chat-b's lines [1, 2, 3] aside, not [2, 3]: it flushed right after appending line 1, and the checkpoint is now never past the journal's durable end (the review's Ahead nit), so line 1 was still past it. The fix's behavior is right; the test now waits until line 1 is durable, so its checkpoint covers it. On the code before the fix the test still fails (the open stops at chat-b's link). * acpmux: replies and events wait for durability, tests (red, S2c) DESIGN.md 3.3d. With the event journal on and every journal sync taking 400 ms (`SimStorage::set_sync_delay`): - an_acked_change_is_answered_once_its_record_is_durable: a rename's reply must come no sooner than one sync after its `renamed` record. Red here: the reply goes at once. - agent_output_is_not_held_behind_another_chats_acked_change: agent output to a connection attached to its chat goes at once while another chat's acknowledged change waits for its sync. Green here (nothing waits); it guards the fix against holding agent output behind changes the connection does not hear. * acpmux: replies and events wait until what they show is durable (S2c) DESIGN.md 3.3d; the tests are the commit before this one. With `store.journal` on, each connection gets an outbound gate (server/durable_gate.rs) between it and its transport: - Every line a connection queues raises the gate's `needs` to the last acknowledged LSN (`Store::durability().acked`, set in the journal store's append before the hub publishes anything of the record), except the lines of a `Stream` record's event (agent output, DESIGN C4): the fan-out delivers those in a thread-local stream scope. - The gate's task takes every queued line at once, reads `needs`, waits (async) until the journal's durable LSN reaches it, then sends the batch in order: one wait per batch, as the journal's group commit. - The durable LSN is a `tokio::sync::watch` fed by `Journal::on_durable`. The listener owns a guard: a writer that ends while the store is not being dropped (a panic on its thread) asks the daemon to stop (`Store::stop_reason`, decision 9.8), and the watch closes, so each gate drops what it holds and ends, which closes its connection. - Without the journal, `Store::durability()` is None: no gate, and `Conn::raise_gate` does nothing. Moves, so server/mod.rs and store.rs stay under their size on the base: the outbound queue helpers to server/outbound.rs, `open_with_journal` to the journal store, and `streamed` to store/durability.rs. * acpmux: the journal lock wait's deadline without an unchecked Instant addition `Instant + Duration` can panic on overflow (clippy::arithmetic_side_effects, tracked by the panic ratchet). The wait now has no deadline only in the case that cannot happen (an Instant 5 s ahead overflowing), where it stops waiting at once. * acpmux: a failed log sync stops the real daemon for a restart, test (red) Lead rule 2026-11-04 (3): a daemon-level test of the store's "stop for a restart" before S2c merges, red first. With store.journal on and every log sync failing (a debug build's ACPMUX_TEST_FAIL_LOG_SYNC; the checkpoint every 200 ms through ACPMUX_TEST_JOURNAL_CHECKPOINT_MS), a rename of a chat on disk makes its log dirty, the checkpoint's sync fails and freezes it, and the store asks the daemon to stop. The test asserts: the WebSocket client gets the restart close (1012), the socket client's pending `_acpmux/wait` ends with the connection (no hang), and the daemon exits with status 74. Red here: the daemon stops and sends 1012, but exits 0 (no exit status says that it stopped to be repaired). This commit adds only the two debug-build seams (release builds have none) and the test. * acpmux: a daemon its store stopped for a restart exits with status 74 The test is the commit before this one. `daemon::run` returned Ok and the process exited 0 for every stop, so a supervisor or a person could not tell a stop to be repaired from a normal one; with systemd `Restart=on-failure` it would not even start again. Now the store's stop (`store_asks_to_stop`: a failed log sync froze the event journal's checkpoint, or the journal's writer ended) marks itself, sets the restart handoff (WebSocket clients get 1012, as before), and the process exits with `daemon::EXIT_REPAIR` (74, EX_IOERR). The units prevent only 75, so launchd KeepAlive and systemd Restart=always start it again; the app does not read the daemon's exit status. README: store.journal names the status. * acpmux: the restart test's request in flight goes through the acpmux client Lead rule (3): the client sees a typed error, not a hang. The socket client is now the acpmux client library (acpmux::client::Client, as the CLI uses it), and the test asserts what the CLI makes of the request that the stop ends: `daemon_closed`, retryable (cli::errors::classify). The first red (fleet 41f2d037, hq run 41804640229, at 80c5fb2b7f8) failed only on the exit status (0, wanted 74), with 1012 and the end of the connection passing. * acpmux: the daemon's disk tier with the journal on, tests (red) Lead rule 2026-11-04 (2): a reply the person waits for never waits on a throttled disk sync. The fleet A/B (STATUS 11554) found that at IOPOL_UTILITY an acknowledged journal append waits about 16 ms p50 under another process's disk stress (0.7 to 1.1 ms at the default tier). - the_daemon_with_the_journal_keeps_the_default_disk_tier: with store.journal on, the daemon process keeps the default tier; without it, utility as before (each case in a child of the test binary). - a_background_thread_yields_its_disk_work: a dedicated background thread (the checkpointer) works at utility; a thread beside it does not. The API they use is stubbed here (`qos::enter_daemon` ignores the journal; `qos::background_thread` does nothing), so both fail on macOS. * acpmux: the restart test borrows no &str twice (clippy needless_borrows_for_generic_args) Found by fleet 7f8afd9d (hq run: the S2c server step at 62b4f68ef8b); the step's other clippy red is the base red in link_push_bench.rs. * acpmux: with the journal on, the daemon keeps the default disk tier (S2c) The tests are 994f74119fb. Lead rule 2026-11-04: a reply the person waits for never waits on a throttled disk sync. Measured on two owned Macs (STATUS 11554): one acknowledged append to durable at IOPOL_UTILITY under another app's disk stress took 16.3-16.7 ms p50 and 37-45 ms p95, against 0.7-1.1 ms p50 and 1.5-1.8 ms p95 at the default tier. - qos::enter_daemon(journal): with store.journal on, the daemon process keeps the default disk tier; without it, utility as before. The switch is read alone from config.json (config::store_journal_enabled), before any thread starts, with no harness discovery. - qos::background_thread: the journal store's checkpointer lowers its own disk I/O to utility for its whole life (a thread that does nothing else: a thread-scope policy drops it out of QoS for good). * acpmux: an agent chunk is a Stream record, test (red) The hub's tap logs an ACP session/update under its sessionUpdate kind (hub/tap.rs: `kind = su.to_owned()`), so a record's kind is `agent_message_chunk`, `tool_call`, ... and never `session/update`; the message's method is. The journal store's `streamed` checked the kind, so every agent chunk was journaled as Ack: a sync per batch at the token rate, and in S2c each token's event waited for that sync (the S2c I/O A/B: STRESS token to client 2.2 ms -> 20.3 ms p50 with the journal on). Red here: the chunk reads back as Ack. * acpmux: an agent chunk is a Stream record (the record's message names session/update) The test is the commit before this one. `streamed` now reads the logged message's method: the hub's tap logs a session/update under its sessionUpdate kind (hub/tap.rs), so the kind never said it. Agent output is again Stream (synced with the next acknowledged batch or a second later), as DESIGN.md 3.3b says. * acpmux: review 2 of #34901, tests (red): a crash model that follows real log syncs The logs' OS crash model now follows the syncs that ran (store/sync_model.rs, tests only): a segment keeps its length at its last sync that counts; a failed sync loses what was written since the one before, and a later sync does not bring it back; a cut ends a loss it removes. Every log segment sync goes through log_sync::sync_segment (same calls as before), where the model and FAIL_NEXT_LOG_SYNC hook in. Red tests, one per finding: - 1a an_os_crash_after_a_restart_from_a_kill_keeps_every_acked_line: the restart's first checkpoint retires records whose log lines no sync made durable; expected [1..5], the head gives [1]. - 1b an_os_crash_after_a_restart_from_a_failed_sync_keeps_every_acked_line: the same after a frozen checkpoint and a restart; expected [1..5]. - 2 a_hole_that_starts_inside_a_line_is_repaired (the reviewer's test): a torn line is read as line 6 of a log that moved on; expected [1..9], the head gives [1, 2, 3, 4, 5, 9]. - 3 a_failed_sync_in_flush_session_freezes_the_checkpoint: a failed sync in flush_session does not freeze the checkpoint. * acpmux: review 2 of #34901: a restart writes the replayed lines again; a torn line is not a line; flush_session freezes Finding 1 (high): the repair now writes every merged line again from the repair start, even when the log already has them all. A killed daemon's log writes may be in the page cache only, and after a failed sync the kernel may have dropped pages that still read back and that a later sync reports as written; the first checkpoint after the open then syncs bytes this process wrote before it retires their records. The lines only the log has from the repair start are journaled and made durable first, as before a cut. The work is bounded by the checkpoint interval. Finding 2: a piece of the log is a line only when it parses as one JSON value (piece_seq). A hole that starts inside a line keeps that line's {"seq":N, head and runs into a later line's tail; it was read as a line of a log that moved on, and the chat's journaled lines went aside. Now its bytes go and the lines are written again. Finding 3: a failed sync in flush_session freezes the checkpoint (its file sync is a SyncFailed now), as a failed sync in a checkpoint or at a log's close does. Nits: cut_at syncs events/ after its removals (and the kind-index folder after its copies go), so a removed segment cannot come back beside the lines written again; the README says that with the switch on an event line over 64 MiB is refused and its log does not get it. Tests: the four red tests of d2c63860ece. * acpmux: chat metadata through the event journal, tests (red, R-DURABILITY S3a) DESIGN.md 3.3e. Red on S2c: - last_seq_never_passes_the_log_after_an_os_crash (E13): session.json is synced at each save while the agent's lines wait for the journal's next sync, so after an OS crash the metadata names lines the log lost. - a_chat_made_after_the_checkpoint_survives_an_os_crash_that_loses_its_folder: nothing syncs the sessions folder when a chat's folder is made, and the replay never makes a folder again, so an acknowledged new chat is lost. - a_metadata_change_is_written_to_session_json_by_the_checkpoint: today each change writes and syncs session.json on the request path. Guards (green before and after): a chat made before the checkpoint and deleted is not made again; the replay never writes an older image over session.json. * acpmux: chat metadata through the event journal (R-DURABILITY S3a) DESIGN.md 3.3e, behind `store.journal` (default off). A metadata change is a `meta` record (kind 4: chat id, a `created` flag, the whole image) instead of a session.json write with two syncs on the request path; the reply waits for the record through S2c's gate, one group commit for a burst. - Overlay: each chat's newest image until a checkpoint writes session.json; load, load_startup and list read it first (list under the projection lock, so a checkpoint cannot drop an entry between its two reads). - Checkpoint: waits on its own thread until every record through its LSN is durable, then writes each image it read (session.json carries the image's LSN as `journalLsn`), then the logs and the sessions folder, as before. An image so names only lines the journal holds (E13). A chat deleted or made again meanwhile is passed (its overlay entry is gone or new). - Replay: per chat after its last `forget`, the newest image, written when newer than the file's `journalLsn`. A missing folder is made again only for a chat a replayed `created` record made; a chat made before the checkpoint whose folder is gone keeps its image in journal-unapplied/. - flush_session also waits until the chat's records are durable. - Delete drops the overlay entry with its `forget`, and removes the folder under the folders lock a checkpoint's write also takes. Tests: the 5 of 0505e9bee04 (3 red there). * acpmux: S3a, tests (red): folders handed out of the store Code outside the store reads and moves chat folders directly (transfer_in puts the live folder aside and installs a transferred chat; session_dir hands the folder to the transfer, the session files view and others). With the metadata in the overlay until a checkpoint: - a_folder_replaced_outside_the_store_keeps_its_own_metadata: a read, the checkpoint and the replay must not put the old folder's image in the new one. - session_dir_hands_out_a_folder_whose_metadata_is_written. - flush_session_writes_the_chats_metadata (the trait says log and metadata). - the_checkpoint_never_makes_a_folder_removed_outside_the_store_again. Red on 71bc09a1e63 (S3a's first implementation). * acpmux: S3a red tests: a replay after a folder swap; flush_session's chat has a log (as its callers' chats do) * acpmux: S3a: each metadata image is bound to its chat folder; session_dir and flush_session write it first Code outside the store reads and moves chat folders directly (a transfer puts the live folder aside and installs a transferred chat under the same id). So a `meta` record now carries its folder's identity (file id and birth time): - reads use an overlay image only while the folder at the path is its folder; - the checkpoint writes an image only into its own folder, never makes a missing one, and keeps the image of a removed or replaced folder in journal-unapplied/; - the replay writes an image only into its own folder (or a folder a replayed `created` record made), else keeps it aside. `session_dir` (the transfer, the session files view) and `flush_session` (the trait: "log and metadata to stable storage") write the chat's image first, once its record is durable. The list reads the overlay before the files (a checkpoint writes a file before it drops the entry), so it needs no lock. Tests: the 5 of 0280ac37f5e and 0e617489731. * acpmux: S2 follow-ups from review 3 of #34901, tests (red) - a_chat_whose_repair_failed_is_repaired_at_the_next_start (follow-up 1): the open's checkpoint retires the records of a chat whose repair failed, so no later start puts its lines back (they are only in journal-unapplied/). - a_hole_that_ends_at_a_line_start_keeps_that_line (follow-up 2): a torn line, zeros, then a whole line only the log holds: the piece does not parse and the whole line is dropped. - an_idle_store_does_not_wake_its_checkpointer (follow-up 4, cmux-ad m9): the checkpointer wakes every period with nothing to do. Test seams only (no behavior change): CHECKPOINTER_WAKES and TEST_CHECKPOINT_EVERY_MS. * acpmux: S2 follow-ups from review 3 of #34901: a failed repair holds its records; a hole's last zero splits; an idle checkpointer sleeps - Follow-up 1: a chat whose repair fails at the open (its log cannot be read or written now) keeps its lines in journal-unapplied/ as before, and the checkpoint now stays below its first journaled LSN (`Order::held`), so the next start repairs it again. Bounded: past 256 MiB of journal after it, the hold goes and the lines stay only in journal-unapplied/ (an error says so). - Follow-up 2: a piece splits at its last zero byte (a whole JSON line holds none), so a hole that ends at a line start, after a torn line, keeps that line (it mattered for a line only the log holds). - Follow-up 4 (cmux-ad m9): with nothing to checkpoint, the checkpointer sleeps until a change wakes it (`Inner::touched`, one atomic read per append), then checkpoints one period later; before, it woke every 30 s. Tests: the 3 of d147057bf85. * acpmux: a metadata change logs its line before its metadata, tests (red, R-DURABILITY S3b) DESIGN.md 3.3e, E6: with the event journal, records survive a crash as a prefix in append order. So a change whose line reaches the store before any metadata image that holds it is never, after a crash, in the chat's metadata without its line in the log. A recording store checks the order of the store calls: - a_tag_change_logs_its_line_before_its_metadata: red (tags saves first). - a_rules_change_logs_its_line_before_its_metadata: red (rules saves first). - a_rename_logs_its_line_before_its_metadata: green (a guard). * acpmux: the qos journal tests module is macOS-only (its Linux build held only an unused import: clippy red, review of #35400 F0) * acpmux: review of #35400 F1 and F2, tests (red) - a_read_that_shows_a_line_covers_its_record (F1): a read sees a log line between its log write and its record (the appends' lock held there), and the reply built from it goes out with no wait for that record. - a_journal_that_syncs_nothing_asks_the_daemon_to_stop (F2): with a journal that makes nothing durable, every reply waits without a limit and nothing asks the daemon to stop. Test seams only: PAUSE_AFTER_LOG_WRITE, TEST_STALL_MS. * acpmux: review of #35400 F1, the streaming read, test (red): session/load's replay sends a line before its record * acpmux: the test seam PAUSE_AFTER_LOG_WRITE is reachable from the server tests * acpmux: review of #35400: a read covers the records of the lines it shows; a stalled journal stops the daemon F1: the append writes a line to the log, then appends its record and sets the acknowledged LSN, all under the appends' lock; reads take no lock. A read could show a line before its record existed, and the reply built from it went out at once. Now every journal-store read of the logs ends with a fence on the appends' lock (`Inner::fenced`): a line the read saw has its record appended and its LSN acknowledged when the read returns, so the reply's gate covers it. The one read that sends as it reads (`Hub::scan_events`, session/load's replay) stops at the chat's last_seq read first. F2: a journal writer that keeps failing (it retries without end) held every reply on every connection with no limit. A connection's gate now waits at most 10 s (`STALL_LIMIT`); past it, `Durability::stalled` asks the daemon to stop for a restart, as a failed log sync does. Tests: the 3 of 3916f84dd68 and c553a19fd8b. * acpmux: S3b red tests: the recording store implements delete (a required Store method) * threading-model: the acpmux event journal's threads and bounds (writer, checkpointer, per-connection gate; review of #35400) * acpmux: the stall test's journal syncs take 3 s, not an hour (a sync under way kept its delay and hung the test's end) * acpmux: a tags or rules change logs its line before its metadata (R-DURABILITY S3b) DESIGN.md 3.3e (revised): with the event journal, records survive a crash as a prefix in append order, so a change whose line reaches the store before any metadata image that holds it is never, after a crash, in the chat's metadata without its line in the log (E6). `renamed`, `policy` and `mode` already did this; `tags` and `rules` saved first. Their reply still waits for the save. Tests: the 3 of 5bb3a27a2da and 7a4f02a56f3 (red at 7a4f02a56f3, fleet 88309e91: tags ["save:tagged", "line:tags"]; rules ["line:permission_chat_allowance", "save:ruled", "line:rules"]). * acpmux: the idle checkpointer's period, test (red): a change seen by its second look must not start a checkpoint at once - a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period: the checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies when it gets the timer lock. The second look saw the change, so the checkpointer is in its period wait by then, and the notify ends that wait: a checkpoint runs at once on the checkpointer's thread. This is the cause of a_failed_log_sync_freezes_the_checkpoint failing in the full store:: run of fleet 0577ac62 (that early checkpoint took the line whose sync the test fails, on a thread where the failure is not set). The test holds both threads at the two points, so the order is exact. Test seams only (no behavior change): the checkpointer's test seams are now by the store's home (period, wake count, the two holds), so tests that run as threads of one process under cargo test do not see each other's. * acpmux: the idle checkpointer's period ends only on its own state, not on a wake meant for its idle wait The checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies once it gets the timer lock. Its change shows in the second look, so the checkpointer has gone to its period wait by then, and that notify ended the period at once: a checkpoint ran right away on the checkpointer's thread (a burst no longer shared one checkpoint), and a spurious wakeup could do the same. In the full store:: run of fleet 0577ac62 such a checkpoint took the line whose sync a_failed_log_sync_freezes_the_checkpoint fails, so that test's own flush had nothing to do and returned Ok. The period wait now ends only on the period, `wanted` or `stop` (`wait_timeout_while`): each of the checkpointer's two waits ends only on the state its waker sets before it notifies under the timer lock. No sleep, no retry. The idle wait is unchanged. Test: a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period (edca3b4823d). * acpmux: S3a review (#35483) red tests: switch-off metadata and deletes, a late settle, a failed session.json folder sync, a purge after the checkpoint F1: metadata written with the switch off survives turning it on again; a chat deleted with the switch off after a failed catch-up is not made again. F2: a late settle never writes an older image over a checkpointed one (the reviewer's test and hold). F3: a failed folder sync after a metadata write freezes the checkpoint. F4: a chat made after the checkpoint and deleted is not made again after a kill. Test seams only; the code is the S3a head merged with feat. * acpmux: S3a review (#35483) red tests: a new chat is acknowledged only once its folder is durable; session_path never waits The root of F4 and of F1's delete case: the replay makes a chat's folder again, so a chat removed by any path comes back. The fix will never make a folder again, which needs an acknowledged chat's folder to survive an OS crash: the test models what each folder's last sync kept (sync_model) and asserts the gate's durability passes a new chat only after its sessions entry is synced. P3 1 of the review: a caller that needs only the path never waits for the journal (Store::session_path, a stub with session_dir's behavior here). * acpmux: S3a review (#35483) fixes: the replay never makes a folder again; switch-off changes are kept; no late or unsynced metadata write F1: each meta record carries the stamp (file id, modification time, length) of the session.json its save saw. The replay writes an image over a file the journal did not write only when one of the chat's records saw that file; otherwise the file moved on without the journal (a run with store.journal off whose catch-up did not replay, or an older build) and the image goes to journal-unapplied/, as S2 does for a log. F1 (deletes) and F4 at the root: the replay never makes a chat's folder again. A save that made a folder notes its record's LSN; the new acpmux-journal-folders thread (the daemon's disk tier) syncs sessions/ at once, and the durable LSN the S2c gate waits for stays below that record until then (journal_store/new_folders.rs). So an acknowledged chat's folder survives an OS crash, and a folder that is gone was removed: a purge whose forget a kill took back, a run with the switch off, the transfer recovery. The created flag and the replay's folder remake are gone. F2: write_through does not write a file that holds the same or a newer journalLsn. F3: session.json is written with its own syncs (file, then folder); a failed sync is SyncFailed and freezes the checkpoint; the replay and the kept-aside images return their errors. P3 1: Store::session_path gives the path with no lock, wait or write; an agent's start, the fixed-id check, the temporary-folder sweep, the host probe and the transfer recovery's root use it. The replaced test (a chat made after the checkpoint survives an OS crash that loses its folder) asserted a remake that cannot be right any more; the new test asserts the guarantee that makes the loss impossible, then the crash. Red (7 of 17 journal_meta_tests, each for its reason; the other 10 pass): macOS 5bc434a9 and 624d5df4, Linux a2207ff0 and 638a44ef. * acpmux: review of #35562 red tests: a failed repair holds no other chat's checkpoint; a checkpoint after it writes only durable images F1 (the reviewer's test, renamed): a checkpoint after a failed repair writes only durable metadata images. F2/F3 at the root: a failed repair does not hold the checkpoint of other chats; a retention rewrite of another chat after a failed repair stays after a restart. Guards: the reviewer's two switch-off tests (green here since the S3a fix 0d246364d90: the stamp rule and no folder remake), and a chat deleted and made again after its failed repair gets none of the old chat's lines. * acpmux: review of #35562: a failed repair keeps its chat's lines in a repair file; no hold of the global checkpoint The hold (Order::held, 49d7e5cef83) kept every chat's records past the checkpoint while one chat's repair failed. That let a checkpoint write metadata images whose records were not durable (F1), made each start with the switch off replay the whole window again and undid a retention rewrite (F2), and grew the journal with each restart (F3, the 256 MiB bound counted one run only). Now a chat whose log cannot be read or written at the open keeps its lines in journal-unapplied/<id>.<lsn>.repair.ndjson (a header with the chat id, then the lines, synced). The next open journals them again, waits until they are durable, removes the file and repairs the chat with them as with any journaled line (a line in both is the same bytes, so a file journaled twice after a crash changes nothing). The checkpoint of every chat goes on as before S2's follow-ups. A delete removes its chat's repair file after the folder, so a chat made again under the same id gets none of the old lines. Red at 23870301064 (macOS a8be0684): the 3 fail for their reasons ("the checkpoint stayed at 1 below chat-a's records (the journal ends at 5)"; "the checkpoint wrote an image to session.json before its record was durable"; "the open kept a copy of a rewritten log aside"); the guards pass; 339 of 342. * acpmux: S3a delta review (N1) red tests: a change to a chat whose session.json another journal wrote reaches it A folder moved in from another home (build adoption), checkpointed and replayed, and a journal made again from nothing: the other journal's journalLsn must not hide a change made here. The first test is the reviewer's (fleet 4d0d98bc and 071ad7c4 at e944a7c21de). * acpmux: S3a delta review (N1): journalLsn is bound to its journal's id; a file another journal wrote follows the stamp rule session.json now carries journalId (16 hex digits, a string) beside journalLsn. The checkpoint's "held" check (F2) and the replay's rule compare an LSN only for a file with this journal's id: a file from another journal (a chat folder build adoption moved in from another home, or a home whose journal was made again) counts as one this journal did not write, so the stamp rule decides, as for a run with the switch off. cmux-journal: Journal::journal_id (the id already in every segment header; kept across opens and retirements), with a unit test. Red at 7c8cb957745: fleet c29b659d (Linux) and 47570494 (macOS) running; the reviewer's red at e944a7c21de: 4d0d98bc (macOS) and 071ad7c4 (Linux).
lawrencecchen
added a commit
that referenced
this pull request
Nov 5, 2026
…ion never waits for a checkpoint in flight (R-DURABILITY, follow-up 4 of review 3 of #34901) (#35866) * cmux-journal: one append-only journal per daemon, with group commit and ack after durable (R-DURABILITY S1) A new crate, no consumer yet. It is stage 1 of the R-DURABILITY rewrite (handoff/rewrites/R-DURABILITY/DESIGN.md): every acknowledged change of the cmux-tui daemon and of acpmux goes into one journal of this format, and a reply goes out only once its record is synced. - Journal::append copies one record into memory (no I/O, no wait) and returns its LSN. One writer thread takes every pending record as one batch, writes it with one write, and syncs it once when it holds an acked record (group commit). Durable listeners then release the replies that wait for those LSNs. Stream records (terminal output, streamed tokens) are written in the next batch and synced with the next acked batch. - Sync levels: F_BARRIERFSYNC per acked batch on macOS (fdatasync on Linux), and F_FULLFSYNC at most 1 s after the first barrier sync, so a power loss takes back at most 1 s of acked batches, always a suffix. - Recovery is the one start path (clean exit, crash, kill -9, OS crash): it syncs the last segment, checks every record's CRC and LSN, and hands the records after the caller's compaction point to its replay. A segment is only ever appended to: a torn tail stays in its file and the journal goes on in a new segment whose header names the last good LSN. Damage that is not a torn tail moves the later segments to quarantine/ (never deleted), and no LSN is used twice. - A failed write is cut back and written again in the same segment; a failed sync is never retried on that segment (its pages may be clean and lost): the records after the durable LSN go into a new segment. - Storage is the file system seam. The `sim` feature's SimStorage keeps synced and unsynced bytes apart and crashes at any storage operation, then keeps what a process crash, an OS crash or a power loss keeps. Tests: unit tests of the format, group commit, ack after durable, listeners, sync levels, idle (no storage operation), backlog, seal, lock, rotation, failed writes and syncs at every operation, torn tails, quarantine and LSN reuse; the crash simulation (200 seeds per crash kind by default, CMUX_JOURNAL_CRASH_SEEDS for the acceptance run); and a round trip with a torn tail on the real file system. OWNERS.md: a row for the crate (R-DURABILITY rewrite owner). * cmux-journal: type a test range; drop an unread segment field (fleet clippy) Fleet job 1a494d9372ec2bb7d71e8e1e at 26e4e71e3f9 refused two lines: E0689 (is_multiple_of on an untyped range in a test) and dead_code (Active::seq in the writer: the next segment's number is next_seq). The acpmux notify::Watcher import error in the same run is a base red. * cmux-journal: a failed F_FULLFSYNC is an error, not a plain fsync; no segment while nothing is owed - macOS sync_file fell back to fsync on ANY fcntl error. An EIO from F_FULLFSYNC or F_BARRIERFSYNC then became a success of a weaker sync, and the writer counted the records durable. Now only "not supported" (ENOTSUP, EOPNOTSUPP, EINVAL, ENOTTY: a network or FAT volume) falls back; any other error reaches the writer, which starts a new segment. - After a rotation the writer made the next segment at its next pass even with nothing to write (the shutdown pass did it too). A segment is now made only when records or a full sync are owed. - Test-only and lint-safety rewrites: a slice parameter in place of &Vec, a type alias for the crash model, a range check, one split match scrutinee. * cmux-journal: the past-the-end test reads back the storage it wrote to Fleet job 7f6bfd5afb225cd1cee9d0f1 at 36f09c87b8d: 24 of 25 crate tests pass (the three 200-seed crash simulations included). The one red, a_compaction_point_past_the_end_never_reuses_an_lsn (left [1], right [1, 10]), was a test fault: it appended LSN 10 in the restarted storage and then read back the first one. The clippy type_complexity hit of the same run is the crash model type, fixed in f55f8c5c53e. * cmux-journal: a sync a signal interrupts is made again (macOS) fcntl(F_BARRIERFSYNC / F_FULLFSYNC) and fsync can return EINTR. The writer counted that as a failed sync and moved to a new segment. Now the call is made again, as std's sync_data and sync_all do (cvt_r). * cmux-journal: crash sim of write-back errors and crash in recovery; S1 review regressions (red) The S1 review (FIX at 89293fb15f39) found that the crash simulation could not find its F1 and F4: it never crashed inside recovery, a failed operation had no partial effect, and a failed sync never lost a page. This commit changes only the simulation and the tests; the next commit fixes the journal. The simulated machine (sim.rs) now has: - a page cache over a disk, per 4 KiB page; a sync writes the dirty pages; a sync that fails writes some of them and drops the rest as clean, so a later sync reports success without them (Linux write-back error behavior); - a write that fails writes a random prefix first; - one ordered log of names (journaling file system): a folder sync makes durable every change up to the last one in that folder; a rename between folders is one change; an unsynced folder can be lost; - a process crash keeps the machine whole, so a later OS crash still loses what it would have lost; - an OS crash writes each dirty page whole, not at all, or a prefix of its sectors; a power loss also cuts back to a common point after the last full sync; - counters of open files and of injected failures. The crash runs (crash_sim_tests.rs) now run several epochs per seed: appends, a crash at a random operation (sometimes after a failed write or sync), and an open that sometimes crashes inside recovery and is tried again. After each open the oracle checks every record byte for byte, that every record an earlier open replayed is still there, that the epoch's records are a prefix of its appends, and that every acked record (fully durable after a power loss; every written one when only processes crashed) survives. Four runs: process, OS, power loss, and mixed. Regression tests for the review findings, red on the code before the fix: - F1 a_crash_inside_the_quarantine_step_never_reuses_a_quarantined_lsn - F2 a_damaged_header_in_the_last_segment_never_reuses_its_lsns - F3 a_panic_on_the_writer_thread_closes_the_journal - F4 a_restart_after_a_failed_sync_keeps_every_later_ack - F6 recovery_holds_at_most_two_files_open * cmux-journal: the F2 regression damages the header of the last of several segments (red) The reviewer's F2 scenario has two segments: S1 holds 1..10, S2 (the last) holds 11..20, and S2's header is damaged. The test now rotates segments (300-byte segments, 20 acked records) and damages the last one's header, so the report must say lost_through 20 and the next append must take 21. On the code before the fix the moved segment's LSNs read as 0, so the next append reuses one. * cmux-journal: the crash sim forces the F4 order (failed sync, kill, OS crash) (red) On the code before the S1 fix, the crash-sim step was green (fleet 184129d8 at 263da86f223): 6 of 7 runs passed, and the mixed run failed only seed 140 in the full run and passed when it ran alone. The random faults reached the order the review found (F4) in 1 of 800 runs. So these tests were no red/green proof. The order: the sync of a write that spans pages fails and the disk gets none of them (Linux keeps them clean in the cache, so a later sync reports success); the process is killed in the writer's backoff; the next open reads the failed segment from the cache, replays its records and goes on; an OS crash or a power loss then takes those pages, and records an open already replayed are gone. - sim.rs: `fail_next_sync(pages, kind)` fails the next sync that has at least that many dirty pages, and the disk gets none of them. `crash_after_failure(n)` makes the n-th operation after the next injected failure the crash point. - crash_sim_tests.rs: a new run forces the order on every seed (a_kill_after_a_failed_sync_then_os_crashes_keep_every_acked_record): epoch 0 aims the fault, the kill is a process crash at the writer's retry, the open after it does not crash, epoch 1 ends in an OS crash, and a third open checks that the records the second open replayed are all there. In the random runs, one epoch in four aims the fault, with the crash 1 to 3 operations after the failed sync and a crash of any kind after it. - A sim test checks the aimed fault itself (the_sim_fails_the_aimed_sync_and_crashes_in_the_backoff). Expected on this commit (old journal code): the forced run fails every seed; the mixed run fails most runs. The fix commit follows. * cmux-journal: fix the S1 review findings F1-F7 (markers, recovery copy, LSN floor, writer end) Review FIX at 89293fb15f39 (PR #33919 comment 6610468202). The regression tests are the two commits before this one. - Sync markers (format.rs): an entry with flags 1 and no payload, whose LSN field says "every record of this segment before me is synced". The writer appends one after each successful sync that covered records (after the durable listeners, so no reply waits for it), and syncs it fully at a seal and at a rotation. - F4, recovery (step 4 and 6): the last segment is trusted only up to its last marker. Its records after the marker may sit in pages a failed sync dropped (a later sync on another descriptor reports success without them), so recovery writes them again into a new segment: header, records and a marker, fully synced under `<name>.seg.new`, renamed into place, folder synced. The journal always goes on in a segment this open made, never in one an earlier process wrote. A restart in the writer's backoff after a failed sync is then safe. - F1, quarantine: every LSN a quarantined segment holds counts in the LSN floor at every open, and a quarantine that no new segment followed (a crash in the middle of an earlier recovery) is reported as damage again. The highest LSN of every moved segment is read before the first rename. - F2, highest LSN: a segment whose header is damaged is read from its first record (each record's checksum covers its LSN), then the bytes after the first bad entry are scanned for records and markers that still check out. The same scan runs after a cut in the last segment: a marker there is media damage (reported), records there are a torn write (`torn_lsns`); their LSNs are never used again. - F3, writer end: a guard closes the journal when the writer thread ends, also by a panic: appends fail with Closed, waits end with WriterEnded, the listeners are dropped (so a channel a held reply waits on closes), and stats.failing says why. The crate does not abort the process; the daemon owns that policy. - F6: recovery closes each segment after its header is read and opens one again only to read it. (Retirement of compacted segments is an S2 API, before the first consumer.) - F5: recovery syncs the parent folder after it makes the journal folder. - F7: format.rs and lib.rs now say what the code does: a failed write is cut back and retried in the same segment; the cut removes only bytes never synced and never acknowledged; nothing shortens a segment past a synced record. Docs: journal.rs says when open's replayed records are durable (on Ok) and what a stopped writer does. * cmux-journal: split the crash sim's machine model into sim/machine.rs (review G5) sim.rs had 924 lines and 63 functions, over the 60-function limit for a new Rust file (the god-file check warned "not in baseline: split it"). The machine model (each file's page cache and disk, the ordered log of names, and what an OS crash or a power loss leaves) moves to sim/machine.rs (297 lines, 19 functions); the process view and the Storage implementation stay in sim.rs (639 lines, 30 functions). No behavior change: the items move as they are, made visible to the sim module. * cmux-journal: re-review regressions G1-G4 and page eviction in the crash sim (red) Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993). Tests and sim only; the fix follows. - sim: the pages a failed sync dropped stay in the cache until an eviction; `evict_dropped_pages_after_read(true)` makes each read evict the dropped pages it read, so a second read of them sees the disk's bytes. - G1 recovery_copies_the_records_it_checked_when_the_cache_drops_them: a sync fails and drops record 2's pages, a kill in the backoff, the next open reads record 2 from the cache (replayed) and the pages are evicted after that read; then an OS crash. Record 2 and the record 3 acked after the restart must survive. On this code recovery's second, unchecked read stages zeros under a vouching marker, and the last open keeps only record 1 and reports damage. - G2: the F1 regression asserts that every open after the damage reports the loss (lost_through 40). On this code a crash at the open's last step (the folder sync after the new segment's rename) loses the report for good. - G3 a_journal_in_a_symlinked_folder_opens_and_round_trips (real file system): a journal in a symlinked state folder, and a journal folder that is a symlink. On this code the parent folder's sync opens with O_NOFOLLOW (ELOOP). - G4 an_open_after_a_finished_quarantine_reads_no_quarantined_segment: on this code every open reads every quarantined segment in full. * cmux-journal: fix the re-review findings G1-G4 and G6 Re-review FIX at 3498975b5267 (PR #33919 comment 6620113993); the regression tests are the commit before this one (red: fleet de35f6d0). - G1: recovery writes again exactly the bytes its checked read returned. The last segment's entries after its last marker are kept from the checked read (header and payload) and staged from memory; the second, unchecked read of the file is gone. A page that a failed sync dropped can be evicted between two reads, and the second read then returned the disk's zeros, which the copy kept under a vouching marker. - G2: the damage an open finds is written to `DAMAGE` (whole, fully synced, renamed into place, folder synced) before the new segment that hides it from later opens. Every open reports the recorded damage first, until the daemon calls the new `Journal::clear_damage`, which moves the record into `quarantine/` (kept, never deleted). - G3: a folder sync follows a symlink in its path (no O_NOFOLLOW): a state folder, or the journal folder itself, may be a symlink. The sync is for the folder that holds the entries. - G4: the quarantined segments are read for their LSNs only while no live segment is numbered above them (an unfinished quarantine); once one is, its first LSN already carries their floor. - G6: `checked_div` in the scan of the bytes after a cut (the crate's one tracked arithmetic hit). - G5 (the sim split) is the commit before the regressions. * cmux-journal: third-review regressions G7 (A, B) and N1 (red) Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018). Tests only. - G7 A a_damaged_header_after_a_quarantine_never_reuses_a_quarantined_lsn: after a finished quarantine, one byte of the empty new segment's header changes. On this code the next open skips the quarantine scan (the listing counts the damaged segment) and the next append reuses a quarantined LSN. - G7 B a_damaged_newest_segment_after_a_quarantine_reports_its_records: the newest segment holds acked 41..43, and its header and its first record are damaged. On this code the moved segment is scanned with a base far below 41, so the loss is not reported through 43 and the next append is not 44. - N1: the F1 regression also asserts that an open reports each loss once (no reported LSN range lies inside another). On this code a crash after the quarantine's renames makes the next open report the same loss twice. * cmux-journal: fix the third-review findings G7, N1, N2; split recovery.rs Third review FIX at cb893c6b64cd (PR #33919 comment 6621681018); the regression tests are the commit before this one. - G7: the quarantine scan is skipped only when a segment whose header checked out is numbered above every quarantined segment (the folder listing also counted a newest segment with a damaged header, which carries no floor). The scans start from the running floor: `quarantined_floor` and `quarantine` pass the highest LSN counted so far as the base of the next file's scan, so a quarantined file with no readable header and no readable first record is scanned for the LSNs right above what is counted. - The "unfinished quarantine" report is made only when the quarantine holds LSNs nothing else counted (an old quarantine behind a segment that a crash tore at its creation holds none, and a crash is not damage). Its reason now also names a damaged newest header. - N1: a loss whose LSN range lies inside a recorded one is the same loss found again after a crash: it is neither recorded nor reported twice. - N2: recovery.rs's docs name `DAMAGE`; two doc lines over 100 columns are wrapped (lib.rs, sim.rs). - recovery.rs was at 987 of its 1000 lines: the damage record moves to recovery/damage.rs, the entry reader and the LSN scans to recovery/scan.rs (moved as they are). * cmux-journal: report each loss once, also when one open finds it twice (N1) The red run of the N1 regression (fleet 27d011ca, op 64 of the F1 test) shows the case the previous commit did not cover: a crash after the quarantine's renames and before DAMAGE is written; the next open finds both the damaged synced record of the last segment (7..14) and the unfinished quarantine (7..40), and reported both. Now every loss inside another (recorded or found by this open) is dropped, the larger range kept; DAMAGE is rewritten only when its set of losses changed. * cmux-journal: fourth-review regressions G8, N3, N4 (red) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919). Tests only. - G8 a_recovery_copy_above_a_quarantine_never_reuses_a_quarantined_lsn (the reviewer's three steps): records 2..5 unsynced and a kill -9; open #1 copies them into C1 and goes on there with 6..10; C1's header is damaged, open #2 quarantines C1, copies 2..5 again into C2 and makes the empty E (first LSN 11); E's header is damaged. On this code open #3 sees C2 numbered above the quarantine, skips its scan and gives 6, an LSN the quarantine holds. - N3: G7 B and the F1 test assert that every file in quarantine/ is named by a reported loss. On this code the absorbed loss's names are dropped (G7 B). - N4: the F1 test asserts that the loss through 40 names the segment where the journal ends (00000000000000000002.seg). On this code a crash after the moves into quarantine/ and before DAMAGE turns it into the vaguer "unfinished quarantine" report. * cmux-journal: the quarantine's floor in its file names; DAMAGE before the moves (G8, N3, N4) Fourth review FIX at 215c8f0485ea (PR #33919 comment 6622740919); the regression tests are the commit before this one. - G8 (the G7 class): the skip of the quarantine scan trusted a live segment numbered above the quarantine to carry its floor, which a recovery copy does not (its records start at vouched + 1). Now each moved segment is named `<segment>.<n>.<floor>`, with the highest LSN counted when it moved (that file's and every one before it, scanned from the open's floor), and every open takes the quarantine's floor from the names alone: no quarantined file is read (G4 stays), and no header can hide the floor. `quarantined_floor` and the `checked_max_seq` skip are gone. The "unfinished quarantine" report is made when the names' floor is above everything else counted. - N4: `quarantine()` is split into `plan_quarantine` (measure and name every file, no change on disk) and `move_quarantined`; the loss, with the planned names, goes to DAMAGE before the moves. A crash after the moves keeps the precise report (the damaged segment and reason). - N3: when one loss absorbs another, the absorbed loss's files and reason go to the kept one, so DAMAGE names every quarantined file. - Docs: recovery step 5 and lib.rs name the quarantine file format. * cmux-journal: sort the quarantine list with sort_by_key (clippy unnecessary_sort_by) Fleet d9f22bed at b589044b852: the 33 journal tests pass, and clippy -D warnings fails at recovery.rs:478 (`consider using sort_by_key`). * cmux-journal: Journal::on_written, a listener per written batch (for S2's projector) R-DURABILITY S2 (acpmux event logs through the journal) projects each chat's log line into its ndjson file once the journal has WRITTEN the batch that holds it, so readers see stream lines at once. The listener runs on the writer thread after the write and before the sync, once per batch that moved the written LSN: a batch written again into a new segment after a failed sync is not heard twice. When the writer stops, these listeners are dropped too. Test: heard while the batch's sync still runs, once. Port of 145faa6faa8 (branch rewrite/r-durability-s2-acpmux-logs) onto the S1 review fixes. * cmux-journal: Journal::retire_through, the one delete (segments a compaction point covers) R-DURABILITY S2: a daemon's projector compacts the journal into its own files (acpmux: the chat ndjson logs, then a synced checkpoint). Without a delete the segment count only grows (review finding F6). `retire_through(compacted)` deletes the segments that hold no record after the caller's compaction point: a segment goes when the next segment's "previous last LSN" is at or before it, so the newest segment never goes. The point is capped at the durable LSN. The deletes and a folder sync run on the caller's thread (the projector, never a request path). A delete that fails leaves its segment live for a later call. Why a crash during a retirement is safe: the caller reopens from a point at or after `compacted`, and recovery reads nothing from a segment the point covers (it skips it on its next segment's header), so any subset of the deletes may reach the disk. A point older than the retired segments is reported as Damage::Missing. Quarantined segments are never deleted. - Storage::remove_file (FsStorage: std::fs::remove_file; SimStorage: one more change in the ordered log of names, durable at the folder sync). - The live segment list: recovery's, then each segment the writer makes. - JournalStats::segments_retired. - Tests: only covered segments go, and a second call deletes none; an OS crash after it keeps every record after the point and an older point is reported; a crash at every operation of a retirement (process, OS crash, power loss) keeps every record after the point and never reuses an LSN; the newest segment stays; a point in the middle of what this writer wrote. This adds a delete of journal data, so its PR is risk:high. * acpmux: move MemoryStore and the store settings into their own files store.rs (2139 lines, 120 fns) and config.rs (1909 lines, 97 fns) are over the god-file limit, and the journal store (R-DURABILITY S2) must not grow them. MemoryStore moves to store/memory.rs and StoreMode, StoreConfig and its defaults to config/store_config.rs; both stay exported where they were (crate::store::MemoryStore, crate::config::{StoreConfig, StoreMode}). No behavior change: the code moves as it is. * acpmux: the event log's journal switch and its crash tests (red) R-DURABILITY S2 (DESIGN.md 3.3b). Today no event-log append is synced: a line a client was told about is on the disk only after the next flush (shutdown), so an OS crash or a power loss takes back every line since (E12). This commit adds only the switch, a seam and the tests; the journal store follows. - `store.journal` (StoreConfig::journal, default false). - `Store::wait_durable(timeout)`: tests and tools wait until every appended record is durable. The default returns at once (the Local and Memory stores never sync an append), which is today's behavior. - `store::open_with_journal` (tests): the store with the journal on a given `cmux_journal::Storage`. Here it ignores the journal and opens LocalStore. - acpmux depends on cmux-journal (dev: with the `sim` feature). - store/journal_store_tests.rs, red on this commit: - a_line_the_store_says_is_durable_survives_an_os_crash (lines 2..20 are lost: nothing synced them); - a_deleted_chat_stays_deleted_after_a_crash and a_line_the_log_refused_is_never_written_later pass here, and must keep passing with the journal store (no replay may bring them back); - a_kill_keeps_every_line_once passes here and must keep passing (a replay after a kill must write no line twice). * acpmux: the event log through one journal, behind `store.journal` (R-DURABILITY S2) DESIGN.md 3.3b. With `store.journal` on, each event-log append is one record in `<home>/journal/` (cmux-journal: group commit, one sync per batch; a power loss takes back at most the last second, always a suffix), then the same line goes to the chat's ndjson as before. Readers, segments and file formats do not change. Default off. - store/journal_store.rs (JournalStore over LocalStore): - append: under one lock, the journal record (a memory copy, no I/O), then the log line; a line the log refuses gets a `cancel` record. Agent stream lines (`in` session/update, claude.*, codex.*) are Class::Stream, every other line Class::Ack. - checkpoint (`<home>/journal-projected.json`): every 30 s while the journal grew, past 64 MiB, at flush (shutdown) and before a retention commit. It syncs each log written since (LocalStore::sync_log, with no lock held while the disk works) and the sessions folder, writes the LSN (temp file, sync, rename, folder sync, every error checked), then retires the journal before it. A log close that failed its sync freezes the checkpoint until a restart (LocalStore::sync_failed). - recovery: the records after the checkpoint; for each chat, the log is cut right after its last readable line below the first journaled seq (later segments removed, the boundary one shortened, their kind-index copies dropped), and the journaled lines are written again in order. `forget` deletes a chat again; `cancel` keeps a refused line out. - delete: `forget` record, folder removal, then a bounded wait until the record is durable. - seal: the logs refuse writes and the journal syncs fully. - LocalStore: `append` is serialize + `append_line`; `sync_log`; a failed sync at a log's close is noted (`sync_failed`). - store::open opens the JournalStore when `store.journal` is on. Not yet (S2c): replies and events do not wait for durability; the daemon does not stop when the journal closes; the writer thread keeps the process's Utility I/O policy. * acpmux: journal store tests for the seal and a chat made again after its delete (red) - a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay: a chat is deleted (its forget record), made again under the same id (a transfer back: session.json and new lines), then a kill -9. On this code the replay deletes the folder again and its session.json is gone. - a_line_after_the_seal_is_refused_and_never_written_later: coverage of the seal (it passes here; the fix also closes a race the test cannot force). * acpmux: journal store: the seal under the appends' lock; a replay keeps a chat made again - seal takes the appends' lock: an append lands in both the journal and the logs, or in neither. Before, an append that raced the seal could leave a journal record whose line the sealed logs refused (its cancel then failed on the closed journal), and the next start would write that line. - append and delete check the logs' seal first, under the same lock, and do not touch the journal after it (no error per append at shutdown). - replay deletes a forgotten chat again only when no line follows its forget record: otherwise the chat was made again under the same id, and its folder holds that chat's own files. * cmux-journal: fifth-review regressions G9 (red) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646). Tests only. - a_skip_hidden_behind_a_recovery_copy_is_never_given_out_again (the reviewer's route): record 1 synced, 2..10 Stream and 11 Ack synced as one run with marker 11, media damage in record 5. Open #1 reports (4, 11], copies 2..4 into segment 2 and makes the empty segment 3 from 12. Segment 3's header is damaged. On this code open #2 reads segment 1 only through the copy's previous last LSN, never scans the damage again, and gives 5. - torn_lsns_hidden_behind_a_recovery_copy_are_never_given_out_again: the same for torn records 4..5 (a tear, then a whole record that no marker vouches for) behind the copy of 2..3: on this code the next append gets 4. * cmux-journal: name a skipped LSN floor in a FLOOR.<lsn> file (G9) Fifth review FIX at 9c47594e1c04 (PR #33919 comment 6623794646); the regression tests are the commit before this one. When an open gives out LSNs above everything its kept and new segments say (damaged synced records or torn records in the last segment, which a copy then hides from every later open), the only record of that floor was the header of the empty segment it makes; a damaged header lost it, and DAMAGE is cleared by `clear_damage` (torn records are not damage at all). Now such an open makes the empty file `FLOOR.<lsn>` and syncs its folder before any new segment, and every open takes the highest floor from the folder listing (no file read), as it does for the quarantine names. The caller's compaction point, the quarantine's names and this open's moves say their floors already: no file is made for them. * acpmux: journal store regressions from the S2 review (red) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824). Each test fails on this commit and passes after the fix that follows it: - 1a lines_written_with_the_switch_off_survive_turning_it_on_again - 1b a_kill_while_the_journal_syncs_keeps_the_lines_the_log_got - 1c a_line_flush_session_made_durable_survives_losing_its_record - 2 a_folder_put_back_after_a_delete_is_not_deleted_by_the_replay - 3 a_kill_in_a_delete_leaves_the_chat_whole_or_gone - 4 a_delete_does_not_wait_for_the_journal_sync - 5 a_failed_log_sync_freezes_the_checkpoint, a_checkpoint_that_keeps_failing_asks_the_daemon_to_stop - 6 one_chat_that_cannot_be_repaired_does_not_stop_the_open - 7 the_journal_storage_refuses_a_lost_home - the repair keeps a log's lines after a hole, also through a crash in it: an_os_crash_hole_is_repaired_and_the_lines_after_it_kept, a_crash_inside_a_repair_loses_no_line - nits: an_unreadable_checkpoint_fails_the_open, the_open_waits_for_the_previous_daemon_to_let_go_of_the_journal, a_crashed_checkpoint_temporary_is_removed_at_open, a_crash_right_after_a_checkpoint_reports_no_damage - the catch-up with the switch off: switching_off_after_a_crash_keeps_every_acknowledged_line - cmux-journal: check_append_says_what_append_would_do So they compile here, this commit adds the seams they use, with no behavior: `Store::stop_reason` (default None), `Journal::check_append` (always Ok), `FencedStorage` (never asks its fence), the open report and a test stop point after a repair's cut. * acpmux: the checkpoint freeze test fails a log sync, not a folder open (red) The freeze guards against a failed fsync (a later fsync of the same file may report success for pages the kernel dropped), not against a folder the store cannot open: a chmod 000 folder is an open error, which the fix retries instead of freezing. The test now fails the log sync itself through a test seam, `FAIL_NEXT_LOG_SYNC`. Red on the code before the fix, like the other review regressions (fleet d8e7037d, 1c22175e, 63604b26 at e030f4fe402: 17 of 17 new acpmux tests and the crate's check_append test fail). * acpmux: the journal store repairs logs, never rebuilds them (S2 review) Review FIX at ab0e9535d9da (PR #34901 comment 6624439824); the regression tests are the two commits before this one. DESIGN.md 3.3c. - 1a/1b/1c: recovery no longer cuts a chat's log at its first journaled line and writes back only the journal's lines. It merges the log's lines and the journal's by seq: a line in both must be equal, a line only the log has stays, a line only the journal has goes in its place. Only bytes that do not read (a torn tail, a hole of zeros) are dropped. When the merge must change the log before its end, the lines only the log holds from there are journaled and made durable first, so a crash inside the repair loses no line. A log that moved on without the journal (a line in both that differs, lines out of order) is left as it is, and the journal's lines go to journal-unapplied/. - Append writes the log first, then the record, under one lock, after `Journal::check_append` (new in cmux-journal: the append's refusals, asked before any write). So a record always has its log line, and the `cancel` record is gone. - 2/3/4: the replay never deletes a folder and never makes a missing one again (a forget only drops the chat's earlier journaled lines). Delete appends its forget under the lock and removes the folder off it, with no wait for the record. - 5: a failed fsync in a checkpoint freezes it (a folder that does not open is retried); a frozen checkpoint returns an error, and the store asks the daemon to stop for a restart (`Store::stop_reason`, wired in daemon.rs), whose replay writes the logs again. Ten failed checkpoints in a row ask the same, so the journal past the checkpoint stays bounded. - 6: an error in one chat keeps its journaled lines in journal-unapplied/ and the open goes on. - 7: the journal's storage refuses writes under a lost home (`FencedStorage`, `config::refuse_lost_home`). - With store.journal off, a journal left by a run with it on is replayed, checkpointed and retired first (`catch_up`). - Nits: the checkpoint is never past the journal's durable end (no `Ahead` after a normal crash); damage is cleared after the next checkpoint; a kind-index copy goes before its segment is cut; an unreadable checkpoint fails the open; the open waits up to 5 s for the journal lock; crashed checkpoint temporaries are removed; `on_written` leaves cmux-journal (no user); README lists store.journal. * acpmux: split the journal store under its size budget (a move) journal_store.rs grew to 1085 lines and 71 functions with the review fix (budget 1000 and 60), and the fix grew store.rs and daemon.rs, which are over budget on the base. A pure move, no behavior change: - store/journal_store/{checkpoint,fenced,repair}.rs: the checkpoint and its thread, `FencedStorage`, the replay and the repair. - store/log_sync.rs: `LocalStore::sync_log`, `SyncFailed` and the test seam `FAIL_NEXT_LOG_SYNC`, out of store.rs (2139 lines on the base, 2113 now). - daemon/stop_causes.rs: `AbortOnDrop`, `next_signal` and `store_asks_to_stop`, out of daemon.rs (2052 on the base, 2046 now). check-no-godfiles.sh --base 51cfdee876b on every changed file: no file over budget grew, and no new file is over budget. * acpmux: the unrepairable-chat test waits for line 1 to be durable before its checkpoint At 03fccdef69c (fleet 32df3c72 on aws-m4pro-9, 144d4ccc on Linux) the test kept chat-b's lines [1, 2, 3] aside, not [2, 3]: it flushed right after appending line 1, and the checkpoint is now never past the journal's durable end (the review's Ahead nit), so line 1 was still past it. The fix's behavior is right; the test now waits until line 1 is durable, so its checkpoint covers it. On the code before the fix the test still fails (the open stops at chat-b's link). * acpmux: replies and events wait for durability, tests (red, S2c) DESIGN.md 3.3d. With the event journal on and every journal sync taking 400 ms (`SimStorage::set_sync_delay`): - an_acked_change_is_answered_once_its_record_is_durable: a rename's reply must come no sooner than one sync after its `renamed` record. Red here: the reply goes at once. - agent_output_is_not_held_behind_another_chats_acked_change: agent output to a connection attached to its chat goes at once while another chat's acknowledged change waits for its sync. Green here (nothing waits); it guards the fix against holding agent output behind changes the connection does not hear. * acpmux: replies and events wait until what they show is durable (S2c) DESIGN.md 3.3d; the tests are the commit before this one. With `store.journal` on, each connection gets an outbound gate (server/durable_gate.rs) between it and its transport: - Every line a connection queues raises the gate's `needs` to the last acknowledged LSN (`Store::durability().acked`, set in the journal store's append before the hub publishes anything of the record), except the lines of a `Stream` record's event (agent output, DESIGN C4): the fan-out delivers those in a thread-local stream scope. - The gate's task takes every queued line at once, reads `needs`, waits (async) until the journal's durable LSN reaches it, then sends the batch in order: one wait per batch, as the journal's group commit. - The durable LSN is a `tokio::sync::watch` fed by `Journal::on_durable`. The listener owns a guard: a writer that ends while the store is not being dropped (a panic on its thread) asks the daemon to stop (`Store::stop_reason`, decision 9.8), and the watch closes, so each gate drops what it holds and ends, which closes its connection. - Without the journal, `Store::durability()` is None: no gate, and `Conn::raise_gate` does nothing. Moves, so server/mod.rs and store.rs stay under their size on the base: the outbound queue helpers to server/outbound.rs, `open_with_journal` to the journal store, and `streamed` to store/durability.rs. * acpmux: the journal lock wait's deadline without an unchecked Instant addition `Instant + Duration` can panic on overflow (clippy::arithmetic_side_effects, tracked by the panic ratchet). The wait now has no deadline only in the case that cannot happen (an Instant 5 s ahead overflowing), where it stops waiting at once. * acpmux: a failed log sync stops the real daemon for a restart, test (red) Lead rule 2026-11-04 (3): a daemon-level test of the store's "stop for a restart" before S2c merges, red first. With store.journal on and every log sync failing (a debug build's ACPMUX_TEST_FAIL_LOG_SYNC; the checkpoint every 200 ms through ACPMUX_TEST_JOURNAL_CHECKPOINT_MS), a rename of a chat on disk makes its log dirty, the checkpoint's sync fails and freezes it, and the store asks the daemon to stop. The test asserts: the WebSocket client gets the restart close (1012), the socket client's pending `_acpmux/wait` ends with the connection (no hang), and the daemon exits with status 74. Red here: the daemon stops and sends 1012, but exits 0 (no exit status says that it stopped to be repaired). This commit adds only the two debug-build seams (release builds have none) and the test. * acpmux: a daemon its store stopped for a restart exits with status 74 The test is the commit before this one. `daemon::run` returned Ok and the process exited 0 for every stop, so a supervisor or a person could not tell a stop to be repaired from a normal one; with systemd `Restart=on-failure` it would not even start again. Now the store's stop (`store_asks_to_stop`: a failed log sync froze the event journal's checkpoint, or the journal's writer ended) marks itself, sets the restart handoff (WebSocket clients get 1012, as before), and the process exits with `daemon::EXIT_REPAIR` (74, EX_IOERR). The units prevent only 75, so launchd KeepAlive and systemd Restart=always start it again; the app does not read the daemon's exit status. README: store.journal names the status. * acpmux: the restart test's request in flight goes through the acpmux client Lead rule (3): the client sees a typed error, not a hang. The socket client is now the acpmux client library (acpmux::client::Client, as the CLI uses it), and the test asserts what the CLI makes of the request that the stop ends: `daemon_closed`, retryable (cli::errors::classify). The first red (fleet 41f2d037, hq run 41804640229, at 80c5fb2b7f8) failed only on the exit status (0, wanted 74), with 1012 and the end of the connection passing. * acpmux: the daemon's disk tier with the journal on, tests (red) Lead rule 2026-11-04 (2): a reply the person waits for never waits on a throttled disk sync. The fleet A/B (STATUS 11554) found that at IOPOL_UTILITY an acknowledged journal append waits about 16 ms p50 under another process's disk stress (0.7 to 1.1 ms at the default tier). - the_daemon_with_the_journal_keeps_the_default_disk_tier: with store.journal on, the daemon process keeps the default tier; without it, utility as before (each case in a child of the test binary). - a_background_thread_yields_its_disk_work: a dedicated background thread (the checkpointer) works at utility; a thread beside it does not. The API they use is stubbed here (`qos::enter_daemon` ignores the journal; `qos::background_thread` does nothing), so both fail on macOS. * acpmux: the restart test borrows no &str twice (clippy needless_borrows_for_generic_args) Found by fleet 7f8afd9d (hq run: the S2c server step at 62b4f68ef8b); the step's other clippy red is the base red in link_push_bench.rs. * acpmux: with the journal on, the daemon keeps the default disk tier (S2c) The tests are 994f74119fb. Lead rule 2026-11-04: a reply the person waits for never waits on a throttled disk sync. Measured on two owned Macs (STATUS 11554): one acknowledged append to durable at IOPOL_UTILITY under another app's disk stress took 16.3-16.7 ms p50 and 37-45 ms p95, against 0.7-1.1 ms p50 and 1.5-1.8 ms p95 at the default tier. - qos::enter_daemon(journal): with store.journal on, the daemon process keeps the default disk tier; without it, utility as before. The switch is read alone from config.json (config::store_journal_enabled), before any thread starts, with no harness discovery. - qos::background_thread: the journal store's checkpointer lowers its own disk I/O to utility for its whole life (a thread that does nothing else: a thread-scope policy drops it out of QoS for good). * acpmux: an agent chunk is a Stream record, test (red) The hub's tap logs an ACP session/update under its sessionUpdate kind (hub/tap.rs: `kind = su.to_owned()`), so a record's kind is `agent_message_chunk`, `tool_call`, ... and never `session/update`; the message's method is. The journal store's `streamed` checked the kind, so every agent chunk was journaled as Ack: a sync per batch at the token rate, and in S2c each token's event waited for that sync (the S2c I/O A/B: STRESS token to client 2.2 ms -> 20.3 ms p50 with the journal on). Red here: the chunk reads back as Ack. * acpmux: an agent chunk is a Stream record (the record's message names session/update) The test is the commit before this one. `streamed` now reads the logged message's method: the hub's tap logs a session/update under its sessionUpdate kind (hub/tap.rs), so the kind never said it. Agent output is again Stream (synced with the next acknowledged batch or a second later), as DESIGN.md 3.3b says. * acpmux: review 2 of #34901, tests (red): a crash model that follows real log syncs The logs' OS crash model now follows the syncs that ran (store/sync_model.rs, tests only): a segment keeps its length at its last sync that counts; a failed sync loses what was written since the one before, and a later sync does not bring it back; a cut ends a loss it removes. Every log segment sync goes through log_sync::sync_segment (same calls as before), where the model and FAIL_NEXT_LOG_SYNC hook in. Red tests, one per finding: - 1a an_os_crash_after_a_restart_from_a_kill_keeps_every_acked_line: the restart's first checkpoint retires records whose log lines no sync made durable; expected [1..5], the head gives [1]. - 1b an_os_crash_after_a_restart_from_a_failed_sync_keeps_every_acked_line: the same after a frozen checkpoint and a restart; expected [1..5]. - 2 a_hole_that_starts_inside_a_line_is_repaired (the reviewer's test): a torn line is read as line 6 of a log that moved on; expected [1..9], the head gives [1, 2, 3, 4, 5, 9]. - 3 a_failed_sync_in_flush_session_freezes_the_checkpoint: a failed sync in flush_session does not freeze the checkpoint. * acpmux: review 2 of #34901: a restart writes the replayed lines again; a torn line is not a line; flush_session freezes Finding 1 (high): the repair now writes every merged line again from the repair start, even when the log already has them all. A killed daemon's log writes may be in the page cache only, and after a failed sync the kernel may have dropped pages that still read back and that a later sync reports as written; the first checkpoint after the open then syncs bytes this process wrote before it retires their records. The lines only the log has from the repair start are journaled and made durable first, as before a cut. The work is bounded by the checkpoint interval. Finding 2: a piece of the log is a line only when it parses as one JSON value (piece_seq). A hole that starts inside a line keeps that line's {"seq":N, head and runs into a later line's tail; it was read as a line of a log that moved on, and the chat's journaled lines went aside. Now its bytes go and the lines are written again. Finding 3: a failed sync in flush_session freezes the checkpoint (its file sync is a SyncFailed now), as a failed sync in a checkpoint or at a log's close does. Nits: cut_at syncs events/ after its removals (and the kind-index folder after its copies go), so a removed segment cannot come back beside the lines written again; the README says that with the switch on an event line over 64 MiB is refused and its log does not get it. Tests: the four red tests of d2c63860ece. * acpmux: chat metadata through the event journal, tests (red, R-DURABILITY S3a) DESIGN.md 3.3e. Red on S2c: - last_seq_never_passes_the_log_after_an_os_crash (E13): session.json is synced at each save while the agent's lines wait for the journal's next sync, so after an OS crash the metadata names lines the log lost. - a_chat_made_after_the_checkpoint_survives_an_os_crash_that_loses_its_folder: nothing syncs the sessions folder when a chat's folder is made, and the replay never makes a folder again, so an acknowledged new chat is lost. - a_metadata_change_is_written_to_session_json_by_the_checkpoint: today each change writes and syncs session.json on the request path. Guards (green before and after): a chat made before the checkpoint and deleted is not made again; the replay never writes an older image over session.json. * acpmux: chat metadata through the event journal (R-DURABILITY S3a) DESIGN.md 3.3e, behind `store.journal` (default off). A metadata change is a `meta` record (kind 4: chat id, a `created` flag, the whole image) instead of a session.json write with two syncs on the request path; the reply waits for the record through S2c's gate, one group commit for a burst. - Overlay: each chat's newest image until a checkpoint writes session.json; load, load_startup and list read it first (list under the projection lock, so a checkpoint cannot drop an entry between its two reads). - Checkpoint: waits on its own thread until every record through its LSN is durable, then writes each image it read (session.json carries the image's LSN as `journalLsn`), then the logs and the sessions folder, as before. An image so names only lines the journal holds (E13). A chat deleted or made again meanwhile is passed (its overlay entry is gone or new). - Replay: per chat after its last `forget`, the newest image, written when newer than the file's `journalLsn`. A missing folder is made again only for a chat a replayed `created` record made; a chat made before the checkpoint whose folder is gone keeps its image in journal-unapplied/. - flush_session also waits until the chat's records are durable. - Delete drops the overlay entry with its `forget`, and removes the folder under the folders lock a checkpoint's write also takes. Tests: the 5 of 0505e9bee04 (3 red there). * acpmux: S3a, tests (red): folders handed out of the store Code outside the store reads and moves chat folders directly (transfer_in puts the live folder aside and installs a transferred chat; session_dir hands the folder to the transfer, the session files view and others). With the metadata in the overlay until a checkpoint: - a_folder_replaced_outside_the_store_keeps_its_own_metadata: a read, the checkpoint and the replay must not put the old folder's image in the new one. - session_dir_hands_out_a_folder_whose_metadata_is_written. - flush_session_writes_the_chats_metadata (the trait says log and metadata). - the_checkpoint_never_makes_a_folder_removed_outside_the_store_again. Red on 71bc09a1e63 (S3a's first implementation). * acpmux: S3a red tests: a replay after a folder swap; flush_session's chat has a log (as its callers' chats do) * acpmux: S3a: each metadata image is bound to its chat folder; session_dir and flush_session write it first Code outside the store reads and moves chat folders directly (a transfer puts the live folder aside and installs a transferred chat under the same id). So a `meta` record now carries its folder's identity (file id and birth time): - reads use an overlay image only while the folder at the path is its folder; - the checkpoint writes an image only into its own folder, never makes a missing one, and keeps the image of a removed or replaced folder in journal-unapplied/; - the replay writes an image only into its own folder (or a folder a replayed `created` record made), else keeps it aside. `session_dir` (the transfer, the session files view) and `flush_session` (the trait: "log and metadata to stable storage") write the chat's image first, once its record is durable. The list reads the overlay before the files (a checkpoint writes a file before it drops the entry), so it needs no lock. Tests: the 5 of 0280ac37f5e and 0e617489731. * acpmux: S2 follow-ups from review 3 of #34901, tests (red) - a_chat_whose_repair_failed_is_repaired_at_the_next_start (follow-up 1): the open's checkpoint retires the records of a chat whose repair failed, so no later start puts its lines back (they are only in journal-unapplied/). - a_hole_that_ends_at_a_line_start_keeps_that_line (follow-up 2): a torn line, zeros, then a whole line only the log holds: the piece does not parse and the whole line is dropped. - an_idle_store_does_not_wake_its_checkpointer (follow-up 4, cmux-ad m9): the checkpointer wakes every period with nothing to do. Test seams only (no behavior change): CHECKPOINTER_WAKES and TEST_CHECKPOINT_EVERY_MS. * acpmux: S2 follow-ups from review 3 of #34901: a failed repair holds its records; a hole's last zero splits; an idle checkpointer sleeps - Follow-up 1: a chat whose repair fails at the open (its log cannot be read or written now) keeps its lines in journal-unapplied/ as before, and the checkpoint now stays below its first journaled LSN (`Order::held`), so the next start repairs it again. Bounded: past 256 MiB of journal after it, the hold goes and the lines stay only in journal-unapplied/ (an error says so). - Follow-up 2: a piece splits at its last zero byte (a whole JSON line holds none), so a hole that ends at a line start, after a torn line, keeps that line (it mattered for a line only the log holds). - Follow-up 4 (cmux-ad m9): with nothing to checkpoint, the checkpointer sleeps until a change wakes it (`Inner::touched`, one atomic read per append), then checkpoints one period later; before, it woke every 30 s. Tests: the 3 of d147057bf85. * acpmux: a metadata change logs its line before its metadata, tests (red, R-DURABILITY S3b) DESIGN.md 3.3e, E6: with the event journal, records survive a crash as a prefix in append order. So a change whose line reaches the store before any metadata image that holds it is never, after a crash, in the chat's metadata without its line in the log. A recording store checks the order of the store calls: - a_tag_change_logs_its_line_before_its_metadata: red (tags saves first). - a_rules_change_logs_its_line_before_its_metadata: red (rules saves first). - a_rename_logs_its_line_before_its_metadata: green (a guard). * acpmux: the qos journal tests module is macOS-only (its Linux build held only an unused import: clippy red, review of #35400 F0) * acpmux: review of #35400 F1 and F2, tests (red) - a_read_that_shows_a_line_covers_its_record (F1): a read sees a log line between its log write and its record (the appends' lock held there), and the reply built from it goes out with no wait for that record. - a_journal_that_syncs_nothing_asks_the_daemon_to_stop (F2): with a journal that makes nothing durable, every reply waits without a limit and nothing asks the daemon to stop. Test seams only: PAUSE_AFTER_LOG_WRITE, TEST_STALL_MS. * acpmux: review of #35400 F1, the streaming read, test (red): session/load's replay sends a line before its record * acpmux: the test seam PAUSE_AFTER_LOG_WRITE is reachable from the server tests * acpmux: review of #35400: a read covers the records of the lines it shows; a stalled journal stops the daemon F1: the append writes a line to the log, then appends its record and sets the acknowledged LSN, all under the appends' lock; reads take no lock. A read could show a line before its record existed, and the reply built from it went out at once. Now every journal-store read of the logs ends with a fence on the appends' lock (`Inner::fenced`): a line the read saw has its record appended and its LSN acknowledged when the read returns, so the reply's gate covers it. The one read that sends as it reads (`Hub::scan_events`, session/load's replay) stops at the chat's last_seq read first. F2: a journal writer that keeps failing (it retries without end) held every reply on every connection with no limit. A connection's gate now waits at most 10 s (`STALL_LIMIT`); past it, `Durability::stalled` asks the daemon to stop for a restart, as a failed log sync does. Tests: the 3 of 3916f84dd68 and c553a19fd8b. * acpmux: S3b red tests: the recording store implements delete (a required Store method) * threading-model: the acpmux event journal's threads and bounds (writer, checkpointer, per-connection gate; review of #35400) * acpmux: the stall test's journal syncs take 3 s, not an hour (a sync under way kept its delay and hung the test's end) * acpmux: a tags or rules change logs its line before its metadata (R-DURABILITY S3b) DESIGN.md 3.3e (revised): with the event journal, records survive a crash as a prefix in append order, so a change whose line reaches the store before any metadata image that holds it is never, after a crash, in the chat's metadata without its line in the log (E6). `renamed`, `policy` and `mode` already did this; `tags` and `rules` saved first. Their reply still waits for the save. Tests: the 3 of 5bb3a27a2da and 7a4f02a56f3 (red at 7a4f02a56f3, fleet 88309e91: tags ["save:tagged", "line:tags"]; rules ["line:permission_chat_allowance", "save:ruled", "line:rules"]). * acpmux: an open that replays many chats flushes the drive a bounded number of times, test (red) Measured (fleet ead05e82, owned Mac aws-m4pro-5): the event journal store's open after a kill takes 29 ms per replayed chat on macOS (8.8 s for 300 chats, past the 8 s start budget; Linux 0.6 ms per chat, 681725fb). Each replayed chat costs about 7 F_FULLFSYNC (every sync_all/sync_data on macOS flushes the drive's cache): the rewrite's truncate and folder syncs, session.json's write_atomic, the checkpoint's log and folder syncs. Test seams only (no behavior change): FULL_SYNCS counts the store's full syncs (`log_sync::sync_full`, `sync_segment`, the folder syncs, session.json). * acpmux: the journal store's checkpoint and recovery sync in write order and flush the drive once Measured (fleet ead05e82, owned Mac aws-m4pro-5): an open after a kill took 29 ms per replayed chat on macOS (8.8 s for 300 chats, past the 8 s start budget), because every sync was an F_FULLFSYNC (std's sync_all and sync_data on macOS), about 7 per chat. Now the checkpoint's log and folder syncs, the repair's cut, and session.json's write sync with F_BARRIERFSYNC (durable against a crash of any process and a kernel panic, in write order; Linux fdatasync as before), and the checkpoint file's own syncs stay full: its F_FULLFSYNC flushes the drive's whole cache once per checkpoint, before any journal record is retired, so everything synced before it is durable against a power loss too (DESIGN.md 3.4: the checkpoint and recovery still end fully durable). cmux-journal: `sync_file(file, SyncLevel)` is public (the crate's tested F_BARRIERFSYNC/F_FULLFSYNC call with its ENOTSUP fallback), so acpmux uses the same sync. Test: an_open_that_replays_many_chats_flushes_the_drive_a_bounded_number_of_times (red at b2240f1b050). * acpmux: the open after a kill syncs nothing per replayed chat, test (red) At 15cfa1421c4 (ordered syncs) an open after a kill still took 6.1 s for 300 replayed chats on an owned Mac (fleet 59175d6b): about 8 ordered syncs per chat at about 2 ms each under load, plus a full flush per log evicted past 64 open logs. The test counts every sync the store makes during the open (seam ALL_SYNCS, tests only) and wants a constant, not one per chat; then the checkpoint (flush) makes every line durable through an OS crash. * acpmux: the open-syncs test counts the opening thread's syncs only (the checkpointer's thread is not the open's cost) * acpmux: the open after a kill syncs nothing per replayed chat; a freeze leaves a durable marker At 15cfa1421c4 an open after a kill still took 6.1 s for 300 replayed chats on an owned Mac (59175d6b), and it synced 283 times for 40 chats (90d87e10): every replayed chat's log was cut and written again (review 2 of #34901, finding 1, the forced rewrite) and its session.json written, then the first checkpoint ran before the open returned. Now: - The repair leaves a log that already holds every merged line, in order and untorn, as it is, and records its segments; the next checkpoint syncs them (ordered) before it retires their records, so finding 1a stays closed. - A freeze writes a durable marker (`journal-frozen`): the next open writes every replayed chat's lines again (finding 1b: the kernel may have dropped pages that still read back); the first checkpoint after it removes the marker, unless a chat's repair failed (the hold keeps it for the next start). - Replayed metadata images go to the overlay; the checkpoint writes session.json. - The open's first checkpoint runs on the checkpointer's thread at once, after the open returns (the daemon binds its socket after the open). Tests: an_open_after_a_kill_syncs_nothing_per_chat_and_the_checkpoint_makes_it_durable (red at 6def18f92c5), a_freeze_marker_makes_the_next_open_write_the_lines_again_then_goes; the restart-then-OS-crash tests of review 2 now flush after the reopen (the open's checkpoint is no longer inside it). * acpmux: the idle checkpointer's period, test (red): a change seen by its second look must not start a checkpoint at once - a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period: the checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies when it gets the timer lock. The second look saw the change, so the checkpointer is in its period wait by then, and the notify ends that wait: a checkpoint runs at once on the checkpointer's thread. This is the cause of a_failed_log_sync_freezes_the_checkpoint failing in the full store:: run of fleet 0577ac62 (that early checkpoint took the line whose sync the test fails, on a thread where the failure is not set). The test holds both threads at the two points, so the order is exact. Test seams only (no behavior change): the checkpointer's test seams are now by the store's home (period, wake count, the two holds), so tests that run as threads of one process under cargo test do not see each other's. * acpmux: the idle checkpointer's period ends only on its own state, not on a wake meant for its idle wait The checkpointer marks itself idle under the timer lock, then looks for work a second time. A change in that window takes the idle flag in `touched` and notifies once it gets the timer lock. Its change shows in the second look, so the checkpointer has gone to its period wait by then, and that notify ended the period at once: a checkpoint ran right away on the checkpointer's thread (a burst no longer shared one checkpoint), and a spurious wakeup could do the same. In the full store:: run of fleet 0577ac62 such a checkpoint took the line whose sync a_failed_log_sync_freezes_the_checkpoint fails, so that test's own flush had nothing to do and returned Ok. The period wait now ends only on the period, `wanted` or `stop` (`wait_timeout_while`): each of the checkpointer's two waits ends only on the state its waker sets before it notifies under the timer lock. No sleep, no retry. The idle wait is unchanged. Test: a_change_seen_by_the_idle_checkpointers_second_look_waits_one_period (edca3b4823d). * acpmux: S3a review (#35483) red tests: switch-off metadata and deletes, a late settle, a failed session.json folder sync, a purge after the checkpoint F1: metadata written with the switch off survives turning it on again; a chat deleted with the switch off after a failed catch-up is not made again. F2: a late settle never writes an older image over a checkpointed one (the reviewer's test and hold). F3: a failed folder sync after a metadata write freezes the checkpoint. F4: a chat made after the checkpoint and deleted is not made again after a kill. Test seams only; the code is the S3a head merged with feat. * acpmux: S3a review (#35483) red tests: a new chat is acknowledged only once its folder is durable; session_path never waits The root of F4 and of F1's delete case: the replay makes a chat's folder again, so a chat removed by any path comes back. The fix will never make a folder again, which needs an acknowledged chat's folder to survive an OS crash: the test models what each folder's last sync kept (sync_model) and asserts the gate's durability passes a new chat only after its sessions entry is synced. P3 1 of the review: a caller that needs only the path never waits for the journal (Store::session_path, a stub with session_dir's behavior here). * acpmux: S3a review (#35483) fixes: the replay never makes a folder again; switch-off changes are kept; no late or unsynced metadata write F1: each meta record carries the stamp (file id, modification time, length) of the session.json its save saw. The replay writes an image over a file the journal did not write only when one of the chat's records saw that file; otherwise the file moved on without the journal (a run with store.journal off whose catch-up did not replay, or an older build) and the image goes to journal-unapplied/, as S2 does for a log. F1 (deletes) and F4 at the root: the replay never makes a chat's folder again. A save that made a folder notes its record's LSN; the new acpmux-journal-folders thread (the daemon's disk tier) syncs sessions/ at once, and the durable LSN the S2c gate waits for stays below that record until then (journal_store/new_folders.rs). So an acknowledged chat's folder survives an OS crash, and a folder that is gone was removed: a purge whose forget a kill took back, a run with the switch off, the transfer recovery. The created flag and the replay's folder remake are gone. F2: write_through does not write a file that holds the same or a newer journalLsn. F3: session.json is written with its own syncs (file, then folder); a failed sync is SyncFailed and freezes the checkpoint; the replay and the kept-aside images return their errors. P3 1: Store::session_path gives the path with no lock, wait or write; an agent's start, the fixed-id check, the temporary-folder sweep, the host probe and the transfer recovery's root use it. The replaced test (a chat made after the checkpoint survives an OS crash that loses its folder) asserted a remake that cannot be right any more; the new test asserts the guarantee that makes the loss impossible, then the crash. Red (7 of 17 journal_meta_tests, each for its reason; the other 10 pass): macOS 5bc434a9 and 624d5df4, Linux a2207ff0 and 638a44ef. * acpmux: review of #35562 red tests: a failed repair holds no other chat's checkpoint; a checkpoint after it writes only durable images F1 (the reviewer's test, renamed): a checkpoint after a failed repair writes only durable metadata images. F2/F3 at the root: a failed repair does not hold the checkpoint of other chats; a retention rewrite of another chat after a failed repair stays after a restart. Guards: the reviewer's two switch-off tests (green here since the S3a fix 0d2…
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stage 2 of the R-DURABILITY rewrite (design:
handoff/rewrites/R-DURABILITY/DESIGN.md3.3, 3.3a, 3.3b and 3.3c). It needs #33919 (thecmux-journalcrate), merged into feat-cmux-next asda881c3b3c1.Today no acpmux event-log append is synced. A line a client was told about (a queue edit, a permission decision,
prompt_accepted, a status) reaches the disk only at the nextflush, which runs at shutdown. So an OS crash or a power loss takes back every line since the daemon started its last flush (DESIGN E12). Withstore.journalon, each line goes to the chat'sevents/*.ndjsonas before and then one record with the same line goes to<home>/journal/(group commit, one sync per batch; a power loss takes back at most the last second, always a suffix). Readers, segment files and every file format stay as they are.The switch
store.journalis off by default. With it off, nothing changes, except that a journal left by a run with it on is first written into the logs (below).What changes
cmux-journal:Journal::retire_through(lsn), the crate's one delete: it removes the journal segments that hold no record after the caller's compaction point (tests: a crash, process, OS or power loss, at every storage operation of a retirement).Journal::check_append(len): whether an append would be refused now (closed, behind), asked before the caller writes anything else.acpmux/src/store/journal_store.rs(new),JournalStoreoverLocalStore:check_append, the line to the chat's log, then the record. So every record's line went to the log, and a refused line is never journaled. A log line may lack its record (a crash between the two, a record still in memory at a kill). Agent stream lines (insession/update,claude.*,codex.*) areStreamrecords, every other line isAck.<home>/journal-projected.json): every 30 s while the journal grew, past 64 MiB of it, atflush(shutdown) and before a retention commit. It syncs every log written since and the sessions folder, writes an LSN at or before the journal's durable end (temporary file, sync, rename, folder sync, every error checked), then retires the journal before it. A failed fsync (in it, at a log's close, or inflush_session) freezes it: it then returns an error (a retention commit does not go ahead), and the store asks the daemon to stop with a restart (Store::stop_reason, a select branch indaemon.rs), whose replay writes the logs again. Ten failed checkpoints in a row ask the same. So the journal past the checkpoint stays bounded.seq(a line in both must be equal; a line only the log has stays; a line only the journal has goes in its place); only bytes that do not parse as a whole JSON line (a torn tail, a hole of zeros, a line a hole ran into a later one) are dropped. Every merged line is written again from the repair start, so the first checkpoint after a restart syncs bytes this process wrote (a killed daemon's writes may be in the page cache only; a failed sync's dropped pages still read back). The lines only the log holds from there are journaled and made durable before the cut, so a crash inside the repair loses no line. A chat whose folder is gone is not written again. A log that moved on without the journal (a line in both that differs, lines out of order: a run with the switch off or an older build after a crash) is left as it is, and the journal's lines go to<home>/journal-unapplied/, as do the lines of a chat whose log cannot be read or written; the open goes on.forgetrecord under the lock (the replay then drops the chat's earlier journaled lines), then the folder's removal off the lock, with no wait.flushthen checkpoints.FencedStorage, which refuses writes under a lost home (config::refuse_lost_home). An unreadable checkpoint fails the open. The open waits up to 5 s for the journal lock (a restart overlaps the old daemon's exit). Crashed checkpoint temporaries are removed.store::openwith the switch off: a journal a run with it on left is replayed, checkpointed and retired first (journal_store::catch_up), so a crash with the switch on and a start with it off keep every acknowledged line.LocalStore:appendis serialize +append_line;sync_log; a failed fsync is aSyncFailederror; a failed sync at a log's close is noted.MemoryStoreand the store settings move tostore/memory.rsandconfig/store_config.rs(a pure move, so the over-limitstore.rsandconfig.rsdo not grow). README listsstore.journal.Review fixes (independent review FIX at ab0e953, comment 6624439824)
lines_written_with_the_switch_off_survive_turning_it_on_again,switching_off_after_a_crash_keeps_every_acknowledged_linea_kill_while_the_journal_syncs_keeps_the_lines_the_log_gotflush_sessionsynced whose records an OS crash took backa_line_flush_session_made_durable_survives_losing_its_recordan_os_crash_hole_is_repaired_and_the_lines_after_it_kept,a_crash_inside_a_repair_loses_no_linea_folder_put_back_after_a_delete_is_not_deleted_by_the_replaya_kill_in_a_delete_leaves_the_chat_whole_or_gonea_delete_does_not_wait_for_the_journal_synca_failed_log_sync_freezes_the_checkpoint,a_checkpoint_that_keeps_failing_asks_the_daemon_to_stopjournal-unapplied/, the open goes onone_chat_that_cannot_be_repaired_does_not_stop_the_openFencedStoragethe_journal_storage_refuses_a_lost_homeAheadafter a normal crasha_crash_right_after_a_checkpoint_reports_no_damagean_unreadable_checkpoint_fails_the_openthe_open_waits_for_the_previous_daemon_to_let_go_of_the_journala_crashed_checkpoint_temporary_is_removed_at_opencancelrecord wrote a refused line againcancela_line_the_log_refused_is_never_written_later(kept)Ok,on_writtenunused, READMEon_writtenremoved; README entrycheck_appendhas its own test in the crate:check_append_says_what_append_would_do.Review 2 fixes (independent review FIX at dcfecad, comment 6629184957)
The logs' OS crash model in the tests now follows the syncs that ran (
store/sync_model.rs, tests only): a segment keeps its length at its last sync that counts; a failed sync loses what was written since the one before, and a later sync does not bring it back; a cut ends a loss it removes. Every log segment sync goes throughlog_sync::sync_segment.an_os_crash_after_a_restart_from_a_kill_keeps_every_acked_linean_os_crash_after_a_restart_from_a_failed_sync_keeps_every_acked_linepiece_seq)a_hole_that_starts_inside_a_line_is_repaired(the reviewer's test)flush_sessiondid not freeze the checkpointSyncFailed; the journal store freezes on ita_failed_sync_in_flush_session_freezes_the_checkpointevents/between a cut's removals and the lines written againcut_atsyncsevents/after its removals, and the kind-index folder after its copies goFound by the S2c A/B, fixed here: the hub's tap logs an ACP
session/updateunder itssessionUpdatekind, so agent chunks wereAckrecords (a sync each). The record's method now saysStream(an_agent_chunk_is_a_stream_record; reda43e979ae39, fleet 2b6becd0; greence8dcba5458, fleet 409c5cfa).Risk: high
retire_through(journal segments that a synced checkpoint covers), and the repair's cut of a chat's log, which removes only bytes that do not read and lines it writes back, after every line only the log held is durable in the journal.store.journaloff. The next start writes the journal's lines into the logs first, then runs as today. An older build that does not know the journal reads the logs as they are; if it ran after a crash with the switch on, it misses at most the lines since the last checkpoint (30 s), and turning the switch on again later keeps every line that build wrote (a log that moved on is left as it is; the journal's lines go tojournal-unapplied/).Known gaps
renamed, so another connection's reply (a session list) can show the new name in that window, and a crash in it loses the rename that the reply showed. S2c's wait does not close it, because the reply can be built before the record is appended. S3 (chat metadata through the journal, due 2026-11-05 18:00Z) appends the metadata image under the chat's meta lock with the change, which closes it.Testing
acpmuxstore::journal_store_tests(the journal on the simulated file systemcmux_journal::SimStorage; the chat logs are real files, and an OS crash of them follows the syncs that ran,store/sync_model.rs), 28 tests: the 6 below, the 17 regressions of review 1, the 4 of review 2 and the Stream class test.a_line_the_store_says_is_durable_survives_an_os_crash,a_kill_keeps_every_line_once,a_deleted_chat_stays_deleted_after_a_crash,a_line_the_log_refused_is_never_written_later,a_line_after_the_seal_is_refused_and_never_written_later,a_chat_made_again_after_its_delete_keeps_its_files_through_a_replay.cmux-journal: the retirement tests andcheck_append_says_what_append_would_do.Red on the code before each fix:
First round, red on
LocalStore(53e933431e0): fleet a736b8cd (hq run 41773493313):a_line_the_store_says_is_durable_survives_an_os_crashfailed (lines 2..20 lost), in the full run and alone. At00512a295dea: fleet dc9dde1c (hq run 41780243300):a_chat_made_again_after_its_delete_keeps_its_files_through_a_replayfailed (None, wantedSome("again")).Review regressions at
e030f4fe402e: 17 of 17 new tests fail, each also alone, the 6 earlier ones pass: owned Mac aws-m4pro-4 fleet d8e7037d, Linux fleet 1c22175e (hq run 41791487676); the crate'scheck_appendtest: fleet 63604b26 (hq run 41791488139). The failures name the findings: the switch-off lines gone ([1..5], wanted[1..9]), a kill lost line 4,flush_session's line 3 gone, the put-back chat deleted ([]), a deleted chat back in part ([2]), the delete waited 5.0 s, the hole's last line gone ([1..9]), the open failed on chat-b's link,Ahead { journal_end: 1, from: 3 }, the lock not waited for, and so on.The freeze test as it is now (it fails the log sync itself,
8b71b4f7a690): fleet 4addcf1e (hq run 41794281768):a_failed_log_sync_freezes_the_checkpointfailed ("a checkpoint after a failed sync went ahead"), also alone.Review 2 regressions at
d2c63860ece(red): owned Mac aws-m4pro-7, fleet bec8251c, and Linux, fleet 17755e95 (hq run 41825080459): the 4 new tests fail, each also alone, the other 24 pass. 1a and 1b give[1](wanted[1..5]), 2 gives[1, 2, 3, 4, 5, 9](wanted[1..9]), 3 "a checkpoint after a failed sync in flush_session went ahead".The Stream class test at
a43e979ae39(red): fleet 2b6becd0, the agent chunk read back asAck.Green at the head
3273b2ade1f81deb90a9367229ffb85040af4d6a(feat-cmux-next merged twice withscripts/merge-main.sh:3b245c15615mergesb43ce2b4ae2, with one conflict indaemon.rs's module list, feat'sterm_senderfrom #35147 and S2'sstop_causes, both kept;3273b2ade1fmergesd3d5a6246abcleanly):all store::: fmt, clippy and 317 of 317 tests pass.journal_store_tests28 of 28.daemon821 of 822; the one failure is the base redacpmux_daemon_idle_has_no_periodic_wakeups(below).Green at the previous head
84f9c266975af5eec7216f751ea055e279ba7467(the head the review 3 SHIP names):journal_store_tests28 of 28. fleet 490a1dae (Linux, hq run 41826018179): 28 of 28.all store::: fmt pass, tests 316 of 316 (filterstore::); clippy red only atcmux-tui/crates/cmux-tui/tests/link_push_bench.rs:59(type_complexity), the base red from Cloud copy: compressed fs.write bytes (fs-write-deflate-v1) and fewer round trips per file #34538.daemontests 810 of 812.acpmux_daemon_idle_has_no_periodic_wakeupsfails the same way on the merge based422b94b60dalone (fleet 6c027917: an idle daemon woke 8, 3 and 4 times per 10 s window): a base red.router_exits_when_its_daemon_is_killedfailed once in the full run (set_read_timeoutEINVAL attests/router_lifetime.rs:248, the router had already closed the socket) and passed on its rerun; it passes alone on the base too (fleet bfb4e48d): a timing flake, not this PR.Green at the previous head
dcfecade4df4421713ec773e0255ccde78d28cf3(the review fix merged with feat-cmux-nextd422b94b60d, which holds #33919 asda881c3b3c1; the cmux-journal files conflicted only as add/add, and feat's copy equals the S1 head51cfdee876b):all store: fmt green, tests 616 of 616 passed. Its only red step is clippy atcmux-tui/crates/cmux-tui/tests/link_push_bench.rs:59(type_complexity): a base red from Cloud copy: compressed fs.write bytes (fs-write-deflate-v1) and fewer round trips per file #34538 (586595362c2), a file this PR does not touch; S2c's server step (335b7dac) has the same single red.journal_store_tests23 of 23,cmux-tui-rust-check: green.cmux-journalhas no line.daemontests 815 of 816. The one failure,a_daemon_whose_home_is_removed_stops(ENOTEMPTY when the test removes the daemon's home), fails the same way on featd422b94b60dalone (fleet e0522147, hq run 41803060724): a base red, owned by BASE RED ACPMUX (a bisect of3a436460eaa1..d422b94b60d); this PR does not touch it. It passed at this branch before the feat merge (fleet 5f5c5b49, hq run 41796145764).7a8bbdce62c: fleet 6798b9c1 (hq run 41798793202)all storegreen (fmt, clippy, 592 of 592); at92b52bfe6f0: fleet 92e85362cmux_journal_testsand ed455835 crash sim green (the crate did not change since).Observed / Not run
storeanddaemonfilters run the store's, the hub's store and the daemon's tests; the PR's CI runs the rest), and a daemon withstore.journalon (no dogfood yet: the switch is off by default). The daemon's stop branch (store_asks_to_stop) has no daemon-level test; the store's side (stop_reasonset on a freeze and after ten failures) has.acpmux: clippy::indexing_slicing 703 > 686was fixed on feat by base red: Rust panic ratchet: teleport, acpmux task plan/turn retry/suggestion gate, and link sync index reads (cmux-automations 1, cmux-tui, acpmux) #34735 and base red: panic ratchet rows from #34798 (link sync history slices) #34941, which this branch now has).Changelog
none
Proof
No capture: a daemon storage change behind a default-off setting, with no UI.
Review receipt
Review (r-durability-s2-merge-reviewer): SHIP at 3273b2a (comment 6632437161): the merge resolution at this head. Review 3 (r-durability-s2-reviewer): SHIP at 84f9c26 (comment 6630992897): the code before the merges. Non-green checks at this head are base findings (comment 6632519884). Its non-blocking follow-ups go in their own PRs: a repair that fails after it changed a log keeps its chat's records past the checkpoint; a hole that ends at a line start; the open's sync cost per replayed chat, measured against the start budget; the idle checkpointer's 30 s wake.