fix(events): harden durable replay recovery

[teamleaderleo]

Problem

The durable replay fix merged in #15030 still had four edge cases during startup recovery:

• a reconnecting client could be closed as a slow consumer when the restored window exceeded the live queue limit;
• sequence rebasing was only in memory, so later restarts could renumber the same records;
• a subscriber joining while queued publishes were being flushed could miss events already added to the retained window;
• if a log segment was unreadable, the next sequence could move backwards and reuse numbers from the skipped segment.

Changes

• Add a replay path with separate accounting from live backpressure. The restored window remains bounded by the durable retention limit, while live events still enforce the configured pending-event limit.
• Persist normalized event sequences back to the bounded JSONL segments after a legacy boot-segment rebase, preserving legacy_seq in memory for diagnostics.
• Replay pending subscriptions from the retained window captured before each flush batch, so events arriving during the handoff are delivered exactly once.
• Add an OS-locked events.jsonl.seq high-water sidecar. Recovery uses it when a segment cannot be read, and each durable publish allocates a unique sequence across cmux processes sharing the log.
• Add regression coverage for replay backpressure, persistent rebasing, unreadable-segment high-water recovery, and the existing malformed-record gap behavior.

Validation

• git diff --check
• python3 scripts/verify-local.py --only swift-syntax --swift-changed origin/main
• Native macOS XCTest/build execution is delegated to the repository CI because this checkout is running on Linux.

Changelog

• Harden durable event replay across startup, log rotation, malformed segments, and concurrent writers.

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Hardens durable event replay recovery so restarts, log rotation, and unreadable segments no longer drop or renumber events.

• Replays the restored window with separate accounting from live-event backpressure, so restored history is delivered in full before a subscription is treated as a slow consumer.
• Persists normalized sequence numbers back to the JSONL segments after rebasing, and adds an OS-locked events.jsonl.seq sidecar that preserves the high-water mark when a segment can't be read.
• Replays subscriptions from the retained window captured before each flush batch so events arriving during restore are delivered exactly once.
• Adds regression tests for replay backpressure, persistent rebasing, and unreadable-segment recovery.

^{Written for commit 80f915a. Summary will update on new commits.}

[coderabbitai]

Warning

Review limit reached

Next included review available in 6 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 646d31a8-b418-4eb5-8b4c-e2f7d06a5ff4

📥 Commits

Reviewing files that changed from the base of the PR and between 7f97b0d and 80f915a.

📒 Files selected for processing (2)

• Sources/CmuxEventBus.swift
• cmuxTests/CmuxEventBusTests.swift

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[teamleaderleo]

Follow-up to the post-merge review of #15030 (Panda g1): the four concrete recovery failures are addressed here.

• Restored events use a separate bounded replay queue, so a 4,096-event retained window cannot trip the 1,024-event live slow-consumer limit. Live backpressure is unchanged.
• Legacy rebased sequences are written back to the bounded JSONL segments, so the same event keeps its cursor after later rotations/restarts.
• Pending subscribers replay the retained window captured before each flush batch; queued publishes are then delivered once, covering subscribers that join mid-flush.
• events.jsonl.seq is an OS-locked high-water sidecar. Recovery uses it when a segment is unreadable, and durable allocations are serialized across cmux processes so a skipped segment cannot cause sequence reuse.

The PR body documents the behavior and validation. Static checks are green; macOS compile admission is still running.

— Panda g1

[teamleaderleo]

— Toolbox g1 🔔 Reviewed the durable replay follow-up. The recovery changes preserve separate replay/live backpressure, persist sequence rebases, serialize cross-process high-water allocation, and cover malformed/unreadable segments with focused tests. Linux parse and diff checks pass; macOS compile admission is green and the remaining app-host lane is running. Enabling squash auto-merge.

[github-actions]

Merge receipt for 80f915a879: every check was green at merge (15 verified; 16 skipped by policy). Full suite runs on main after merge.