Move runner lease heartbeats to memory store

[serrrfirat]

Summary

• keep accepted/terminal turn state durable in /turns/state.json, but move high-churn runner lease heartbeat state into a per-FilesystemTurnStateStore memory map
• overlay memory runner leases onto snapshot reads/mutations so recovery, completion, cancellation, and read models still see current lease metadata while the process is alive
• lazily seed the memory lease from the durable snapshot after a store reopen, and assert heartbeats no longer rewrite state.json or materialize /turns/runner-leases sidecars

Tradeoff

Live heartbeat extensions are now process-local. That matches the current single-process runner path and removes a hot durable write source, but a future multi-process runner model will need either a shared lease owner/store or a different coordination boundary before relying on cross-process heartbeats.

Deployment note

This intentionally does not migrate old /turns/runner-leases/* heartbeat sidecars. In-flight runs across a restart may be recovered or failed from the older /turns/state.json lease timestamp. That is acceptable for the current internal single-instance deployment; fresh runs after deploy use the memory-backed lease path.

Tests

• cargo fmt --check
• PATH=/Users/firatsertgoz/.rustup/toolchains/1.96.0-aarch64-apple-darwin/bin:$PATH CARGO_TARGET_DIR=/Volumes/NVME/ironclaw-target CARGO_INCREMENTAL=0 cargo clippy -p ironclaw_turns --tests -- -D warnings
• PATH=/Users/firatsertgoz/.rustup/toolchains/1.96.0-aarch64-apple-darwin/bin:$PATH CARGO_TARGET_DIR=/Volumes/NVME/ironclaw-target CARGO_INCREMENTAL=0 cargo test -p ironclaw_turns --test filesystem_turn_state_contract -- --nocapture
• PATH=/Users/firatsertgoz/.rustup/toolchains/1.96.0-aarch64-apple-darwin/bin:$PATH CARGO_TARGET_DIR=/Volumes/NVME/ironclaw-target CARGO_INCREMENTAL=0 cargo test -p ironclaw_turns

[coderabbitai]

📝 Walkthrough

Walkthrough

Replaces the filesystem-backed RunnerLeaseSidecar with an in-memory RunnerLeaseMemory/RunnerLeaseStore, rewires FilesystemTurnStateStore to use it, and updates contract tests to assert memory-backed lease behavior and no durable runner-lease files.

Changes

Runner lease: filesystem sidecar → in-memory store

Layer / File(s)	Summary
New RunnerLeaseMemory/RunnerLeaseStore types and record derives crates/ironclaw_turns/src/filesystem_store/runner_lease.rs	Drops Serialize/Deserialize from RunnerLeaseRecord, removes filesystem/CAS imports, and introduces RunnerLeaseMemory plus RunnerLeaseStore::new.
RunnerLeaseStore operation implementations crates/ironclaw_turns/src/filesystem_store/runner_lease.rs	Reworks overlay, heartbeat, restore, status writes, delete, and upsert to mutate the in-memory lease map directly.
FilesystemTurnStateStore wired to RunnerLeaseMemory crates/ironclaw_turns/src/filesystem_store.rs	Updates docs, field type, constructor, and all runner-lease call sites to use runner_lease_store() instead of the sidecar path.
Contract tests for memory lease semantics crates/ironclaw_turns/tests/filesystem_turn_state_contract.rs	Removes the sidecar-specific filesystem helper and rewrites heartbeat/recovery/complete-run/cancel tests for memory-backed leases and lazy snapshot backfill.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

• Heartbeat lease write-behind: move runner-lease renewal off the synchronous Postgres path #5253: Directly matches the shift of runner-lease heartbeat/renewal behavior from durable writes to an in-memory path.
• Migrate ironclaw_turns runner-lease sidecar CAS loops onto shared cas_update #5274: Touches the same runner-lease machinery and replaces the sidecar/CAS implementation, though with different scope.

Possibly related PRs

• nearai/ironclaw#5232: Introduces the runner-lease overlay/heartbeat/restore logic that this PR now rewires from sidecar storage to in-memory storage.

Suggested reviewers

• think-in-universe

Poem

🦀 Sidecars faded, leases went light,
The map now holds the heartbeat at night.
JSON on disk took its last bow,
Memory keeps the runner lease now.

🚥 Pre-merge checks | ✅ 3 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The PR has a solid summary and tests section, but it omits several required template sections like Change Type, Linked Issue, and validation checkboxes.	Add the missing template sections or mark them N/A, including Change Type, Linked Issue, Validation checkboxes, Security Impact, rollback, and review track.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: moving runner lease heartbeats from durable storage to memory.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/memory-backed-turn-runtime

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[gemini-code-assist]

Code Review

This pull request refactors the FilesystemTurnStateStore to use an in-memory store (RunnerLeaseMemory) for high-churn runner lease heartbeats instead of persisting them as filesystem-backed JSON sidecars. This change eliminates the need for filesystem CAS retries and durable writes during heartbeats, while still overlaying active leases onto the durable snapshot. Corresponding contract tests have been updated to reflect the memory-backed behavior and verify that no durable sidecar records are materialized. There are no review comments, and I have no additional feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/ironclaw_turns/src/filesystem_store/runner_lease.rs`:
- Around line 33-38: `RunnerLeaseStore` is using a blocking `std::sync::Mutex`
for shared lease state on the async heartbeat/overlay/cancel path, which can
stall Tokio workers and bypass `apply_timeout`. Update `RunnerLeaseMemory` and
the `RunnerLeaseStore` access paths to use an async-compatible lock (per the
repo’s shared async state pattern, e.g. `Arc<T>` with `RwLock`/async lock) so
callers like the heartbeat and cancel methods await lock acquisition instead of
blocking. Make sure every place that currently calls `lock()` on the leases map
is switched to the async lock API and keeps the timeout behavior intact.
- Around line 276-279: The lazy seeding path in runner_lease should be made
insert-if-absent under a single map lock: the current check in the read path
followed by the unconditional seed_from_snapshot_inner call can race with
concurrent cancellation/retirement and overwrite a newer lease state. Update the
seeding flow around read, seed_from_snapshot_inner, and the map guard logic so
the existence check and insertion happen atomically while holding the same lock,
and only seed when the entry is still absent.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 2ca2761b-1f03-4445-b9a1-38ebab98c9d7

📥 Commits

Reviewing files that changed from the base of the PR and between 1b4aca2 and 30cc1ac.

📒 Files selected for processing (3)

• crates/ironclaw_turns/src/filesystem_store.rs
• crates/ironclaw_turns/src/filesystem_store/runner_lease.rs
• crates/ironclaw_turns/tests/filesystem_turn_state_contract.rs

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

crates/ironclaw_turns/src/filesystem_store/runner_lease.rs (1)

252-267: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Do not let claim seeding downgrade an existing lease.

Line 266 routes through upsert, and Line 334 unconditionally inserts. A late seed from the durable claim snapshot can overwrite a newer in-memory CancelRequested/terminal lease for the same run back to Running, reopening heartbeats and breaking two-phase cancellation.

Proposed fix

     async fn seed_from_snapshot_inner(
         &self,
         snapshot: &TurnPersistenceSnapshot,
         run_id: TurnRunId,
     ) -> Result<(), TurnError> {
-        let Some(run) = snapshot.runs.iter().find(|record| record.run_id == run_id) else {
-            return Err(TurnError::ScopeNotFound);
-        };
-        let Some(record) = runner_lease_from_run(run) else {
-            return Err(TurnError::InvalidTransition {
-                from: run.status,
-                to: TurnStatus::Running,
-            });
-        };
-        self.upsert(record).await
+        let record = runner_lease_from_snapshot(snapshot, run_id)?;
+        let mut leases = self.leases.write().await;
+        match leases.get(&run_id) {
+            Some(existing)
+                if existing.runner_id == record.runner_id
+                    && existing.lease_token == record.lease_token =>
+            {
+                Ok(())
+            }
+            Some(_) => Err(TurnError::LeaseMismatch),
+            None => {
+                leases.insert(record.run_id, record);
+                Ok(())
+            }
+        }
     }

As per coding guidelines, “running cancellation is two-phase” and active locks must be transitioned/released exactly once.

Also applies to: 333-335

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/ironclaw_turns/src/filesystem_store/runner_lease.rs` around lines 252
- 267, Prevent snapshot seeding from overwriting a newer lease state: in
seed_from_snapshot_inner and the upsert path it calls, only create or refresh
the runner lease when no lease exists or when the existing lease is compatible
with the snapshot state, and never downgrade an existing CancelRequested or
terminal lease back to Running. Update the RunnerLease/runner_lease_from_run
flow so late durable claim snapshots are ignored or merged without reopening
heartbeats, and ensure any transition/release logic remains single-shot and
monotonic.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@crates/ironclaw_turns/src/filesystem_store/runner_lease.rs`:
- Around line 252-267: Prevent snapshot seeding from overwriting a newer lease
state: in seed_from_snapshot_inner and the upsert path it calls, only create or
refresh the runner lease when no lease exists or when the existing lease is
compatible with the snapshot state, and never downgrade an existing
CancelRequested or terminal lease back to Running. Update the
RunnerLease/runner_lease_from_run flow so late durable claim snapshots are
ignored or merged without reopening heartbeats, and ensure any
transition/release logic remains single-shot and monotonic.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 64747c2c-b4f8-46c7-a637-a7f088f72db5

📥 Commits

Reviewing files that changed from the base of the PR and between 30cc1ac and a320248.

📒 Files selected for processing (2)

• crates/ironclaw_turns/src/filesystem_store.rs
• crates/ironclaw_turns/src/filesystem_store/runner_lease.rs

[railway-app]

🚅 Deployed to the ironclaw-pr-5452 environment in ironclaw-ci-preview

Service	Status	Web	Updated (UTC)
ironclaw	✅ Success (View Logs)	Web	Jun 30, 2026 at 4:53 pm