Skip to content

docs(reborn): reconcile Tier B self-repair with lease recovery - #6458

Open
italic-jinxin wants to merge 3 commits into
mainfrom
issue-6455-self-repair-reconciliation
Open

italic-jinxin wants to merge 3 commits into
mainfrom
issue-6455-self-repair-reconciliation

Conversation

@italic-jinxin

Copy link
Copy Markdown
Contributor

Summary

  • reconcile the v1 stuck-job self-repair behavior with Reborn's existing heartbeat, lease recovery, checkpoint, and bounded reclaim contracts
  • correct the stale TurnRunner contract and crash-consistency test commentary to reflect the current [EPIC] error-recoverability endgame — the model recovers from 100% of the errors it sees #6284 behavior
  • document that stuck-run recovery is already satisfied without adding another scheduler or recovery loop
  • split broken-tool auto-rebuild into separate future extension-lifecycle policy work

Linked Issue

Closes #6455
Part of #6369

Validation

  • cargo fmt --all -- --check
  • cargo test -p ironclaw_runner --test turn_scheduler_contract — 31 passed
  • cargo test -p ironclaw_turns --test row_store_crash_consistency lease_expiry_ — 3 passed
  • cargo test -p ironclaw_turns --test retry_failed_turn_store_contract lease_recovery_preserves_retryability — 2 passed
  • terminal lease-expiry and cancellation contract tests — 2 passed
  • cargo test --test reborn_integration_lease_wedge — 13 passed
  • bash scripts/pre-commit-safety.sh

Security Impact

No runtime or authority changes. The documentation preserves bounded recovery and explicitly excludes automatic rebuilding of untrusted tool source.

Database Impact

None. No persistence schema or serialized status vocabulary changes.

Blast Radius

Documentation and test commentary only. Runtime behavior, recovery defaults, scheduler configuration, and production wiring are unchanged.

Risk Assessment

Low. The primary risk is documentation becoming inconsistent with runtime behavior; the updated contract links directly to existing caller-level, persistence, and whole-path regression coverage.

Rollback Plan

Revert commit 293ade150. No data or runtime rollback is required.


Review track: A

@italic-jinxin italic-jinxin added size: M 50-199 changed lines risk: low Changes to docs, tests, or low-risk modules contributor: core 20+ merged PRs labels Jul 22, 2026
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@ironloopai

ironloopai Bot commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor

🔎 IronLoop Review Status

Head: acfb2ec9ac401ac9e4f12de2f73d2917752a97fc
Result: One or more review results were superseded by a newer PR head.
Next: Run @ironloopai review on the latest PR head.
Updated: 2026-07-22T12:52:30.522Z

Current reviewers:

Reviewer State Verdict Findings Last update
ironloop/common-reviewer (reviewer) Superseded N/A N/A 2026-07-22T12:40:18.566Z
Reviewer summaries
Reviewer Detail
ironloop/common-reviewer (reviewer) Superseded by a newer PR head. New head: 405b567. Previous verdict: Changes requested.
Recent activity
Time Reviewer State Detail
2026-07-22T12:17:57.485Z ironloop/common-reviewer (reviewer) Queued Accepted review request for head 293ade1.
2026-07-22T12:17:57.485Z ironloop/common-reviewer (reviewer) Queued Waiting for this reviewer lane to become available.
2026-07-22T12:17:57.581Z ironloop/common-reviewer (reviewer) Started Reviewer worker started.
2026-07-22T12:18:00.515Z ironloop/common-reviewer (reviewer) Workspace ready Prepared isolated checkout (merge_ref) at 8b1a888.
2026-07-22T12:22:06.612Z ironloop/common-reviewer (reviewer) Result captured Changes requested; 1 blocking finding.
2026-07-22T12:22:06.612Z ironloop/common-reviewer (reviewer) Completed Review completed and terminal status was persisted.
2026-07-22T12:40:18.566Z ironloop/common-reviewer (reviewer) Superseded A newer PR head replaced this review (405b567).
Available commands
  • @ironloopai help
  • @ironloopai agents
  • @ironloopai review
  • @ironloopai review --agent <agent>
Run metadata

Admission: webhook accepted the request and IronLoop persisted reviewer state before this projection.

@github-actions github-actions Bot added the scope: docs Documentation label Jul 22, 2026
@railway-app
railway-app Bot temporarily deployed to ironclaw-ci-preview / ironclaw-pr-6458 July 22, 2026 12:18 Destroyed
@github-actions github-actions Bot added size: XS < 10 changed lines (excluding docs) and removed size: M 50-199 changed lines labels Jul 22, 2026
@coderabbitai

coderabbitai Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Summary by CodeRabbit

  • Documentation
    • Clarified crash-recovery expectations for durable running state, including bounded lease-recovery outcomes and checkpoint-aware behavior.
    • Updated TurnRunner, turn persistence, and turn lifecycle contracts with refined cancellation/heartbeat semantics and explicit lease-expiration terminal outcomes.
    • Added verification steps and evidence references for contract validation.
    • Documented Tier B self-repair reconciliation scope, mappings, compatibility/rollback constraints, and why existing coverage suffices.
  • Tests
    • Updated documentation for the crash-consistency test expectations and how to validate outcomes.

Walkthrough

The PR updates TurnRunner persistence, lifecycle, and lease-recovery documentation, records Tier B self-repair scope and verification evidence, and revises crash-consistency test commentary. No executable production logic or public declarations change.

Changes

Turn recovery contract

Layer / File(s) Summary
Lifecycle and persistence outcomes
docs/reborn/contracts/turn-persistence.md, docs/reborn/contracts/turns-agent-loop.md
Defines cancel_requested, checkpoint-aware lease expiry, bounded checkpointless requeueing, terminal failure, and legacy-only RecoveryRequired handling.
Runner lease and exit handling
docs/reborn/contracts/turn-runner.md
Clarifies heartbeat cancellation behavior, lease reconciliation, loop-exit mappings, and verification commands.
Self-repair scope and crash evidence
docs/reborn/tier-b-self-repair-reconciliation.md, crates/ironclaw_turns/tests/row_store_crash_consistency.rs
Separates stuck-run recovery from broken-tool rebuilding and aligns crash commentary with existing caller-level tests.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Suggested reviewers: serrrfirat

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title uses Conventional Commits style and accurately summarizes the docs-only lease-recovery reconciliation.
Description check ✅ Passed The description covers summary, validation, impact, rollback, and review track; only some template sections are unfilled.
Linked Issues check ✅ Passed The changes document existing lease-recovery behavior, bounded reclaim, and broken-tool separation, matching #6455 without runtime changes.
Out of Scope Changes check ✅ Passed All changed files are docs or test commentary aligned to #6455; no unrelated code paths or production behavior were added.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@italic-jinxin

Copy link
Copy Markdown
Contributor Author

@claude review

ironloopai[bot]

This comment was marked as resolved.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/ironclaw_turns/tests/row_store_crash_consistency.rs`:
- Around line 2018-2022: Update the documentation above the crash-consistency
test to describe the actual reopen/compare behavior: state that reopening the
durable bytes preserves the Running state, then the test reads the reopened
store and uses assert_recovered_matches_model to compare snapshots and prefix
invariants with the reference model. Remove the inaccurate claim about a later
lease-recovery pass.

In `@docs/reborn/tier-b-self-repair-reconciliation.md`:
- Around line 25-32: Update the “Stop and report terminal recovery outcomes” row
to state that only expired leases persisted as CancelRequested transition to
Cancelled; document that an observed interrupt racing ahead of persisted
cancellation may instead become sanitized terminal Failed. Keep the existing
mappings for exhausted pre-checkpoint recovery and checkpointed expiry
unchanged.
- Around line 25-32: The evidence cells in the reconciliation table use bare
test names that are not directly auditable. Update the table’s Evidence entries
to include the exact source file paths and, where practical, line ranges for
each referenced test or implementation symbol, including entries such as
scheduler_heartbeats_long_running_executor_until_completion,
wedged_tool_call_is_reaped_by_lease_expiry_not_left_running_forever, and the
lease recovery tests. Preserve the existing claims and dispositions while
ensuring every load-bearing citation points to implementation-level definitions
or write sites.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 038e3a40-0e16-457f-8412-4d103b6d2ecf

📥 Commits

Reviewing files that changed from the base of the PR and between d4ef6e9 and 293ade1.

📒 Files selected for processing (3)
  • crates/ironclaw_turns/tests/row_store_crash_consistency.rs
  • docs/reborn/contracts/turn-runner.md
  • docs/reborn/tier-b-self-repair-reconciliation.md

Comment thread crates/ironclaw_turns/tests/row_store_crash_consistency.rs Outdated
Comment thread docs/reborn/tier-b-self-repair-reconciliation.md
@claude

This comment was marked as resolved.

@github-actions

github-actions Bot commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor

Coverage ratchet

Ratchet mode: ENFORCING

RATCHET PASS: global
  observed: 86.24% (304712 / 353327 lines)
  floor:    85.3% (tolerance 0.5pp -> effective floor 84.8%)
  denominator: 353327 lines now vs 320188 at floor capture (+33139 lines, +10.35%) — material change (>5%)

⚠️ 2 Reborn crate(s) have 0 int-tier coverage (target: 0) — ironclaw_prompt_envelope, ironclaw_scripts

Reborn integration-tier coverage

Line coverage (Reborn crates): 86.24% — 304712 / 353327 lines

Per-crate breakdown (62 crates, lowest-covered first)
Crate Line % Covered / Total
ironclaw_prompt_envelope 0% 0 / 88
ironclaw_scripts 0% 0 / 345
ironclaw_telegram_extension 34.88% 60 / 172
ironclaw_event_projections 43.31% 673 / 1554
ironclaw_dispatcher 60% 72 / 120
ironclaw_observability 61.54% 16 / 26
ironclaw_authorization 62.98% 609 / 967
ironclaw_memory 69.2% 773 / 1117
ironclaw_trust 72.88% 661 / 907
ironclaw_filesystem 73.5% 4543 / 6181
ironclaw_capabilities 74.07% 2717 / 3668
ironclaw_wasm_limiter 74.6% 47 / 63
ironclaw_extractors 74.72% 538 / 720
ironclaw_projects 76.48% 400 / 523
ironclaw_triggers 77.33% 2531 / 3273
ironclaw_mcp 77.56% 736 / 949
ironclaw_reborn_cli 78.25% 10379 / 13264
ironclaw_product_context 78.57% 11 / 14
ironclaw_llm 78.63% 20824 / 26485
ironclaw_wasm 79.72% 735 / 922
ironclaw_process_sandbox 80.46% 671 / 834
ironclaw_memory_native 81.17% 3195 / 3936
ironclaw_events 81.95% 1594 / 1945
ironclaw_first_party_extensions 82.58% 6608 / 8002
ironclaw_reborn_event_store 83.03% 1169 / 1408
ironclaw_telegram_v2_adapter 83.07% 2017 / 2428
ironclaw_product_adapter_registry 83.43% 574 / 688
ironclaw_reborn_identity 83.59% 433 / 518
ironclaw_processes 83.78% 940 / 1122
ironclaw_secrets 83.8% 2550 / 3043
ironclaw_reborn_config 84.17% 1962 / 2331
ironclaw_product_workflow 84.79% 13158 / 15518
ironclaw_auth 85.01% 4011 / 4718
ironclaw_common 85.18% 1741 / 2044
ironclaw_product_adapters 85.29% 3351 / 3929
ironclaw_run_state 85.61% 458 / 535
ironclaw_network 85.97% 913 / 1062
ironclaw_hooks 86.6% 9931 / 11468
ironclaw_extensions 87.02% 3795 / 4361
ironclaw_threads 87.17% 4849 / 5563
ironclaw_host_api 87.52% 5394 / 6163
ironclaw_skills 87.58% 4470 / 5104
ironclaw_reborn_traces 88.11% 11972 / 13587
ironclaw_reborn_composition 88.38% 57807 / 65404
ironclaw_turns 88.53% 14467 / 16341
ironclaw_host_runtime 88.64% 18227 / 20562
ironclaw_webui 88.95% 8064 / 9066
ironclaw_reborn_openai_compat 89.03% 3627 / 4074
ironclaw_extension_host 89.59% 2856 / 3188
ironclaw_slack_extension 89.7% 2439 / 2719
ironclaw_approvals 90.18% 1598 / 1772
ironclaw_conversations 90.39% 3123 / 3455
ironclaw_resources 90.85% 4477 / 4928
ironclaw_runner 91.18% 16908 / 18544
ironclaw_event_streams 91.24% 1063 / 1165
ironclaw_loop_host 92.22% 16133 / 17494
ironclaw_attachments 93.06% 630 / 677
ironclaw_agent_loop 94.81% 9467 / 9985
ironclaw_safety 95.15% 3749 / 3940
ironclaw_first_party_extension_ports 95.62% 3672 / 3840
ironclaw_outbound 95.77% 3513 / 3668
ironclaw_runtime_policy 96.55% 811 / 840

This table itself is informational and never gates the PR on its own — not the percentage, not the per-crate holes, not the 0-coverage callout. A separate coverage ratchet (dry-run until enforce=true; see tests/integration/coverage-floor.toml) can fail the build on specific configured floors.

Exemptions (3 entry/entries excluded from the accounting above)
Module / Crate Reason Issue
crate: ironclaw_embeddings v1-only: consumed only by root ironclaw (src/app.rs, src/tools/builtin/memory.rs, src/workspace/mod.rs, src/config/{mod,embeddings}.rs); no crates/* dependents. Covered by "Tests (Legacy)". #5657
crate: ironclaw_gateway v1-only: consumed only by root ironclaw (src/channels/web/platform/static_files.rs, src/channels/web/handlers/frontend.rs); no crates/* dependents. Covered by "Tests (Legacy)". #5657
crate: ironclaw_tui v1-only: consumed only by root ironclaw (src/main.rs, src/channels/tui.rs); no crates/* dependents. Crate's own doc comment confirms it bridges INTO v1, not Reborn. Covered by "Tests (Legacy)". #5657

@railway-app
railway-app Bot temporarily deployed to ironclaw-ci-preview / ironclaw-pr-6458 July 22, 2026 12:40 Destroyed
@railway-app

railway-app Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

🚅 Deployed to the ironclaw-pr-6458 environment in ironclaw-ci-preview

Service Status Web Updated (UTC)
ironclaw ✅ Success (View Logs) Web Aug 21, 2026 at 1:32 am

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/ironclaw_turns/tests/row_store_crash_consistency.rs`:
- Around line 2018-2023: Update the documentation comment above the
crash-consistency test to call the tests below “persistence-level tests,” since
they directly invoke the row store’s recover_expired_leases method; do not
describe them as caller-level tests.

In `@docs/reborn/contracts/turn-persistence.md`:
- Line 117: Reconcile the documentation for RecoveryRequired with the
active-lock rules by defining one consistent behavior for legacy rows, including
whether the lock is retained or released. Update the relevant statements around
the legacy terminal-status description and active-lock guidance, and cite the
actual transition or load-path symbol that implements this behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 8b9217fd-21a5-45d3-bc82-739455e2a4e3

📥 Commits

Reviewing files that changed from the base of the PR and between 293ade1 and 405b567.

📒 Files selected for processing (5)
  • crates/ironclaw_turns/tests/row_store_crash_consistency.rs
  • docs/reborn/contracts/turn-persistence.md
  • docs/reborn/contracts/turn-runner.md
  • docs/reborn/contracts/turns-agent-loop.md
  • docs/reborn/tier-b-self-repair-reconciliation.md

Comment thread crates/ironclaw_turns/tests/row_store_crash_consistency.rs
Comment thread docs/reborn/contracts/turn-persistence.md Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
docs/reborn/contracts/turn-persistence.md (1)

103-103: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add implementation and test references for the changed recovery rules.

These are load-bearing reservation and lease semantics, but unlike the active-lock section they do not cite the enforcing code or caller-level tests. Reference the reservation transition implementation and the concrete heartbeat/recover_expired_leases paths, plus tests covering requeue versus terminal outcomes, so this contract cannot drift from behavior.

Also applies to: 113-118

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/reborn/contracts/turn-persistence.md` at line 103, Update the
reservation and lease recovery rules in turn-persistence.md to include
references to the implementation enforcing reservation transitions, the concrete
heartbeat and recover_expired_leases paths, and caller-level tests covering
requeue versus terminal outcomes. Apply the same references to the related
statements at lines 113–118, without changing the documented semantics.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@docs/reborn/contracts/turn-persistence.md`:
- Line 103: Update the reservation and lease recovery rules in
turn-persistence.md to include references to the implementation enforcing
reservation transitions, the concrete heartbeat and recover_expired_leases
paths, and caller-level tests covering requeue versus terminal outcomes. Apply
the same references to the related statements at lines 113–118, without changing
the documented semantics.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 3fd277ea-7fc5-4245-acbc-165cb58b02b5

📥 Commits

Reviewing files that changed from the base of the PR and between 405b567 and acfb2ec.

📒 Files selected for processing (2)
  • crates/ironclaw_turns/tests/row_store_crash_consistency.rs
  • docs/reborn/contracts/turn-persistence.md

This branch was successfully deployed

1 active deployment
ironclaw-ci-preview / ironclaw-pr-6458 — acfb2ec9 Deployed Jul 22, 2026 by railway-app[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor: core 20+ merged PRs risk: low Changes to docs, tests, or low-risk modules scope: docs Documentation size: XS < 10 changed lines (excluding docs)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reconcile Tier B self-repair gap with existing lease recovery

1 participant