Skip to content

fix(coverage): declare RUST_MIN_STACK on the push-to-main coverage lane - #6660

Merged
BenKurrek merged 2 commits into
mainfrom
fix/coverage-lane-stack-headroom
Jul 24, 2026
Merged

BenKurrek merged 2 commits into
mainfrom
fix/coverage-lane-stack-headroom

Conversation

@BenKurrek

@BenKurrek BenKurrek commented Jul 24, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Code Coverage has been red on every push to main for 20+ consecutive commits, aborting with has overflowed its stack / fatal runtime error: stack overflow (SIGABRT, exit 101) — on a different test each time, as unrelated PRs shifted which future sat deepest:

commit test that overflowed suite
096e8f843 unbound_telegram_actor_pairs_via_web_minted_code_… extension_delivery
8f4d832f1 duplicate_and_restart_replay_converge_exactly_once::case_1 extension_ingress
d06bde940 extension_install_survives_independent_reopen durable

Root cause

A workflow gap, not test depth. libtest gives each test thread a 2 MiB stack.

reborn-tests.yml splits this package's suites across two jobs and gives each the headroom it needs:

coverage.yml runs cargo llvm-cov --workspace — both tiers in one job — and declared neither. This sets it to the union's requirement, 64 MiB.

This also explains why the earlier per-test fixes did not converge: the depth lives in shared harness code (group build → submit_turn → composition), so #6609's Box::pin lowered one test below the ceiling and the next-deepest test simply became the new failure. Boxing individual futures can't fix a lane-wide ceiling.

Verification

Local reproduction, same command and instrumentation, only the env differs:

# without (2 MiB default)
$ cargo llvm-cov --no-report -p ironclaw_reborn_integration_tests --test reborn_integration_durable
thread 'extension_install_survives_independent_reopen' has overflowed its stack
fatal runtime error: stack overflow, aborting
(signal: 6, SIGABRT) ... exit status: 101

# with the value in this PR
$ RUST_MIN_STACK=67108864 cargo llvm-cov --no-report -p ... --test reborn_integration_durable
test extension_install_survives_independent_reopen ... ok
test result: ok. 14 passed; 0 failed

The same contrast is visible in CI at d06bde940: Reborn integration coverage (1) ran reborn_integration_durable instrumented with RUST_MIN_STACK=8388608 and passed, while Coverage (all-features) / Coverage (default) ran the same suite under the same instrumentation with no setting and SIGABRT'd.

Regression coverage

tests/reborn_coverage_lane_stack_headroom.rs pins the invariant across both workflows, sized per tier (whole-workspace lanes need 64 MiB; integration-tier-only lanes need 8 MiB).

  • Red before this change: coverage.yml:coverage runs this package's test targets under llvm-cov but declares no job-level RUST_MIN_STACK; green after.
  • Mutation-tested three ways — a below-floor value, a whole-workspace lane set to the integration tier's 8 MiB, and dropping reborn-tests.yml's own value — each fails the guard. (The second mutation is not hypothetical: it caught an 8 MiB value in an earlier draft of this very PR.)
  • Non-vacuity assertions keep a renamed job or a reworded run: line from silently emptying the scan and leaving the check trivially true.

Reviewer notes

  • coverage.yml triggers only on push: branches: [main], so this PR's own CI cannot exercise the fixed lane. It is covered by the guard test plus the evidence above; the lane itself is proven by the next push to main.

  • On the 64 MiB choice — one honest caveat. I verified the integration-tier need directly (repro above). I did not reproduce the QA-tier need: on macOS/aarch64, reborn_qa_smoke_scenarios_e2e passed instrumented at 8 MiB (26 passed). The 64 MiB therefore rests on the repo's own documented measurement (tests/reborn_qa_smoke_scenarios_e2e.rs header: ~10 MiB uninstrumented, i.e. already over 8) and on matching root-reborn-parity-tests, whose scope this lane subsumes — not on a measurement of mine on CI's platform. Since RUST_MIN_STACK reserves virtual address space per thread rather than committing pages, sizing to the sibling lane is the cheap and safe direction. If a reviewer prefers the tighter 8 MiB, the guard's WHOLE_WORKSPACE_BYTES constant is the single place to change.

  • The e2e-coverage job is deliberately untouched: it runs pytest against a built binary rather than libtest test threads, and has not been failing. Its llvm-cov calls (show-env, clean, report) spawn no test threads and are excluded from the guard by name.

  • Per-crate lanes (llvm-cov -p <pkg>) are also out of scope — they never reach this package's tests/.

  • Second commit (fix(ci): route the coverage-headroom guard …): the guard's first filename put it outside classify-test-scope.sh's tests/reborn_* routing, so the first push classified as has_reborn_tests=false and skipped every Reborn test lane — the guard was dead weight on exactly the PR shape it polices. Renamed to the convention every other root suite uses; verified the classifier now reports has_reborn_tests=true for this PR's real file set.

🤖 Generated with Claude Code

`Code Coverage` has been red on every push to main for 20+ consecutive
commits, aborting with `has overflowed its stack` / `fatal runtime error:
stack overflow` (SIGABRT, exit 101) — on a *different* test each time as
unrelated PRs shifted which future sat deepest:

  096e8f8  unbound_telegram_actor_pairs_via_web_minted_code_… (extension_delivery)
  8f4d832  duplicate_and_restart_replay_converge_exactly_once::case_1 (extension_ingress)
  d06bde9  extension_install_survives_independent_reopen (durable)

Root cause is a workflow gap, not test depth. libtest gives each test
thread a 2 MiB stack. `reborn-tests.yml` splits this package's suites
across two jobs and gives each the headroom it needs —
`reborn-integration-coverage` carries 8 MiB (llvm-cov inflates the
integration harness's async frames; #6609) and `root-reborn-parity-tests`
carries 64 MiB (reborn_qa_smoke_scenarios_e2e drives whole turns on the
libtest stack, ~10 MiB uninstrumented). `coverage.yml` runs
`cargo llvm-cov --workspace`, i.e. BOTH tiers in one job, and declared
neither. Set it to the union's requirement, 64 MiB.

This also explains why the per-test fixes did not converge: the depth
lives in shared harness code (group build -> submit_turn -> composition),
so #6609's `Box::pin` lowered one test below the ceiling and the
next-deepest test simply became the new failure.

The controlled comparison at d06bde9: `Reborn integration coverage (1)`
ran reborn_integration_durable instrumented with RUST_MIN_STACK=8388608
and passed, while `Coverage (all-features)`/`Coverage (default)` ran the
same suite under the same instrumentation with no setting and SIGABRT'd.
Same code, same instrumentation — only the stack size differed.

Regression coverage: tests/coverage_lane_stack_headroom.rs pins the
invariant on both workflows, sized per tier (whole-workspace lanes need
64 MiB; integration-tier-only lanes need 8 MiB). Verified red before this
change (`coverage.yml:coverage … declares no job-level RUST_MIN_STACK`)
and green after. Mutation-tested three ways: a below-floor value, a
whole-workspace lane set to the integration tier's 8 MiB, and dropping
reborn-tests.yml's own value each fail the guard. Non-vacuity assertions
keep a renamed job or reworded `run:` line from silently emptying the scan.

Note: coverage.yml triggers only on `push: branches: [main]`, so this
PR's own CI cannot exercise the fixed lane — it is validated by the
guard test plus the CI evidence above, and proven by the next push to main.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ironloopai

ironloopai Bot commented Jul 24, 2026 •

Copy link
Copy Markdown
Contributor

🔎 IronLoop Review Status

Head: 5e69f90061537bd894519dbc528838b871a9eb60
Result: One or more review results were superseded by a newer PR head.
Next: Run @ironloopai review on the latest PR head.
Updated: 2026-07-24T18:09:53.067Z

Current reviewers:

Reviewer State Verdict Findings Last update
ironloop/common-reviewer (reviewer) Superseded N/A N/A 2026-07-24T18:09:53.056Z
Reviewer summaries
Reviewer Detail
ironloop/common-reviewer (reviewer) Superseded by a newer PR head. New head: 5e69f90.
Recent activity
Time Reviewer State Detail
2026-07-24T18:08:24.339Z ironloop/common-reviewer (reviewer) Queued Accepted review request for head 3556424.
2026-07-24T18:08:24.339Z ironloop/common-reviewer (reviewer) Queued Waiting for this reviewer lane to become available.
2026-07-24T18:08:24.542Z ironloop/common-reviewer (reviewer) Started Reviewer worker started.
2026-07-24T18:08:27.417Z ironloop/common-reviewer (reviewer) Workspace ready Prepared isolated checkout (merge_ref) at fe968ff.
2026-07-24T18:09:53.056Z ironloop/common-reviewer (reviewer) Superseded A newer PR head replaced this review (5e69f90).
Available commands
  • @ironloopai help
  • @ironloopai agents
  • @ironloopai review
  • @ironloopai review --agent <agent>
Run metadata

Admission: webhook accepted the request and IronLoop persisted reviewer state before this projection.

@railway-app
railway-app Bot temporarily deployed to ironclaw-ci-preview / ironclaw-pr-6660 July 24, 2026 18:08 Destroyed
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@github-actions github-actions Bot added scope: ci CI/CD workflows size: S 10-49 changed lines labels Jul 24, 2026
@coderabbitai

coderabbitai Bot commented Jul 24, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Coverage CI now sets a 64 MiB Rust thread stack, and a new test validates stack thresholds for instrumented coverage jobs in both workflow files.

Changes

Coverage headroom enforcement

Layer / File(s) Summary
Coverage workflow stack configuration
.github/workflows/coverage.yml
The coverage job adds job-level RUST_MIN_STACK: "67108864" configuration and explanatory comments.
Workflow headroom validation
tests/reborn_coverage_lane_stack_headroom.rs
A test parses coverage workflows, classifies instrumented lanes, validates tier-specific stack floors, and checks that expected lanes are found.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • nearai/ironclaw#5430: Adds the Reborn integration-tier coverage workflow targeted by this stack-headroom validation.

Suggested reviewers: think-in-universe

🚥 Pre-merge checks | ✅ 3 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The PR text is informative, but it doesn't follow the repo's required template and omits sections like Change Type, Linked Issue, and Rollback Plan. Rewrite the PR description to match the template and fill in all required sections, especially Summary, Change Type, Linked Issue, Validation, and Rollback Plan.
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title uses conventional commit style and accurately summarizes the coverage-stack fix.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added risk: medium Business logic, config, or moderate-risk modules contributor: core 20+ merged PRs labels Jul 24, 2026
…lanes

The guard added in the previous commit never ran: `root-reborn-parity-tests`
gates on `has_reborn_tests`, and `classify-test-scope.sh`'s
`is_reborn_test_path` matches root suites by the `tests/reborn_*` prefix.
`tests/coverage_lane_stack_headroom.rs` did not match, so a PR touching only
it and a workflow classified as `has_reborn_tests=false` and skipped every
Reborn test lane — the guard was dead weight on exactly the PR shape it
exists to police (a workflow edit).

Caught on PR CI for this branch: `Reborn root tests` reported `skipping`.

Rename to `tests/reborn_coverage_lane_stack_headroom.rs`, matching the
convention every other root suite already uses. Verified with the real
staged file set: the classifier now reports `has_reborn_tests=true`, and the
suite passes under its new target name.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@railway-app
railway-app Bot temporarily deployed to ironclaw-ci-preview / ironclaw-pr-6660 July 24, 2026 18:09 Destroyed

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/coverage_lane_stack_headroom.rs`:
- Around line 1-3: Update the module documentation and related guard wording
around WORKFLOWS to state that validation covers only the explicitly enumerated
workflow files and their jobs, rather than every CI job. Do not claim universal
coverage unless the implementation is changed to discover all workflow files;
preserve the existing job classification behavior, including its independence
from Job::name.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 2526a42f-4227-4012-9a3e-3953f15895c4

📥 Commits

Reviewing files that changed from the base of the PR and between 5efb0e2 and 3556424.

📒 Files selected for processing (2)
  • .github/workflows/coverage.yml
  • tests/coverage_lane_stack_headroom.rs

Comment on lines +1 to +3
//! Guards the coverage-lane stack-overflow class: every CI job that runs this
//! package's test targets under `cargo llvm-cov` must declare `RUST_MIN_STACK`
//! headroom at the job level, sized to the tier it actually executes.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Bound the guard’s documented coverage.

The test scans only the two fixed paths in WORKFLOWS, not every CI job; also, renaming a job does not empty this scan because classification ignores Job::name. State that it covers enumerated workflows, or discover all workflow files before retaining the universal guarantee.

As per coding guidelines, “Comments promising cross-layer guarantees must be enforced by code or tests, or softened to describe intent.”

Also applies to: 234-236

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/coverage_lane_stack_headroom.rs` around lines 1 - 3, Update the module
documentation and related guard wording around WORKFLOWS to state that
validation covers only the explicitly enumerated workflow files and their jobs,
rather than every CI job. Do not claim universal coverage unless the
implementation is changed to discover all workflow files; preserve the existing
job classification behavior, including its independence from Job::name.

Source: Coding guidelines

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/reborn_coverage_lane_stack_headroom.rs`:
- Around line 168-191: Update job_level_rust_min_stack to distinguish a declared
but malformed RUST_MIN_STACK from an absent variable instead of converting parse
failures to None with .ok(). Return or propagate an explicit parsing error for
malformed values, and update the caller and violation message to report that
error while preserving the existing absent-variable behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 196afb6d-bfb9-4aab-94ba-e3e697b4ae60

📥 Commits

Reviewing files that changed from the base of the PR and between 3556424 and 5e69f90.

📒 Files selected for processing (2)
  • .github/workflows/coverage.yml
  • tests/reborn_coverage_lane_stack_headroom.rs

Comment thread tests/reborn_coverage_lane_stack_headroom.rs
@BenKurrek
BenKurrek merged commit 3714b66 into main Jul 24, 2026
57 of 58 checks passed
@BenKurrek
BenKurrek deleted the fix/coverage-lane-stack-headroom branch July 24, 2026 18:20
@github-actions

Copy link
Copy Markdown
Contributor

Coverage ratchet

Ratchet mode: ENFORCING

RATCHET PASS: global
  observed: 86.23% (314392 / 364600 lines)
  floor:    86.27% (tolerance 0.5pp -> effective floor 85.77%)
  denominator: 364600 lines now vs 354049 at floor capture (+10551 lines, +2.98%) — not a material change

⚠️ 2 Reborn crate(s) have 0 int-tier coverage (target: 0) — ironclaw_prompt_envelope, ironclaw_scripts

Reborn integration-tier coverage

Line coverage (Reborn crates): 86.23% — 314392 / 364600 lines

Per-crate breakdown (61 crates, lowest-covered first)
Crate Line % Covered / Total
ironclaw_prompt_envelope 0% 0 / 88
ironclaw_scripts 0% 0 / 345
ironclaw_host_ingress 42.5% 17 / 40
ironclaw_event_projections 43.71% 684 / 1565
ironclaw_observability 61.54% 16 / 26
ironclaw_telegram_v2_adapter 62.35% 631 / 1012
ironclaw_authorization 62.98% 609 / 967
ironclaw_dispatcher 64.75% 79 / 122
ironclaw_memory 70.15% 919 / 1310
ironclaw_trust 73.21% 664 / 907
ironclaw_filesystem 73.7% 4587 / 6224
ironclaw_wasm_limiter 74.6% 47 / 63
ironclaw_extractors 74.72% 538 / 720
ironclaw_capabilities 75.36% 2593 / 3441
ironclaw_mcp 76.2% 775 / 1017
ironclaw_projects 76.48% 400 / 523
ironclaw_reborn_cli 77.97% 10549 / 13529
ironclaw_telegram_extension 78.59% 962 / 1224
ironclaw_llm 78.73% 20989 / 26659
ironclaw_wasm 79.72% 735 / 922
ironclaw_process_sandbox 80.46% 671 / 834
ironclaw_memory_native 80.97% 3114 / 3846
ironclaw_first_party_extensions 82.26% 6624 / 8053
ironclaw_events 82.47% 1604 / 1945
ironclaw_operator 83.2% 5625 / 6761
ironclaw_processes 83.3% 933 / 1120
ironclaw_host_api 83.5% 8942 / 10709
ironclaw_secrets 83.79% 2548 / 3041
ironclaw_reborn_identity 83.8% 450 / 537
ironclaw_reborn_config 85.24% 2102 / 2466
ironclaw_auth 85.31% 4026 / 4719
ironclaw_common 85.53% 2252 / 2633
ironclaw_run_state 85.77% 458 / 534
ironclaw_triggers 85.92% 2783 / 3239
ironclaw_network 85.97% 913 / 1062
ironclaw_reborn_event_store 86.51% 1251 / 1446
ironclaw_hooks 86.58% 9930 / 11469
ironclaw_product 86.81% 21683 / 24977
ironclaw_extensions 87.2% 3678 / 4218
ironclaw_threads 87.2% 4851 / 5563
ironclaw_skills 87.77% 4480 / 5104
ironclaw_turns 88.06% 14312 / 16253
ironclaw_reborn_traces 88.13% 11986 / 13600
ironclaw_reborn_composition 88.36% 49089 / 55557
ironclaw_slack_extension 88.47% 1934 / 2186
ironclaw_host_runtime 88.57% 18997 / 21448
ironclaw_extension_host 89.12% 3745 / 4202
ironclaw_reborn_openai_compat 89.32% 3780 / 4232
ironclaw_resources 90.84% 4474 / 4925
ironclaw_runner 90.87% 17192 / 18920
ironclaw_webui 91.02% 9035 / 9926
ironclaw_conversations 91.1% 3202 / 3515
ironclaw_approvals 91.12% 1682 / 1846
ironclaw_event_streams 91.24% 1063 / 1165
ironclaw_loop_host 91.9% 16503 / 17958
ironclaw_attachments 93.06% 630 / 677
ironclaw_outbound 94.43% 3987 / 4222
ironclaw_agent_loop 94.88% 9837 / 10368
ironclaw_safety 95.15% 3749 / 3940
ironclaw_first_party_extension_ports 95.62% 3672 / 3840
ironclaw_runtime_policy 96.55% 811 / 840

This table itself is informational and never gates the PR on its own — not the percentage, not the per-crate holes, not the 0-coverage callout. A separate coverage ratchet (dry-run until enforce=true; see tests/integration/coverage-floor.toml) can fail the build on specific configured floors.

Exemptions (3 entry/entries excluded from the accounting above)
Module / Crate Reason Issue
crate: ironclaw_embeddings v1-only: consumed only by root ironclaw (src/app.rs, src/tools/builtin/memory.rs, src/workspace/mod.rs, src/config/{mod,embeddings}.rs); no crates/* dependents. Covered by "Tests (Legacy)". #5657
crate: ironclaw_gateway v1-only: consumed only by root ironclaw (src/channels/web/platform/static_files.rs, src/channels/web/handlers/frontend.rs); no crates/* dependents. Covered by "Tests (Legacy)". #5657
crate: ironclaw_tui v1-only: consumed only by root ironclaw (src/main.rs, src/channels/tui.rs); no crates/* dependents. Crate's own doc comment confirms it bridges INTO v1, not Reborn. Covered by "Tests (Legacy)". #5657

This branch was successfully deployed

No deployments
ironclaw-ci-preview / ironclaw-pr-6660 — 5e69f900 Deployed Jul 24, 2026 by railway-app[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor: core 20+ merged PRs risk: medium Business logic, config, or moderate-risk modules scope: ci CI/CD workflows size: S 10-49 changed lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant