Skip to content

[Hermes Agent][hermes-pr35642-split-snowball][1/n] Split dflash canary hardening loop - #37160

Closed
OmarB97 wants to merge 7 commits into
NousResearch:mainfrom
OmarB97:codex/pr35642-dflash-canary-hardening-loop
Closed

[Hermes Agent][hermes-pr35642-split-snowball][1/n] Split dflash canary hardening loop#37160
OmarB97 wants to merge 7 commits into
NousResearch:mainfrom
OmarB97:codex/pr35642-dflash-canary-hardening-loop

Conversation

@OmarB97

@OmarB97 OmarB97 commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Split the dflash stability canary tooling out of fix(agent): allow bounded repeated stall retries #35642 into its own reviewable PR.
  • Add a bounded canary runner that drives representative hermes -z dflash workflows and writes JSONL evidence.
  • Add a supervised hardening loop that can file sanitized MeshBoard repair tasks when a canary case fails.
  • Install opt-in oneshot faulthandler diagnostics so timeout supervisors can request thread dumps before terminating the process.

Why

#35642 grew into a multi-feature recovery stack. This PR is the measurement/ownership slice only, so it can be reviewed independently from the behavior-changing stall-retry and local-backend failover changes.

Validation

  • scripts/run_tests.sh tests/scripts/test_dflash_stability_canary.py
  • python3 scripts/check-windows-footguns.py scripts/dflash_stability_canary.py scripts/dflash_hardening_loop.py hermes_cli/oneshot.py tests/scripts/test_dflash_stability_canary.py
  • python3 -m py_compile scripts/dflash_stability_canary.py scripts/dflash_hardening_loop.py hermes_cli/oneshot.py tests/scripts/test_dflash_stability_canary.py
  • git -c core.fsmonitor=false diff --check upstream/main..HEAD

Relationship To Existing PRs

Split from #35642. This PR intentionally does not include the stall-retry policy changes, local first-chunk failover, or local backend recovery hook.

@OmarB97
OmarB97 marked this pull request as ready for review June 2, 2026 03:26
@OmarB97 OmarB97 changed the title test(dflash): add stability canary hardening loop [Hermes Agent][hermes-pr35642-split-snowball][1/n] Split dflash canary hardening loop Jun 2, 2026
@alt-glitch alt-glitch added type/refactor Code restructuring, no behavior change comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have labels Jun 2, 2026
@OmarB97

OmarB97 commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

Closing: the canary/hardening loop shells out to MeshBoard tooling (.mesh/tools/meshctl.py and --meshboard-* flags) that doesn't exist for upstream users, so this belongs in our own tooling repo rather than upstream. The one upstream-generic piece — opt-in faulthandler/SIGUSR1 thread-dump diagnostics for oneshot — will be resubmitted standalone as a small feat(cli) PR.

@OmarB97 OmarB97 closed this Jun 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have type/refactor Code restructuring, no behavior change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants