Multiplexed housekeeping: one profile's failure no longer skips the rest or leaks its home (salvage #110405)

[austinpickett]

One gap left in the multiplexed gateway's per-profile housekeeping after #117746 and #118006 moved every served profile's state.db sweep onto that loop.

What changed

• gateway/run_profile_reconcile.py::_for_each_served_profile gives each served profile its own try/except. Before this, one profile's unreadable store or broken .env ended the loop, and _housekeeping_chore only catches at the tick level. Every profile after the failing one lost its state.db archive/prune, curator, skill-sync and MCP-reconcile pass on every tick. This can happen in practice: _init_session_db tolerates a failed launch store and keeps running, and hermes serve hands every served profile's auto-archive to this loop (state.db residuals: serve stands down under a live gateway, recovery survives the first resume, import refuses a held deleted database #117746 C1). A satellite sitting behind a broken launch store had nothing sweeping it. Failures log at debug, like _housekeeping_chore.
• The companion _profile_runtime_scope unwinding fix (a failed secret hydration leaving the profile's home installed) landed on main independently as 6aaa1c4 while this was open, so it was dropped on rebase. The regression test for it is kept, since this loop makes that path reachable on every tick.

Tests

Two cases added to tests/gateway/test_housekeeping_profile_scope.py. Both use real stores and the real housekeeping loop:

• test_a_failing_profile_does_not_strand_the_profiles_after_it: the launch store raises in acquire(), and the served profile B must still be swept.
• test_profile_scope_setup_failure_restores_the_callers_home: secret hydration raises, and the caller's home and secret scope must be restored.

Red on origin/main before the scope fix landed, with both production files reverted:

E       AssertionError: assert [] == [PosixPath('/.../b/state.db')]
E       AssertionError: unpinned store escaped the sandbox: /.../test_profile_scope_setup_failu0/home/.hermes/profiles/b/state.db

The second line comes from a different test in the same worker. The leaked override carried profile B's home into it. With only the isolation fix applied, the scope test still fails with profiles/b == .hermes.

Check	Result
tests/gateway/test_housekeeping_profile_scope.py	7 passed
Every tests/gateway + tests/tui_gateway file that references _profile_runtime_scope / _for_each_served_profile / housekeeping (66 files)	745 passed, 1 skipped
ruff, check-windows-footguns.py, check_no_tmp_literals.py, check_compat_pointers.py, git diff --check	clean

Not changed

• The fail-closed ownership gate from fix(dashboard): fail closed on unknown gateway ownership, isolate per-profile maintenance #110405 is not carried: stand down when gateway liveness is unknown, the strict PID-identity probe, and the GATEWAY_HEALTH_URL rung. state.db residuals: serve stands down under a live gateway, recovery survives the first resume, import refuses a held deleted database #117746 already makes serve stand down when _check_gateway_running says a gateway owns the store. Under one-backend-per-host, serve is a legitimate SessionDB writer anyway, since Desktop chat turns write through it. The close hazard that made a stray writable open dangerous is handled by the WAL lock guard (state.db WAL survives sibling closes and stray lock cancellation on every runtime (Desktop chat dies after update; salvage #109758) #110544 / state.db WAL lock guard follows the handle lifecycle: reopen, checkpoint-vs-close, fd reuse (review of #110544) #110872), not by this gate. Merged onto current main, the fix(dashboard): fail closed on unknown gateway ownership, isolate per-profile maintenance #110405 head also fails two existing main tests (test_web_server_auto_archive_gateway_lock.py, test_web_server_auto_archive_profile_config.py).
• No per-chore try/except. The boundary sits in the shared loop, so every profile-scoped chore gets it.

Refs #100896

Supersedes #110405. Thanks @isair: the broken-launch-store isolation case comes from your review rounds on that PR (you're co-author on the commit). Your scope-unwinding fix is also on main now, via 6aaa1c4.

Rebased onto main 2026-09-23: test_housekeeping_profile_scope.py 7/7 pass; the isolation test fails on main's run_profile_reconcile.py and passes with the fix.

[github-actions]

૮ >ﻌ< ა ci review

_{ran on 8511560 — fix(gateway): one served profile's failing chore no longer s}

debug info

CI timings

CI timings · View report · View job

Wall time 5m54s vs 6m42s (-11.9%). 4 job(s) slower, 8 faster,

• Python lints / Windows footguns (blocking): -108.0s
• OS-specific tests / Windows-only tests: -48.0s
• Python tests / Run tests: -46.0s
• Python tests / e2e: +25.0s
• Python lints / ruff enforcement (blocking): -17.0s