fix(cron): isolate multiplex tick failures per profile

[OYLFLMH]

Problem

Under multiplex_profiles, InProcessCronScheduler._start_multiplex() ticks every served profile's cron store (#69377). Two per-profile blocks in that function are try/finally with no except, so one profile's failure escapes into the others.

1. The tick loop

The per-profile loop sits inside a single try/except BaseException for the whole cycle, while each profile's own block is try/finally with no except:

try:
    for entry in profile_homes:
        home_token = set_hermes_home_override(str(home))
        try:
            with use_cron_store(home):
                cron_tick(...)
        finally:                    # no except here
            reset_hermes_home_override(home_token)
    ok = True
except BaseException as e:          # one profile raising unwinds the whole loop
    ...

cron.scheduler.tick() has no catch-all of its own — its body in cron/scheduler.py is try/finally — so anything raised inside it propagates to this caller. Two consequences:

1. Profiles ordered after the raising one are skipped for that cycle. A transient error self-heals on the next cycle; a persistent per-profile failure does not. The jobs.json ownership class of problem fixed on the owner side in 722bf5d51 is exactly such a failure, and it starves every profile after it indefinitely — with no signal that they were never reached.

2. The heartbeat pass writes one cycle-wide ok into every profile's store. hermes cron status then reports profiles as failing that ticked cleanly, and record_ticker_error() — added so the reason is visible (cron: CLI run as root silently locks out the uid-1000 ticker (jobs.json rewritten root:600); zombie ticker never surfaced #68483) — files the unrelated profile's error message against them.

2. The startup recovery pass

The same gap exists in the pass that runs before the loop is entered:

for entry in profile_homes:
    home_token = set_hermes_home_override(str(home))
    try:
        with use_cron_store(home):
            recovered = self.recover_interrupted()
            ...
            record_ticker_heartbeat()
    finally:                        # no except here either
        reset_hermes_home_override(home_token)

recover_interrupted() lands in recover_interrupted_executions(), which opens a transaction against that profile's own ledger; neither it nor _transaction() has a catch-all, so an unreadable or locked store raises straight out of start().

The blast radius here is wider than in the tick loop, precisely because this pass runs before the loop. The gateway runs start() as a bare thread target (gateway/run.py::start_gateway) with nothing above it but threading.excepthook, which logs the traceback and neither restarts the thread nor stops the process. One bad ledger therefore kills the ticker before a single cycle runs, and every profile stops firing — including the ones whose recovery succeeded, and including the failing profile's own jobs, which may well still be tickable — until the gateway is restarted. Nothing in hermes cron status attributes the silence to the profile that caused it.

Solution

cron/scheduler_provider.py:

• each profile's tick gets its own try/except BaseException. BaseException for the same reason the single-profile path catches it ([Bug]: Cron ticker dies silently — no error log, no watchdog, misleading status #32612): a SystemExit out of a misbehaving provider SDK must not kill the ticker thread.
• each cycle collects one (profile, ok, error) outcome, and the heartbeat pass records each profile's own result rather than a shared flag.
• each profile's startup recovery gets the same handler: log, record the reason via record_ticker_error() into that profile's own store, continue to the next profile. That error-recording is itself guarded — the store is exactly what may be unreachable, and letting the report of a failure raise would reintroduce the failure it reports.
• a profile whose recovery failed stays in profile_homes. A ledger that cannot be recovered may still hold tickable jobs, and if it cannot be ticked either, the per-profile tick handler records that on its own terms.

Unchanged on purpose:

• the single-profile path (profile_homes unset) is untouched, including its recovery call. There is no other profile to starve there and the process owns exactly one store, so failing loudly stays the right contract;
• a paused can_dispatch cycle keeps its current meaning — nothing attempted, recorded as clean — matching the single-profile path.

Tests

Three regression tests in tests/cron/test_scheduler_provider.py:

• test_multiplex_tick_error_does_not_stop_other_profiles — three profiles, the first raises every cycle; the other two must still be ticked.
• test_multiplex_heartbeat_success_is_per_profile — a profile that ticked cleanly must not be recorded as failing because a different profile raised.
• test_multiplex_startup_recovery_error_does_not_stop_other_profiles — three profiles, the first raises sqlite3.OperationalError from recovery; all three must still be ticked.

All three fail on current main and pass with this change. Verified by reverting only the source file and keeping the tests:

# tests only, cron/scheduler_provider.py reverted to main
3 failed, 20 passed

# with this change
23 passed

On the unfixed code the recovery test fails the way the analysis predicts rather than by a bare assertion: the ticker thread dies on the unhandled sqlite3.OperationalError — pytest reports it as an unhandled thread exception — and no profile is ticked at all.

Run locally on Python 3.13.14. The 20 pre-existing tests in that file pass unchanged in both runs.

Context

This is the one gap that survived from #54913, which I closed as obsolete after #69377 and a61183b56 landed the same cross-profile design.

The startup-recovery half was raised in review on this PR and is fixed in 011180e7b.

Note on scope: I do not run a multiplex production deployment. The failure modes are derived from reading _start_multiplex(), tick(), recover_interrupted_executions() and the gateway's thread setup rather than from an observed incident, and are validated by the three tests above. Feedback from multiplex users is welcome.

[teknium1]

Thanks for tracing this through the current multiplex path. The steady-state fix matches the defect on current main: cron_tick() is inside one cycle-wide handler at cron/scheduler_provider.py:321-340, while cron.scheduler.tick() can propagate exceptions from its main try/finally body (cron/scheduler.py:4140). The per-profile outcomes and heartbeat assertions are a sound direction.

Problems

• The same profile-isolation guarantee is not applied during startup. cron/scheduler_provider.py:298-313 invokes recover_interrupted() for each profile with only finally; recover_interrupted_executions() opens a transaction at cron/executions.py:204. An exception for an earlier profile still exits the scheduler before later profiles reach their first tick.

Suggested changes

• Catch and log the per-profile recovery failure, continue to the next profile, and add a regression test proving a later profile is ticked when the first profile's recovery raises.

Automated hermes-sweeper review.

[OYLFLMH]

Confirmed and fixed in 011180e7b.

The path you pointed at holds. The recovery pass at cron/scheduler_provider.py:298-313 is try/finally with no except, and recover_interrupted() lands in recover_interrupted_executions(), whose _transaction() (cron/executions.py:66) is also try/finally — nothing between the ledger and start() catches.

One correction to the impact, in the direction of worse. The consequence is not that later profiles miss their first tick; it is that no profile ticks at all. The recovery pass runs before the while loop is entered, and the gateway runs start() as a bare thread target (gateway/run.py::start_gateway) with nothing above it but threading.excepthook, which logs the traceback and neither restarts the thread nor stops the process. So one unreadable ledger kills the ticker before a single cycle runs, and every profile stops firing — including the ones whose recovery already succeeded — until the gateway is restarted. The new test observes exactly that on the unfixed code: the thread dies on the sqlite3.OperationalError, pytest reports an unhandled thread exception, and nothing is ticked.

Changes:

• the per-profile recovery block gets its own try/except BaseException, logs, records the reason via record_ticker_error() into that profile's own store, and continues to the next profile. The error-recording is itself guarded — the store is exactly what may be unreachable, so letting the report of a failure raise would reintroduce the failure it reports.
• the failing profile stays in profile_homes. A ledger that cannot be recovered may still hold tickable jobs, and if it cannot be ticked either, the per-profile tick handler already records that on its own terms.
• test_multiplex_startup_recovery_error_does_not_stop_other_profiles — three profiles, the first raises sqlite3.OperationalError from recovery, all three must still be ticked. 23 passed with the change; reverting only cron/scheduler_provider.py to main and keeping the tests gives 3 failed, 20 passed.
• the docstring's isolation guarantee now covers startup as well as steady state. It previously read "a tick that raises", which was accurate for what the code delivered but narrower than the sentence it opened with.

Left alone on purpose: the single-profile path (profile_homes unset) still lets a recovery failure propagate. There is no other profile to starve there and the process owns exactly one store, so failing loudly stays the right contract.

PR body updated to cover both halves.

[teknium1]

Thanks for this PR. Merged via #101245 (eed481b) on current main — multiplex cron ticks isolate per profile, fire the bound port, deliver satellites fail-closed.

#101245 won as the consolidated fix because it covers the whole multiplex-profile bug class in one change (with tests) rather than the single symptom addressed here; this PR is superseded by it.
You are credited via Co-authored-by / in the PR body of #101245 as noted there.

If anything from your original change is still missing on main >= eed481b, please open a fresh PR/issue against main and tag it. Thanks again.