fix(cron): enforce one scheduler owner per home

[opita04]

Summary

Hermes Desktop and Gateway could both schedule the same stateful cron job for one HERMES_HOME. Whichever process won a tick could advance the job before discovering that its process lacked the configured delivery credentials, so generated output disappeared from the intended platform.

This change makes scheduler ownership fail closed and exact-home scoped while preserving standalone Desktop compatibility, multiplex Gateway profiles, hosted callbacks, and external providers.

Decision	Behavior
Missing owner	auto: Gateway is preferred when present; Desktop-only installs continue scheduling.
Explicit owner	gateway and desktop are strict; malformed or unavailable configurations do not silently fall back.
Ownership boundary	One interprocess lease per canonical HERMES_HOME; provider routing and module caches use that same boundary.
Shutdown/handoff	Admission closes first; provider borrowers, lifecycle threads, and running jobs drain before resources are stopped or the lease is released.
External providers	Resolution is strict and case-preserving; reconciliation failures remain dirty and retryable.
Hosted callbacks	Job ID resolves to exactly one profile before token/config/provider lookup. Unknown, ambiguous, or unavailable ownership fails closed.

The runtime also avoids Windows-dangerous process probes, detects PID reuse through process identity, keeps housekeeping independent from scheduler suppression, and reports configured policy, active provider, and lease state through the CLI.

Compatibility and safety

• Gateway, Desktop, hosted dashboard, and multiplex profile supervisors all use the same ownership runtime.
• Gateway Chronos callbacks route by exact job ID to one owning profile; duplicate IDs are rejected instead of selecting the first profile.
• Hosted scale-to-zero callbacks authenticate and execute within the owning profile’s exact HERMES_HOME.
• Built-in scheduling cannot bypass ownership through an external-provider fallback.
• Startup rollback and shutdown retain the lease until published borrowers and accepted callback work drain.
• A cancelled async caller cannot release provider admission while its worker thread is still executing.
• Chronos authoritative-state, provision, cancellation, and signature-read failures propagate or retry rather than becoming false success.
• Gateway liveness checks reuse platform-safe process utilities instead of os.kill(pid, 0).

Validation

Tests ran with isolated HOME/HERMES_HOME; the real cron registry was SHA-256 checked before and after each authoritative run.

• 1235 passed across cron, Chronos, Gateway callback/ownership/drain, dashboard cron, config, and cron-tool suites.
• 519 passed in the complete tests/hermes_cli/test_web_server.py suite.
• Focused teardown, exact-profile callback, ambiguity, cancellation, Windows probe, transient signature, and one-worker executor regressions passed.
• Ruff passed for every changed Python file.
• All six new Python files are Ruff-format clean.
• Python compilation and git diff --check passed.
• Production ~/.hermes/cron/jobs.json remained unchanged during each guarded run.

One pre-existing mocked multi-target delivery test still emits an un-awaited coroutine warning; the affected suites otherwise pass.

New concepts

Admission barriers for resource-safe shutdown

Closing a scheduler’s public registry prevents new work, but already admitted callbacks may still hold and use its provider. The runtime therefore treats shutdown as a barrier:

flowchart LR
  A[Close admission] --> B[Unpublish provider]
  B --> C[Signal lifecycle stop]
  C --> D[Drain borrowers and jobs]
  D --> E[Stop provider resources]
  E --> F[Release exact-home lease]

Loading

This is preferable to stopping the provider first because callbacks cannot safely finish against closed network clients or schedulers. It is not needed for immutable, stateless resources with no concurrent borrowers.

Exact-home ownership

HERMES_HOME is the scheduler’s tenancy boundary: policy, jobs, credentials, provider instance, callback authentication, and lease ownership all resolve from the same canonical home. That lets multiplex profiles share a process without sharing scheduler authority. A process-global singleton is simpler, but is unsafe whenever multiple profiles or homes coexist.

[alt-glitch]

This was generated by AI during triage.

Related: #70286 and #44049 address the same scheduler-owner problem with different mechanisms. This patch adds a canonical-home kernel lease and provider lifecycle; maintainer consolidation is needed rather than treating the patches as duplicates.

[teknium1]

Thanks for carrying the ownership problem through Gateway, Desktop, callbacks, and profile routing. The underlying bug is still present on current main: hermes_cli/web_server.py:150-168 starts a Desktop scheduler, while gateway/run.py:25491-25542 independently starts the Gateway scheduler; cron/scheduler.py:4099-4145 only serializes ticks.

Problems

• cron/scheduler_runtime.py:44 adds an independent behavioral config parser. Current main commit ed33ebca1d60ae5069f871e4e5f13d97df384e4c established canonical config loaders/shared raw-read primitives specifically to avoid managed-scope, ${VAR}, and profile-path drift. Salvage this reader onto that current mechanism.
• The user-facing cron guidance remains stale: website/docs/guides/cron-troubleshooting.md:39-41 says automatic cron requires a Gateway, but this change introduces Desktop ownership and cron.scheduler_owner without a docs update.

Suggested changes

• Rework the exact-home policy read using the current shared config-read path before integration.
• Document owner modes, Desktop-only behavior, and the new status semantics.

Automated hermes-sweeper review.

[teknium1]

Current main's ed33ebca1d60ae5069f871e4e5f13d97df384e4c centralized behavioral config reads to prevent managed-overlay, ${VAR}, and profile-home drift. Please salvage this onto read_user_config_raw() plus the established exact-profile overlay/expansion path rather than adding a new parser.