fix(cron): harden degraded jobs saves against conflicts

[poisdahl]

Summary

Experimental follow-up to #81000. That merged change fixes the filed #80624 concurrent-create loss by recovering unexpected on-disk job IDs. This branch explores a stronger conflict-aware jobs.json contract:

• bind each in-lock load_jobs() result to its loaded base
• reconcile base/desired/current snapshots for disjoint creates, edits, and deletes
• use a short .jobs.commit.lock around final reconciliation and atomic publication when supported
• bound generation retries instead of doing an unchecked terminal write
• keep the new lock file out of backup/import

Why this is draft

The implementation is correctness-reviewed and test-green, but it is substantially larger than the demonstrated production incident: roughly 407 net production lines plus 407 test lines. #81000 already fixed the reported concurrent-create failure. The remaining motivation is a stronger same-ID/degraded-writer storage guarantee, for which there is not currently a linked Hermes production report.

This should not merge as a routine small bugfix. Keep it draft until maintainers explicitly want the stronger storage-semantics contract and agree that its complexity is justified. If that contract is not desired, this PR should be closed rather than partially trimmed into a lock-only or generation-only patch that would not deliver the claimed invariant.

Availability policy

The broad .jobs.lock deliberately degrades to process-local protection when its backend is unavailable or a sibling is wedged (#60703/#60855). The short publication lock follows the same policy: missing backends, unsupported lock operations, lock-file preparation failures, or timeout log a warning and continue with the process lock, three-way reconciliation, and bounded generation checks.

When the short lock is available it closes the final generation-check-to-replace window for cooperating Hermes writers. In degraded mode that small final window remains. This preserves Hermes' established scheduler-availability tradeoff.

Deliberate boundaries

• this is best-effort conflict hardening, not a due-job dispatch CAS
• legacy/manual writers that ignore Hermes locks remain outside the cooperating-writer guarantee
• no hardlink/inode/path-swap framework is introduced
• do not represent this as fixing no_agent script cron jobs disappear from jobs.json while gateway is running (CLI-created jobs clobbered by scheduler state) #80624; that issue is already fixed by fix(cron): preserve concurrent creates when saving jobs.json (salvage of #80687 + #80703) #81000

Validation

Refreshed conflict-free against main@715d26cdf on 2026-08-13.

• full tests/cron plus backup coverage: 582 passed, 0 failed
• focused conflict, availability-fallback, Cron ticker silently stalls after gateway restart #60703 liveness, ownership, and native-OS lock tests pass
• Ruff and git diff --check: clean
• correctness review: PASS
• merge-readiness/minimality review: keep draft or drop unless upstream chooses the stronger storage contract

[poisdahl]

Refreshed against main@222465d84. The cron/jobs.py conflict was resolved by retaining this PR’s bounded, fail-closed publication retry while preserving current main’s JSON serialization change. Current validation: the full tests/cron directory plus backup regressions passed (585 passed, 0 failed, 1 Windows-only skip); Ruff and git diff --check pass.

[poisdahl]

Independent and adversarial audit found a real degraded-publication gap: if the short commit lock was missing or unsupported, two writers could both pass the generation check and race the final replace (including replace=True). The new head now fails closed before publication in that state. Regressions cover no backend plus ENOSYS/ENOTSUP/EOPNOTSUPP, both replace modes, unchanged bytes, and no temp publication. Final validation: 593 passed, 1 Windows-only skip; final reviews found no remaining P0–P3 issue.