fix(delegation): stop re-running schema DDL and add jittered retry to async_delegation writes

[pierrenode]

Summary

tools/async_delegation.py (backing delegate_task(background=True)'s durable completion persistence, added earlier today) opens a fresh ad hoc sqlite3 connection on every single call, against the SAME state.db file hermes_state.py's SessionDB owns — its own class docstring says "gateway + cron processes share one state.db" and documents that unnecessary write-lock hold time "starves competing writers." Two problems against that shared file:

1. Every call re-ran schema DDL. _connect() executed CREATE TABLE IF NOT EXISTS + PRAGMA table_info + up to 5 conditional ALTER TABLE statements on every single call (read or write), even though the schema never changes at runtime. Fixed by recording "schema already ensured" per resolved db path and skipping the DDL once it's already run for that path. (Keyed by path, not a single flag, since HERMES_HOME can change across profiles within one process — and does across test fixtures.)
2. No retry on lock contention. Write operations relied solely on the connection's own busy-timeout (a single attempt). SessionDB._execute_write (hermes_state.py) was deliberately built with a short per-connection timeout plus app-level jittered retry specifically because — per that method's own docstring — SQLite's built-in busy-timeout backoff is deterministic and convoy-prone under real concurrent load. async_delegation.py writes into the exact same contended file but had none of that mitigation. Added a local _execute_write() mirroring SessionDB's pattern (BEGIN IMMEDIATE + jittered retry on SQLITE_BUSY) and moved the 10 actual write functions onto it. The 2 read-only functions (restore_undelivered_completions, get_durable_delegation) are untouched — wrapping a SELECT in BEGIN IMMEDIATE would itself add unnecessary write-lock contention.

Also lowered _connect()'s own timeout from 10s to 1.0s (matching SessionDB's connection timeout): now that retries happen at the app level, a fresh connection per retry attempt with a 10s built-in timeout would multiply into a much longer worst-case stall than the retry loop (15 attempts × short jitter) is meant to bound.

Test plan

• Added test_connect_only_runs_schema_ddl_once_per_db_path — spies on CREATE TABLE/ALTER TABLE execution via a sqlite3.Connection subclass (the base type is immutable and can't be monkeypatched directly) and asserts the DDL runs on the first _connect() call for a path but not the second.
• Added test_connect_reruns_ddl_for_a_different_db_path — a different resolved path is correctly treated as a cache miss (guards against a no such table regression from over-aggressive caching).
• Added test_execute_write_retries_on_database_locked — a fake connection wrapper raises sqlite3.OperationalError("database is locked") on the first two BEGIN IMMEDIATE attempts, then succeeds; asserts _execute_write retries transparently rather than propagating the error. Deterministic (no real wall-clock lock contention), so it isolates the retry logic itself.
• Mutation-verify: reverted the fix locally, confirmed all 3 new tests fail against pre-fix code (AttributeError for the now-removed constants / DDL re-running), then restored the fix and confirmed all pass.
• uv run --frozen --extra dev python -m pytest tests/tools/test_async_delegation.py tests/gateway/test_async_delegation_session_binding.py — 35 passed, 1 pre-existing failure (test_crashed_runner_produces_error_completion) confirmed unrelated: reproduces identically against pre-fix code in the same run order (test-order-dependent flake, not a regression from this change).
• uv run --frozen --extra dev python -m pytest tests/cli/test_cli_background_status_indicator.py tests/tui_gateway/test_delegation_session_lifecycle.py tests/gateway/test_scale_to_zero_watcher.py — 48 passed (no regression in dependent consumers).
• ruff check on both changed files — clean.

Update (rebase onto current main)

Rebasing onto current main surfaced that this exact area had independently evolved upstream (connection-leak fix via _transaction(), an origin_session_id column, and _connect() already routing through apply_wal_with_fallback() instead of a bare PRAGMA). Reconciled by keeping all of upstream's additions and layering this PR's schema-caching + jittered-retry on top via a shared _initialize_schema(conn, path) used by both the pre-existing _transaction() call sites and this PR's _execute_write().

While here, also unified _connect()'s apply_wal_with_fallback() call to use the same db_label="state.db" SessionDB itself uses for this physical file — this folds in the fix from #64491 (opened separately before the rebase surfaced they'd collide), which is now redundant and can be closed. A distinct label would have double-logged the same NFS/SMB WAL-fallback warning for one physical file. Added #64491's own 2 regression tests (test_connect_routes_through_shared_wal_fallback_helper, test_connect_falls_back_gracefully_on_wal_incompatible_filesystem), mutation-verified.

[tonydwb]

Code Review Summary

Verdict: LGTM

Stopping redundant schema DDL re-runs and adding jittered backoff addresses real inefficiency in delegation hot paths. The fix shape (idempotent + retry-then-jitter) is standard.

Strengths

• Adds jittered backoff — clearly the right move to avoid lockstep retries from concurrent delimiters.
• Two-file change keeps the diff readable.
• Includes a negative test for the lock-retry path.

---

Reviewed by Hermes Agent in batch mode

[teknium1]

Thanks for targeting a real shared-state.db contention path. Current main confirms that tools/async_delegation.py:87-124 performs WAL setup and schema work on every connection, while durable writes such as tools/async_delegation.py:139-216 have no application-level retry.

Problems

• In c332073879616468dd2fef421bb5f9c350f8c744, the new _execute_write() calls _connect() before entering its try block. _connect() performs PRAGMA journal_mode=WAL and, on a schema-cache miss, DDL, so a SQLITE_BUSY from either operation still escapes rather than retrying. The new test only exercises a failure from BEGIN IMMEDIATE.
• The retry sleep remains inside _DB_LOCK. This blocks all same-process durable-delegation writes during jitter, unlike hermes_state.py:1250-1301, whose lock is released before sleeping.

Suggested changes

• Include _connect() in the retry-protected scope and close a successfully created connection conditionally in finally.
• Release _DB_LOCK before jittering, and add a deterministic _connect()-raises-locked-then-succeeds test.

Automated hermes-sweeper review.

[pierrenode]

Substantially rewritten against current `upstream/main`, not a mechanical rebase. Since this PR was opened, an unrelated commit (`#69594`, fixing a connection-leak/FD-exhaustion bug — #69567) refactored every write call site in this file from bare `with _DB_LOCK, _connect() as conn:` onto a new `_transaction()` context manager that guarantees the connection is always closed. That refactor did not touch the two problems this PR fixes — schema DDL still re-ran on every single `_connect()` call, and there was still no retry on `SQLITE_BUSY`/`database is locked` anywhere in the file — so both bugs were still fully live, just underneath a different call-site shape than this PR originally targeted.

Re-integration, not reapplication:

• `_execute_write()` now wraps `_transaction()` (`with _DB_LOCK, _transaction() as conn: conn.execute("BEGIN IMMEDIATE"); return fn(conn)`, retried) instead of duplicating its connect/commit/rollback/close logic the way the original diff did — so the connection-leak fix and this PR's retry/schema-cache fix compose cleanly rather than one undoing the other.
• All 13 `with _DB_LOCK, _transaction() as conn:` call sites (up from whatever count existed when this PR was first opened) now route through `_execute_write()`.
• `restore_undelivered_completions()` needed one additional care point I hadn't had to consider originally: it now collects events into a list and calls `target_queue.put()` only after `_execute_write` returns (transaction committed) instead of in-loop — a `queue.put()` isn't rolled back the way the DB writes around it are, so enqueueing before commit would let a retried attempt (after a busy error on a later statement in the same transaction) re-deliver the same completion twice.
• The regression tests' fake `_FlakyConnection` wrapper needed `enter`/`exit` added (delegating to the real connection) since `_transaction()`'s `with conn:` now requires the context-manager protocol on whatever `_connect()` returns — a bare method-forwarding fake was sufficient against the old code but not this one.

Targeted suite (25 tests, 5 new) passes, mutation-verified both fixes independently (removing the schema-cache check makes the DDL-runs-once test fail; removing the retry loop makes all 3 retry tests fail with the injected `OperationalError` propagating instead of being retried). Broader sweep across every test file that references `async_delegation` (133 tests: CLI delivery, gateway completion/relay routing, TUI session lifecycle, restored-delegation ownership, cronjob background, FD-leak) plus `api_server`/`journal_mode`/`process_registry` (221 more) all pass clean. Ruff clean. Fresh competitor search found no other PR touching this file's write path.