feat: move the durable job substrate to PostgreSQL (#6809) by mikemcdougall · Pull Request #7245 · honua-io/honua-server

mikemcdougall · 2026-11-10T08:46:18Z

Pull Request

Issue Link

Refs #6809 (this is packages P1 and P2 for the four job-side interfaces; the rest of the Redis removal is on the follow-on branch feat/postgres-control-plane-state)

Stage B and C verification (head `6e362c99ee` or later)

All runs through the build-slot shim, Release, PostgreSQL through Testcontainers. One command at a time.

What	Result
`dotnet build src/Honua.Server/Honua.Server.csproj /p:TreatWarningsAsErrors=true`, same for `src/Honua.Worker.Gdal`	`Build succeeded. 0 Warning(s) 0 Error(s)`
Server tests: the `Postgres*` store classes, `JobReconciliationServiceTests`, `JobExecutionServiceTests`, `ExecutionJobReconcilerTests`, `ExecutionWorkerProfileTests`, `DurableJobSubstrateRegistrationTests`, `ControlPlaneTriggerTests`, `ControlPlaneLambdaEventTargetsTests`, `GpDeploymentHandoffTests`, `CustomCodeArtifactIngestorTests`, `CapabilityManifestEndpointTests`, `GeoprocessingOutputStagingOptionsTests`	first run `Failed: 4, Passed: 367, Skipped: 2, Total: 373` (two test-side assertions, fixed); rerun of the affected classes `Passed: 16, Failed: 0`
Server tests: import durable runner (with and without Redis), tile export recovery, geometry buffer, data enrichment, GP row and field security, vector parity, admission multi-node, tile admission lease, Redis durability attestation	`Failed: 0, Passed: 58, Total: 58`
No Redis, protocol level: `GPServerDurableRuntimeTests`, `GPServerDefaultServiceTests`, `RoutingAsyncWebToolTests`	`Failed: 2, Passed: 81, Total: 83`; the two failures were `RoutingAsyncWebToolTests` hosts that failed to initialize (the fixture's dispose hid the cause); rerun of those two theories `Passed: 6, Failed: 0`. Cause of the first-run failures not identified.
No Redis, protocol level: `OgcProcessesDurableRuntimeTests`, `OgcProcessesCiteEchoFixtureTests`	`Failed: 0, Passed: 23, Total: 23`
GDAL worker: `GdalWorkerJobSubstrateRegistrationTests`, `GdalWorkerLicensingTests`, `OgcProcessesExecutionProofTests`	`Failed: 1, Passed: 10, Total: 11`. The failure, `Execute_RasterSurfaceProcess_CompletesAndReturnsAnalyticalSlope`, is the build host failing to pull `ghcr.io/osgeo/gdal:ubuntu-full-3.13.1`; the job was claimed, failed in `gdaldem`, and retried on the PostgreSQL store. Not verified here.
Outage grace, with and without: `WorkersHadTimeToHeartbeatSinceLastStoreInterruption` forced true	without the grace period: 1 of 2 runs failed, 1 passed; with it: passed in every run (3). The test does not fail every time without the grace period.
`RedisDurabilityInformationalTests` (Core)	`Passed: 15`
`PostgresDataSourceFactoryTests`	`Passed: 52`
`Honua.Architecture.SourceTests`	`Passed: 117`
`Honua.Architecture.Tests`, filter `DatabaseMigrationSafety`	`Passed: 45`
`Honua.Architecture.Tests`, all	`Failed: 2, Passed: 295, Total: 297`; both failures are the GP operation matrix digests in the open items
`dotnet format --verify-no-changes` on `Honua.Jobs`, `Honua.Worker.Gdal`, `Honua.TestKit`	exit 0

How the no-Redis tests guarantee it: NoRedisJobHost (one per protocol test project) builds the host with WithProductionJobSubstrate() and UseSetting("ConnectionStrings:redis", ""), registers nothing job-related itself, and asserts after start that the redis connection string and the Aspire Redis key are empty, that no IConnectionMultiplexer is registered, that IExecutionJobStore is PostgresExecutionJobStore, IJobQueue is PostgresJobQueue, and a hosted JobExecutionService exists. The import journey is ImportEndpointDurableRunnerWithoutRedisTests.UploadImport_ForcedBackgroundOnHostWithoutRedis_SucceedsOnDurableRunnerAndRowsAreReadable. The three per-node fallbacks are exercised in GPServerDurableRuntimeTests.NodeLocalCoordination.cs.

GP operation matrix re-audit (eng/gp-operation-matrix.v1.json)

The matrix has no regeneration script: each evidence receipt pins the SHA-256 of the test method body (for a forwarding test, the expression plus the helper it calls), and audit.catalogSourceContentDigest pins the catalog sources. The digests were recomputed with the test's own algorithm; the recomputation reproduces the 137 unchanged recorded digests and the pre-change digests of the five changed ones.

Row	Evidence test	What changed in the body	Re-run on the PostgreSQL store
`geometry.buffer`	`GeometryBufferJobExecutorIntegrationTests.BufferProcess_CompletesAndReturnsBufferedGeometry`	Redis key cleanup became `ClearJobSubstrateAsync`; the host is built with `WithProductionJobSubstrate()`	passed locally
`geometry.buffer`	`OgcProcessesExecutionProofTests.Execute_FirstSliceVectorProcess_CompletesAndReturnsBufferGeometry`, `...Execute_FirstSliceProcessWithValueOutputSelection_CompletesAndReturnsInlineBuffer`	`fixture.InitializeAsync()` became `StartAsync(fixture)` in the shared `ProveBuffer` helper	passed locally
`surface.slope`	`OgcProcessesExecutionProofTests.Execute_RasterSurfaceProcess_CompletesAndReturnsAnalyticalSlope`	`fixture.InitializeAsync()` became `StartAsync(fixture)`	not re-run: the build host cannot pull the GDAL image; audited by diff only
`routing.closest-facilities`, `routing.od-cost-matrix`, `routing.location-allocation`	`RoutingAsyncWebToolTests.SubmitJob_StockFeatureSet_CompletesWithNamedResultsAndInputIdentity`	`fixture.InitializeAsync()` became `NoRedisJobHost.StartAsync(fixture)`	passed locally

No assertion changed in any of them. RoutingPgRoutingQueuedJobTests (the pgRouting evidence for the routing rows) is opt-in and its three evidence test bodies are unchanged. audit.catalogSourceContentDigest changed because GeoprocessingServiceCollectionExtensions.cs now registers the PostgreSQL job store and result-package store; the executor registrations in that file are untouched.

Checks that cannot pass while this is a draft

Review Gate and Publish SHA-bound review attestation fail with pull request is draft (.github/workflows/review-gate.yml). After the draft flag is removed they also need an attesting reviewer's review of the head commit and no unresolved reviewer thread. Marking the PR ready is the founder's call.

Open items

Not run: scripts/ci/pre-pr-check.sh (its dry run selects FULL mode, a solution-wide build and the whole suite, because Honua.Architecture.Tests changed); the full Honua.Server.Tests suite; LayerPublishingIntegrationTests (remote publish), RoutingPgRoutingQueuedJobTests (opt-in), NativeWorkerPublicApiTests and ProductionWorkerContainerHandoffTests (need the worker image); the retention sweeper hosted service in a running host; the longer heartbeat and TOAST measurement.
Event trigger mode on a control-plane Lambda handler still needs Redis to reconcile execution jobs (ControlPlaneLambdaEventTargets.EnsureDurableControlPlane, ControlPlaneEventHandler).
Test hosts built by WebAppFixture have no job store unless they call WithProductionJobSubstrate(). A test that set a Redis connection string and relied on the job store that came with it now has none.
Jobs:RequireDurableStore=true still refuses to start without an attested Redis on the serving host (DurableJobSubstrateStartupGate); left for feat: PostgreSQL shared state and Redis removal (#6809) #7250.

Summary

Moves the durable job system (execution jobs, the job queue, job logs, geoprocessing result packages) from Redis to PostgreSQL. Each PostgreSQL store replaces its Redis store outright; there is no selection setting and no dual-store period.

Landing. This PR stays a draft. It enters rc.2 only through the release session's batch process and on the founder's go. The branch is never half-swapped: Stage A adds code without wiring it in, Stage B is one coherent swap.

Stage status

Stage	Content	State
Design	ADR-0081 (`docs/internal/contributor/adr/0081-postgres-job-substrate.md`), reproduced below	pushed, with design review changes, the pipelines fit section and the rollout rule
A	Migration 147, the four PostgreSQL stores, converted tests against PostgreSQL; nothing wired in	pushed and run; see the open items below
B	Registrations swapped at every site, gating and manifest reworked, Redis job classes and their tests deleted, protocol-level durable-runtime tests green with no Redis configured	pushed and run; see the verification table and open items below. The server has no Redis job store.
C	Retention sweeper with bounded deletes, result-package expiry test, docs, ADR supersession notes	pushed and run (sweeper tests pass). Longer heartbeat and TOAST measurement not done.

What Stage A changes at run time. Nothing is registered, so the Redis stores are still the ones in use. Three consumers gained optional constructor arguments that default to today's behaviour (IJobStoreClock, IReconcilableJobSource): JobReconciliationService, ExecutionJobReconciler and its background service, AssignedExecutionJobQueue. Two behaviours are new even on Redis: a heartbeat write that fails is retried after 2 s instead of at the next interval, and the reconciliation sweep does not expire heartbeats for one heartbeat timeout after a sweep that failed or stalled. ExecutionJobRecord and ExecutionJobSpec now derive from ExtensibleJsonRecord, so unknown JSON members survive a rewrite.

Verification (PostgreSQL 18 through Testcontainers, image postgis/postgis:18-3.6)

Command for every test line: dotnet test tests/dotnet/Honua.Server.Tests/Honua.Server.Tests.csproj --configuration Release --filter <the Postgres* classes in Features/Infrastructure/ControlPlane>, through the build-slot shim. All results below are for commit 8f4b9aad59. The head commit 971d66ffba (outage grace on stalled sweeps, unknown record fields, TOAST autovacuum thresholds) has not been built or run: the build slots were held by other lanes.

What	Result
Store, queue, contention, log, restart and outage classes, first run	`Total tests: 63, Passed: 63`
Skewed node clock, +60 and -60 minutes	`Passed! - Failed: 0, Passed: 2, Skipped: 0, Total: 2`
Stale-attempt fencing with a paused and killed second process, backoff, cancellation during backoff	`Total tests: 3, Passed: 3`
Everything together, second run	`Failed! - Failed: 1, Passed: 67, Skipped: 2, Total: 70` (skipped: the two evidence tests)
Migration safety (`Honua.Architecture.Tests`, filter `DatabaseMigrationSafety`)	`Passed! - Failed: 0, Passed: 45`

The one failure is JobReconciliation_AfterAnOutageLongerThanTheHeartbeatTimeout_DoesNotExpireALiveWorkersJob, which passed in the first run. The sweep blocked through the outage inside the store's connect retries instead of failing, so the grace period never started. The fix is in 971d66ffba and is unverified. The check that this test fails with the grace period removed has not been done.

Two-worker contention (TryClaimAsync_EightWorkersRaceOneJob_ExactlyOneWinsWithOneAttempt, TryClaimAsync_WorkersDrainContestedQueue_EveryJobIsClaimedExactlyOnce), kill-mid-job recovery (WorkerLoss_NaturalHeartbeatLapse_..., StaleAttempt_IsFencedAcrossReclaimDuplicateDeliveryAndTerminalization) and the PostgreSQL restart during a running job (JobExecutionService_WhenPostgresRestartsMidJob_JobCompletesExactlyOnce: one execution, attempt 1) passed in every run.

Heartbeat load (PostgresJobStoreEvidenceTests, HONUA_JOB_STORE_EVIDENCE=1): 200 running jobs for 3 minutes, heartbeat every 30 s, with the 5 s candidate query and the 30 s sweep running, before the TOAST thresholds were added.

Measure	Value
Heartbeat writes	1,200
Average stored record	5,584 bytes
Row versions written	1,200 heartbeats (the counter read 1,600 because it also caught the 400 setup writes)
Heap-only updates	0
Heap	73,728 to 81,920 bytes (+8 kB)
TOAST	2,269,184 to 3,473,408 bytes (+1.2 MB)
Indexes	442,368 to 491,520 bytes (+49 kB)
Autovacuum runs on the table	3

Every heartbeat rewrites the out-of-line record, so TOAST grew about 0.4 MB per minute. Three minutes does not show whether it levels off; a longer run with the TOAST autovacuum thresholds is still to do. If it does not level off, heartbeats move to a statement that does not carry the record.

EXPLAIN (ANALYZE, BUFFERS) on 200,000 seeded rows, 11,400 queued

seeded rows by status: Cancelled=2000 Failed=6000 Provisioning=200 Queued=11400 Running=400 Succeeded=180000
== claim, one runtime profile (rolled back)
Update on execution_jobs job (actual rows=1.00 loops=1)
  Buffers: shared hit=47 read=6 dirtied=4
  CTE next_job
    ->  Limit (actual rows=1.00 loops=1)
          Buffers: shared hit=6 read=2 dirtied=1
          ->  LockRows (actual rows=1.00 loops=1)
                Buffers: shared hit=6 read=2 dirtied=1
                ->  Index Scan using idx_execution_jobs_claimable on execution_jobs (actual rows=1.00 loops=1)
                      Index Cond: ((runtime_profile = 'native'::text) AND (queue_visible_at <= now()))
                      Filter: ((kind = ANY ('{Geoprocessing,FileImport}'::text[])) AND (status = 'Queued'::text))
                      Rows Removed by Filter: 1
                      Index Searches: 1
                      Buffers: shared hit=5 read=2
  ->  Nested Loop (actual rows=1.00 loops=1)
        Buffers: shared hit=10 read=2 dirtied=1
        ->  CTE Scan on next_job (actual rows=1.00 loops=1)
              Storage: Memory  Maximum Storage: 17kB
              Buffers: shared hit=6 read=2 dirtied=1
        ->  Index Scan using execution_jobs_pkey on execution_jobs job (actual rows=1.00 loops=1)
              Index Cond: (operation_id = next_job.operation_id)
              Index Searches: 1
              Buffers: shared hit=4
  Buffers: shared hit=154 read=4
Execution Time: 0.542 ms
== list, no filter, second page
Limit (actual rows=51.00 loops=1)
  Buffers: shared hit=2 read=5
  ->  Index Scan using idx_execution_jobs_created on execution_jobs (actual rows=51.00 loops=1)
        Index Cond: (ROW(created_at, operation_id) < ROW((now() - '01:00:00'::interval), 'job-0003600'::text))
        Index Searches: 1
        Buffers: shared hit=2 read=5
  Buffers: shared hit=27
Execution Time: 0.051 ms
== list by status Failed
Limit (actual rows=51.00 loops=1)
  Buffers: shared hit=9 read=62
  ->  Index Scan using idx_execution_jobs_created on execution_jobs (actual rows=51.00 loops=1)
        Filter: (status = ANY ('{Failed}'::text[]))
        Rows Removed by Filter: 1029
        Index Searches: 1
        Buffers: shared hit=9 read=62
  Buffers: shared hit=16
Execution Time: 0.398 ms
== list by kind
Limit (actual rows=51.00 loops=1)
  Buffers: shared hit=12
  ->  Index Scan using idx_execution_jobs_created on execution_jobs (actual rows=51.00 loops=1)
        Filter: (kind = 'FileImport'::text)
        Rows Removed by Filter: 100
        Index Searches: 1
        Buffers: shared hit=12
Execution Time: 0.051 ms
== list by requester
Limit (actual rows=51.00 loops=1)
  Buffers: shared hit=342 read=63 written=63
  ->  Sort (actual rows=51.00 loops=1)
        Sort Key: created_at DESC, operation_id COLLATE "C" DESC
        Sort Method: top-N heapsort  Memory: 38kB
        Buffers: shared hit=342 read=63 written=63
        ->  Bitmap Heap Scan on execution_jobs (actual rows=400.00 loops=1)
              Recheck Cond: (lookup_keys @> '{requested-by:USER-42}'::text[])
              Heap Blocks: exact=400
              Buffers: shared hit=342 read=63 written=63
              ->  Bitmap Index Scan on idx_execution_jobs_lookup_keys (actual rows=400.00 loops=1)
                    Index Cond: (lookup_keys @> '{requested-by:USER-42}'::text[])
                    Index Searches: 1
                    Buffers: shared hit=5
  Buffers: shared hit=14 read=1 written=1
Execution Time: 1.747 ms
== list by backend (not selective)
Limit (actual rows=51.00 loops=1)
  Buffers: shared hit=6
  ->  Index Scan using idx_execution_jobs_created on execution_jobs (actual rows=51.00 loops=1)
        Filter: (lookup_keys @> '{backend:LOCAL}'::text[])
        Rows Removed by Filter: 1
        Index Searches: 1
        Buffers: shared hit=6
  Buffers: shared hit=15
Execution Time: 0.031 ms
== list by correlation id
Limit (actual rows=1.00 loops=1)
  Buffers: shared hit=6
  ->  Sort (actual rows=1.00 loops=1)
        Sort Key: created_at DESC, operation_id COLLATE "C" DESC
        Sort Method: quicksort  Memory: 25kB
        Buffers: shared hit=6
        ->  Bitmap Heap Scan on execution_jobs (actual rows=1.00 loops=1)
              Recheck Cond: (lookup_keys @> '{correlation:CORR-777}'::text[])
              Heap Blocks: exact=1
              Buffers: shared hit=6
              ->  Bitmap Index Scan on idx_execution_jobs_lookup_keys (actual rows=1.00 loops=1)
                    Index Cond: (lookup_keys @> '{correlation:CORR-777}'::text[])
                    Index Searches: 1
                    Buffers: shared hit=5
  Buffers: shared hit=1
Execution Time: 0.045 ms
== all jobs of one orchestration run
Limit (actual rows=50.00 loops=1)
  Buffers: shared hit=47 read=8 written=8
  ->  Sort (actual rows=50.00 loops=1)
        Sort Key: created_at DESC, operation_id COLLATE "C" DESC
        Sort Method: quicksort  Memory: 37kB
        Buffers: shared hit=47 read=8 written=8
        ->  Bitmap Heap Scan on execution_jobs (actual rows=50.00 loops=1)
              Recheck Cond: (lookup_keys @> '{run:WF-77}'::text[])
              Heap Blocks: exact=50
              Buffers: shared hit=47 read=8 written=8
              ->  Bitmap Index Scan on idx_execution_jobs_lookup_keys (actual rows=50.00 loops=1)
                    Index Cond: (lookup_keys @> '{run:WF-77}'::text[])
                    Index Searches: 1
                    Buffers: shared hit=5
  Buffers: shared hit=1
Execution Time: 0.467 ms
== active jobs
Limit (actual rows=100.00 loops=1)
  Buffers: shared hit=859 read=141 written=141
  ->  Sort (actual rows=100.00 loops=1)
        Sort Key: updated_at DESC
        Sort Method: top-N heapsort  Memory: 50kB
        Buffers: shared hit=859 read=141 written=141
        ->  Bitmap Heap Scan on execution_jobs (actual rows=12000.00 loops=1)
              Recheck Cond: (status = ANY ('{Queued,Provisioning,Running}'::text[]))
              Heap Blocks: exact=867
              Buffers: shared hit=859 read=141 written=141
              ->  Bitmap Index Scan on idx_execution_jobs_status_created (actual rows=12001.00 loops=1)
                    Index Cond: (status = ANY ('{Queued,Provisioning,Running}'::text[]))
                    Index Searches: 1
                    Buffers: shared hit=95 read=38 written=38
  Buffers: shared hit=3
Execution Time: 5.345 ms
== 30 s sweep candidates
Bitmap Heap Scan on execution_jobs (actual rows=0.00 loops=1)
  Recheck Cond: ((status = ANY ('{Provisioning,Running}'::text[])) OR ((backend = 'local'::text) AND (status = 'Queued'::text) AND (queue_visible_at IS NULL)))
  Filter: (((status = ANY ('{Provisioning,Running}'::text[])) AND ((COALESCE(last_heartbeat_at, claimed_at) <= (now() - heartbeat_timeout)) OR (claimed_at <= (now() - max_duration)))) OR ((status = 'Queued'::text) AND (queue_visible_at IS NULL) AND (backend = 'local'::text)))
  Rows Removed by Filter: 600
  Heap Blocks: exact=223
  Buffers: shared hit=225
  ->  BitmapOr (actual rows=0.00 loops=1)
        Buffers: shared hit=2
        ->  Bitmap Index Scan on idx_execution_jobs_reconcilable (actual rows=601.00 loops=1)
              Index Cond: (status = ANY ('{Provisioning,Running}'::text[]))
              Index Searches: 1
              Buffers: shared hit=1
        ->  Bitmap Index Scan on idx_execution_jobs_undispatched (actual rows=0.00 loops=1)
              Index Cond: (backend = 'local'::text)
              Index Searches: 1
              Buffers: shared hit=1
  Buffers: shared hit=13
Execution Time: 0.415 ms
== 5 s backend reconciler candidates
Index Scan using idx_execution_jobs_reconcilable on execution_jobs (actual rows=800.00 loops=1)
  Index Searches: 1
  Buffers: shared hit=626
  Buffers: shared hit=8
Execution Time: 0.449 ms
== queue depth
Aggregate (actual rows=1.00 loops=1)
  Buffers: shared hit=6473 read=31 written=31
  ->  Index Only Scan using idx_execution_jobs_claimable on execution_jobs (actual rows=11200.00 loops=1)
        Heap Fetches: 11200
        Index Searches: 1
        Buffers: shared hit=6473 read=31 written=31
  Buffers: shared hit=3
Execution Time: 3.910 ms
== retention sweep batch (rolled back)
Delete on execution_jobs job (actual rows=0.00 loops=1)
  Buffers: shared hit=6052 read=8 dirtied=58 written=8
  CTE expired
    ->  Limit (actual rows=1000.00 loops=1)
          Buffers: shared hit=1056 read=4 dirtied=58 written=4
          ->  LockRows (actual rows=1000.00 loops=1)
                Buffers: shared hit=1056 read=4 dirtied=58 written=4
                ->  Index Scan using idx_execution_jobs_expired on execution_jobs (actual rows=1000.00 loops=1)
                      Index Cond: (expires_at <= (now() + '8 days'::interval))
                      Filter: (status = ANY ('{Succeeded,Failed,Cancelled}'::text[]))
                      Index Searches: 1
                      Buffers: shared hit=56 read=4 written=4
  ->  Nested Loop (actual rows=1000.00 loops=1)
        Buffers: shared hit=5052 read=8 dirtied=58 written=8
        ->  CTE Scan on expired (actual rows=1000.00 loops=1)
              Storage: Memory  Maximum Storage: 71kB
              Buffers: shared hit=1056 read=4 dirtied=58 written=4
        ->  Index Scan using execution_jobs_pkey on execution_jobs job (actual rows=1.00 loops=1000)
              Index Cond: (operation_id = expired.operation_id)
              Index Searches: 1000
              Buffers: shared hit=3996 read=4 written=4
  Buffers: shared hit=10
Execution Time: 2.448 ms
[testcontainers.org 00:03:21.51] Delete Docker container dfdee75347c6
[xUnit.net 00:03:24.63]   Finished:    Honua.Server.Tests

Converted tests: what was kept and what changed

Every behavioural assertion of the six Redis-bound test classes is kept in a Postgres* class in tests/dotnet/Honua.Server.Tests/Features/Infrastructure/ControlPlane/, with these changes:

Assertions on Redis keys (sorted-set membership, key TTLs) became assertions on the same fact through the store or the table (queue membership, log expiry).
Jobs that are enqueued use backend local; the Redis tests used integration-test or integration, which the queue's backend constraint now rejects.
Tests that aged a heartbeat by writing an old timestamp through TrySetAsync now age it with SQL, because a compare-and-set stamps database time. Tests that aged it through SetAsync or TryCreateAsync are unchanged. One assertion changed for the same reason: "heartbeat equals the value the test wrote" became "heartbeat advanced and is close to database time".
The mock-based RedisJobQueueTests (which asserted Redis calls) became integration tests of the same behaviours: delayed and other-kind jobs do not block a ready job, the runtime-profile fence (six cases), a locked candidate is skipped, a job that became terminal is not claimed.
Dropped as Redis-specific, with what replaces them: the malformed queue member test (replaced by "enqueue of an unknown id does nothing"); the legacy oversized log entry placeholder (replaced by the table's 4 KiB check constraint); the stale-claim repair atomicity tests (replaced by the dispatch repair and return-to-queue tests); "every secondary index expires" (retention is one column; the sweeper is Stage C).
The stale-attempt fencing test still runs a second process and pauses it with SIGSTOP. Its Redis scratch keys became a test_signals table.
The Redis-bound originals are still in the tree and still run against Redis until Stage B deletes them with the Redis classes.

Open items in Stage A

971d66ffba is unbuilt and unrun (see above).
The outage-longer-than-timeout guarantee (design review finding 4) is not met until that commit is verified.
Not run: Honua.Architecture.SourceTests, dotnet format --verify-no-changes, scripts/ci/pre-pr-check.sh, and the existing reconciler and worker unit tests against the consumer changes.
Not written: a result-package expiry test; contract runs against the in-memory doubles.

Release note text

Durable jobs now live in PostgreSQL. Geoprocessing, OGC API Processes and background jobs no longer need Redis or the caching.redis entitlement. Jobs in progress are not carried over from earlier pre-release builds; start from a clean deployment. The GDAL worker now needs ConnectionStrings:DefaultConnection (the system database) instead of ConnectionStrings:redis.

Where the implementation differs from the design review as written

5 s reconciler candidates. The reconciler also acts on local Queued jobs that have a cancellation request (it cancels them) or an earlier attempt (it resets their progress), so those stay in. Only first-attempt local jobs waiting in the queue are excluded.
Per-job lease for local Queued rows. Not changed separately: the rows in question no longer reach the reconciler.
Stuck Queued repair. Every local kind is enqueued after 60 s rather than failed or opted in per kind.
One retry time. next_retry_at and queue_visible_at both remain, written from one expression; different readers compare them.
Autovacuum thresholds are kept on the job table and its TOAST table; the fill factor and TOAST storage settings are gone.
Single-job worker pool cap of 2. Not adopted; the default cap is 8 until a test shows 2 holds (Stage B).
Clock and candidate queries are two small interfaces in Honua.Jobs that the store implements, not default methods on IExecutionJobStore: NSubstitute returns a zero date and an empty list for an unconfigured default interface method, which would have silently disabled expiry in about eight existing test files.

Follow-ups, not in this PR

A database role for job hosts (GDAL and single-job workers) limited to the job tables.
A switch to hold claiming and reconciliation after a point-in-time restore.
Deployment changes outside this repository: the Helm chart and Terraform must give the GDAL worker the database connection.
PostgresOperateFixtureExecutionStore (a fixture-only second job store with its own tables) will never be selected once the real store is always registered.

ADR-0081: PostgreSQL is the durable job substrate

Status

Proposed (2026-11-09). Supersedes the Redis-as-system-of-record statements in
ADR-0025 ("Redis and Durability") and
ADR-0031 ("Redis remains the coordination layer for
job state") for execution jobs, the job queue, job logs and geoprocessing result packages.
Workflow operations, proposals, operation instances and operation secrets stay on Redis until their
own change. Decision record: honua-server#6809 (founder decision and amendment, 2026-11-09).

Context

Execution jobs, their queue, their logs and geoprocessing result packages were Redis strings, sorted
sets, lists and Lua scripts. A deployment without Redis could not run geoprocessing, OGC API
Processes or background imports, and losing Redis lost every job. PostgreSQL is already required,
already holds the catalog, and already runs a FOR UPDATE SKIP LOCKED outbox (migration 024).

The four interfaces do not change shape: IExecutionJobStore, IJobQueue (with
IQueueClaimReconciler), IExecutionLogStore, IGeoprocessingResultPackageStore. The guarantees do
not change: at-least-once execution with fencing by version compare-and-set, lease-owned updates, the
per-attempt job token and CommittedEffectReferences.

Decision

One table holds a job and its queue position. The queue is a query over that table. Each PostgreSQL
store replaces its Redis store outright; there is no selection setting and no dual-store period.

The stores are PostgresExecutionJobStore, PostgresJobQueue, PostgresExecutionLogStore
(src/Honua.Jobs, raw Npgsql on an NpgsqlDataSource) and PostgresGeoprocessingResultPackageStore
(src/Honua.Geoprocessing). They live beside the consumers because Honua.Worker.Gdal references
Honua.Jobs and must not reference Honua.Server (ADR-0038), and the module policy forbids a
Honua.Postgres to Honua.Jobs reference in either direction.

SQL below is written with the default system schema honua; the migration uses $HonuaSchema$ and
the stores quote the configured Database:Schema. The migration only creates tables and indexes.

1. Tables

CREATE TABLE honua.execution_jobs (
    operation_id      TEXT COLLATE "C" PRIMARY KEY CHECK (length(operation_id) BETWEEN 1 AND 512),
    -- attempt state: authoritative, stamped by the database, overlaid on record when read
    version           BIGINT NOT NULL CHECK (version > 0),
    status            TEXT NOT NULL CHECK (status IN
                          ('Queued', 'Provisioning', 'Running', 'Succeeded', 'Failed', 'Cancelled')),
    attempt_count     INTEGER NOT NULL CHECK (attempt_count >= 0),
    claimed_by        TEXT,
    claimed_at        TIMESTAMPTZ,
    last_heartbeat_at TIMESTAMPTZ,
    next_retry_at     TIMESTAMPTZ,
    completed_at      TIMESTAMPTZ,
    percent_complete  DOUBLE PRECISION,
    current_phase     TEXT,
    updated_at        TIMESTAMPTZ NOT NULL,
    -- local queue position: authoritative; NULL means "not in the local queue"
    queue_priority    SMALLINT,
    queue_visible_at  TIMESTAMPTZ,
    -- retention
    expires_at        TIMESTAMPTZ NOT NULL,
    -- the whole ExecutionJobRecord as last written
    record            JSON NOT NULL,
    -- projections of record, written in the same statement as record
    kind                   TEXT NOT NULL,
    runtime_profile        TEXT NOT NULL,
    backend                TEXT NOT NULL,
    priority               SMALLINT NOT NULL,
    created_at             TIMESTAMPTZ NOT NULL,
    tenant_id              TEXT,
    cancellation_requested BOOLEAN NOT NULL,
    heartbeat_timeout      INTERVAL NOT NULL,
    max_duration           INTERVAL NOT NULL,
    lookup_keys            TEXT[] NOT NULL,
    CHECK ((queue_visible_at IS NULL) = (queue_priority IS NULL)),
    CHECK (queue_visible_at IS NULL OR backend = 'local')
) WITH (autovacuum_vacuum_scale_factor = 0.02, autovacuum_vacuum_threshold = 200,
        autovacuum_analyze_scale_factor = 0.05, autovacuum_analyze_threshold = 200,
        toast.autovacuum_vacuum_scale_factor = 0.02, toast.autovacuum_vacuum_threshold = 200);

CREATE INDEX idx_execution_jobs_created ON honua.execution_jobs (created_at DESC, operation_id DESC);
CREATE INDEX idx_execution_jobs_status_created
    ON honua.execution_jobs (status, created_at DESC, operation_id DESC);
CREATE INDEX idx_execution_jobs_kind_created
    ON honua.execution_jobs (kind, created_at DESC, operation_id DESC);
CREATE INDEX idx_execution_jobs_lookup_keys ON honua.execution_jobs USING GIN (lookup_keys);
CREATE INDEX idx_execution_jobs_claimable
    ON honua.execution_jobs (runtime_profile, queue_priority, queue_visible_at, operation_id)
    WHERE status = 'Queued' AND queue_visible_at IS NOT NULL;
CREATE INDEX idx_execution_jobs_undispatched ON honua.execution_jobs (backend)
    WHERE status = 'Queued' AND queue_visible_at IS NULL;
CREATE INDEX idx_execution_jobs_reconcilable ON honua.execution_jobs (status)
    WHERE status IN ('Queued', 'Provisioning', 'Running')
      AND NOT (status = 'Queued' AND backend = 'local' AND attempt_count = 0
               AND NOT cancellation_requested);
CREATE INDEX idx_execution_jobs_expired ON honua.execution_jobs (expires_at)
    WHERE status IN ('Succeeded', 'Failed', 'Cancelled');

CREATE TABLE honua.leases (
    lease_key  TEXT COLLATE "C" PRIMARY KEY,
    owner_id   TEXT NOT NULL,
    expires_at TIMESTAMPTZ NOT NULL
);
CREATE INDEX idx_leases_expires ON honua.leases (expires_at);

CREATE TABLE honua.execution_job_log_streams (
    operation_id         TEXT COLLATE "C" PRIMARY KEY,
    appended_count       BIGINT NOT NULL DEFAULT 0,
    dropped_count        BIGINT NOT NULL DEFAULT 0,
    event_identity_count INTEGER NOT NULL DEFAULT 0,
    expires_at           TIMESTAMPTZ NOT NULL,
    CONSTRAINT execution_job_log_streams_event_identity_limit CHECK (event_identity_count <= 8192)
);
CREATE INDEX idx_execution_job_log_streams_expires ON honua.execution_job_log_streams (expires_at);

CREATE TABLE honua.execution_job_log_entries (
    operation_id TEXT COLLATE "C" NOT NULL
        REFERENCES honua.execution_job_log_streams (operation_id) ON DELETE CASCADE,
    sequence     BIGINT NOT NULL,
    entry        JSON NOT NULL CHECK (octet_length(entry::text) <= 4096),
    PRIMARY KEY (operation_id, sequence)
);

CREATE TABLE honua.execution_job_log_event_identities (
    operation_id TEXT COLLATE "C" NOT NULL
        REFERENCES honua.execution_job_log_streams (operation_id) ON DELETE CASCADE,
    event_id     TEXT COLLATE "C" NOT NULL CHECK (length(event_id) BETWEEN 1 AND 160),
    PRIMARY KEY (operation_id, event_id)
);

CREATE TABLE honua.geoprocessing_result_packages (
    job_id     TEXT COLLATE "C" PRIMARY KEY,
    package    JSON NOT NULL,
    expires_at TIMESTAMPTZ NOT NULL
);
CREATE INDEX idx_geoprocessing_result_packages_expires
    ON honua.geoprocessing_result_packages (expires_at);

record is JSON, not JSONB: the stores never query inside it, and JSONB rejects the \^@
escape that a caller-supplied job parameter can carry.

lookup_keys holds one dimension:VALUE entry per filterable value of the record: backend,
queue, requested-by, correlation, trace, definition (definition id and geoprocessing plan
id), resource (one per resource reference), environment, server, release, change-set,
alert. The value is trimmed and upper-cased, which reproduces the Redis store's trimmed,
case-insensitive match. These are the dimensions the Redis store indexed.

Autovacuum starts at 2 % dead rows (plus 200) instead of the default 20 %: the claim index is a small
partial index over a table that churns, and dead entries in it lengthen every claim. The TOAST table
gets the same thresholds because every write of a job replaces its out-of-line record.

2. Record of truth and projections

Columns	Role	Kept consistent by
`version`, `status`, `attempt_count`, `claimed_by`, `claimed_at`, `last_heartbeat_at`, `next_retry_at`, `completed_at`, `percent_complete`, `current_phase`, `updated_at`	Record of truth for those eleven `ExecutionJobRecord` fields.	A read deserializes `record` and overlays these columns. `record` holds the writer's copy of them, which a reader never sees.
`queue_priority`, `queue_visible_at`	Record of truth for the local queue. Not part of `ExecutionJobRecord`.	Written only by the queue statements and by the compare-and-set statement (3.3).
`expires_at`	Record of truth for retention.	Set by every write of the job.
`record`	Record of truth for every other field.	Rewritten by every write of the job, in the one statement that writes the columns.
`kind`, `runtime_profile`, `backend`, `priority`, `created_at`, `tenant_id`, `cancellation_requested`, `heartbeat_timeout`, `max_duration`, `lookup_keys`	Projections of `record`, used only in `WHERE` and `ORDER BY`.	Computed from the record in the store and written in the same statement as `record`.

There is one write path: every create and every compare-and-set is a single statement that writes
record, the attempt state and the projections together. The claim (3.2) and the queue statements
(3.6) are the only writes that do not carry the record; they change only columns whose record of
truth is the column.

3. Statements

Every statement runs as its own transaction on a pooled connection. now() is therefore the
statement start time on the database server.

3.1 Create if absent (`TryCreateAsync`)

INSERT INTO honua.execution_jobs (<all columns except the queue columns>)
VALUES (@operation_id, 1, @status, ..., now() + @retention, @record, <projections>)
ON CONFLICT (operation_id) DO NOTHING;

One affected row means created. SetAsync (used only by the operate fixture seeder) is the same
INSERT with ON CONFLICT (operation_id) DO UPDATE over every column except the queue columns.
These two are the only statements that store the caller's timestamps as given.

3.2 Claim (`IJobQueue.TryClaimAsync`)

WITH next_job AS (
    SELECT operation_id FROM honua.execution_jobs
    WHERE status = 'Queued' AND queue_visible_at <= now()
      AND runtime_profile = @runtime_profile
      AND (@kinds::text[] IS NULL OR kind = ANY (@kinds))
    ORDER BY queue_priority, queue_visible_at, operation_id
    LIMIT 1
    FOR UPDATE SKIP LOCKED)
UPDATE honua.execution_jobs AS job
SET status = 'Provisioning', version = job.version + 1, attempt_count = job.attempt_count + 1,
    claimed_by = @worker_id, claimed_at = now(), last_heartbeat_at = now(), next_retry_at = NULL,
    current_phase = 'Claimed', updated_at = now()
FROM next_job WHERE job.operation_id = next_job.operation_id
RETURNING <attempt-state columns>, record;

A worker's accepted runtime profiles are {managed} when it declares none, as
RuntimeProfiles.CanClaim defines. A worker with one profile, which is every worker today, runs the
statement with runtime_profile = @runtime_profile, so the scan is the ordered range of
idx_execution_jobs_claimable for that profile and a native worker never reads managed rows. A
worker with several profiles runs it with runtime_profile = ANY (@runtime_profiles).

queue_priority is 0 Critical, 1 High, 2 Normal, 3 Low. Within a band the order is the time the job
became visible, which is the Redis score: a requeued or retrying job goes to the back of its band, so
a job that keeps being handed back (partition-lease contention requeues with a 5 s delay and does
not consume an attempt) cannot hold the head of the band.

Lost reply. If the claim fails after it was sent, the queue remembers that for its worker id
(unique per process). The next TryClaimAsync first runs

SELECT <attempt-state columns>, record FROM honua.execution_jobs
WHERE claimed_by = @worker_id AND status = 'Provisioning' AND current_phase = 'Claimed'
ORDER BY claimed_at LIMIT 1;

and returns that job if there is one. A worker runs one job at a time and moves a job out of the
Claimed phase before it claims again, so such a row can only be the claim whose reply was lost.

3.3 Version compare-and-set (`TrySetAsync`), heartbeat, progress, cancel request

UPDATE honua.execution_jobs SET
    version = version + 1,
    status = @status, attempt_count = @attempt_count, claimed_by = @claimed_by,
    percent_complete = @percent_complete, current_phase = @current_phase, updated_at = now(),
    claimed_at        = <stamp>(claimed_at, @claimed_at),
    last_heartbeat_at = <stamp>(last_heartbeat_at, @last_heartbeat_at),
    completed_at      = <stamp>(completed_at, @completed_at),
    next_retry_at = CASE WHEN @next_retry_at IS NULL THEN NULL
                         WHEN next_retry_at IS NOT DISTINCT FROM @next_retry_at THEN next_retry_at
                         ELSE now() + @retry_delay END,
    queue_visible_at = CASE
        WHEN @status IN ('Succeeded', 'Failed', 'Cancelled') THEN NULL
        WHEN @status = 'Queued' AND status <> 'Queued' AND queue_visible_at IS NOT NULL
            THEN now() + @retry_delay
        ELSE queue_visible_at END,
    queue_priority = CASE WHEN @status IN ('Succeeded', 'Failed', 'Cancelled') THEN NULL
                          ELSE queue_priority END,
    expires_at = now() + @retention,
    record = @record, <projections>
WHERE operation_id = @operation_id AND version = @expected_version;

where <stamp>(column, @value) is
CASE WHEN @value IS NULL THEN NULL WHEN column IS NOT DISTINCT FROM @value THEN column ELSE now() END:
a timestamp the writer left as it read it is kept, one the writer changed becomes the database time,
and null clears it. No row affected is a version conflict and returns false.

@retry_delay is the writer's NextRetryAt - UpdatedAt (zero when NextRetryAt is null): the
writer states a delay on its own clock and the database turns it into a time on the database clock.
next_retry_at and queue_visible_at come from that one expression. next_retry_at is what the
record reports and what the provider reconciler waits on for a job no local worker runs;
queue_visible_at is the only time a claim compares.

The heartbeat, progress, cancel-request and terminal writes are all this statement. A job that was
claimed from the local queue and is written back to Queued becomes claimable again in this
statement; there is no window in which it is Queued and absent from the queue.

3.4 Lease-owned update (`TrySetIfLeaseOwnedAsync`)

Statement 3.3 with the lease row locked for the statement:

WITH owned_lease AS MATERIALIZED (
    SELECT 1 FROM honua.leases
    WHERE lease_key = @lease_key AND owner_id = @owner_id AND expires_at > now()
    FOR UPDATE)
UPDATE honua.execution_jobs SET ...
WHERE operation_id = @operation_id AND version = @expected_version
  AND EXISTS (SELECT 1 FROM owned_lease);

A concurrent takeover of the lease waits for this statement, and this statement re-checks owner and
expiry if the takeover committed first.

3.5 Leases (`TryAcquireLeaseAsync`, `RenewLeaseAsync`, `ReleaseLeaseAsync`)

INSERT INTO honua.leases AS lease (lease_key, owner_id, expires_at)
VALUES (@lease_key, @owner_id, now() + @duration)
ON CONFLICT (lease_key) DO UPDATE SET owner_id = EXCLUDED.owner_id, expires_at = EXCLUDED.expires_at
WHERE lease.expires_at <= now();                                              -- acquire

UPDATE honua.leases SET expires_at = now() + @duration
WHERE lease_key = @lease_key AND owner_id = @owner_id AND expires_at > now(); -- renew

DELETE FROM honua.leases WHERE lease_key = @lease_key AND owner_id = @owner_id; -- release

As with the Redis lock, the holder cannot re-acquire a lease it still holds.

3.6 Enqueue, requeue with delay, remove, depth

UPDATE honua.execution_jobs SET queue_visible_at = now(), queue_priority = @priority
WHERE operation_id = @operation_id AND status = 'Queued' AND backend = 'local'
  AND queue_visible_at IS NULL;                                                          -- enqueue

UPDATE honua.execution_jobs SET queue_visible_at = now() + @visible_after, queue_priority = @priority
WHERE operation_id = @operation_id AND backend = 'local'
  AND status NOT IN ('Succeeded', 'Failed', 'Cancelled');                                -- requeue

UPDATE honua.execution_jobs SET queue_visible_at = NULL, queue_priority = NULL
WHERE operation_id = @operation_id AND status = 'Queued' AND queue_visible_at IS NOT NULL; -- remove

SELECT count(*) FROM honua.execution_jobs
WHERE status = 'Queued' AND queue_visible_at IS NOT NULL;                                -- depth

A repeated enqueue keeps the original position, as the Redis ZADD NX did. Only a job whose backend
is local can be in the queue; the table constraint enforces it. A job that a remote backend runs
(AWS Batch, Azure Batch, Kubernetes Jobs, the local process pool) sits Queued with
queue_visible_at NULL until that backend submits it.

3.7 Dispatch repair (`IQueueClaimReconciler.ReconcileStaleClaimsAsync`)

Every producer creates the job and then enqueues it in a second call: geoprocessing, imports, tile
cache, layer publish, share export, provisioner build, topology rebuild, multidimensional scan, and
the console's manual retry. A process that dies between the two leaves a local job Queued and not
in the queue. JobReconciliationService repairs first-attempt geoprocessing jobs immediately
(NeedsLocalDispatchRepair); every other case is repaired by this statement, which the service
already calls each sweep with a 60 s threshold:

UPDATE honua.execution_jobs SET queue_visible_at = now(), queue_priority = priority
WHERE status = 'Queued' AND queue_visible_at IS NULL AND backend = 'local'
  AND NOT cancellation_requested AND updated_at <= now() - @stale_threshold;

Enqueueing is the step the producer was about to take and the job has not started, so the repair
enqueues for every local kind instead of failing the job. A cancellation-requested job is left for
the cancellation path.

3.8 Reconciler candidates

ListActiveAsync returned every non-terminal job to two loops. With a deep queue that is the whole
backlog every 5 s on every serving node. The two loops now read candidates instead:

-- JobReconciliationService, every 30 s: claims that may have expired, and local dispatch repair
SELECT <attempt-state columns>, record FROM honua.execution_jobs
WHERE (status IN ('Provisioning', 'Running')
       AND (COALESCE(last_heartbeat_at, claimed_at) <= now() - heartbeat_timeout
            OR claimed_at <= now() - max_duration))
   OR (status = 'Queued' AND queue_visible_at IS NULL AND backend = 'local');

-- ExecutionJobReconcilerBackgroundService, every 5 s: jobs the reconciler acts on
SELECT operation_id FROM honua.execution_jobs
WHERE status IN ('Queued', 'Provisioning', 'Running')
  AND NOT (status = 'Queued' AND backend = 'local' AND attempt_count = 0
           AND NOT cancellation_requested);

The second excludes exactly the first-attempt local jobs waiting in the queue, for which the
reconciler only reads progress. It keeps local Queued jobs with a cancellation request or an
earlier attempt, because the reconciler cancels the former and resets the progress of the latter.
The reconciler's per-job lease is unchanged; the rows that made it expensive no longer reach it.
Both services keep their own checks on each record; the queries only narrow what they look at. They
are IReconcilableJobSource, implemented by the job store; a store without it (the in-memory
doubles) is read through ListActiveAsync as before.

3.9 List and query with cursor

SELECT <attempt-state columns>, record FROM honua.execution_jobs
WHERE (@statuses::text[] IS NULL OR status = ANY (@statuses))
  AND (@kind::text IS NULL OR kind = @kind)
  AND (NOT @apply_tenant_scope OR tenant_id IS NOT DISTINCT FROM @tenant_id)
  AND (@created_from::timestamptz IS NULL OR created_at >= @created_from)
  AND (@created_to::timestamptz IS NULL OR created_at < @created_to)
  AND lookup_keys @> @required_lookup_keys
  AND (@cursor_created_at::timestamptz IS NULL
       OR (created_at, operation_id) < (@cursor_created_at, @cursor_operation_id))
ORDER BY created_at DESC, operation_id DESC
LIMIT @limit + 1;                                             -- QueryAsync, limit clamped to 1..200

SELECT <attempt-state columns>, record FROM honua.execution_jobs
WHERE status IN ('Queued', 'Provisioning', 'Running') AND (@kind::text IS NULL OR kind = @kind)
ORDER BY updated_at DESC LIMIT 2 * @limit;                    -- ListActiveAsync

The store builds the text with only the predicates a query uses. The cursor is the opaque base64 of
<created_at in Unix microseconds>|<operation_id> of the last row returned.

3.10 Log append with dedupe and trim, tail, page, retention

Append is one batch (one round trip, one implicit transaction):

INSERT INTO honua.execution_job_log_streams (operation_id, expires_at)
VALUES (@operation_id, now() + @retention)
ON CONFLICT (operation_id) DO UPDATE SET expires_at = EXCLUDED.expires_at;      -- locks the stream

WITH accepted AS (
    INSERT INTO honua.execution_job_log_event_identities (operation_id, event_id)
    SELECT @operation_id, @event_id WHERE @event_id IS NOT NULL
    ON CONFLICT DO NOTHING RETURNING 1),
advanced AS (
    UPDATE honua.execution_job_log_streams
    SET appended_count = appended_count + 1,
        dropped_count = GREATEST(dropped_count, appended_count + 1 - @max_entries),
        event_identity_count = event_identity_count + (SELECT count(*) FROM accepted)::int
    WHERE operation_id = @operation_id AND (@event_id IS NULL OR EXISTS (SELECT 1 FROM accepted))
    RETURNING appended_count - 1 AS sequence, dropped_count),
inserted AS (
    INSERT INTO honua.execution_job_log_entries (operation_id, sequence, entry)
    SELECT @operation_id, sequence, @entry FROM advanced RETURNING 1)
DELETE FROM honua.execution_job_log_entries AS entry USING advanced
WHERE entry.operation_id = @operation_id AND entry.sequence < advanced.dropped_count;

sequence is the absolute zero-based position, which is what the existing offset cursor encodes.
@max_entries is 2,000 and an entry is bounded to 4 KiB before it is sent, as today. An event id
already recorded for the job appends nothing, including after its entry was trimmed. The limit of
8,192 event identities per job is the named CHECK constraint.

Tail (@start = -1) and page (@start from the cursor) are one statement, so counts and entries
come from one snapshot:

WITH stream AS (
    SELECT appended_count, dropped_count,
           CASE WHEN @start < 0 THEN GREATEST(dropped_count, appended_count - @limit)
                ELSE GREATEST(dropped_count, @start) END AS window_start
    FROM honua.execution_job_log_streams WHERE operation_id = @operation_id)
SELECT stream.appended_count, stream.dropped_count, stream.window_start, entry.entry
FROM stream LEFT JOIN LATERAL (
    SELECT sequence, entry FROM honua.execution_job_log_entries
    WHERE operation_id = @operation_id AND sequence >= stream.window_start
    ORDER BY sequence LIMIT @limit + 1) AS entry ON TRUE
ORDER BY entry.sequence;

Every append moves the stream's expiry to Jobs:Store:LogRetention from now, and
SetRetentionAsync, which the execution and reconciliation services call when a job reaches a
terminal state, sets it from that moment. The sweep (3.12) does not delete the log of a job that is
not terminal. A log therefore never expires while its job runs.

3.11 Result packages

INSERT INTO honua.geoprocessing_result_packages (job_id, package, expires_at)
VALUES (@job_id, @package, now() + @retention)
ON CONFLICT (job_id) DO UPDATE SET package = EXCLUDED.package, expires_at = EXCLUDED.expires_at;

SELECT package FROM honua.geoprocessing_result_packages WHERE job_id = @job_id AND expires_at > now();

3.12 Retention sweep

JobStoreRetentionSweepService (a hosted service on the serving host, modelled on
AuditRetentionPruneService) runs every Jobs:Store:RetentionSweepInterval and repeats each
statement while it deletes a full batch:

WITH expired AS (
    SELECT operation_id FROM honua.execution_jobs
    WHERE status IN ('Succeeded', 'Failed', 'Cancelled') AND expires_at <= now()
    ORDER BY expires_at LIMIT @batch_size FOR UPDATE SKIP LOCKED)
DELETE FROM honua.execution_jobs AS job USING expired WHERE job.operation_id = expired.operation_id;

WITH expired AS (
    SELECT stream.operation_id FROM honua.execution_job_log_streams AS stream
    WHERE stream.expires_at <= now()
      AND NOT EXISTS (SELECT 1 FROM honua.execution_jobs AS job
                      WHERE job.operation_id = stream.operation_id
                        AND job.status IN ('Queued', 'Provisioning', 'Running'))
    ORDER BY stream.expires_at LIMIT @log_stream_batch_size FOR UPDATE SKIP LOCKED)
DELETE FROM honua.execution_job_log_streams AS stream USING expired
WHERE stream.operation_id = expired.operation_id;

and the first shape for geoprocessing_result_packages and leases. A log stream cascades to up to
2,000 entries and 8,192 event identities, so @log_stream_batch_size is 10: one statement deletes at
most about 100,000 small rows. The sweep runs on every serving node without leader election: the
deletes are bounded, idempotent and skip rows another node has locked.

Setting	Default	Was
`Jobs:Store:JobRetention`	7 days	Redis key TTL of 7 days, hard-coded
`Jobs:Store:LogRetention`	7 days	Redis key TTL of 7 days, hard-coded
`Geoprocessing:Executors:ResultRetention`	7 days, unchanged	unchanged
`Jobs:Store:RetentionSweepInterval`	10 minutes	Redis expiry
`Jobs:Store:RetentionSweepBatchSize`	1,000	n/a

A job that is not in a terminal state is never swept. Redis silently dropped a job left Queued for
seven days; here it stays visible and, if local, is repaired by 3.7.

3.13 Database time

IJobStoreClock.GetUtcNowAsync, implemented by the job store, returns SELECT now(). A consumer
given no clock uses the node clock, which is right for a store that stamps with the node clock (the
in-memory doubles).

4. Invariants and the tests that prove them

Tests are in tests/dotnet/Honua.Server.Tests/Features/Infrastructure/ControlPlane/ and run against
PostgreSQL through Testcontainers.

#	Invariant	Mechanism	Test
1	A job is claimed by at most one worker per attempt.	Claim is one statement that row-locks the candidate with `SKIP LOCKED` and re-checks `status = 'Queued'` on the locked row.	`PostgresJobClaimContentionTests.TryClaimAsync_EightWorkersRaceOneJob_ExactlyOneWinsWithOneAttempt`, `...TryClaimAsync_WorkersDrainContestedQueue_EveryJobIsClaimedExactlyOnce`
2	A stale attempt cannot write after it is superseded.	Every write carries the version it read; a requeue or a new claim increments it. Finalization under a partition lease also requires the lease row.	`PostgresStaleAttemptFencingIntegrationTests.StaleAttempt_IsFencedAcrossReclaimDuplicateDeliveryAndTerminalization`, `PostgresJobClaimContentionTests.WorkerLoss_NaturalHeartbeatLapse_ReExecutesOnceAndSuppressesTheStaleWorkersTerminal`, `PostgresExecutionSubstrateIntegrationTests.ExecutionJobStore_FinalizesOnlyWhileLeaseIsOwned`
3	A local `Queued` job is always eventually claimable.	A return to `Queued` restores the queue position in the same statement; a local `Queued` job that is not in the queue is enqueued by the next sweep after 60 s (3.7); a job is never swept before it is terminal.	`PostgresJobQueueTests.TrySetAsync_ClaimedJobReturnedToQueued_IsClaimableWithoutRequeue`, `PostgresJobQueueTests.ReconcileStaleClaimsAsync_LocalJobCreatedButNeverEnqueued_IsEnqueued` (one case per job kind), `...ReconcileStaleClaimsAsync_ManualRetryThatNeverRequeued_IsEnqueued`, `...ReconcileStaleClaimsAsync_RemoteBackendJob_IsLeftForItsBackend`, `PostgresExecutionSubstrateIntegrationTests.JobReconciliation_PersistedLocalJobWithoutDelivery_ReplacementWorkerExecutesOnce`
4	Cancellation is observed.	Durable `CancellationRequestedAt`; read on each heartbeat and by the reconciler.	`PostgresExecutionSubstrateIntegrationTests.JobReconciliationService_CancelsDurablyRequestedStaleJob`, `PostgresStaleAttemptFencingIntegrationTests.CancellationDuringBackoff_PreventsTheNextAutomaticAttempt`
5	Retries respect the delay and go to the back of their band.	`queue_visible_at = now() + delay`; claim filters and orders on it.	`PostgresJobQueueTests.RequeueAsync_WithDelay_HidesTheJobUntilTheDelayElapses`, `...TryClaimAsync_RequeuedJob_GoesBehindJobsAlreadyWaitingInItsBand`, `PostgresStaleAttemptFencingIntegrationTests.AutomaticTransientFailure_UsesConfiguredBackoffAndStopsAtExactMaxAttempts`
6	A worker killed mid-job is recovered within the heartbeat timeout plus one sweep.	`last_heartbeat_at` stops advancing; the 30 s sweep requeues or fails the job once it is 90 s old.	`PostgresJobClaimContentionTests.WorkerLoss_NaturalHeartbeatLapse_ReExecutesOnceAndSuppressesTheStaleWorkersTerminal`, `PostgresStaleAttemptFencingIntegrationTests.StaleAttempt_IsFencedAcrossReclaimDuplicateDeliveryAndTerminalization` (kills the worker process)
7	Log events are not duplicated.	Primary key on `(operation_id, event_id)`, checked in the append transaction.	`PostgresExecutionLogStoreTests.BoundedLog_TrimsWithAbsoluteCursorsAndDeduplicatesExternalEvents`, `...AppendOnceAsync_ConcurrentDeliveriesOfOneEvent_AppendExactlyOneEntry`
8	Correctness depends only on database time.	Section 5.	`PostgresJobStoreClockTests.ClaimLeaseAndHeartbeatExpiry_UnderSkewedNodeClocks_FollowDatabaseTime`
9	A database restart during a running job loses nothing and does not run the job twice.	Section 8.	`PostgresJobExecutionResilienceTests.JobExecutionService_WhenPostgresRestartsMidJob_JobCompletesExactlyOnce`
10	A database outage longer than the heartbeat timeout does not expire healthy workers' jobs.	Section 8.	`PostgresJobExecutionResilienceTests.JobReconciliation_AfterAnOutageLongerThanTheHeartbeatTimeout_DoesNotExpireALiveWorkersJob`
11	A claim whose reply was lost is taken by the worker that made it and costs no attempt.	3.2.	`PostgresJobQueueTests.TryClaimAsync_AfterAClaimWhoseReplyWasLost_ReturnsThatJobWithOneAttempt`

5. Time

Every lease expiry, visibility delay, heartbeat timeout, execution timeout and retention comparison
uses the database clock.

Stamped in SQL with now(): claimed_at, last_heartbeat_at, next_retry_at, completed_at,
updated_at, queue_visible_at, expires_at and every lease expires_at. A compare-and-set
write replaces a timestamp the writer changed with the database time (3.3); it never stores the
writer's value. A duration between two of them cannot be negative.
Compared in SQL with now(): claim visibility, lease acquire and renew, the lease-owned update,
the reconciler candidate queries, result-package expiry on read, the retention sweep.
Compared in C# against IJobStoreClock: the places that today compare a stored timestamp against
the node clock.

Today (node clock)	Change
`JobReconciliationService`: heartbeat and timeout expiry against `DateTimeOffset.UtcNow`	Reads the store clock once per sweep, and once more for each job it is about to expire.
`ExecutionJobReconciler`: `NextRetryAt > DateTimeOffset.UtcNow` for provider-run retries	Reads the store clock. `NextRetryAt` is database-stamped.
`AssignedExecutionJobQueue`: heartbeat expiry of another worker's claim against its `TimeProvider`	Reads the store clock.
`RedisJobQueue`: visibility and stale-claim age against `DateTimeOffset.UtcNow`	Deleted; SQL `now()`.
Redis key TTLs (server clock already)	`expires_at` and SQL `now()`.

CreatedAt and CancellationRequestedAt remain the writer's values: they are shown and sorted,
never compared with a clock to make a decision. A worker's own execution timeout and drain deadline
are durations on its monotonic clock. TryCreateAsync and SetAsync store timestamps as given,
which is how a job is first written and how tests age a record.

6. Write volume and reconciler load

Every write of a job rewrites its row including record. A heartbeat is one read and one such
write: 200 running jobs at the 30 s interval is about 14 statements and 7 row versions per second.
A record over about 2 kB is stored out of line, so each of those writes also replaces its TOAST
chunks; autovacuum reclaims them. The measured row versions and heap, TOAST and index growth for that
load are in the pull request. If they show a problem, the fix is a heartbeat statement that does not
carry the record, which the column layout already allows.

Reconciler load per process, with R running or provisioning jobs and any number queued:

Loop	Where	Statements
Claim poll, every 5 s when idle	each worker	1 index-ordered claim
Heartbeat, every 30 s per running job	the worker running it	2
`JobReconciliationService`, every 30 s	each serving node, each GDAL worker	1 clock read, 1 candidate query (3.8, returns only expired claims and undispatched local jobs), 1 dispatch repair (3.7); then work only for jobs it recovers
`ExecutionJobReconcilerBackgroundService`, every 5 s	each serving node in poll mode	1 candidate query; per candidate a lease insert, a read, a write only if the observation changed something, a lease delete. Candidates are R plus remote-backend and retrying jobs, not the local backlog

With 200 running jobs that last loop is about 120 to 160 statements per second per serving node,
which is what it costs on Redis today. A backlog of 10,000 queued local jobs adds nothing to any
loop.

7. Connections

The stores hold no connection between statements and use no session state of their own. They are
free of connection pinning behind RDS Proxy only when Database:SessionSettings:Mode is
role-defaults: in the default mode the shared data source runs SET in its connection
initializer, for every user of the pool.

A single-job execution worker (the server image started with HONUA_OPERATION_ID) defaults to a
pool of at most 8 and at least 0 connections when Limits:Connections sets neither; the GDAL worker
opens its own data source for the job store with at most 4 and at least 0
(Jobs:Store:WorkerMaxPoolSize), from ConnectionStrings:DefaultConnection, which it now requires
in place of ConnectionStrings:redis. The serving host is unchanged (200 and 20).

8. Failure behaviour

A statement that fails with a transient error (connection refused or reset, 57P01, 57P03, class
08, 53300) is retried with backoff 0.2 s, 0.5 s, 1 s, 2 s when repeating it is safe:

Always safe: failures while opening the connection; reads; enqueue, remove, lease renew and
release, AppendOnceAsync, result-package writes.
Safe with confirmation: TryCreateAsync and the compare-and-set writes. If an attempt failed after
the statement was sent and the retry affects no row, the store reads the row; version
expected + 1 holding exactly the record that was sent means the first attempt committed, and the
call returns true.
Claim: not repeated blindly; recovered as in 3.2.
Not repeated after send: lease acquire, AppendAsync without an event id. The error reaches the
caller.

Beyond that budget the existing loops carry on: the claim loop retries at its 5 s poll and the
reconciler at its next sweep. A heartbeat pump whose write failed retries every 2 s instead of
waiting for its next interval, so heartbeats resume as soon as the database is back. Nothing is held
in memory that the database does not have, so an outage loses no job.

After an outage every heartbeat in the table is older than the outage, and a sweep that compared
them with the current database time would expire healthy workers' jobs, failing those with
MaxAttempts = 1. JobReconciliationService therefore does not expire a heartbeat until it has
seen the store without interruption for that job's heartbeat timeout. A sweep that fails, or whose
reads take longer than one second (the store retries through a short outage and then answers),
starts the period again. Execution-timeout expiry is not delayed.

An outage shorter than the retry budget is invisible to jobs.
A longer outage, of any length, does not by itself expire a job whose worker is alive.
A worker that died during the outage is recovered one heartbeat timeout after the reconciler next
sees the database.
The API returns its existing store-unavailable error for a job request it cannot persist.

Capability manifest: jobs.runner is available when the job store and queue are registered, which
is whenever the system database is configured. It no longer depends on Redis or on the
caching.redis entitlement. DurableJobSubstrateCause becomes Available, JobStoreNotConfigured,
JobQueueNotRegistered. The manifest does not flap with reachability. Readiness: the job store has
no dependency of its own; the existing database readiness check reports DatabaseUnavailable and
takes the node out of rotation. The Redis durability attestation leaves the job path;
Jobs:RequireDurableStore is accepted and always satisfied. operations.proposals keeps its Redis
requirement and is gated on its own store instead of on the job-store classification.

9. Upgrade and rollouts

Jobs in progress are not carried over from earlier pre-release builds; start from a clean
deployment.

A rollout runs two releases side by side behind the load balancer (canary and primary), and the
workers of both claim from the same table. A job record written by release N must therefore be
readable and executable by release N-1 for the length of a rollout:

fields are only added to ExecutionJobRecord and ExecutionJobSpec, never removed or renamed
within one release step;
a field is never reused with a changed meaning;
a release that rewrites a record keeps the fields it does not know. ExecutionJobRecord and
ExecutionJobSpec carry unknown JSON members through deserialization and back
(ExtensibleJsonRecord), so a heartbeat from an N-1 worker does not strip what release N wrote.
Test: PostgresExecutionSubstrateIntegrationTests.TrySetAsync_RecordWithFieldsThisReleaseDoesNotKnow_KeepsThem.
Unknown members of the other nested records (audit, policies) are not preserved;
a new column of execution_jobs is nullable or has a default, so an N-1 writer that does not set
it still inserts.

The existing rollout step "drain in-flight jobs before the switch" still applies: the rule above
covers the overlap, not a job that needs behaviour only release N has.

What still uses Redis after this change, each owned by the follow-on branch
feat/postgres-control-plane-state: proposals, operation instances, operation secrets and workflow
operations; the data-protection key ring; sessions and the distributed cache (portal tokens, scoped
job tokens, replay protection, idempotency); leader election; catalog change notification across
nodes; the import, export, tile and version job managers; the orchestration run, definition and
watermark stores; and, on the job path, the cross-node submission lease
(ExecutionAdmissionCoordinator), the per-principal submission rate limit
(ExecutionAdmissionEvaluator) and job progress (UniversalProgressStore), each of which falls
back to per-node state when Redis is absent.

10. Limits

Expected envelope: hundreds of concurrently running jobs, a backlog of the order of 100,000 queued or
retained rows, tens of submissions per second, one database.

What would be added if that is exceeded, each behind the same interfaces: a heartbeat statement that
does not carry the record; claiming several jobs per statement (LIMIT n); an optional wake-up
signal so an idle worker does not wait for its 5 s poll.

Not reachable without a redesign: time partitioning of execution_jobs (a partitioned table cannot
keep operation_id unique across partitions, which create-if-absent relies on), and active-active
operation across regions (every guarantee here rests on one writable database and its clock).

Known follow-ups: a point-in-time restore of the database restores job state with it, so a job that
had finished can come back as Queued or Running and run again; there is no switch yet to hold
claiming while an operator reviews them. Job hosts (GDAL and single-job workers) use the system
database role; a role limited to the job tables is not implemented.

Deliberately not done: LISTEN/NOTIFY, session advisory locks, SET, any second queue structure,
any cross-statement transaction.

11. Fit for pipelines

The 2026.2 pipelines work (#7151) moves the orchestration stores to PostgreSQL on this substrate.
How a run uses jobs today: each step, and each fan-out item, becomes one geoprocessing job through
the normal submit path (WorkflowOrchestrationEngine.SubmitStepAsync,
GeoprocessingJobService.SubmitJobCoreAsync); the job id is derived from
{runId}:{stepId}:{attempt} (JobOwnershipSecurity); the run learns a step finished by reading that
job every 5 s (WorkflowOrchestrationEngine.ObserveStepAsync); there is no terminal callback and no
parent job.

Correlation. A job carries its run in Spec.Parameters["orchestration.runId"] and its step in
orchestration.stepId. The run id is projected now as a run:<id> entry in lookup_keys, so all
jobs of a run are one GIN lookup. No query field uses it yet; adding RunId to
ExecutionJobQuery is one more required lookup key. A step is a filter within a run's jobs
(at most 1,000 today, WorkflowForEachSpec.HardIterationLimit).
Fan-out. 5,000 child jobs of one run cost the reconcilers nothing while they wait: they are
first-attempt local jobs in the queue, which neither candidate query (3.8) returns. They cost what
any job costs once claimed. The run's own polling is the load to change: one read per unfinished
step every 5 s is 1,000 statements per second for 5,000 steps, where one run:<id> listing would
do.
Scheduling. A firing that creates a job with an id derived from the schedule and fire time is
exactly-once across nodes: TryCreateAsync is ON CONFLICT DO NOTHING on the primary key and
returns true to one caller. A job can be created already delayed through the existing interface:
TryCreateAsync, then RequeueAsync(id, priority, visibleAfter), which puts a local job into the
queue with a future visibility whether or not it was in the queue. If the second call is lost, the
dispatch repair (3.7) enqueues the job without the delay after 60 s; a caller that cannot accept
that would need the delay on the record at creation, which is one optional field.
Leases. leases is (lease_key, owner_id, expires_at) with acquire, renew and release
compared against the database clock. Run leases (IOperationStore on the run store) and schedule
and trigger claims (set-if-absent with an expiry) fit it unchanged, under their own key prefixes.
Retention. expires_at is per row and every write takes a retention (ttl), so retention can
differ by kind or follow the owning run without a schema change. Run history itself belongs to the
run store, not to job rows.
Live status. A job transition and an outbox row are both rows in this database, but the stores
run each statement on its own pooled connection, so today they are two transactions. Writing them
together later is additive: the compare-and-set becomes the first branch of a WITH ... INSERT INTO <outbox> statement, or the store gains an overload that takes the caller's transaction.
To undo. Nothing in the schema. One assumption would have to change if a worker ever claims
several jobs at once under one worker id: recovery of a claim whose reply was lost (3.2) relies on
a worker holding at most one job in the Claimed phase.

Alternatives considered

A separate queue table. Two rows per job have to change together in every transition, and the
reconciler has to repair the cases where they did not; that is the Redis design's weakness
re-created. One row cannot disagree with itself.
Advisory locks for claims or leases. Session-level locks pin RDS Proxy connections and do not
survive a frozen Lambda (Lambda + RDS Proxy: session advisory locks and bound-source SET pin pooled connections; frozen sandboxes exhaust the proxy (POST /ogc/styles Lambda timeout, run 40961202761) #6760, bug: import and publish session advisory locks still pin RDS Proxy sessions on Lambda #6765); transaction-level locks would hold a transaction open for
the length of a job.
Keeping Redis for the queue only. Keeps the two-store gap between queue and record, and keeps
Redis as a requirement for running a job at all, which is what the decision removes.
An external broker (SQS, RabbitMQ). A third required service, per-cloud variants, and still no
atomic move between the broker and the job record.
A narrow heartbeat write that leaves record untouched (a second statement, a record hash and
heap-only-update tuning). Rejected in design review: two write paths can drift, and the volume one
path writes is small. It stays available as the first step in section 10.

Consequences

A deployment with PostgreSQL and no Redis runs durable jobs. Community edition gets the durable job
runtime.
The GDAL worker needs database credentials and network access to PostgreSQL; it no longer needs
Redis.
Job state is queryable with SQL and is covered by database backup and restore.
Job writes and reconciliation add load to the system database (section 6).

Changes Made

ADR-0081: design of the PostgreSQL job substrate.
Migration 147_CreateJobSubstrate.sql and its ledger hash.
PostgresExecutionJobStore, PostgresJobQueue, PostgresExecutionLogStore (src/Honua.Jobs), PostgresGeoprocessingResultPackageStore (src/Honua.Geoprocessing); not registered.
IJobStoreClock and IReconcilableJobSource as optional dependencies of the reconcilers and the single-job queue; heartbeat write retry; post-outage heartbeat grace.
ExecutionJobRecord and ExecutionJobSpec keep unknown JSON members.
Converted tests, the PostgreSQL stale-attempt worker process, clock, restart, outage and evidence tests.

Testing

Unit tests added/updated
Integration tests added/updated
Architecture tests pass
Manual testing performed

Gate Impact

PR gates (build, test, governance)
Nightly gates (conformance, performance, security)
Release gates (packaging, publishing)
Deploy gates (promotion, post-apply validation)
None — no gate impact

Docs or Contract Impact

OpenAPI spec changed
Protobuf/gRPC contract changed
Control plane SDK surface changed
Documentation updated
None — no docs or contract impact

Release/Deploy Impact

Requires coordinated release across repos
Requires database migration
Requires infrastructure changes
Requires environment variable or secret changes
None — standard merge-and-release flow

Breaking Changes

Jobs in progress are not carried over from earlier pre-release builds.
The GDAL worker requires ConnectionStrings:DefaultConnection and no longer reads ConnectionStrings:redis.

Pre-PR Checklist

Ran scripts/ci/pre-pr-check.sh and all checks passed
Commit messages follow conventional format: type: description (#issue)
PR title matches main commit message
Issue number linked above
Tests added for new functionality
If protocol/auth behavior changed: updated compatibility contract
If breaking admin/control-plane API changes: updated migration guide
OPENAPI_BREAKING_CHANGE_APPROVED — intentional Admin API break acknowledged for this PR
If breaking gRPC/proto wire changes: confirmed with explicit review

…w) (#6809) Stores, queue, log store, result package store and five converted test classes written against the first ADR-0081 design. Not built as a whole and not run. The migration in this commit already follows the reviewed design, so the stores do not match it yet.

github-actions · 2026-11-10T09:11:15Z

Clean code failed on 7161f48 (run 42787674848). This is a code-quality failure, not a flake, and the fix is too big for a small automated change, so I've added ci-stuck and pushed nothing.

Cause: the WIP commit (stores plus converted tests) triggers 69 blocking findings across 16 files:

Rule	Count	What it flags
C1	33	`///` XML doc blocks, history wording ("no longer"), ticket references (`#4403`) in comments
G5	22	test blocks duplicated from the Redis tests (`JobClaimContentionRedisTests.cs`, etc.)
G30	8	functions newly over 60 lines (e.g. `WorkerLoss_NaturalHeartbeatLapse_...` is 135 lines)
F1	3	more than 3 parameters
Ch10	3	ordering/vertical layout

Most findings are in PostgresExecutionSubstrateIntegrationTests.cs (25) and PostgresJobClaimContentionTests.cs (15). The rest are in src/Honua.Jobs/Features/ControlPlane/* (JobStoreDatabase.cs, PostgresExecutionJobStore.cs, ...) and PostgresJobSubstrateHarness.cs.

Log line that proves it:

| G30 | block | `tests/dotnet/Honua.Server.Tests/Features/Infrastructure/ControlPlane/PostgresJobClaimContentionTests.cs:165` | Function newly over 60 lines, or grew by 10+ while over: WorkerLoss_NaturalHeartbeatLapse_ReExecutesOnceAndSuppressesTheStaleWorkersTerminal is 135 lines; extract steps |

Suggested fix (when Stage A is reworked to the reviewed design):

Delete the /// blocks. Rewrite or drop comments that describe history or cite tickets.
Fix the G5 findings by extracting shared contention and fencing scenarios that both the Redis and PostgreSQL fixtures use. In Stage B, deleting the Redis test classes also removes the duplicates.
Split the long test methods into arrange/act/assert helpers. Group the parameters of the 4+ parameter methods into a small record.

ci-doctor: 7161f48

Generated by CI doctor · claude · opus · ~61 AIC · ⌖ ~19 AIC · ⊞ 2.8K · ◷

…verted tests (#6809) One write path per job, database-stamped attempt state, claim ordered by visibility time per runtime profile, dispatch repair, lost-reply claim recovery, reconciler candidate queries, post-outage heartbeat grace and heartbeat write retry. Nothing is registered yet; the Redis stores remain in use.

github-actions · 2026-11-10T09:25:29Z

Clean code failed again on 23a5843 (run 42789664985). This is a code-quality failure, not a flake. The fix is a refactor across about 20 files, too big for an automated patch, so ci-stuck stays and I pushed nothing.

Cause: Failed: 78 blocking, 0 advisory. That is up from 69 on 7161f48:

Rule	Count	What it flags
C1	39	`///` XML doc blocks (37), plus history or ticket wording in comments
G5	22	test blocks duplicated from `RedisExecutionSubstrateIntegrationTests.cs` and the other Redis tests
G30	10	functions newly over 60 lines (e.g. `ExecutionJobStore_QueriesByFiltersAndCursor` is 181 lines, `BuildQueryCommand` is 70)
Ch10	4	new files over 400 lines: `PostgresExecutionJobStore.cs` (629), `PostgresExecutionSubstrateIntegrationTests.cs` (1173), `PostgresJobClaimContentionTests.cs` (510), `PostgresJobQueueTests.cs` (475)
F1	3	methods with more than 4 parameters

Log lines that prove it:

| Ch10 | block | `tests/dotnet/Honua.Server.Tests/Features/Infrastructure/ControlPlane/PostgresExecutionSubstrateIntegrationTests.cs:1` | New file over 400 lines: 1173 lines; split it by responsibility |
Failed: 78 blocking, 0 advisory. Fix them, or add `quality-exception: <reason>` to the PR body.

Suggested fix:

Delete the /// blocks in src/Honua.Jobs/Features/ControlPlane/* and PostgresGeoprocessingResultPackageStore.cs. Rewrite or drop comments that describe history or cite tickets.
Split PostgresExecutionJobStore.cs by responsibility (for example, put query/command building in its own type). Split the large test classes by scenario: queries, reconciliation, contention, queue.
Remove the G5 duplicates by extracting the shared scenarios into a helper that both the Redis and PostgreSQL fixtures use. Deleting the Redis tests in Stage B also removes them.
Break the long tests into arrange/act/assert helpers. Group the parameters of the 4+ parameter methods into a record.
If a finding is deliberate for this staged migration, add quality-exception: <reason> to the PR body instead.

ci-doctor: 23a5843

Generated by CI doctor · claude · opus · ~40 AIC · ⌖ ~19 AIC · ⊞ 2.8K · ◷

…vidence tests (#6809)

github-actions · 2026-11-10T09:37:54Z

Clean code failed again on 8f4b9aa (run 42791492528). This is a code-quality failure, not a flake. The fix is a refactor across about 20 files, too big for an automated patch, so I pushed nothing and ci-stuck stays.

Cause: Failed: 95 blocking, 0 advisory. That is up from 78 on 23a5843. The new stale-attempt fencing, clock and evidence test files account for the increase:

Rule	Count	What it flags
C1	42	`///` XML doc blocks, plus history or ticket wording in comments
G5	30	blocks duplicated from the Redis tests. `PostgresStaleAttemptFencingIntegrationTests.cs` alone adds 8
G30	15	functions newly over 60 lines. New ones include `QueryPlans_OnASeededTable_ArePrinted` (103 lines), `StaleAttempt_IsFencedAcrossReclaim...`, and the skewed-clock and heartbeat-load tests
Ch10	5	new files over 400 lines. New: `PostgresStaleAttemptFencingIntegrationTests.cs` (819). Still there: `PostgresExecutionJobStore.cs` (629), `PostgresExecutionSubstrateIntegrationTests.cs` (1173), `PostgresJobClaimContentionTests.cs` (510), `PostgresJobQueueTests.cs` (475)
F1	3	methods with more than 4 parameters (`TrySetIfLeaseOwnedAsync`, `ExecuteLeaseStatementAsync`, `ExecuteQueueStatementAsync`)

Log lines that prove it:

| Ch10 | block | `tests/dotnet/Honua.Server.Tests/Features/Infrastructure/ControlPlane/PostgresStaleAttemptFencingIntegrationTests.cs:1` | New file over 400 lines: 819 lines; split it by responsibility |
Failed: 95 blocking, 0 advisory. Fix them, or add `quality-exception: <reason>` to the PR body.

Suggested fix: the same as before, now also covering the three new test files.

Delete the /// blocks, including the ones at the top of PostgresStaleAttemptFencingIntegrationTests, PostgresJobStoreClockTests and PostgresJobStoreEvidenceTests.
Pull the fencing, contention and queue scenarios out into a shared helper that both the Redis and PostgreSQL fixtures use. That clears the G5 findings. Removing the Redis tests in Stage B would also clear them.
Split the files over 400 lines by scenario, and move the query/command building out of PostgresExecutionJobStore.
Break the long tests into arrange/act/assert helpers. Group the parameters of the 5–6-parameter methods into a record.
If a finding is deliberate for this staged migration, add quality-exception: <reason> to the PR body instead.

ci-doctor: 8f4b9aa

Generated by CI doctor · claude · opus · ~37 AIC · ⌖ ~19 AIC · ⊞ 2.8K · ◷

…ovacuum (unbuilt) (#6809) Not built and not run when committed. Changes migration 147 and its ledger hash, and makes ExecutionJobRecord and ExecutionJobSpec derive from ExtensibleJsonRecord.

github-actions · 2026-11-10T10:27:35Z

Clean code failed again on 971d66f (run 42798431905). This is a code-quality failure, not a flake. The fix is a refactor across about 20 files, too big for an automated patch, so I pushed nothing and ci-stuck stays.

Cause: still Failed: 95 blocking, 0 advisory, the same count as on 8f4b9aa. The rules are unchanged: C1 42, G5 30, G30 15, Ch10 5, F1 3. This commit made PostgresExecutionSubstrateIntegrationTests.cs longer (1173 → 1206 lines). The other findings carry over:

Ch10, new files over 400 lines: PostgresExecutionJobStore.cs (629), PostgresExecutionSubstrateIntegrationTests.cs (1206), PostgresStaleAttemptFencingIntegrationTests.cs (819), PostgresJobClaimContentionTests.cs (510), PostgresJobQueueTests.cs (475).
F1, too many parameters: TrySetIfLeaseOwnedAsync (5), ExecuteLeaseStatementAsync (6), ExecuteQueueStatementAsync (5).
G30: BuildQueryCommand is 70 lines.

Log lines that prove it:

| Ch10 | block | `tests/dotnet/Honua.Server.Tests/Features/Infrastructure/ControlPlane/PostgresExecutionSubstrateIntegrationTests.cs:1` | New file over 400 lines: 1206 lines; split it by responsibility
Failed: 95 blocking, 0 advisory.

Suggested fix: the same as for 8f4b9aa.

Delete the /// blocks and comments that describe history or cite tickets.
Extract the fencing, contention and queue scenarios into a helper that both the Redis and PostgreSQL fixtures use, which clears G5. Removing the Redis tests in Stage B also clears it.
Split the files over 400 lines by scenario, and move query/command building out of PostgresExecutionJobStore.
Break the long methods into steps, and group the parameters of the 5–6-parameter methods into a record.
If a finding is deliberate for this staged migration, add quality-exception: <reason> to the PR body instead.

ci-doctor: 971d66f

Generated by CI doctor · claude · opus · ~41 AIC · ⌖ ~19 AIC · ⊞ 2.8K · ◷

…d) (#6809) TryAddPostgresExecutionJobStore and TryAddPostgresJobQueueAndLogStore over the host's system NpgsqlDataSource. Nothing calls them yet; the Stage B swap starts from here.

…econciliation out of the Redis block (unbuilt) (#6809) The serving host, the single-job execution worker and the GDAL worker compose the PostgreSQL job store, queue, log store and result-package store over the system data source. The GDAL worker opens its own small pool from ConnectionStrings:DefaultConnection and no longer connects to Redis. The job reconciler, backstop sweep and queue-depth collector run whenever the job store is registered. Tests are not converted yet and the Redis job classes are still in the tree.

…e registered (unbuilt) (#6809) Classify reports Available for a registered store and queue whatever the Redis state. operations.proposals is gated on its own Redis-backed store.

…rness through the test kit (unbuilt, tests not converted) (#6809)

github-actions · 2026-11-10T10:55:43Z

Clean code failed again on d2b982f (run 42802369281). This is a code-quality failure, not a flake. The fix is a refactor across about 20 files, too big for an automated patch, so I pushed nothing and ci-stuck stays.

Cause: Failed: 101 blocking, 0 advisory, up from 95 on 971d66f. By rule: C1 46, G5 30, G30 15, Ch10 5, F1 4, F3 1. The findings from 971d66f are all still there. This commit adds:

F1 + F3 in src/Honua.Server/Features/Capabilities/CapabilityManifestService.cs:799: Capability takes 16 parameters, including the boolean flag requiresProposalStore.
4 more C1 comment findings: /// blocks, plus history (no longer) and ticket (#4403) wording in PostgresJobClaimContentionTests.cs.

These findings carry over unchanged:

Ch10, files over 400 lines: PostgresExecutionJobStore.cs (629), PostgresExecutionSubstrateIntegrationTests.cs (1206), PostgresStaleAttemptFencingIntegrationTests.cs (819), PostgresJobClaimContentionTests.cs (510), PostgresJobQueueTests.cs (475).
G5: blocks duplicated from the Redis test classes.

Log lines that prove it:

| F1 | block | `src/Honua.Server/Features/Capabilities/CapabilityManifestService.cs:799` | More than 4 parameters: Capability takes 16; introduce a parameter object |
| F3 | block | `src/Honua.Server/Features/Capabilities/CapabilityManifestService.cs:799` | Boolean flag parameter: Capability(requiresProposalStore): split into two intention-revealing functions |
Failed: 101 blocking, 0 advisory. Fix them, or add `quality-exception: <reason>` to the PR body.

Suggested fix:

CapabilityManifestService.Capability: pass the 16 arguments as one descriptor record. Replace requiresProposalStore with two named factories, or with a requirement value on that record.
Delete the /// blocks and the comments that describe history or cite tickets.
G5: the duplicates are against the Redis test classes. Deleting those classes, as this commit's Redis store removal implies, or extracting shared scenarios would clear them.
Split the files over 400 lines by scenario, and move query/command building out of PostgresExecutionJobStore. Break the long methods into steps, and group the 5–6-parameter lease/queue statement methods into a record.
If a finding is deliberate for this staged migration, add quality-exception: <reason> to the PR body instead.

ci-doctor: d2b982f

Generated by CI doctor · claude · opus · ~38 AIC · ⌖ ~13 AIC · ⊞ 2.8K · ◷

…ompose wiring, test conversion in progress (unbuilt) (#6809)

…rker smoke on PostgreSQL (unbuilt) (#6809)

…ession notes (unbuilt) (#6809)

github-actions · 2026-11-10T11:20:12Z

Clean code failed again on 2afd3cf (run 42805358202). This is a code-quality failure, not a flake. The fix is a refactor across about 40 files, too big for an automated patch, so I pushed nothing and ci-stuck stays.

Cause: Failed: 86 blocking, 0 advisory, down from 101 on d2b982f. By rule: C1 67, G30 8, G5 4, F1 4, Ch10 2, F3 1.

Fixed since the last run: most of the G5 duplicates against the Redis tests, and the over-400-line Postgres test files except PostgresJobQueueTests.cs.
C1 grew (46 → 67). This commit adds new /// blocks in the converted tests (ExecutionWorkerProfileTests, TileExportDurableRecoveryTests, ImportEndpointDurableRunnerIntegrationTests, GPServerJobWait, PostgresJobSubstrateHarness, ...) and in src/Honua.Worker.Gdal/GdalWorkerServiceCollectionExtensions.cs. It also adds ticket references (issue #2689, honua-server#5916) and one no longer history comment.
New G5: the two NoRedisJobHost.cs copies (GeoServices and OgcApi tests) duplicate each other. ExecutionAdmissionMultiNodeTests and TileExportAdmissionSharedLeaseTests share a node-setup block.
New G30: PostgresDataSourceFactory.ResolveConnectionLimits grew from 72 to 92 lines.
Still there: PostgresExecutionJobStore.cs (629 lines; F1 on TrySetIfLeaseOwnedAsync/ExecuteLeaseStatementAsync; BuildQueryCommand is 70 lines), PostgresJobQueue.ExecuteQueueStatementAsync (F1), CapabilityManifestService.Capability (16 parameters plus the requiresProposalStore flag), and PostgresJobQueueTests.cs (476 lines).

Log lines that prove it:

| G5 | block | `tests/dotnet/Honua.Protocols.GeoServices.Tests/Source/GPServer/NoRedisJobHost.cs:15` | Duplicated block (15+ lines, 100+ tokens): repeats tests/dotnet/Honua.Protocols.OgcApi.Tests/Source/Processes/NoRedisJobHost.cs
Failed: 86 blocking, 0 advisory. Fix them, or add `quality-exception: <reason>` to the PR body.

Suggested fix:

Delete the /// blocks. Drop the ticket references and the history wording.
Move NoRedisJobHost into a shared test helper, for example in Honua.TestKit, and use it from both protocol test projects. Extract the shared node setup used by the two admission test classes.
Pull the new branches of ResolveConnectionLimits into their own methods.
Split PostgresExecutionJobStore (move query/command building into its own type) and PostgresJobQueueTests. Group the lease/queue statement parameters into a record. Pass Capability a descriptor record in place of the 16 parameters and the boolean flag.
If a finding is deliberate for this staged migration, add quality-exception: <reason> to the PR body instead.

ci-doctor: 2afd3cf

Generated by CI doctor · claude · opus · ~51 AIC · ⌖ ~13 AIC · ⊞ 2.8K · ◷

github-actions · 2026-11-10T11:29:00Z

Clean code failed again on 7f9408b (run 42806626558). This is a code-quality failure, not a flake. I pushed nothing and ci-stuck stays.

Cause: Failed: 86 blocking, 0 advisory. The count and the breakdown are the same as on 2afd3cf: C1 67, G30 8, G5 4, F1 4, Ch10 2, F3 1. The latest commit (a raw string literal in a test, plus ADR notes) neither fixed nor added any finding. Every finding listed in the previous comment is still there.

Log line that proves it:

Failed: 86 blocking, 0 advisory. Fix them, or add `quality-exception: <reason>` to the PR body.

Suggested fix: the same as for 2afd3cf. The fix is a refactor across about 40 files, too big for an automated patch:

Delete the /// blocks. Drop the ticket references (issue #2689, honua-server#5916) and the no longer history comment.
Move NoRedisJobHost into a shared test helper. Extract the node setup that the two admission test classes share.
Split PostgresExecutionJobStore.cs (629 lines) and PostgresJobQueueTests.cs (476 lines). Extract steps from BuildQueryCommand, ResolveConnectionLimits and the long tests.
Group the lease/queue statement parameters into a record. Pass Capability a descriptor record in place of its 16 parameters and the requiresProposalStore flag.
If a finding is deliberate for this staged migration, add quality-exception: <reason> to the PR body instead.

ci-doctor: 7f9408b

Generated by CI doctor · claude · opus · ~39 AIC · ⌖ ~13 AIC · ⊞ 2.8K · ◷

…ilt) (#6809)

github-actions · 2026-11-10T11:48:08Z

Clean code failed again on 6ea9d18 (run 42809480927). This is a code-quality failure, not a flake. I pushed nothing and ci-stuck stays.

Cause: still Failed: 86 blocking, 0 advisory, the same count and breakdown as on 7f9408b and 2afd3cf: C1 67, G30 8, G5 4, F1 4, Ch10 2, F3 1. The latest commit only fixes a raw string literal in the unknown-record-fields test. It neither fixed nor added any finding.

Log line that proves it:

Failed: 86 blocking, 0 advisory. Fix them, or add `quality-exception: <reason>` to the PR body.

Suggested fix: the same as in the 2afd3cf comment. It is a refactor across about 40 files, too big for an automated patch:

Delete the /// blocks. Drop the ticket references and the no longer history comment.
Move NoRedisJobHost into a shared test helper. Extract the node setup that ExecutionAdmissionMultiNodeTests and TileExportAdmissionSharedLeaseTests share.
Split PostgresExecutionJobStore.cs and PostgresJobQueueTests.cs. Extract steps from BuildQueryCommand, ResolveConnectionLimits and the long tests (skewed clocks, heartbeat load, outage reconciliation, worker profile).
Group the lease/queue statement parameters into a record. Pass Capability a descriptor record in place of its 16 parameters and the requiresProposalStore flag.
If a finding is deliberate for this staged migration, add quality-exception: <reason> to the PR body instead.

ci-doctor: 6ea9d18

Generated by CI doctor · claude · opus · ~33 AIC · ⌖ ~13 AIC · ⊞ 2.8K · ◷

mikemcdougall · 2026-11-10T17:42:52Z

Stopped overlapping #3606 / #7227 work and preserved WIP at feat/s-front-door-durable-proposals, head 850a021729. No PR will be opened.

Coverage to reuse beyond the job-store scope here:

PostgreSQL operation-instance/proposal persistence, restart/idempotency coverage, secret-at-rest protection, and unavailable-store behavior when PostgreSQL is absent.
PostgresProposalApprovalRestartTests: authenticated human approval across restarts, stable proposal/audit identity, exactly one actuator invocation.
StudioPostgresPublicationTests plus shared StudioPostgresSchema: real Redis-free Studio publication and retained operation/publication after restart. This tests the configured admin direct-publication path; independent approval coverage is in the test above.

Latest focused remote run passed all 3 acceptance tests (Studio publication, existing PostgreSQL Studio HTTP journey, proposal approval restart). Broader affected tests have failures around durable proposal availability/idempotent read or audit; that run has not completed. Sol fixture delta passed, but the branch is not merge-ready: database-restore continuity of PostgreSQL approval authority still needs resolution and overall review/qualification are incomplete. Please port selectively rather than merge the WIP wholesale.

geometry.buffer, surface.slope and the three routing rows: the evidence test bodies changed only in how the host and job tables are prepared (PostgreSQL job substrate, no Redis); no assertion changed. catalogSourceContentDigest follows GeoprocessingServiceCollectionExtensions.cs, whose job store registration moved to PostgreSQL.

chatgpt-codex-connector · 2026-11-10T19:05:13Z

Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-10T20:50:57.807364Z	`0480f15`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 04855f6967

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

A serverless deployment hosts no in-process timer, so expired jobs, logs, result packages and leases accumulated. JobStoreRetention is a scheduled-tick kind whose handler drives the same JobStoreRetentionSweeper as the hosted loop.

honua-io-bot · 2026-11-10T19:52:42Z

Commit pushed: 6de4de7

Generated by Sync doctor · claude · opus · ~48 AIC · ⌖ ~32 AIC · ⊞ 2.8K

honua-io-bot · 2026-11-10T19:52:44Z

Merged trunk dadaaa3; conflicts in docs/internal/contributor/adr/README.md, eng/schema/schema-migration-hashes.v1.json resolved by:

adr/README.md: kept both index rows (this PR's ADR 0081 and trunk's ADR 0083).
schema-migration-hashes.v1.json: union of entries (this PR's 147_CreateJobSubstrate.sql and trunk's 151_CreateCdnReleaseRecord.sql); both hashes re-verified against the files.

Generated by Sync doctor · claude · opus · ~48 AIC · ⌖ ~32 AIC · ⊞ 2.8K · ◷

…greSQL lease rows (#7283) * feat: multi-node hosts start without Redis; version lock on lease rows (unbuilt, not run) (#6809) Removes the startup rule that a MultiNode deployment needs a Redis connection. MultiNode still refuses local file storage. The branch-versioning lock moves from Redis to a lease row in the system database, through a small ILeaseStore over the job store's lease statements. Jobs:RequireDurableStore=true is satisfied by the system database. Tests: MultiNodeWithoutRedisStartupTests (validation and a host that starts and answers liveness), LeaseVersionLockTests. Not compiled and not run on this branch; the same changes passed in the stacked branch. * test: fixture configuration overrides for the multi-node startup test (unbuilt) (#6809)

mikemcdougall · 2026-11-10T20:35:33Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fdab761829

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…otects (#6809) LeaseHolder stops renewing after a failed renewal and cancels its lease-lost token, but version maintenance never observed it, so it could keep running after another node took the expired lease. IVersionLockHandle now exposes LockLostToken; RunExclusiveAsync runs the work under it and reports a lost lock as VersionLockedException, which the job runner retries and the endpoint answers with 409.

mikemcdougall · 2026-11-10T20:48:38Z

@codex review

chatgpt-codex-connector · 2026-11-10T20:50:55Z

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 0480f15772

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…wner holds the lease (#6809)

docs: add ADR-0081 for the PostgreSQL job substrate (#6809)

cec61fe

mikemcdougall added the release/2026.1-rc.2 Plumbing/security/truth cut label Nov 10, 2026

mikemcdougall added 2 commits November 9, 2026 23:03

docs: design review changes to ADR-0081 (#6809)

dde8ec4

github-actions Bot added the ci-stuck label Nov 10, 2026

test: convert stale-attempt fencing to PostgreSQL and add clock and e…

8f4b9aa

…vidence tests (#6809)

mikemcdougall mentioned this pull request Nov 10, 2026

Epic: run clustered without Redis: Postgres substrate for durable jobs, coordination and security state #6809

Open

github-code-quality Bot found potential problems Nov 10, 2026

View reviewed changes

Comment thread src/Honua.Jobs/Features/ControlPlane/JobStoreDatabase.cs Fixed

Comment thread ...er.Tests/Features/Infrastructure/ControlPlane/PostgresStaleAttemptFencingIntegrationTests.cs Fixed

mikemcdougall mentioned this pull request Nov 10, 2026

feat: PostgreSQL shared state and Redis removal (#6809) #7250

Draft

15 of 19 tasks

fix: outage grace on stalled sweeps, unknown record fields, TOAST aut…

971d66f

…ovacuum (unbuilt) (#6809) Not built and not run when committed. Changes migration 147 and its ledger hash, and makes ExecutionJobRecord and ExecutionJobSpec derive from ExtensibleJsonRecord.

mikemcdougall added 4 commits November 10, 2026 00:36

wip: registration of the PostgreSQL job substrate (unbuilt, not calle…

690e226

…d) (#6809) TryAddPostgresExecutionJobStore and TryAddPostgresJobQueueAndLogStore over the host's system NpgsqlDataSource. Nothing calls them yet; the Stage B swap starts from here.

feat: the job runner is available whenever the job store and queue ar…

7409004

…e registered (unbuilt) (#6809) Classify reports Available for a registered store and queue whatever the Redis state. operations.proposals is gated on its own Redis-backed store.

refactor: delete the Redis job stores and share the PostgreSQL job ha…

d2b982f

…rness through the test kit (unbuilt, tests not converted) (#6809)

github-code-quality Bot found potential problems Nov 10, 2026

View reviewed changes

Comment thread tests/dotnet/Honua.TestKit/Extensions/WebAppFixtureJobSubstrateExtensions.cs Fixed

mikemcdougall added 3 commits November 10, 2026 01:05

wip: retention sweeper, single-job worker pool default, GDAL worker c…

4a53128

…ompose wiring, test conversion in progress (unbuilt) (#6809)

wip: converted job tests, no-Redis durable runtime tests, docs and wo…

2afd3cf

…rker smoke on PostgreSQL (unbuilt) (#6809)

fix: raw string literal in the unknown-record-fields test; ADR supers…

7f9408b

…ession notes (unbuilt) (#6809)

github-code-quality Bot found potential problems Nov 10, 2026

View reviewed changes

Comment thread ...t/Honua.Server.Tests/Features/Geoprocessing/Execution/VectorProcessParityIntegrationTests.cs Resolved

Comment thread src/Honua.Jobs/Features/ControlPlane/JobStoreRetentionSweepService.cs Fixed

fix: triple-dollar raw string in the unknown-record-fields test (unbu…

6ea9d18

…ilt) (#6809)

mikemcdougall mentioned this pull request Nov 10, 2026

feat: stage geoprocessing outputs in object storage (#6809) #7258

Merged

10 of 13 tasks

fix: missing using for OwnedJobLease (#6809)

c98d37a

mikemcdougall added 2 commits November 10, 2026 07:47

refactor: GDAL worker registration is internal to its consumers (#6809)

d1e5206

honua-io-bot Bot mentioned this pull request Nov 10, 2026

[aw] Review doctor: is missing required data #7266

Open

mikemcdougall mentioned this pull request Nov 10, 2026

fix: enforce rc.2 migration compatibility boundary #7281

Merged

mikemcdougall marked this pull request as ready for review November 10, 2026 19:05

chatgpt-codex-connector Bot reviewed Nov 10, 2026

View reviewed changes

Comment thread src/Honua.Server/Startup/ControlPlaneReconcileRegistration.cs Resolved

mikemcdougall mentioned this pull request Nov 10, 2026

feat: MultiNode deployments start without Redis; version lock on PostgreSQL lease rows #7283

Merged

2 tasks done

Merge trunk into feat/postgres-job-substrate (#6809)

40745e8

mikemcdougall enabled auto-merge (squash) November 10, 2026 19:36

Merge origin/trunk (dadaaa3) into feat/postgres-job-substrate

6de4de7

mikemcdougall mentioned this pull request Nov 10, 2026

aws-serverless: schedule the JobStoreRetention control-plane tick honua-io/honua-iac#411

Open

chatgpt-codex-connector Bot reviewed Nov 10, 2026

View reviewed changes

Comment thread src/Honua.Core/Core/Features/Infrastructure/Coordination/LeaseVersionLock.cs Resolved

This was referenced Nov 10, 2026

Epic: task-first asset management for small governments (rc.3) #7300

Open

feat: asset profiles, task contracts and recurring programmes (rc.3) #7301

Open

mikemcdougall added 4 commits November 10, 2026 10:58

test: wait for lock loss through the handle (CA2016) (#6809)

ccd0a86

test: nullable flow in the lock-loss test (#6809)

75802a9

test: the scripted lease store refuses re-acquisition while another o…

d8a4f4f

…wner holds the lease (#6809)

Merge trunk into feat/postgres-job-substrate (#6809)

1aed2df

mikemcdougall merged commit 3fc5621 into trunk Nov 10, 2026
64 of 68 checks passed

Conversation

mikemcdougall commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request

Issue Link

Stage B and C verification (head 6e362c99ee or later)

Summary

ADR-0081: PostgreSQL is the durable job substrate

Status

Context

Decision

1. Tables

2. Record of truth and projections

3. Statements

3.1 Create if absent (TryCreateAsync)

3.2 Claim (IJobQueue.TryClaimAsync)

3.3 Version compare-and-set (TrySetAsync), heartbeat, progress, cancel request

3.4 Lease-owned update (TrySetIfLeaseOwnedAsync)

3.5 Leases (TryAcquireLeaseAsync, RenewLeaseAsync, ReleaseLeaseAsync)

3.6 Enqueue, requeue with delay, remove, depth

3.7 Dispatch repair (IQueueClaimReconciler.ReconcileStaleClaimsAsync)

3.8 Reconciler candidates

3.9 List and query with cursor

3.10 Log append with dedupe and trim, tail, page, retention

3.11 Result packages

3.12 Retention sweep

3.13 Database time

4. Invariants and the tests that prove them

5. Time

6. Write volume and reconciler load

7. Connections

8. Failure behaviour

9. Upgrade and rollouts

10. Limits

11. Fit for pipelines

Alternatives considered

Consequences

Changes Made

Testing

Gate Impact

Docs or Contract Impact

Release/Deploy Impact

Breaking Changes

Pre-PR Checklist

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

mikemcdougall commented Nov 10, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

honua-io-bot Bot commented Nov 10, 2026

Uh oh!

honua-io-bot Bot commented Nov 10, 2026

Uh oh!

mikemcdougall commented Nov 10, 2026 •

edited

Loading

Stage B and C verification (head `6e362c99ee` or later)

3.1 Create if absent (`TryCreateAsync`)

3.2 Claim (`IJobQueue.TryClaimAsync`)

3.3 Version compare-and-set (`TrySetAsync`), heartbeat, progress, cancel request

3.4 Lease-owned update (`TrySetIfLeaseOwnedAsync`)

3.5 Leases (`TryAcquireLeaseAsync`, `RenewLeaseAsync`, `ReleaseLeaseAsync`)

3.7 Dispatch repair (`IQueueClaimReconciler.ReconcileStaleClaimsAsync`)

chatgpt-codex-connector Bot commented Nov 10, 2026 •

edited

Loading