diff --git a/CHANGELOG.md b/CHANGELOG.md index c05b5bfb8..24c2b76cc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ - Add a Noema-owned exact-claim evidence receipt contract whose execution and research producers serialize one canonical artifact that binds every receipt semantic field, including command/result/isolation/network or source revision/excerpt/retrieval policy. Admission accepts only a receipt ID from untrusted model output. The owner API first verifies the exact authenticated OpenCode-handoff manifest digest, canonical envelope bytes, reviewed producer-to-kind policy, and repository/head/workflow/run/attempt identity before it can construct an immutable typed index; admission then reconstructs each canonical artifact and verifies time/claim/artifact identity. The version-2 manifest now binds a separate producer-authenticated `ClaimEvidenceRequirement` containing the exact claim, independently required evidence kind, and `context` or `finding` publication authority. Raw current-head source lines are context only: they are withheld from finding-reference prompts and cannot publish a finding or `request_changes`; an explicitly producer-authorized source finding remains usable and retains exact path/line checks. Finding-free model `request_changes` and `blocked` verdicts cannot bypass receipt admission to publish a vacuous blocking review. Requirement/receipt kind mismatch, fixed-artifact semantic substitution, caller-supplied receipt dictionaries, model self-classification, stale identities, cross-kind receipts, marker-only sandbox output, noncanonical artifact bytes, and expired receipts fail closed before the GitHub publisher. This remains the owner prerequisite for ContextualWisdomLab/.github#1641 and issue #555. The reviewed `sandboxed_verify` adapter exists in owner source, but its actual central stdout/stderr/marker-to-manifest wiring and the trusted research producer are not yet integrated; exact-head hosted GREEN, immutable release, and the verified central consumer bump remain required. ## Unreleased +- Workflow / Task Execution의 기존 execution-scoped `NOEMA_WORKFLOW_STATE` private Durable Object command surface에 observation-only `read_operability`를 추가한다. Exact admitted execution/plan과 동일한 hashed object identity를 재사용하고 SQLite `databaseSize`를 `{ database_size_bytes }`로만 반환한다. Caller-only field는 private command transport projection에서 제거하고, 성공 응답에는 admitted plan·workflow/task payload·raw execution identity를 포함하지 않는다. Foreign object identity와 unavailable/throwing/negative/non-integer storage metadata는 실패-폐쇄한다. 이 값은 exact-object storage-growth 증거를 만들기 위한 source-level producer일 뿐 unit/fake bytes가 deployed transaction/restart/recovery, representative workload denominator, p95, PITR/rollback, deployment 또는 immutable release 증거가 되지는 않는다. issue #541, PR #605, ADR 0013. - Noema Policy / Approval에 procedural graph **publication preflight**를 추가했다. PR #603에서 current State / Checkpoint history와 Policy / Approval snapshot을 stable double-read로 다시 읽어 moving authority와 current revoke를 실패-폐쇄하고 exact graph/history/evaluator handoff/signer/approval identity를 결합한다. 반환되는 process-local preflight receipt는 `publicationAuthorized:false`와 `activationAuthorized:false`를 유지한다. 실제 graph publication, current lifecycle/revocation, live Keyverse/owner trust, immutable released `context-graph-contracts`, canary/rollback 및 production outcome authority는 별도다. - Policy / Approval에 procedural graph lineage의 독립 승인·취소 CAS 원장을 추가한다. 기존 Agent Runtime이 admit한 exact graph와 State / Checkpoint가 provenance-preserving read authority로 반환한 verified evaluation history를 exact candidate/history/evaluator handoff identity에 결합하고, monotonic approval-version CAS·exact decision replay·explicit revoke·bounded digest-chain integrity를 검증한다. structural clone history/approval snapshot, stale writer, malformed 또는 mismatched independent decision과 retained-byte integrity 위반은 실패-폐쇄하며 모든 event/snapshot은 `activationAuthorized:false`를 유지한다. Keyverse signer/key custody, graph publication, immutable release/deployment/canary, tool/model/provider routing과 product/domain truth는 이 경계 밖이다. issue #584, PR #601. - State / Checkpoint의 durable procedural evaluation history에 provenance-preserving read authority를 추가한다. canonical repository가 retained bytes와 graph lineage를 검증해 반환한 immutable snapshot만 process-local admission으로 표시하고, structural clone·deserialized lookalike·append-return snapshot·caller-created object·null은 downstream Policy / Approval anti-corruption boundary에서 authority로 오인되지 않도록 실패-폐쇄한다. 이 표시는 읽은 시점의 verified provenance만 증명하며 이후 append에 대한 currentness, Policy / Approval decision, graph publication/promotion 또는 activation authority를 부여하지 않는다. issue #584, PR #599. diff --git a/docs/OPERABILITY.md b/docs/OPERABILITY.md index b24bf61d1..7b44f83e1 100644 --- a/docs/OPERABILITY.md +++ b/docs/OPERABILITY.md @@ -12,6 +12,7 @@ | Independent review | correct reviewer identity/model route가 exact head를 검토하는가 | workflow run, formal review, evidence manifest | | Commercial maintenance | Maintainer App이 정확한 policy 아래 안전하게 dispatch/merge하는가 | governance audit, loop report, merge/downstream-run evidence | | Product development | OpenCode proposal이 bounded/uncredentialed이고 publication이 분리되는가 | proposal artifact, verifier, publisher run evidence | +| Workflow / Task Execution | exact execution-scoped state authority가 claim/checkpoint/recovery와 같은 객체에서 관측 가능한가 | `NOEMA_WORKFLOW_STATE` exact-object state evidence, bounded `database_size_bytes`, deployed transaction/recovery/p95 receipts | | External-extension lifecycle | exact admitted artifact의 Noema lifecycle authority가 restart/CAS/replay/rollback 뒤에도 보존되는가 | Durable Object current projection, append-only audit/recovery receipt, contention/storage-growth evidence | | Procedural graph advisory | execution-local procedural context와 candidate screening이 activation authority로 오인되지 않는가 | exact source/tests, local graph/session admission evidence, explicit abstention/rejection reason, `activationAuthorized: false` | | Release | protected integrated source에서 immutable artifact가 만들어졌는가 | package/SBOM/provenance/publication receipt | @@ -47,6 +48,7 @@ Worker/runtime category: - immutable allowed workflow SHA; - rate-limiter and replay-guard Durable Object namespaces; - configured request-rate policy; +- `NOEMA_WORKFLOW_STATE` Durable Object binding/namespace when Workflow / Task state is deployed for operational acceptance; - external-extension lifecycle Durable Object binding/namespace when that slice is deployed for operational acceptance. GitHub automation category: @@ -161,6 +163,8 @@ Structured operational events use bounded fields such as: Do not log bearer tokens, GitHub installation token, private key, raw body, raw `jti`, authorization header or provider secret. +Workflow / Task operability observation is execution-object-local. It may expose only bounded platform metadata such as `database_size_bytes` through the already-private `NOEMA_WORKFLOW_STATE` capability after exact execution/plan admission and object-name verification. It must not export workflow/task payloads, raw execution identity, claims, checkpoints, secrets, foreign-owner truth, or turn observation into mutation/retry/recovery authority. + External-extension lifecycle observability may expose bounded stream identity, version/state, transition ID, CAS/replay/conflict reason, digest/reference identities, latency and storage-growth metrics. It must not emit plugin prompt plaintext, raw product data, hidden reasoning, raw secrets, provider credentials, or editable copies of foreign-owner verdict/policy state. Procedural graph diagnostics may retain bounded tenant/task/execution-safe identifiers, graph/structure digest, advisory availability, abstention/rejection reason and evaluation-case counts required for troubleshooting. They must not turn raw graph guidance/pitfall text, product payload, evaluator hidden reasoning, provider credentials or unauthenticated score material into durable operational evidence. `eligibleForApproval` and digest equality are not activation metrics. @@ -175,6 +179,8 @@ Current operational materials define KPI/alert tooling for exchange failure and A non-strict `SKIP` because no production log exists is not production SLO proof. +For Workflow / Task state, an object-local byte counter is only an evidence producer. Production storage-growth evidence must bind the exact immutable release, deployment, execution-scoped object, observation window, request/transition workload, retention behavior and before/after measurements. Unit/fake `database_size_bytes`, namespace/global aggregates, reduced samples, or warm-cache timing are not capacity, recovery or p95 evidence. Measure the real synchronous buyer/runtime path under representative concurrency and retain the transaction/restart/recovery context with the observation. + For external-extension lifecycle evidence, record current-projection and contended-append latency separately. The target is p95 ≤20 ms where that path is synchronous buyer/runtime authority. O(1) storage cardinality, unit timing, a reduced sample, or cache-only warmup is not that evidence. Record the actual Durable Object backend, request count/window, contention pattern, stream cardinality, storage size and exact source/deployment identity used for the measurement. The protected procedural graph library source has no production p95 claim because it has no deployed synchronous buyer path. If procedural graph lookup later enters such a path, measure the real end-to-end path under representative graph sizes and concurrency rather than promoting unit timing to production latency evidence. @@ -232,6 +238,8 @@ Queued/pending runs are not success. RCA should distinguish runner/billing/provi Malformed/unavailable state decision fails credential issuance or lifecycle mutation. Before deleting state, distinguish current active claim/window from stale cleanup and preserve rollback implications. +For Workflow / Task state, inability to read object-local operability metadata is a storage-unavailable observation, not permission to fabricate a zero or reuse another execution's metric. Preserve the exact execution/object/release identity and diagnose the storage/runtime boundary separately from retained-state conflicts. An operability read never authorizes state mutation, claim release, retry, checkpoint replacement, or recovery. + For external-extension lifecycle state, never “repair” corruption by editing/deleting prior events, copying current mutable owner truth into historical events, auto-rebasing a failed CAS, or truncating early history to recover capacity. Quarantine the affected stream from new activation/invocation as applicable, retain exact head/tail/version/storage evidence, run complete audit-chain verification, and recover only from a verified snapshot/event prefix or platform recovery point whose continuity can be proved. PITR can restore storage but does not become the canonical audit ledger. ## 11. Rollback @@ -353,11 +361,14 @@ This canonical operability document does not duplicate every command. Use: Runtime health/exchange, readiness/security state, maintenance/development workflows, external-extension admission/lifecycle, procedural graph advisory/session/screening and lifecycle-gated projection, authenticated procedural evaluator handoff, #597 State / Checkpoint evaluation/rejection history, #601 Policy / Approval CAS, and evidence scripts exist in protected source. Exact deployed revision is always live-verified rather than inferred from this document. Protected source does not by itself prove real-backend p95/recovery, live signer trust, current non-workflow lifecycle/revocation, publication-time cross-authority reconciliation, graph publication, immutable release, canary/rollback, product outcome, or deployment. +The `read_operability` source path adds a bounded Workflow / Task operability producer to the already-existing `NOEMA_WORKFLOW_STATE` adapter. Source integration establishes only this observation contract. `database_size_bytes` is not proof of deployed transaction compatibility, restart/recovery, representative storage growth, p95, PITR/rollback, or immutable release. + ### External / not yet proven by source - issue #27 enforced `main` governance; - issue #29 Maintainer/Reviewer App provisioning and activation; - production environment independent governance; +- actual Workflow / Task `NOEMA_WORKFLOW_STATE` deployment with representative transaction/restart/recovery behavior, exact-object storage-growth series and denominator, realistic synchronous-path p95, PITR/equivalent rollback and immutable release/deployment identity; - actual Durable Object external-extension lifecycle deployment, realistic current-projection/contended-append p95, partition/storage-growth evidence, snapshot rebuild and recovery rehearsal; - procedural graph released cross-service schema, live owner/Keyverse signer trust selection, non-workflow current-lifecycle/revocation authority, deployed State / Checkpoint and Policy / Approval compatibility/p95/recovery, fresh publication-time State / Checkpoint plus Policy / Approval reconciliation, graph publication, canary/rollback operation and product outcome improvement; - current production KPI/deployment/release acceptance; @@ -418,3 +429,16 @@ The protected procedural graph source has a deliberately short operating contrac 8. Before any graph publication or activation, require a released owner contract, live owner/Keyverse signer-trust selection, current non-workflow lifecycle/revocation evidence where applicable, deployed State / Checkpoint and Policy / Approval compatibility, a fresh publication-time read/reconciliation of both current authorities, and canary/rollback evidence. None of these may be inferred from a prior #601 CAS success. There is therefore no current procedural-graph production traffic, rollback metric or durability SLO to claim. A future activation change must add those operational evidence classes rather than retrospectively interpreting source/unit-test integration as production acceptance. + +## 20. Workflow / Task state operability acceptance + +The Workflow / Task execution state is already owned by the execution-scoped `NOEMA_WORKFLOW_STATE` Durable Object. An operability observation must therefore reuse that exact authority rather than create a second metrics/state store or scan another execution's storage. + +1. Re-admit the exact workflow plan before selecting the Durable Object and derive the same privacy-preserving execution-scoped object identity used by state mutations. +2. Treat `read_operability` as observation-only. Its success shape contains only `database_size_bytes`; caller-only fields and retained workflow/task payloads do not cross the private command boundary. +3. Reject an object-name mismatch before reading storage metadata. Do not reuse another execution object's byte count as a fallback. +4. Treat unavailable, throwing, negative, non-integer or otherwise non-canonical SQLite size metadata as storage unavailable. Do not manufacture zero or normalize malformed values into success. +5. Bind any production storage-growth claim to exact release/deployment/object identity, observation window and workload/retention denominator. A unit/fake byte count is source-semantic evidence only. +6. Verify deployed transaction behavior, restart/recovery, contention, synchronous-path p95, PITR/equivalent rollback and release provenance separately before ADR-0013 can become `Accepted`. + +The `read_operability` path supplies the source-level bounded observation contract but does not itself satisfy steps 5–6. Source integration cannot be cited as deployment, recovery, SLO or release evidence. diff --git a/docs/adr/0013-durable-workflow-execution-authority.md b/docs/adr/0013-durable-workflow-execution-authority.md index 8f639bac1..e49395913 100644 --- a/docs/adr/0013-durable-workflow-execution-authority.md +++ b/docs/adr/0013-durable-workflow-execution-authority.md @@ -3,7 +3,7 @@ - **Status:** Proposed - **Scope:** Agent Runtime / Workflow & Task Execution / State & Checkpoint / Recovery - **Supersedes:** none -- **Related:** ADR-0012, issue #541, active stacked PR #542 +- **Related:** ADR-0012, issue #541, protected source lineage from PR #542, operability extension PR #605 ## Context @@ -22,6 +22,7 @@ Noema owns this runtime execution authority. It does not own LLM provider routin - Runtime evidence must distinguish claim, effect start, completion, cancellation, recovery, blocked descendants and checkpoint commits without storing prompts, tool payloads, provider credentials, foreign domain data or security verdicts. - Provenance retained in the execution record must be bounded; durable execution state is not an unbounded audit warehouse. - One execution must resolve to one production serialization authority and one admitted plan identity before any repository mutation is attempted. Tests that serialize only an in-memory fake are insufficient deployment evidence. +- Operability observation must remain bound to that same execution-scoped object. Its success result may expose only bounded platform metadata required to produce later operational evidence and must not return workflow/task payloads, raw execution identity, foreign-owner truth or secrets. ## Considered options @@ -39,13 +40,15 @@ Rejected. It would create cross-service authority coupling or cross-service SQL ### Cloudflare Durable Object storage behind a Noema repository boundary -Selected for the current implementation candidate. It is already part of Noema's runtime technology, provides a transaction boundary, and can remain hidden behind the Noema-owned `DurableWorkflowStateRepository`. This decision is about the port and invariants, not permanent vendor lock-in; a future adapter may replace the storage technology while preserving the same domain/application contract. +Selected for the protected Workflow / Task execution authority. It is already part of Noema's runtime technology, provides a transaction boundary, and remains hidden behind the Noema-owned `DurableWorkflowStateRepository`. This decision is about the port and invariants, not permanent vendor lock-in; a future adapter may replace the storage technology while preserving the same domain/application contract. -The active implementation now adds the missing production composition. `workflowStateObjectName` validates the canonical execution identity and maps it to a SHA-256-derived `workflow:` Durable Object name. `routeWorkflowStateCommand` therefore sends every plan revision and scheduler caller for the same execution to the same `NOEMA_WORKFLOW_STATE` object. `NoemaWorkflowState` independently re-admits the plan, re-derives the expected object name, verifies it against the object's retained `DurableObjectState.id.name`, and admits authority-bearing checkpoint/claim data before delegating storage mutations to `DurableWorkflowStateRepository`. A command delivered through another execution's object identity, or through an unnamed object identity, fails closed before storage mutation. `src/runtime-entrypoint.ts` exports the class and `wrangler.toml` declares the `NOEMA_WORKFLOW_STATE` binding plus SQLite-backed `NoemaWorkflowState` export. Raw execution identity is not embedded in the Durable Object name. +The protected implementation maps `workflowStateObjectName` from a validated canonical execution identity to a SHA-256-derived `workflow:` Durable Object name. `routeWorkflowStateCommand` therefore sends every plan revision and scheduler caller for the same execution to the same `NOEMA_WORKFLOW_STATE` object. `NoemaWorkflowState` independently re-admits the plan, re-derives the expected object name, verifies it against the object's retained `DurableObjectState.id.name`, and admits authority-bearing checkpoint/claim data before delegating storage mutations to `DurableWorkflowStateRepository`. A command delivered through another execution's object identity, or through an unnamed object identity, fails closed before storage mutation. `src/runtime-entrypoint.ts` exports the class and `wrangler.toml` declares the `NOEMA_WORKFLOW_STATE` binding plus SQLite-backed `NoemaWorkflowState` export. Raw execution identity is not embedded in the Durable Object name. -Inside that execution-scoped object, the repository now retains an execution-scoped `workflow-state-plan-authority:v1:` record in the same initialization transaction as the plan-specific workflow state. The authority record binds the execution to exactly one `planId`; initialization of a second plan identity is rejected before another state record can become active. Every read and mutation requires this authority and still independently validates the retained workflow record against the complete admitted plan revision, including task dependencies. The existing plan-specific state key is retained as a storage-layout detail rather than as permission to run multiple plans for one execution. +Inside that execution-scoped object, the repository retains an execution-scoped `workflow-state-plan-authority:v1:` record in the same initialization transaction as the plan-specific workflow state. The authority record binds the execution to exactly one `planId`; initialization of a second plan identity is rejected before another state record can become active. Every read and mutation requires this authority and still independently validates the retained workflow record against the complete admitted plan revision, including task dependencies. The existing plan-specific state key is retained as a storage-layout detail rather than as permission to run multiple plans for one execution. -The private adapter currently uses an internal JSON `fetch` command boundary instead of making the Durable Object protocol part of Noema's public API. Cloudflare documents that Durable Objects do not receive requests directly from the Internet; callers require a Durable Object binding configured at upload time, so the `NOEMA_WORKFLOW_STATE` namespace binding is the current caller capability boundary rather than a public HTTP endpoint. Noema does not add a second shared-secret protocol inside that binding unless a future service/tenant trust boundary makes it necessary. Cloudflare's current invocation guidance says new projects, and existing projects with compatibility date `2024-04-03` or later, should prefer Durable Object RPC methods. That is a future adapter refinement, not authority to bypass the current repository contract or postpone the single-authority repair. A future RPC migration must preserve the same command validation, one-execution routing, failure mapping, tests, and rollback semantics. +The private adapter uses an internal JSON `fetch` command boundary instead of making the Durable Object protocol part of Noema's public API. Cloudflare documents that Durable Objects are reached through a configured binding, so the `NOEMA_WORKFLOW_STATE` namespace binding is the current caller-capability boundary rather than a public HTTP endpoint. Noema does not add a second shared-secret protocol inside that binding unless a future service/tenant trust boundary makes it necessary. Cloudflare's invocation guidance recommends RPC for newer compatibility dates; that remains an adapter refinement, not authority to bypass the repository contract. Any future RPC migration must preserve command validation, one-execution routing, failure mapping, tests and rollback semantics. + +The same private adapter also exposes `read_operability` as an observation-only operation. It reuses the admitted plan and exact hashed Durable Object identity, then requires the existing repository `readState(plan)` path to validate the current retained execution-plan authority and retained workflow record before reading platform storage metadata. Caller-only properties outside the canonical command schema are projected out before serialization; the admitted plan intentionally crosses the private boundary so the object can re-admit and verify execution/plan authority. A command routed to a different execution object, an uninitialized object, a mismatched retained plan, or malformed retained state conflicts before `DurableObjectStorage.sql.databaseSize` is read. An unreadable, negative, non-integer or otherwise non-canonical byte count is normalized to the existing storage-unavailable failure contract. The validated retained workflow state is never returned by this operation: the success result remains only `{ database_size_bytes }`, with no workflow/task payload or raw execution identity, and grants no mutation, retry, recovery, deployment or release authority. The value is a source-level evidence producer only; an in-memory/fake byte count is not production storage-growth evidence. ## Decision @@ -67,9 +70,9 @@ The state record retains a monotonic transition sequence and at most `MAX_TRANSI Legacy state records that predate the transition ledger remain readable only when the ledger is entirely absent. A partially present or malformed ledger fails closed. Missing historical effect-start evidence is exposed as unknown (`null`) rather than fabricated as false, so legacy side-effecting attempts without affirmative pre-effect evidence cannot be treated as safely replayable. -Retained bytes are not trusted merely because the Durable Object storage operation succeeded. A malformed root record, task vector/task record, checkpoint object, transition receipt, execution-plan authority, or other impossible retained state is classified as a `WorkflowStateConflictError`, not as `WorkflowStateStoreUnavailableError`. The latter is reserved for actual storage-operation failure. This distinction prevents durable data corruption from being presented to callers as a transient 503 that invites blind retry. +Retained bytes are not trusted merely because the Durable Object storage operation succeeded. A malformed root record, task vector/task record, checkpoint object, transition receipt, execution-plan authority, or other impossible retained state is classified as a `WorkflowStateConflictError`, not as `WorkflowStateStoreUnavailableError`. The latter is reserved for an unavailable storage operation or platform storage metadata required by an observation-only operation. This distinction prevents durable data corruption from being presented to callers as a transient 503 that invites blind retry. -The workflow-state Durable Object binding and this execution-plan authority are first introduced by the active Proposed change; there is no protected or released production workflow-state dataset to migrate. Candidate records created before the execution-plan authority existed are not silently trusted. Only exact-plan `initialize` may backfill a missing authority when the retained plan-specific record independently validates against the same admitted plan and checkpoint; ordinary reads/mutations fail closed while authority is absent. A different-plan candidate record is never promoted by that compatibility path. The first accepted deployment must not reuse ungoverned pre-merge candidate namespace data as production authority. +The workflow-state Durable Object binding and execution-plan authority are already present in protected source, but Noema still has no released production workflow-state dataset whose successful deployment can be inferred from source alone. Candidate records created before the execution-plan authority existed are not silently trusted. Only exact-plan `initialize` may backfill a missing authority when the retained plan-specific record independently validates against the same admitted plan and checkpoint; ordinary reads/mutations fail closed while authority is absent. A different-plan candidate record is never promoted by that compatibility path. The first accepted deployment must not reuse ungoverned pre-merge candidate namespace data as production authority. ## State and authority sequence @@ -105,6 +108,11 @@ sequenceDiagram R->>C: admit successor against retained checkpoint C-->>R: accepted/replay or conflict R-->>S: checkpoint_committed receipt or conflict + S->>O: read_operability(plan) + O->>R: readState(plan) / validate retained authority + R-->>O: current retained state validated or conflict + O->>O: read SQLite databaseSize + O-->>S: bounded database_size_bytes or fail closed ``` ## Consequences @@ -118,7 +126,8 @@ sequenceDiagram - Structurally corrupt retained state fails as a state conflict instead of masquerading as a transient storage outage. - Evidence size is bounded, so this ledger is suitable for operational provenance but not a substitute for a separately governed long-term audit/event store. - Adding an effect-start marker creates a caller obligation: production composition must persist it immediately before crossing the actual effect boundary. Merely exposing the method is not production acceptance. -- Durable Object routing is explicit deployment configuration rather than an implicit assumption in an in-memory test harness. The active PR still needs exact-head hosted/runtime-compatible execution before this becomes protected truth. +- Durable Object routing is explicit deployment configuration rather than an implicit assumption in an in-memory test harness. +- The observation-only storage-size command lets a deployed evidence producer bind later storage-growth measurements to the exact execution-scoped object without returning retained workflow state. Source/fake values alone still prove neither deployment nor a growth denominator. ## Risks and rejected shortcuts @@ -126,24 +135,28 @@ sequenceDiagram - A caller that crosses the external effect boundary without first persisting `effectStarted=true` violates the authority protocol and can make restart recovery unsafe; this ordering must remain an executable application-boundary invariant. - Treating `idempotent` as equivalent to `pure` during cancellation is unsafe: the effect may have changed external state even though a repeated invocation would converge to the same result. Cancellation must not invent that first invocation's outcome. - Durable Object transaction behavior must be verified in the deployed/runtime-compatible environment; a serialized in-memory backing store proves adapter composition but does not substitute for Cloudflare/workerd transaction and restart evidence. +- A `database_size_bytes` value from a fake or isolated unit test must not be promoted to deployed storage-growth, capacity or SLO evidence. Operational evidence needs exact release/deployment/object identity, observation window and workload/retention denominator. - An execution-plan revision is not implemented by creating another plan-specific record under the same execution. A future migration protocol must explicitly quiesce the prior plan, preserve recovery/checkpoint invariants, and atomically replace the execution-scoped plan authority. - A future RPC migration must not create a second authority path beside the private fetch adapter. One migration replaces the adapter only after parity tests and rollback evidence are present. - The transition ledger must not accumulate foreign payloads in future extensions. New receipt fields require a privacy/authority review. -- `queued` GitHub checks, predecessor-head results, or this ADR's existence do not make the implementation protected truth. +- `queued` GitHub checks, predecessor-head results, or this ADR's existence do not make an implementation or operational claim protected truth. ## Verification and acceptance -The current candidate is exercised by state-store tests for concurrent claims, checkpoint races, cancellation, bounded retry, blocked descendants, restart claim reconstruction and transition provenance. The cancellation regressions additionally require a started idempotent task to retain its exact running claim after cancellation until explicit reconciliation/outcome evidence exists, while preserving the existing safe cancellation path for work proven not to have crossed its effect boundary. The provenance regression requires distinct `task_claimed` and `effect_started` receipts and verifies bounded receipt retention. The application-runner regressions verify that durable claim and effect-start authority precede effect invocation, that effect-start persistence failure invokes no external effect, that a side-effecting claim proven unstarted can be recovered and re-claimed, and that an effect-started uncertain side effect remains running for explicit reconciliation rather than implicit retry. +Protected source is exercised by state-store tests for concurrent claims, checkpoint races, cancellation, bounded retry, blocked descendants, restart claim reconstruction and transition provenance. The cancellation regressions additionally require a started idempotent task to retain its exact running claim after cancellation until explicit reconciliation/outcome evidence exists, while preserving the existing safe cancellation path for work proven not to have crossed its effect boundary. The provenance regression requires distinct `task_claimed` and `effect_started` receipts and verifies bounded receipt retention. The application-runner regressions verify that durable claim and effect-start authority precede effect invocation, that effect-start persistence failure invokes no external effect, that a side-effecting claim proven unstarted can be recovered and re-claimed, and that an effect-started uncertain side effect remains running for explicit reconciliation rather than implicit retry. + +`test/workflow-state-durable-object-routing.test.ts` exercises the production adapter class and namespace routing contract: two concurrent routed side-effect claims for one execution must reach one object and produce one 200 winner plus one 409 conflict; distinct executions derive distinct hashed object names; commands delivered to a foreign or unnamed object identity must fail before durable mutation; all repository command families cross the private adapter; malformed plans/checkpoints/claims and unavailable storage fail closed. `test/workflow-state-durable-object-plan-authority.test.ts` additionally routes two plan identities for one execution through the same object and requires the second initialization and claim to conflict while the first plan remains readable. `test/workflow-state-store-plan-authority.test.ts` verifies malformed authority is a durable-state conflict rather than a retryable storage outage and that missing authority can be backfilled only by exact retained-plan reinitialization. `test/workflow-state-store-malformed-record-shape.test.ts` corrupts the retained root record, task vector, task entry, checkpoint, and transition receipt and requires each case to remain a state conflict rather than being normalized into storage-unavailable retry evidence. -`test/workflow-state-durable-object-routing.test.ts` exercises the production adapter class and namespace routing contract: two concurrent routed side-effect claims for one execution must reach one object and produce one 200 winner plus one 409 conflict; distinct executions derive distinct hashed object names; commands delivered to a foreign or unnamed object identity must fail before durable mutation; all repository command families cross the private adapter; malformed plans/checkpoints/claims and unavailable storage fail closed. `test/workflow-state-durable-object-plan-authority.test.ts` additionally routes two plan identities for one execution through the same object and requires the second initialization and claim to conflict while the first plan remains readable. `test/workflow-state-store-plan-authority.test.ts` verifies malformed authority is a durable-state conflict rather than a retryable storage outage and that missing authority can be backfilled only by exact retained-plan reinitialization. `test/workflow-state-store-malformed-record-shape.test.ts` corrupts the retained root record, task vector, task entry, checkpoint, and transition receipt and requires each case to remain a state conflict rather than being normalized into storage-unavailable retry evidence. These tests close the source-level binding/routing, parallel-plan, and malformed-retained-state classification gaps while leaving deployed workerd/Cloudflare transaction evidence as an exact-head acceptance requirement. +`test/workflow-state-operability-evidence.test.ts` exercises only the bounded operability extension: deterministic routing reaches the same hashed execution-scoped object; caller-only fields outside the command schema are projected out while the canonical admitted plan intentionally crosses the private boundary for re-admission; an uninitialized or foreign execution object is rejected before platform storage metadata is read; the returned success payload contains only `database_size_bytes`; and invalid or throwing SQLite metadata fails as storage unavailable. These tests establish source semantics only. They do not substitute for a deployed object, representative transactions, restart/recovery, contention, storage-growth, latency or release evidence. Before this ADR can become `Accepted`: -- the exact implementation head must pass repository typecheck/tests, owned production statement/branch coverage, review, security and applicable image/SBOM/provenance gates; +- the exact protected implementation must pass repository typecheck/tests, owned production statement/branch coverage, review, security and applicable image/SBOM/provenance gates; - production composition must use the declared `NOEMA_WORKFLOW_STATE` binding, the execution-scoped plan authority, and durable claim → effect-start evidence → effect/outcome under the exact claim; - restart/recovery and real Durable Object transaction behavior must have executable runtime-compatible acceptance evidence; -- PRD/TRD/Architecture/UML/TEST_STRATEGY/OPERABILITY/TRACEABILITY/CHANGELOG and the product technical gap baseline must describe the same boundary without presenting the active PR as protected truth; -- the stacked foundation must integrate normally and this work must be non-force restacked/revalidated against the resulting protected base. +- deployed storage-growth evidence must be sampled from the exact execution-scoped object under an identified workload/window using the bounded observation contract rather than namespace aggregates or synthetic/fake values; +- PRD/TRD/Architecture/UML/TEST_STRATEGY/OPERABILITY/TRACEABILITY/CHANGELOG and the product technical gap baseline must describe the same boundary without converting source capability into deployed/released evidence; +- immutable version/tag/package/SBOM/provenance/reproducibility and rollback evidence must bind the accepted deployment to the exact protected source. ## References @@ -151,4 +164,6 @@ Cloudflare. (2026). *Invoke methods*. Cloudflare Durable Objects documentation. Cloudflare. (2026). *Getting started*. Cloudflare Durable Objects documentation. https://developers.cloudflare.com/durable-objects/get-started/ -Cloudflare. (2026). *Durable Object Namespace*. Cloudflare Durable Objects documentation. https://developers.cloudflare.com/durable-objects/api/namespace/ \ No newline at end of file +Cloudflare. (2026). *Durable Object Namespace*. Cloudflare Durable Objects documentation. https://developers.cloudflare.com/durable-objects/api/namespace/ + +Cloudflare. (2026). *SQLite-backed Durable Object Storage*. Cloudflare Durable Objects documentation. https://developers.cloudflare.com/durable-objects/api/sqlite-storage-api/ diff --git a/src/workflow-task-execution/workflow-state-durable-object.ts b/src/workflow-task-execution/workflow-state-durable-object.ts index ecf939114..9a1c5bc99 100644 --- a/src/workflow-task-execution/workflow-state-durable-object.ts +++ b/src/workflow-task-execution/workflow-state-durable-object.ts @@ -30,6 +30,7 @@ const workflowTaskTerminalOutcomes = new Set([ const workflowStateOperations = new Set([ "initialize", "read", + "read_operability", "claim_next", "claim_runnable", "mark_effect_started", @@ -45,10 +46,16 @@ export interface WorkflowStateDurableObjectEnv { NOEMA_WORKFLOW_STATE: DurableObjectNamespace; } +/** Bounded storage observation from exactly one execution-scoped workflow-state authority. */ +export interface WorkflowStateOperabilitySnapshot { + readonly database_size_bytes: number; +} + /** Serializable command surface used only between Noema's scheduler adapter and its private Durable Object. */ export type WorkflowStateCommand = | { readonly operation: "initialize"; readonly plan: WorkflowTaskPlan; readonly checkpoint: ExecutionCheckpoint } | { readonly operation: "read"; readonly plan: WorkflowTaskPlan } + | { readonly operation: "read_operability"; readonly plan: WorkflowTaskPlan } | { readonly operation: "claim_next"; readonly plan: WorkflowTaskPlan; readonly claimId: string } | { readonly operation: "claim_runnable"; @@ -78,6 +85,7 @@ const workflowStateCommandPayloadFields: Readonly< > = Object.freeze({ initialize: ["checkpoint"], read: [], + read_operability: [], claim_next: ["claimId"], claim_runnable: ["taskId", "claimId"], mark_effect_started: ["claim"], @@ -97,7 +105,7 @@ const workflowStateNestedPayloadFields: Readonly(); + readonly sql: { databaseSize: number }; + + constructor(databaseSize = 4096) { + this.sql = { databaseSize }; + } + + async get(key: string): Promise { + return structuredClone(this.records.get(key)) as T | undefined; + } + + async put(key: string, value: T): Promise { + this.records.set(key, structuredClone(value)); + } + + async list(options: { prefix?: string; limit?: number } = {}): Promise> { + const prefix = options.prefix ?? ""; + const limit = options.limit ?? Number.POSITIVE_INFINITY; + return new Map( + [...this.records.entries()] + .filter(([key]) => key.startsWith(prefix)) + .sort(([left], [right]) => left.localeCompare(right)) + .slice(0, limit) + .map(([key, value]) => [key, structuredClone(value) as T] as const), + ); + } + + async transaction(callback: (txn: OperabilityStorage) => Promise): Promise { + return callback(this); + } +} + +class CapturingWorkflowNamespace { + readonly objects = new Map(); + readonly objectNames: string[] = []; + lastBody: string | null = null; + + idFromName(name: string): DurableObjectId { + this.objectNames.push(name); + return { name, toString: () => name } as unknown as DurableObjectId; + } + + get(id: DurableObjectId): DurableObjectStub { + const name = id.toString(); + let object = this.objects.get(name); + if (!object) { + object = new NoemaWorkflowState({ + id, + storage: new OperabilityStorage(), + } as unknown as DurableObjectState); + this.objects.set(name, object); + } + return { + fetch: async (input: RequestInfo | URL, init?: RequestInit) => { + this.lastBody = typeof init?.body === "string" ? init.body : null; + return object!.fetch(new Request(input, init)); + }, + } as unknown as DurableObjectStub; + } +} + +const digest = (character: string): string => character.repeat(64); + +const plan = (executionId = "exec-operability-001"): WorkflowTaskPlan => ({ + executionId, + planId: "plan-operability-001", + maxConcurrency: 1, + tasks: [{ taskId: "observe", dependsOn: [], effect: "pure" }], +}); + +const initialCheckpoint = (executionId = "exec-operability-001"): ExecutionCheckpoint => ({ + executionId, + sequence: 0, + stateDigest: digest("a"), +}); + +async function responseData(response: Response): Promise { + return (await response.json()) as T; +} + +async function initializeObject( + object: NoemaWorkflowState, + candidatePlan = plan(), +): Promise { + return object.fetch(new Request("https://noema-workflow-state.internal/command", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + operation: "initialize", + plan: candidatePlan, + checkpoint: initialCheckpoint(candidatePlan.executionId), + }), + })); +} + +describe("Workflow state Durable Object operability evidence", () => { + it("routes a bounded exact-object SQLite size observation only after retained execution authority exists", async () => { + const namespace = new CapturingWorkflowNamespace(); + const env = { + NOEMA_WORKFLOW_STATE: namespace as unknown as DurableObjectNamespace, + } satisfies WorkflowStateDurableObjectEnv; + const candidatePlan = plan(); + + expect((await routeWorkflowStateCommand(env, { + operation: "initialize", + plan: candidatePlan, + checkpoint: initialCheckpoint(), + })).status).toBe(200); + + const response = await routeWorkflowStateCommand(env, { + operation: "read_operability", + plan: candidatePlan, + secret: "must-not-cross-boundary", + } as const); + + expect(response.status).toBe(200); + expect(await responseData(response)).toEqual({ + ok: true, + data: { database_size_bytes: 4096 }, + }); + expect(new Set(namespace.objectNames)).toEqual(new Set([ + await workflowStateObjectName(candidatePlan.executionId), + ])); + expect(namespace.objectNames[0]).not.toContain(candidatePlan.executionId); + expect(namespace.lastBody).not.toBeNull(); + const privateCommand = JSON.parse(namespace.lastBody!) as Record; + expect(privateCommand).not.toHaveProperty("secret"); + expect(privateCommand).toHaveProperty("plan.executionId", candidatePlan.executionId); + }); + + it("rejects an exact-object operability observation before retained execution-plan authority exists", async () => { + const candidatePlan = plan(); + const object = new NoemaWorkflowState({ + id: { name: await workflowStateObjectName(candidatePlan.executionId) } as DurableObjectId, + storage: new OperabilityStorage(), + } as unknown as DurableObjectState); + + const response = await object.fetch(new Request("https://noema-workflow-state.internal/command", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ operation: "read_operability", plan: candidatePlan }), + })); + + expect(response.status).toBe(409); + expect(await responseData(response)).toEqual({ ok: false, error: "conflict" }); + }); + + it("rejects an operability observation routed to another execution authority", async () => { + const storage = new OperabilityStorage(); + const object = new NoemaWorkflowState({ + id: { name: await workflowStateObjectName("exec-operability-001") } as DurableObjectId, + storage, + } as unknown as DurableObjectState); + + const response = await object.fetch(new Request("https://noema-workflow-state.internal/command", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ operation: "read_operability", plan: plan("exec-operability-002") }), + })); + + expect(response.status).toBe(409); + expect(await responseData(response)).toEqual({ ok: false, error: "conflict" }); + }); + + it.each([-1, 1.5, Number.NaN])( + "fails closed when exact-object SQLite size is unavailable (%s)", + async (databaseSize) => { + const candidatePlan = plan(); + const objectName = await workflowStateObjectName(candidatePlan.executionId); + const object = new NoemaWorkflowState({ + id: { name: objectName } as DurableObjectId, + storage: new OperabilityStorage(databaseSize), + } as unknown as DurableObjectState); + + expect((await initializeObject(object, candidatePlan)).status).toBe(200); + const response = await object.fetch(new Request("https://noema-workflow-state.internal/command", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ operation: "read_operability", plan: candidatePlan }), + })); + + expect(response.status).toBe(503); + expect(await responseData(response)).toEqual({ ok: false, error: "storage_unavailable" }); + }, + ); + + it("fails closed when the SQLite size accessor throws", async () => { + const candidatePlan = plan(); + const storage = new OperabilityStorage(); + Object.defineProperty(storage.sql, "databaseSize", { + configurable: true, + get() { + throw new Error("storage metadata unavailable"); + }, + }); + const object = new NoemaWorkflowState({ + id: { name: await workflowStateObjectName(candidatePlan.executionId) } as DurableObjectId, + storage, + } as unknown as DurableObjectState); + + expect((await initializeObject(object, candidatePlan)).status).toBe(200); + const response = await object.fetch(new Request("https://noema-workflow-state.internal/command", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ operation: "read_operability", plan: candidatePlan }), + })); + + expect(response.status).toBe(503); + expect(await responseData(response)).toEqual({ ok: false, error: "storage_unavailable" }); + }); +});