Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
- Removed the repository-owned 900-second repair deadline and duplicate model repair call from Noema. The GitHub Actions caller now issues one structured-output request while `contextual-orchestrator` owns repair/failover/timeouts.
- Hardened serving-model telemetry against control-character/workflow-command injection and lone-surrogate encoding failures, restored actionable exact changed-line diagnostics, and constrained local trailing-comma repair to complete JSON values.
- Added permanent single-request/no-fixed-timeout regressions and retired obsolete deadline/retry fixtures.
- Documented the RCA boundary for the historical Noema 900-second repair deadline and distinguished it from the three 900-second sandboxed test-command limits in `opencode-review-dispatch.yml`; future telemetry must retain phase and failure class for request-too-large, discovery, rate-limit, provider transport, malformed-output, stale-head, and sandbox-command failures.

# Changelog

Expand Down
44 changes: 26 additions & 18 deletions docs/doctoring/noema-model-output-repair-boundary.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,41 @@
# Noema model-output repair boundary

## Incident
## Current contract (2026-09-02)

On 2026-09-01 the required Noema review for `ContextualWisdomLab/naruon#1505` reached deterministic verdict validation, rejected an adversarial-probe `outcome` outside the closed `falsified|confirmed` domain, then spent the repair path on a long second model call that ultimately surfaced only `HTTP 502 Bad Gateway`. That final transport symptom erased the more informative first trusted-validator failure from the top-level diagnostic.
`.github` owns pull-request review orchestration, exact-head evidence, deterministic verdict validation, and publication. `contextual-orchestrator` owns provider discovery, capability routing, the `orchestrator/free` pool, structured-output repair, failover, and provider completion.

## Decision
After `.github#1672` merged as `a28fc2f4e185df7847e2f2f5f6ec561d1e84805d`, Noema issues exactly one structured-output request for a review. The repository caller no longer performs a second model repair request and no longer installs a 900-second process-level repair deadline. There is no caller-owned fixed inference wall-clock deadline or sampling-temperature override; gateway/provider completion and the outer workflow lifecycle remain separate concerns.

1. Model-produced JSON/envelope/schema/semantic-contract failures are `NoemaModelOutputError`; they remain fail-closed and are not consumer-source findings.
2. The primary review keeps the accepted contextual-orchestrator no-fixed-inference-timeout contract. The *single corrective attempt* is different: it repairs an already-completed verdict and therefore has one 900-second process-level wall-clock deadline across open/read/decode/validation. It deliberately does not use `urllib`'s renewable socket-operation timeout.
3. A corrective transport failure is `NoemaTransportError` and carries the sanitized first validator diagnostic plus the later transport exception class/status. Raw model output is never copied into public Actions diagnostics.
4. Exact-head validation before retry and before publication remains mandatory. All model traffic remains on contextual-orchestrator `orchestrator/free`.
The gateway response is still validated locally. A malformed or semantically invalid response fails closed with a bounded diagnostic containing the phase, elapsed duration, stable failure category, and served-model metadata when available. Raw model output and credentials are not written to Actions logs.

## Historical incident and the 900-second distinction

On 2026-09-01, `ContextualWisdomLab/html4tree` reached the old Noema corrective path after malformed JSON. The old caller then reported `NoemaRepairDeadlineExceeded` after a 900-second absolute wall-clock boundary. That boundary belonged to the superseded caller-side repair implementation; it is not a current Noema inference policy.

The same incident family also exposed real upstream failures: HTTP 413 `request_too_large`, Bytez discovery HTTP 500, NVIDIA timeout/429/404 responses, and malformed structured output. These are different failure classes and must remain visible as separate telemetry events rather than being collapsed into a generic timeout.

Three `timeout --kill-after=20 900` commands remain in `opencode-review-dispatch.yml`. They cap individual untrusted test-measurement shell commands in the coverage evidence job. They are not model requests, not Noema repair, and not a 900-second GitHub job timeout. Operational logs should describe them as sandbox command containment (for example, `sandbox_command_limit_seconds=900`) so an operator cannot mistake them for inference termination.

## Diagnostic and concurrency invariants

1. Model-produced JSON, envelope, schema, and semantic-contract failures remain fail-closed and are not consumer-source findings.
2. Every provider attempt reports a phase such as connecting, reading, decoding, or validating, its elapsed duration, a stable failure category, and the served model if known. Provider status classes such as 413, 429, 500, and 502 are retained as categories without copying provider secrets or raw model output.
3. The triggering pull-request head is checked before model work and again before publication. A push to the same PR makes the old head obsolete; the old run must not publish a verdict or spend a second repair call.
4. All model traffic for required review remains on contextual-orchestrator `orchestrator/free` and is subject to its discovery, capability, failover, and privacy policy.
5. A workflow shell timeout is evidence about that shell command only. It must never be used as evidence that the gateway or provider ended inference.

## Verification

The #1617 regression first proved RED because `NoemaModelOutputError` did not exist. The repair adds focused cases for malformed-verdict typing, malformed-then-502 evidence preservation with the 900-second repair-only timeout, and repeated malformed output remaining typed and non-passing. The repository full coverage/docstring gate is run before the one-shot repair workflow commits the result.
The merged #1672 regression suite proves one gateway request, no caller-side retry/deadline/sampling machinery, sanitized model telemetry, strict local validation, bounded trailing-comma normalization, and exact changed-line diagnostics. A fresh exact-head Actions run is still required to establish hosted runtime evidence; queued or cancelled checks do not count as a pass.

## References
Incident replay acceptance requires the log to distinguish at least: request_too_large, discovery_failure, rate_limited, provider_transport, malformed_model_output, stale_head, and sandbox_command_timeout. Each category must include phase and duration, while raw response bytes, credentials, and unbounded provider text remain excluded.

Fielding, R., Nottingham, M., & Reschke, J. (2022). *HTTP semantics* (RFC 9110). Internet Engineering Task Force.
## References

Python Software Foundation. (2026). *urllib.request — Extensible library for opening URLs*. Python 3 documentation.
Fielding, R., Nottingham, M., & Reschke, J. (2022). HTTP semantics (RFC 9110). Internet Engineering Task Force.

Python Software Foundation. (2026). urllib.request — Extensible library for opening URLs. Python 3 documentation.

## Actionable diagnostic boundary

Corrective prompts need the deterministic *class* of a malformed verdict to repair it,
but do not need arbitrary model-produced values. Trusted structural validator messages
(such as a missing required field or an invalid adversarial-probe outcome class) remain
available after secret scrubbing. Unsupported decision values and unknown model-output
text are redacted to stable diagnostics, and a repeated invalid-model exception is raised
without retaining the raw model exception as an explicit cause. Tests use a sentinel value
to prove it reaches neither the retry prompt nor the final diagnostic.
Corrective prompts, when implemented by the gateway, may use the deterministic class of a malformed verdict but do not need arbitrary model-produced values. Trusted structural validator messages remain available after secret scrubbing. Unsupported decision values and unknown model-output text are represented by stable diagnostics, and raw model exceptions are not retained as public causes.
12 changes: 7 additions & 5 deletions docs/product-technical-gap-baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -2616,12 +2616,14 @@ Higgins, S. S., Crepalde, N., & Fernandes, L. (2021). Segmented multiplexity: A

## Noema single-request model-control ownership — PR #1672 (2026-09-02)

**Status:** Proposed / exact-head verification required before merge.
**Status:** Merged into protected `main` as `a28fc2f4e185df7847e2f2f5f6ec561d1e84805d`; fresh exact-head hosted evidence remains an operational acceptance item.

**Root cause.** Noema duplicated `contextual-orchestrator` structured-output repair by making a second model request and wrapped that request in an unmeasured 900-second repository wall-clock deadline. This created a self-hosting admission failure: the required review could terminate valid long inference using policy that the gateway already owns.
**Root cause.** Noema duplicated contextual-orchestrator structured-output repair by making a second model request and wrapped that request in an unmeasured 900-second repository wall-clock deadline. This created a self-hosting admission failure: valid long inference could be terminated by a policy that the gateway already owns.

**Context Map / responsibility boundary.** `.github` owns CI review orchestration, exact-revision evidence, deterministic verdict validation and publication. `contextual-orchestrator` owns provider discovery, capability routing, `orchestrator/free`, structured-output repair/failover and provider completion. No provider/model-specific fallback or caller wall-clock timeout crosses that boundary.
**Context Map / responsibility boundary.** `.github` owns CI review orchestration, exact-revision evidence, deterministic verdict validation, and publication. `contextual-orchestrator` owns provider discovery, capability routing, `orchestrator/free`, structured-output repair/failover, and provider completion. No provider/model-specific fallback or caller wall-clock timeout crosses that boundary.

**Action.** Replace recursive caller repair with one structured-output gateway request; remove fixed deadline/signal machinery and sampling temperature; retain exact-head checks before and after model work; sanitize serving-model telemetry; restore exact changed-line diagnostics; retain bounded non-heuristic evidence cardinality and strict local JSON parsing.
**Action delivered.** The recursive caller repair and fixed deadline/signal machinery were removed. Noema now sends one structured-output request, keeps exact-head checks before and after model work, sanitizes serving-model telemetry, restores exact changed-line diagnostics, and retains bounded non-heuristic evidence cardinality with strict local JSON parsing.

**Evidence / acceptance.** Permanent tests forbid retry/deadline/sampling symbols and prove one gateway request, one attempt annotation, control-character-safe telemetry, missing-value rejection, valid trailing-comma normalization, and exact changed-line guidance. Fresh exact-head repository checks/reviews remain the admission authority; predecessor-head evidence is not transferable.
**900-second clarification.** The historical `NoemaRepairDeadlineExceeded` from the html4tree incident came from the retired caller repair path. The three literal `timeout --kill-after=20 900` invocations still present in `opencode-review-dispatch.yml` are separate containment limits for untrusted test-measurement commands; they are not model or Noema inference timeouts. Telemetry and runbooks must report the command class and phase separately.

**Evidence / acceptance.** Permanent tests forbid retry/deadline/sampling symbols in the caller and prove one gateway request, one attempt annotation, control-character-safe telemetry, missing-value rejection, valid trailing-comma normalization, and exact changed-line guidance. Fresh exact-head repository checks and reviews remain the admission authority; predecessor-head evidence is not transferable. The remaining runtime work is to preserve distinct `request_too_large`, discovery, rate-limit, provider transport, malformed-output, stale-head, and sandbox-command-timeout categories in hosted logs.
Loading