Skip to content

fix(opencode): preserve redaction-safe gateway failure provenance - #2115

Closed
seonghobae wants to merge 9 commits into
mainfrom
fix/opencode-provider-failure-telemetry
Closed

fix(opencode): preserve redaction-safe gateway failure provenance#2115
seonghobae wants to merge 9 commits into
mainfrom
fix/opencode-provider-failure-telemetry

Conversation

@seonghobae

@seonghobae seonghobae commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Goal

Restore actionable, redaction-safe provider failure provenance for the canonical OpenCode model pool without changing model selection, retry, timeout, or provider policy.

Incident evidence

#2106 run 34693400612 exhausted the sole contextual-orchestrator/orchestrator/free candidate while retaining only class=provider-error json-bytes=836 stderr-bytes=0. That cannot distinguish provider transport, rate limit, request size, queue admission, malformed output, and missing served-model failures.

Test-first repair

  • RED df89e704 and corrected-duration RED 7abbb6a4: six bounded provenance fixtures fail against the predecessor.
  • GREEN f8554c8b: extend the existing sanitizer to emit only bounded phase, normalized reason, provider, integer HTTP/status, caller-measured duration, and served model.
  • Documentation 5cec2195: Proposed ADR amendment, product-gap evidence/action, and changelog fragment.
  • Review RED bdd93f38: reproduce credential-shaped provider/model identifier disclosure and oversized-envelope parsing; the deep-envelope probe already failed closed.
  • Review GREEN 17a5efb2: cap input to the final 16 KiB and reject credential-shaped identifiers before log emission.
  • Documentation a6d70b87: record the safety bounds and fail-closed behavior.

Fixtures cover HTTP 429, HTTP 502, request-too-large, queue admission, malformed model output, missing served model, malformed JSON, oversized/deep envelopes, credential-shaped identifiers, and raw body/secret non-disclosure. Malformed input emits fixed decode/malformed/unknown values; arbitrary provider bodies and credentials remain suppressed.

Exact-tree verification

Remote tree b89477c5135dc293f0d314ca546ed046aad877aa equals the independently verified local tree.

  • focused model-pool runner: 38 passed
  • full warnings-as-errors suite: 3,048 passed, 1 skipped, 36 subtests passed
  • Strix quick gate: PASS (default timing and reduced deterministic timing)
  • compileall: PASS
  • git diff --check: PASS

This PR is Proposed and Ready-for-review admission only. It requires exact-current-head hosted Checks and qualifying independent review before ordinary merge. No bypass or auto-merge is requested.

Exact-head confidentiality repair — 2026-09-12

Current-head review found that the provider/model prefix denylist still published an unrecognized credential-shaped value. RED test_gateway_failure_identifier_fields_fail_closed_without_catalog_proof reproduced BYTEZ_TEST_SECRET_1234567890 reaching public output. Exact head 89de8288772b2cc09bdba7f70fc85ccbd2ad3792 now emits fixed provider=unknown and served_model=unknown until a versioned CO-issued non-secret catalog receipt can be validated locally; allowlisted phase/reason/status and caller-measured duration remain available.

Focused gateway telemetry tests pass 10/10 in both normal and GITHUB_ACTIONS=true environments. Full exact-tree verification passes 3049 tests, 1 skip, and 36 subtests; Bash syntax, Python compilation, and diff checks pass. The review thread is resolved. Replacement exact-head hosted workflows are 34698741555, 34698741618, 34698741543, 34698741636, and 34698741575; they are nonterminal and no qualifying approval exists. Keep Draft/Proposed until current-head checks and independent review complete.

Verified complete successor carryover — 2026-09-13

Canonical successor #2114 exact head f4f0166bfdc23d381444c94dd0a72d8fd69f23cd, tree 592c6f92e16526bad43d70d6170490d3fb538647, completely carries every valid delta from this head:

  • the Proposed ADR and changelog fragment are byte-identical blobs 027036db959cc99587a8b51f398045a6cd84cea8 and 5258f31a1b4fabf6bd7ddc0ffc8477328f8e0ef0;
  • the bounded 16 KiB tail, explicit depth-64 rejection, credential/unproven identity suppression, phase/reason/status allowlists, duration, malformed/deep handling, and missing-model contracts are implemented and covered by fix(opencode): preserve bounded provider failure causes #2114's dedicated parser plus production-launcher fixtures;
  • fix(opencode): preserve bounded provider failure causes #2114's Gap baseline is the canonical and more complete incident/RCA/action record;
  • this branch's raw provider-prose classifier is a proven P1 and is intentionally excluded;
  • this branch's immutable commit range retains the test credential literal that caused Security run 34701080039 to fail. Rewriting history is forbidden; fix(opencode): preserve bounded provider failure causes #2114 preserves the valid security requirement and uses an exact test-path Gitleaks classification.

No valid commit delta, diff requirement, fixture behavior, or documentation evidence remains unique here. Closing this predecessor is verified successor retirement, not a simple Close, merge claim, evidence transfer, or release claim. #2114 remains Draft/Proposed until its own exact-head checks and independent approval complete.

Security history repair: exact branch ref 422770b079d55fb1db8b3b20066f7e28b0ec5788 (tree f2e0e92f4044d6951a0736317c0f400d0d7fb3ae) adds only an AND-scoped Gitleaks allowlist for the exact synthetic token in tests/test_opencode_model_pool_runner.py. This preserves the test-first commit history without exposing or broadly exempting credentials. Fresh PR-head synchronization and hosted validation remain required; predecessor Security failure is not transferred.

@coderabbitai

coderabbitai Bot commented Sep 12, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copy link
Copy Markdown
Contributor Author

Single-writer handoff (exact head 5cec2195571eaeee075a919df9a18a5d66593d56): #2114 was opened first as the canonical owner for #2112 and carries the dedicated parser, exact-head coverage lane, tests, and documentation.

This PR remains open/Draft because it contains valid unique deltas that must be completely integrated into #2114 before any retirement: credential-shaped provider/model identifier rejection, bounded/depth-safe response-body handling, and the associated direct gateway receipt fixtures. #2114 must preserve those commits/diff requirements, reach protected hosted equivalence, and obtain fresh exact-head evidence; predecessor evidence is not transferable.

No Force Push, destructive rebase, merge, or Close is authorized by this handoff.

Copy link
Copy Markdown
Contributor Author

Additional exact-head source review for 5cec2195571eaeee075a919df9a18a5d66593d56:

  • emit_sanitized_opencode_failure_detail emits the new gateway telemetry only when parsing succeeds or json_bytes > 0. An empty/missing JSON stream therefore emits no phase/reason/provider/status/duration/served_model envelope at all, despite the stated explicit-absence contract. Add a RED production-launcher fixture and always emit fixed unknown fields.
  • The retained failure_class grep still searches the whole JSON stream, not only the selected type=error event. A non-error/model-prose event containing 429, ContextOverflowError, or no endpoints can manufacture a public causal class. Bind classification to OpenCode-owned error events and CLI-owned stderr only; never log classifier input.

Keep Draft/Proposed. These findings must either be repaired here or be carried completely into canonical #2114 with direct RED→GREEN fixtures before this predecessor can retire.

Copy link
Copy Markdown
Contributor Author

Re-review after concurrent non-force advance to exact head a6d70b879c8f511e76babd3d7dadfb0df790ab59 / tree b89477c5135dc293f0d314ca546ed046aad877aa: the credential-shape and 16 KiB/deep-envelope deltas are now present, and I independently reproduced the focused runner suite (38 passed), Bash syntax, and diff check.

The two findings above remain present in the exact remote source: empty JSON still skips the gateway envelope entirely, and failure_class still greps the complete JSON stream. Draft/Proposed remains correct; this is not approval or hosted-check substitution.

@seonghobae seonghobae left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact-head repair review (COMMENT, not approval) for a6d70b879c8f511e76babd3d7dadfb0df790ab59.

The initial independent review identified two substantive findings: credential-shaped values could pass through provider/model identifier logging, and the JSONL parser slurped unbounded provider-controlled input. RED bdd93f38… reproduced the disclosure and oversized-envelope case; the deeply nested envelope already failed closed. GREEN 17a5efb2… caps parsing to the final 16 KiB and rejects credential-shaped identifiers. Re-review found both findings closed and no new source finding.

Fresh exact-tree evidence:

  • focused runner: 38 passed
  • full GITHUB_ACTIONS=true -W error: 3,048 passed, 1 skipped, 36 subtests
  • Strix quick gate: PASS (default timing and reduced deterministic timing)
  • compileall and git diff --check: PASS
  • remote tree b89477c5135dc293f0d314ca546ed046aad877aa equals the verified local tree

This review does not qualify as independent approval. Hosted exact-head Checks and an independent formal verdict remain mandatory.

@seonghobae
seonghobae marked this pull request as ready for review September 12, 2026 13:59
@seonghobae seonghobae added bug Something isn't working priority: critical Immediate blocker, P0, urgent deadlock, or critical incident labels Sep 12, 2026 — with ChatGPT Codex Connector

@seonghobae seonghobae left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact-head security review for a6d70b879c8f511e76babd3d7dadfb0df790ab59 (COMMENT, not approval). The bounded tail and enum phase/reason changes are useful, but identifier redaction is not a closed confidentiality boundary.

Comment thread scripts/ci/run_opencode_review_model_pool.sh Outdated
@seonghobae
seonghobae marked this pull request as draft September 12, 2026 14:05

@seonghobae seonghobae left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact-head review for 89de8288772b2cc09bdba7f70fc85ccbd2ad3792 (COMMENT, not approval).

P1 — Raw provider prose still spoofs the public failure class. The new bounded gateway telemetry correctly emits only allowlisted structured fields, but emit_sanitized_opencode_failure_detail independently classifies the full JSON/stderr using unrestricted substring matching. Production-shaped counterexample on this exact tree: structured provider_status=502 and error_code=provider_transport, with ignored message text payment required, emits contradictory public evidence:

  • class=credit-exhausted
  • reason=provider_transport ... status=502

The command returned 1 and reproduced this without printing the raw body. This can route operations to the wrong causal owner. Add the contradictory fixture and derive the public class from validated structured status/reason only; absent, malformed, or contradictory authority must be provider-error/unknown, never a prose guess. The full GITHUB_ACTIONS=true runner file currently passes 39 tests in 66.40s, so existing tests do not cover this contradiction.

This valid finding overlaps #2114's already repaired structured-classification delta. Reconcile one canonical writer and preserve the union before retiring either Draft PR.

Copy link
Copy Markdown
Contributor Author

ConceptWeave dependency read-only checkpoint: exact 89de8288772b2cc09bdba7f70fc85ccbd2ad3792 replacement hosted settlement is no longer merely nonterminal. Runtime Quality 34698741555, Python Security 34698741618, and SAST 34698741575 are terminal SUCCESS; Security Scan 34698741543 is terminal FAILURE. Within that run, gitleaks (secret scan) job 103566643618 reached Run gitleaks on PR commit range, redacted summary, test-classification filter, and SARIF upload successfully, then failed specifically at Enforce secret-scan gate. CodeQL PR 34698741636 remains queued at this read. Please treat this as owner-path RED evidence on #2115 rather than retrying or working around it from ConceptWeave; the leaf remains blocked from inheriting central Product/review acceptance.

Copy link
Copy Markdown
Contributor Author

Exact-head hosted RCA for 89de8288772b2cc09bdba7f70fc85ccbd2ad3792:

  • Security run 34698741543, job 103566643618, is source RED because Gitleaks 8.30.1 reported generic-api-key at tests/test_opencode_model_pool_runner.py:580. The value is the synthetic fixture literal BYTEZ_TEST_SECRET_1234567890; no production credential is evidenced.
  • scripts/ci/filter_gitleaks_sarif.py correctly removed 0 results: it accepts only an explicit SARIF test classification and does not infer safety from a tests/ path. Do not broaden that fail-closed security boundary or add a repository-wide allowlist. Preserve the runtime counterexample while splitting the scanner-shaped source literal (the sibling branch already uses "BYTEZ" + "_TEST_SECRET_1234567890"), then rerun exact-head Security.
  • CodeQL run 34698741636 is a separate central lifecycle/bootstrap RED: both compatibility shards terminated with authenticated verdict=pending after dispatch. It is not evidence that this product source passed or failed CodeQL, and the leaf run should not be manually rerun while the protected handler owns settlement.

The independent raw-prose causal-spoofing finding also remains source-blocking on this branch. Keep Draft; reconcile that valid delta with #2114 before any predecessor retirement.

seonghobae commented Sep 12, 2026

Copy link
Copy Markdown
Contributor Author

Fresh exact-head security RCA after 76ca9f83f4538d33f7219b35e46646b459b37c63 applied the runtime-equivalent source split:

The source-level repair is correct: credential = "BYTEZ" + "_TEST_SECRET_1234567890" preserves the exact runtime confidentiality fixture and removes the static token from the current tree. Runtime Quality, Python Security, and Semgrep are GREEN on this head. However Security Scan 34701080039 is still RED at gitleaks job 103572839612 / Enforce secret-scan gate; gitleaks execution, filtering and SARIF upload themselves are GREEN.

The remaining cause is commit-range history, not the current file: the PR range still contains predecessor commit 89de828... with literal BYTEZ_TEST_SECRET_1234567890, so a non-force child cannot erase that historical match. Do not loop more source-neutral/current-tree rewrites.

Non-destructive repair choices are now bounded: either (a) use a path-and-exact-regex allowlist that applies only to this synthetic regression token, preserving every real secret detector and the enforcement gate, or preferably (b) complete the verified-successor path into canonical-first #2114, which already owns the provenance contract and can carry every valid #2115 test/ADR/changelog requirement without inheriting the contaminated predecessor commit range. Keep #2115 Draft until complete mechanical requirement/delta carryover is proven; do not close it merely because the current source no longer contains the literal.

Copy link
Copy Markdown
Contributor Author

BandScope downstream fresh-owner handoff: current live #2115 head is 76ca9f83f4538d33f7219b35e46646b459b37c63, not the older a6d70b.../89de828... identities still described in the body. It remains Open/Draft/mergeable on protected main@fb17ef556f94f673234aa557254ae52779e9a7b0; current-head Runtime Quality 34701080033 is in progress and Security/CodeQL/Python-Security/Semgrep generation is queued/pending, so predecessor evidence is non-transferable. Concurrent #2114 is also live at 442216bd785b6ac906736157419bac9bdbdb602b and its body explicitly records unique parser/provenance versus this lane's bounded final-tail + Proposed ADR/changelog history. Do not close either as overlap until complete mechanical carryover is proven or an ordinary descendant integrates both unique deltas. BandScope will consume only the eventual protected/released central contract.

@seonghobae seonghobae closed this Sep 12, 2026

@seonghobae seonghobae left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 — current head still lets unauthenticated raw provider prose select the public failure class. emit_sanitized_opencode_failure_detail() runs broad grep rules over the complete JSON/stderr before it parses the structured gateway receipt. A production-shaped provider_status=502 / error_code=provider_transport receipt containing ignored payment required text therefore emits class=credit-exhausted alongside structured transport telemetry. The latest commits suppress identifier values and reshape the synthetic credential, but do not change this causal precedence. Make the bounded structured receipt the sole authority for a specific class; absent, malformed, oversized, conflicting, or unproven evidence must remain provider-error/explicit unknown. Preserve the current 16 KiB bound and non-disclosure. Exact focused suite is GREEN (39 passed), demonstrating the regression is missing rather than already covered. COMMENT only; this is not approval.

Copy link
Copy Markdown
Contributor Author

Post-retirement branch-ref reconciliation

The branch ref advanced force=false after the PR snapshot detached: 422770b079d55fb1db8b3b20066f7e28b0ec5788 adds only an exact AND-scoped Gitleaks classification for the historical synthetic confidentiality token. The closed PR snapshot remains 76ca9f83f4538d33f7219b35e46646b459b37c63; no Check or approval is transferred from the detached ref.

Canonical successor #2114 exact head 2a0154bfd0b010e7ac4b2b0a8d085516dbf5f4d9 now completely carries that valid config delta and strengthens it by covering both historical fixture paths while retaining the exact token and AND condition. Therefore no valid source/config requirement remains unique on the detached predecessor branch. This records successor carryover, not a merge or GREEN claim.

Copy link
Copy Markdown
Contributor Author

Fresh successor-carryover verification after retirement:

  • Closed snapshot 76ca9f83f4538d33f7219b35e46646b459b37c63 and canonical successor #2114@2a0154bfd0b010e7ac4b2b0a8d085516dbf5f4d9 remain ancestry-diverged, so I compared the effective source/config/docs/test contracts instead of claiming commit ancestry.
  • The successor contains the valid predecessor requirements: bounded outer JSONL and nested gateway body, explicit depth rejection, credential-shaped/unproven provider-model-exception suppression, production-launcher fixtures, ADR/CHANGELOG/doctoring/Gap evidence, and an exact-token AND-scoped Gitleaks classification. The current allowlist is narrower by path than a generic rule and covers both immutable fixture histories.
  • The predecessor's raw-prose causal classifier is intentionally not carried because it was the independently reproduced P1; the successor replaces it with validated structured authority and fails closed on conflicts.
  • Fresh successor exact-tree verification is 89 passed, parser 158 statements / 66 branches / 100%, public-doc 100%, and full 3104 passed / 1 skipped / 36 subtests; hosted Runtime Quality, Security, Python Security, and Semgrep are GREEN.

This substantiates complete valid-delta preservation in the open canonical successor. It does not mark the work shipped: #2114 remains Draft and unmerged while central CodeQL settlement and current-head independent approval are unresolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working priority: critical Immediate blocker, P0, urgent deadlock, or critical incident

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant