fix(strix): treat LLM-backend-unavailable outcomes as neutral, not required-check failures - #349
Conversation
…eck failures
The Strix required check exits 1 (blocking merges) when its LLM backend is
unavailable — GitHub Models "Too many requests" rate limits or the
413 tokens_limit_reached token cap — even though no scan actually ran and no
vulnerability was found. That is a CI infrastructure outage masquerading as a
security finding.
The strix_quick_gate.sh gate returns exit 1 for BOTH genuine blocking
vulnerabilities and backend-unavailable fail-closed outcomes (with
STRIX_FAIL_ON_PROVIDER_SIGNAL=1). This wraps the "Run Strix (quick)" step to
capture the gate exit code and console output and disambiguate:
* exit 0 -> pass; exit 2 (config) / any non-1 code -> hard fail (unchanged)
* exit 1 -> neutral skip (exit 0 + ::warning) ONLY when a recognized
backend-unavailability signal is present AND the scan confirmed zero
vulnerabilities AND no vulnerability/severity was reported anywhere.
Otherwise the failure is preserved.
Real security gating is untouched: any reported finding, config error, or
non-backend failure still fails the check. The gate script and its test suite
are unchanged; the smoke-test contract (literal `bash "$TRUSTED_STRIX_GATE"`)
is preserved.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AxU2xaupAjp912oDNFuWyd
OpenCode Review Overview
Pull request overviewOpenCode reviewed the current-head bounded evidence and found no blocking issues. FindingsNo blocking findings. SummaryApproval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: strix.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: strix.yml"]
R1 --> V1["actionlint plus required checks"]
|
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode reviewed the current-head bounded evidence and found no blocking issues.
Findings
No blocking findings.
Summary
Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including .github/workflows/strix.yml.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports test coverage as not applicable because no supported changed source files or package manifests were found.
Docstring coverage: coverage execution evidence reports docstring coverage as not applicable because no supported changed source files or package manifests were found.
DAG: CodeGraph/source-backed behavior map connects .github/workflows/strix.yml to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source checks are delegated to configured OpenCode web_search/Context7/DeepWiki sources when applicable; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: Playwright visual, DOM locator, ARIA snapshot, console, and responsive evidence were checked when a web UI surface was present; for non-web surfaces, API/CLI/log/docs/workflow interaction evidence was reviewed instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.
- Result: APPROVE
- Reason: Improves workflow robustness by treating LLM-backend-unavailable outcomes as neutral
- Head SHA:
a0b9e9d7699204791085b32455a2bc14157bc51d - Workflow run: 29059595620
- Workflow attempt: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: strix.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: strix.yml"]
R1 --> V1["actionlint plus required checks"]
1 participant
Problem
The strix required check fails merges (exit 1) whenever its LLM backend is unavailable rather than when it actually finds a vulnerability:
openai.RateLimitError: Too many requests. For more on scraping GitHub...Error code: 413 ... tokens_limit_reached/Max size: 4000 tokensIn these cases no scan completed and zero vulnerabilities were reported, yet the required check goes red and blocks every dependent repo's PRs. This is a CI infrastructure outage masquerading as a security finding.
Root cause:
scripts/ci/strix_quick_gate.shreturns exit 1 for both genuine blocking vulnerabilities and backend-unavailable fail-closed outcomes (the workflow setsSTRIX_FAIL_ON_PROVIDER_SIGNAL: "1", so provider-signal failures fail closed). TheRun Strix (quick)step could not tell the two apart.Fix (minimal, workflow-only)
Wrap the
Run Strix (quick)step to capture the gate's exit code and console output, then disambiguate:Vulnerabilities 0and noseverity:/Vulnerabilities [1-9]appears anywhereexit 0with a::warningThe neutral path fires only when all three conditions hold, so it can never downgrade a run that reported a finding.
Why this does NOT weaken security gating
Vulnerabilities 0or the gate's own "reported zero vulnerabilities before provider infrastructure failure" message) and the absence of anyseverity:/ non-zeroVulnerabilitiesline. Any real finding keeps the failure.RateLimitError,RESOURCE_EXHAUSTED,tokens_limit_reached,Error code: 413,LLM CONNECTION FAILED, the gate's own "unavailable"/"provider infrastructure" verdicts) — bare numeric429/413were intentionally excluded to avoid spurious matches.strix_quick_gate.shand its test harness are not touched. The required-workflow smoke-test contract (literalbash "$TRUSTED_STRIX_GATE") is preserved and the smoke test still passes.Validation
python3 -c "import yaml; yaml.safe_load(open('.github/workflows/strix.yml'))"→ okbash scripts/ci/strix_required_workflow_smoke.sh→ passedbash -non the extracted step body → ok🤖 Generated with Claude Code