fix(strix): sanitize benign MODEL QUALITY WARNING banner before infra-failure detection

[seonghobae]

Problem

Strix prints a box-drawn MODEL QUALITY WARNING startup disclaimer for models outside its hardcoded recommendation list. The literal heading can satisfy the generic provider-failure matcher and turn a complete zero-finding scan into a false non-passing result.

Exact repair

Current exact head: 095422b40e9abd22b985bf94aa0a57ae85b6e72f (fix(strix): preserve raw last-attempt evidence).

The classifier removes only the exact box-content heading line after normalizing ANSI CSI color sequences for comparison. It no longer deletes the surrounding box. A timestamp, arbitrary prefix, target output, or any line that is not the authenticated exact heading remains unsuppressed and fail-closed.

Raw and classification evidence are now separated:

• RAW_STRIX_LOG remains the immutable per-attempt console transcript;
• STRIX_CLASSIFICATION_LOG is a private copy used for sanitization and failure classification;
• gate-last-attempt.log is published from the untouched raw transcript;
• gate-attempts/ remains raw; and
• real provider/fatal/warning evidence before or inside the banner box remains fail-closed.

TDD and verification

RED evidence:

• parent bc194048… erased same-box console/report provider warnings and returned exit 0;
• parent c38a61f… published a gate-last-attempt.log that had lost the raw MODEL QUALITY WARNING line.

GREEN on the current tree:

• five focused classification controls plus raw last/per-attempt artifact assertions;
• 1406 passed, 1 skipped, 16 subtests passed;
• full scripts/ci/test_strix_quick_gate.sh PASS using the hosted 3s/5s timeout-fixture budget;
• bash -n and git diff --check PASS;
• unresolved review threads: 0 after source-backed dispositions.

The earlier hosted run 32803460871 is predecessor evidence only. Current-head hosted workflows must regenerate.

Current governance state

This body is not approval or merge evidence. Do not admin-merge, bypass protection, or treat queued/predecessor checks as passing. Merge eligibility requires terminal exact-current-head workflows, a qualifying formal review on this head, and reconciliation with the latest protected-main Strix model contract owned separately by #1316.

[coderabbitai]

📝 Walkthrough

Walkthrough

Strix 보고서와 콘솔 로그에서 MODEL QUALITY WARNING 제목만 제거합니다. 같은 박스에 있는 provider 실패 신호는 유지합니다. 로그 정제 시 ANSI 이스케이프도 처리합니다. 관련 실패 폐쇄 시나리오를 추가합니다.

Changes

Strix 경고 처리

Layer / File(s)	Summary
보고서 및 콘솔 로그 정제 scripts/ci/strix_quick_gate.sh	로그를 줄 단위로 처리합니다. ANSI 이스케이프를 제거한 뒤 MODEL QUALITY WARNING 제목만 필터링합니다. 다른 박스 내용과 provider 경고는 유지합니다.
실행 연결 및 실패 신호 검증 scripts/ci/strix_quick_gate.sh, scripts/ci/test_strix_quick_gate.sh	run_strix_once가 실패 신호 판정 전에 콘솔 로그를 정제합니다. 같은 박스의 provider 경고가 보존되고 실패 폐쇄가 적용되는 보고서 및 콘솔 시나리오를 추가합니다.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: 🟡 Moderate · up to c38a6

The sanitizer is narrowly scoped, but the current implementation may alter the source console log before archiving it, weakening failure diagnosis and auditability. Merge should wait until archives are generated from an untouched log copy.

Suggested reviewers: actions

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 1 files. (1 skipped: 1…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	제목은 Strix의 benign MODEL QUALITY WARNING 배너를 정리하여 인프라 실패 감지 오류를 수정하는 핵심 변경을 정확히 설명합니다. 변경 목적과 영향이 명확하고 관련 없는 내용이 없습니다.

Full details: Docstring Coverage

Explanation

Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 1 files. (1 skipped: 1 too large.)

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/strix-model-quality-banner-false-fail-closed

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

Note on the failing `strix` check here: it's the exact bug this PR fixes, reproducing on itself. `strix_quick_gate.sh` resolves its "trusted Strix source" from `main` (by design — a PR can't be allowed to rewrite its own security gate to auto-pass), so this PR's own scan still runs the pre-fix script and hits the same MODEL QUALITY WARNING false fail-closed described above (confirmed in the job log: clean "Vulnerabilities 0" summary, then "Strix run emitted provider infrastructure or failure-signal output; failing closed."). This is a chicken-and-egg case — the fix can't pass its own gate until it's merged.

Every other check has passed: CodeQL, Semgrep (SAST + OSS), gitleaks, trivy-fs, dependency-review, SBOM generation, pip-audit, osv-scan, Scorecard, Devin Review, CodeRabbit, coverage-evidence/coverage-source-tree, noema-review, and opencode-review. Full local regression run of `test_strix_quick_gate.sh` (120 cases, including the new red/green-verified `console-model-quality-warning-banner-sanitized` case) also passes with no other-scenario regressions.

Flagging for whoever has merge authority here, consistent with the same one-off-intervention pattern used for prior self-referential CI-pipeline fixes in this repo.

[seonghobae]

Fresh downstream canary for this exact false-fail boundary (2026-08-25 KST): canonical Security Scan owner PR ContextualWisdomLab/.github#897 is still open at exact head 6406b4663b216e4a85fbf96a7cbd33bf9c0a990e on protected main@613a33e0cb1c6db9790fae99f6253445712ac37a. Its current Strix run/job 32710533347/97549503711 reaches a complete fallback scan with 0 vulnerabilities, then rejects that completed result with Strix run emitted provider infrastructure or failure-signal output; failing closed. Current protected-main scripts/ci/strix_quick_gate.sh still uses the broad Fatal|Denied|Warn|Warning|Timeout report matcher while sanitize_known_strix_report_warnings() does not remove the benign MODEL QUALITY WARNING banner. This is therefore not only the TEPP/fast-mlsirm reproduction in this PR body; it is directly blocking #897, which is the canonical Security Scan exact-head repair needed by ContextualWisdomLab/scopeweave#1222.

Acceptance after this repair integrates: rerun unchanged/current #897. A complete authoritative fallback that reports zero findings must no longer be rejected solely because of the model-quality disclaimer, while genuine provider/incomplete-scan signals and real vulnerability findings must remain fail-closed. Once #897 is green on its unchanged exact head and its exact-head binding fix integrates, regenerate ScopeWeave #523/#553 Security evidence against their then-current contributor heads. Later fallback/provider-routing errors (for example direct-provider 404/rate-limit paths) remain separate owner lanes; this comment is specifically the reproduced benign-warning false-failure boundary.

[seonghobae]

Second live canary from the other canonical ScopeWeave #1222 owner path: ContextualWisdomLab/.github#941 is currently at exact head f5d3224404a9dcd04a1b4e1650c51bc256876586 on protected main@613a33e0cb1c6db9790fae99f6253445712ac37a; current Strix run/job 32718660538/97549126961 checks out that exact PR head correctly. In that job, the NVIDIA fallback nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 reaches Penetration test completed and Vulnerabilities 0 (No exploitable vulnerabilities detected), but the trusted gate immediately rejects the attempt with Strix run emitted provider infrastructure or failure-signal output; failing closed. The same attempt prints the benign MODEL QUALITY WARNING banner handled by this PR. The chain then unnecessarily falls through to openai-direct/gpt-5.6-luna, which separately returns 404.

This matters directly to ScopeWeave because #941 is the canonical SAST Semgrep owner needed to repair the synthetic-checkout defect tracked by .github#1222. Acceptance for this PR therefore includes rerunning unchanged/current #941 after integration: the completed zero-vulnerability NVIDIA fallback must be accepted rather than rejected solely by the model-quality disclaimer, so a clean completed attempt does not fall through to a later broken provider. Genuine rate-limit/incomplete-scan/provider failures and real vulnerability findings must remain fail-closed; the direct-OpenAI 404 remains a separate owner lane if it is reached for a legitimately incomplete earlier attempt.

[seonghobae]

Current-head repair note after verifying the two unresolved Devin findings against 7c6d80c27b7be017d46560c7b4cc0f7fa2714288 (base/protected main still 613a33e0cb1c6db9790fae99f6253445712ac37a; current hosted workflows are green): both findings are valid and should be fixed before treating this owner PR as review-ready.

1. Fail-closed preservation bug in the banner matcher. Both new Python regexes begin at the first ╭─...╮ and allow arbitrary intervening lines before MODEL QUALITY WARNING. If another box precedes the model-quality box, the substitution can consume that preceding box and text between it and the warning box. That can erase a real Fatal/Denied/Warning/Timeout signal before deterministic fail-closed classification. Smallest robust remedy: parse/remove one complete box at a time (opening ╭ through its matching first closing ╰) and delete the block only when that same block contains MODEL QUALITY WARNING; do not use a cross-box wildcard. RED acceptance: prepend a separate box containing a genuine provider-failure signal, then the benign model-quality box, followed by an otherwise clean zero-finding result. Sanitization must remove only the benign box and the gate must still fail on the preserved genuine signal. Add the equivalent structured-report regression as well as the console regression.

2. Audit-artifact fidelity. preserve_attempt_log archives the raw transcript, but sanitize_strix_console_log mutates $STRIX_LOG in place; the cleanup trap later publishes that mutated file as gate-last-attempt.log. Keep the detector input sanitized without destroying the raw last-attempt artifact (for example, classify a sanitized temporary/in-memory copy, or publish the corresponding raw preserved attempt). RED acceptance: successful non-frontier-model attempt must leave gate-last-attempt.log byte-faithful to the raw console including the disclaimer, while deterministic classification ignores only the benign banner and gate-attempts/ remains raw.

After those regressions are GREEN, rerun the full Strix gate suite and then the live #897/#941 canaries. This preserves the intended false-fail repair without creating a fail-open erasure path or weakening evidence provenance.

[cursor]

Exact-head reproduction from #1314@3f0c3ac5c018770c32a80663a28c1f499ddfa1ba (Strix run 32792844719).

Trusted-base Strix on that current-main loopback PR failed closed after:

1. Primary nvidia_nim/nvidia/nemotron-3-super-120b-a12b RateLimitError 429.
2. Fallback nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 completed with Vulnerabilities 0, then strix_quick_gate.sh classified the static MODEL QUALITY WARNING banner as provider infrastructure/failure-signal output.
3. openai-direct/gpt-5.6-luna returned 404 page not found.

That is the same catch-22 this PR describes: a clean 0-finding scan on the org NIM models cannot pass while the trusted base still treats the banner WARNING substring as infra failure. #1314 cannot land until same-head Strix succeeds, and same-head Strix executes the trusted base gate — not #1314's tree — so folding this sanitizer into #1314 would not repair that check.

This comment is not OpenCode approval and not merge evidence for #1311. #1311's own required strix failure is expected until this gate is on protected main. Quality/Noema/security Checks on 7c6d80c27b7be017d46560c7b4cc0f7fa2714288 are not re-validated here. Wait for current-head OpenCode App APPROVE on #1311; do not treat the stub opencode-review check as that approval.

View PR

  

_{Sent by Cursor Automation: Fix Issues}

[seonghobae]

Revalidated after protected main moved to a724582 (direct push, gpt-5.6-luna → gpt-5.4 rename only). That tip does not sanitize the MODEL QUALITY WARNING banner and does not clear the NIM LLM_API_BASE for openai-direct models.

Same-head reproduction after that rename was already queued: #1300 Strix attempt 2 (32718111634, started 00:55Z on the pre-rename workflow) still did 49b Vulnerabilities 0 → banner classified as infra → openai-direct/gpt-5.6-luna 404 → STRIX_PROVIDER_UNAVAILABLE. #1314@3f0c3ac5 remains the same catch-22.

Local filtered proof on this PR's exact head 7c6d80c27b7be017d46560c7b4cc0f7fa2714288:
STRIX_TEST_CASE_FILTER=console-model-quality-warning-banner-sanitized bash scripts/ci/test_strix_quick_gate.sh → exit 0.

This comment is not OpenCode approval and not merge evidence. Required trusted-base Strix on this PR (and on every public .github PR) will keep failing until this sanitizer is the gate on protected main. Folding it into #1314 would not change that check. After it is on main, retry same-head Strix on #1314@3f0c3ac5 (do not push a successor SHA there just to rerun).

[seonghobae]

Revalidated 2026-08-25T01:32Z. Head still 7c6d80c2. This comment is not OpenCode approval and not merge evidence.

Hosted Strix Changed Path Quality CI 32748314314 is success on this exact SHA. Required Strix remains the expected trusted-base catch-22.

git merge --no-commit origin/main onto this head is a clean auto-merge. Combined tree keeps sanitize_strix_console_log and the a724582 openai-direct/gpt-5.4 pins in strix.yml.

Land hazard: a724582 already has test_strix_quick_gate.sh asserting openai/gpt-5.4 in opencode-review-dispatch.yml while that workflow still has openai/gpt-5.6-luna (workflow_has_luna=True, test_asserts_gpt54=True). Merging this PR without also fixing that pair will make the next Strix Changed Path Quality CI fail on the successor SHA. Fix (a) the workflow needle or (b) the test needle in the same land. Do not fold that into #1314.

After this sanitizer is the gate on main, retry same-head Strix on #1314@3f0c3ac5.

[seonghobae]

Correction 2026-08-25T01:34Z, not merge evidence. Org-queue-sweep 32794227373 completed cleanly and did not inspect this PR. The org walk explicitly skips ContextualWisdomLab/.github; the one review dispatch went to ContextualWisdomLab/OriginWeave#45. Same-repo MAX_PRS=100 + CREATED_AT ASC still never reaches this head. Operator land path unchanged: this sanitizer plus the a724582 luna/gpt-5.4 needle pair on main.

[seonghobae]

Same-head Strix re-run after a724582, not merge evidence.

Required run 32748314313 job 97653261171 (started 01:37:01Z on exact head 7c6d80c2) passed Self-test Strix required workflow contract and proceeded to Run Strix (quick). Overlay strix.yml on this branch still has openai-direct/gpt-5.6-luna, which matches the trusted smoke on main. That is the opposite of #1316@c3eb391b, whose overlay is gpt-5.4 and whose required Strix failed at that smoke.

The gate executing this scan is still the trusted pre-sanitizer script from main. A clean 0-finding NIM result can still fail closed on the MODEL QUALITY WARNING banner. This re-run does not land the sanitizer and is not OpenCode approval.

Operator path remains: land these two files onto protected main (direct push if required Strix stays catch-22). Do not fold into #1314. After the sanitizer is the trusted gate, retry same-head Strix on #1314@3f0c3ac5.

[seonghobae]

Follow-up from #1316@e4f9865a required Strix 32799128752 (not merge evidence for this PR).

After #1318, trusted smoke passes on current-main gpt-5.4 overlays. The scan then failed closed on primary NIM 429 → 502 without invoking 49b or openai-direct/gpt-5.4. Last-attempt NVIDIA 502 Upstream request failed is not is_model_retryable_error, so run_current_target_scan takes the fail-closed branch even when distinct fallbacks are configured.

Landing this sanitizer still matters for clean 0-finding NIM scans. It does not by itself make that 502 path retry fallbacks. Do not fold a 502 classifier into #1314.

[seonghobae]

Same-head required Strix 32748314313 job 97653261171 finished FAILURE at 02:19:21Z (started 01:37:01Z). This is not OpenCode approval and not merge evidence. The executing gate is still the trusted pre-sanitizer script on main.

Observed on exact head 7c6d80c2:

• Smoke passed (luna overlay matched pre-fix(strix): align required-path smoke fallback assertions with the gpt-5.4 contract #1318 smoke at job start).
• Primary nemotron-3-super-120b-a12b produced Vulnerabilities 0 and also NIM 429. The gate printed Strix report artifacts emitted warning/fatal/denied/timeout output; failing closed (banner WARNING in report .log) and retried the same model 3/3.
• Fallback llama-3.3-nemotron-super-49b-v1.5 completed a gray-box assessment with Vulnerabilities 0 / No exploitable vulnerabilities detected, then fail-closed on the same banner: Strix run emitted provider infrastructure or failure-signal output.
• Next fallback openai-direct/gpt-5.6-luna (this branch's overlay) died in 4s. Final annotation: STRIX_PROVIDER_UNAVAILABLE / zero vulnerabilities before provider infrastructure failure.

That is the live TEPP#214-class false fail-closed this PR repairs. The 49b scan was complete and clean; the trusted matcher treated the cosmetic banner as infra. Landing these two files onto main (direct push if required Strix stays catch-22) is what lets a later same-head 0-finding NIM fallback count as evidence.

Do not fold into #1314.

[seonghobae]

Revalidated merge onto current protected main 8fd471a (after #1318). Not OpenCode approval and not merge evidence.

git merge --no-commit origin/main onto 7c6d80c2: clean auto-merge of scripts/ci/test_strix_quick_gate.sh (no conflict). Combined tree keeps:

• sanitize_strix_console_log + console-model-quality-warning-banner-sanitized
• strix.yml / smoke / test workflow needles on openai-direct/gpt-5.4 (from a724582 + fix(strix): align required-path smoke fallback assertions with the gpt-5.4 contract #1318)

opencode-review-dispatch.yml on that merge tree still has openai/gpt-5.6-luna in OPENCODE_MODEL_CANDIDATES (pre-existing on main; #1316 carries the pool alignment). Landing these two files does not worsen that mismatch.

The 02:19Z same-head Strix run is the live 0-vuln banner false fail-closed. Operator path is unchanged: land these two files onto main (same style as #1318 if required Strix stays catch-22). Do not fold into #1314.

[seonghobae]

Second live reproduction, not merge evidence. #941 required Strix 32718660538 (job 97654998253, finished 02:26:52Z) is the same trusted-gate failure:

• Fallback llama-3.3-nemotron-super-49b-v1.5 reported Vulnerabilities 0.
• Gate: Strix run emitted provider infrastructure or failure-signal output; failing closed (banner WARNING in the log).
• Next fallback openai-direct/gpt-5.6-luna died in 3s.
• Final: zero vulnerabilities before provider infrastructure failure.

Same class as this PR's own 32748314313 and TEPP#214. Two hub PRs in one hour discarded a completed 0-finding 49b scan on the cosmetic banner. Land these two files onto main (direct push if required Strix stays catch-22). Do not fold into #1314.

[seonghobae]

Do not admin-merge or direct-push this exact head 7c6d80c27b7be017d46560c7b4cc0f7fa2714288. This comment is not OpenCode approval and not merge evidence.

Local proof just now that the current banner regex is fail-open. The matcher is:

╭─[^\n]*╮\n(?:[^\n]*\n)*?[^\n]*MODEL QUALITY WARNING[^\n]*\n(?:[^\n]*\n)*?╰─[^\n]*╯

A preceding complete box that contains a genuine Fatal: LLM CONNECTION FAILED signal, followed by the benign MODEL QUALITY WARNING box, is consumed as one match. After sub("") the Fatal text is gone. Classification then sees only Vulnerabilities 0 and would pass.

The cleanup trap also copies the already-mutated $STRIX_LOG to gate-last-attempt.log, so the published last-attempt artifact is not byte-faithful to the raw console. gate-attempts/ stays raw because preserve_attempt_log runs first.

Those are the two Devin findings from 2026-08-24T19:56Z; they are still unfixed on this SHA. Live 0-finding NIM scans this hour still prove the false fail-closed this PR intends to repair. That repair must be box-scoped (delete only a box that itself contains MODEL QUALITY WARNING) and must classify a sanitized copy while publishing the raw last-attempt log.

Protected main is still 8fd471a. After #1318, do not retry same-head Strix on #1314@3f0c3ac5 (luna overlay vs trusted gpt-5.4 smoke). Do not land these two files as-is.

[seonghobae]

Revalidated the new head bc19404852f6908ce8b95c06ebfdac1f21929bbd. This comment is not OpenCode approval and not merge evidence.

The successor regex is box-bounded. Local proof just now against the exact compiled pattern in scripts/ci/strix_quick_gate.sh:

• Preceding complete box with Fatal: LLM CONNECTION FAILED, then a cosmetic MODEL QUALITY WARNING box, then Vulnerabilities 0: Fatal text remains; banner is removed.
• Banner-only: banner removed; Vulnerabilities 0 remains.
• Hosted exact-head-path-policy 97668848899 succeeded on this SHA, including console-model-quality-warning-preserves-prior-failure.

Still do not admin-merge this SHA.

1. sanitize_strix_console_log still writes back to $STRIX_LOG (after preserve_attempt_log). gate-last-attempt.log is therefore sanitized, not byte-faithful. gate-attempts/ stays raw.
2. Overlay strix.yml on this branch still emits openai-direct/gpt-5.6-luna. Trusted smoke on main 8fd471a wants gpt-5.4.

Parent 7c6d80c remains fail-open. Land #1314@0ef296dcb88d2740bb390b181bfd0fe49979ac84 instead (box-scoped strip, classification copy, raw last-attempt log, plus the loopback SSRF gate). Hosted quality and test_strix_quick_gate.sh are already green on that exact SHA.

[devin-ai-integration]

Devin Review found 1 new potential issue.

[seonghobae]

Superseded by #1314 at exact head 6b06a9239adce28532cd2d225a8f346184592f04; closing this duplicate without merge.

Fresh source/test verification shows #1314 now contains the current raw-evidence contract from this PR as well as its bounded SSRF repair: gate-last-attempt.log and gate-attempts/ retain the raw console, classification reads a private sanitized copy, cosmetic-only MODEL QUALITY WARNING output is ignored, and same/prior-box Fatal, provider warning, and rate-limit evidence remains fail-closed. Five focused sanitizer controls, shell syntax, and diff integrity completed successfully on the exact #1314 tree; its hosted exact-head substantive workflows are terminal success and both PRs have zero unresolved threads.

This closure does not transfer approval or merge evidence. #1314 still requires a qualifying exact-current-head formal review under live governance.