superseded by #1263: make Azure and cross-provider fallbacks executable

.github/workflows/strix.yml

@@ -858,15 +858,24 @@ jobs:
             exit 0
           fi
 
+          # Exit 3 is emitted only by the trusted gate after its exact PR-scope
+          # mapper excludes every reported finding as unchanged and all model
+          # providers are unavailable. Vulnerability text remains in the audit
+          # log, so this cannot use the generic no-finding regex path below.
+          if [ "$strix_rc" -eq 3 ]; then
+            echo "::warning title=Strix backend unavailable after baseline scan::Strix excluded all reported findings from the pull-request diff, then exhausted its model providers. Treating this exact-head run as a neutral infrastructure skip; changed or unmapped findings still fail closed."
+            exit 0
+          fi
+
           # Preserve configuration failures (exit 2) and any unexpected exit
           # code as hard failures — only the scan-failure code (1) can be an
           # infrastructure/backend-unavailability outcome.
           if [ "$strix_rc" -ne 1 ]; then
             exit "$strix_rc"
           fi
 
-          # Recognized signals that the LLM backend was unavailable / starved.
-          backend_unavailable_signal='RateLimitError|Too many requests\. For more on scraping GitHub|exceeded your current quota|insufficient_quota|billing details|"status"[[:space:]]*:[[:space:]]*"RESOURCE_EXHAUSTED"|tokens_limit_reached|Request body too large|Max size:[[:space:]]*[0-9]+[[:space:]]+tokens|Error code:[[:space:]]*413|LLM CONNECTION FAILED|Could not establish connection to the language model|LLM warm-up failed|Configured model and fallback models were unavailable|Configured Vertex model and fallback models were unavailable|emitted provider infrastructure or failure-signal output|before provider infrastructure failure|litellm(\.exceptions)?\.NotFoundError[^[:cntrl:]]*Nvidia_nimException[^[:cntrl:]]*Error code:[[:space:]]*404'
+          # Recognized signals that the scanner or LLM backend was unavailable.
+          backend_unavailable_signal='RateLimitError|Too many requests\. For more on scraping GitHub|exceeded your current quota|insufficient_quota|billing details|"status"[[:space:]]*:[[:space:]]*"RESOURCE_EXHAUSTED"|tokens_limit_reached|Request body too large|Max size:[[:space:]]*[0-9]+[[:space:]]+tokens|Error code:[[:space:]]*413|LLM CONNECTION FAILED|Could not establish connection to the language model|LLM warm-up failed|Configured model and fallback models were unavailable|Configured Vertex model and fallback models were unavailable|emitted provider infrastructure or failure-signal output|before provider infrastructure failure|litellm(\.exceptions)?\.NotFoundError[^[:cntrl:]]*Nvidia_nimException[^[:cntrl:]]*Error code:[[:space:]]*404|RuntimeError:[[:space:]]*loginAsGuest failed after [0-9]+ attempts:[^[:cntrl:]]*curl exit 7:[^[:cntrl:]]*Failed to connect to 127\.0\.0\.1 port 48080'
           # Any evidence that a vulnerability was actually reported. Its presence
           # forces a hard failure so real findings are NEVER downgraded. Keep the
           # severity branch anchored away from identifiers so environment lines
@@ -879,7 +888,7 @@ jobs:
           # from blocking current-head merge progress.
           if grep -Eiq "$backend_unavailable_signal" "$strix_run_log" \
             && ! grep -Eiq "$reported_vulnerability_signal" "$strix_run_log"; then
-            echo "::warning title=Strix backend unavailable::Strix could not complete because its LLM backend was unavailable (rate limit / token cap / connection or warm-up failure) before producing a vulnerability report. Treating as a neutral skip so an infrastructure outage does not block merges; genuine findings still fail the check. See the strix-reports artifact and the run log."
+            echo "::warning title=Strix infrastructure unavailable::Strix could not complete because its scanner bootstrap or LLM backend was unavailable before producing a vulnerability report. Treating as a neutral skip so an infrastructure outage does not block merges; genuine findings still fail the check. See the strix-reports artifact and the run log."
             exit 0
           fi
 

CHANGELOG.md

@@ -22,6 +22,13 @@ Semantic Versioning where the repository publishes a release.
 
 ### Changed
 
+- Preserve the trusted PR-scope decision when Strix finds only unchanged-file
+  baseline issues and then exhausts every LLM provider, so provider retirement
+  or outage is a neutral infrastructure result; recognize the exact local Caido
+  bootstrap connection failure the same way only when no vulnerability exists,
+  while changed and unmapped findings and all other runtime failures still fail
+  closed. The reproduced incidents and gate boundary are recorded in
+  `docs/doctoring/strix-pr-baseline-provider-exhaustion.md`.
 - Emit completed repository pull-list requests as they finish in the five-minute
   agent-mention sweep, while retaining the four-worker ceiling, rotation, and
   exact-name dispatch ledger, so one slow repository cannot hide ready sibling

docs/doctoring/strix-pr-baseline-provider-exhaustion.md

@@ -0,0 +1,41 @@
+# Strix PR baseline/provider-exhaustion incident
+
+## Observed failure
+
+LineageWeave PR 392 run `32530198775` reported a critical secret in the
+nonexistent `frontend/src/config.ts`. The trusted changed-file mapper correctly
+classified that report as unchanged, but later fallback attempts ended in
+provider HTTP 410 retirement brownouts. The gate retained the earlier severity
+rank and returned the same exit code used for changed-file findings, so the
+outer workflow could not distinguish the cleared baseline report from a real
+pull-request vulnerability.
+
+A later central PR run reported zero vulnerabilities and then failed before
+scanning when Strix's local Caido process did not accept connections on
+`127.0.0.1:48080`; the outer workflow did not yet recognize that exact scanner
+bootstrap outage as infrastructure.
+
+## Root cause and repair
+
+The quick gate already owns the exact PR-head changed-file mapping decision.
+After it has classified every report as `allow_baseline`, a later provider
+exhaustion now returns the dedicated status 3. The trusted reusable workflow
+maps only that status to a neutral infrastructure warning. Changed, unmapped,
+or manifest findings still return the blocking status, and configuration or
+unexpected statuses still fail closed.
+
+The outer workflow also recognizes only the observed `loginAsGuest` retry
+exhaustion with curl exit 7 against Strix's fixed local Caido port. It is neutral
+only when no positive vulnerability or severity signal exists; every other
+runtime failure remains blocking.
+
+## Verification
+
+- A three-attempt regression reproduces an unchanged critical report followed
+  by two provider failures and requires status 3.
+- Existing source tests require changed findings to remain blocking and clean
+  unchanged findings to remain admissible.
+- A workflow regression requires the exact Caido bootstrap outage to be neutral
+  with zero findings and blocking when any vulnerability is reported.
+- The central Python suite, native workflow validation, Bash syntax checks, and
+  the complete Strix shell regression suite run on the final tree.

scripts/ci/strix_quick_gate.sh

@@ -4040,6 +4040,18 @@ run_current_target_scan() {
 		return 1
 	fi
 
+	# The trusted PR-scope mapper has already proved these reports belong only
+	# to unchanged files. Keep that decision distinct from a real changed-file
+	# finding when every remaining provider is unavailable.
+	if [ "$PR_FINDINGS_DECISION" = "allow_baseline" ] && [ "$INFRA_ERROR_DETECTED" -eq 1 ]; then
+		if is_vertex_model "$PRIMARY_MODEL"; then
+			echo "Configured Vertex model and fallback models were unavailable after unchanged-file findings were excluded." >&2
+		else
+			echo "Configured model and fallback models were unavailable after unchanged-file findings were excluded." >&2
+		fi
+		return 3
+	fi
+
 	local threshold_rank
 	threshold_rank="$(severity_rank "$STRIX_FAIL_ON_MIN_SEVERITY")"
 	if [ "${STRIX_MAX_SEVERITY_RANK:--1}" -ge "$threshold_rank" ]; then

scripts/ci/test_strix_quick_gate.sh

@@ -4650,6 +4650,28 @@ EOS
 		echo "Penetration test failed: baseline critical finding"
 		exit 1
 		;;
+	pr-baseline-provider-exhausted)
+		case "${STRIX_LLM:-}" in
+		vertex_ai/baseline-primary)
+			mkdir -p "$STRIX_REPORTS_DIR/fake-pr-baseline-provider/vulnerabilities"
+			cat >"$STRIX_REPORTS_DIR/fake-pr-baseline-provider/vulnerabilities/vuln-0001.md" <<'EOS'
+Severity: CRITICAL
+Location 1:
+sync-module-system/smart-crawling-biz/src/main/java/org/empasy/sync/modules/system/service/impl/SysUserServiceImpl.java:5
+EOS
+			echo "litellm.APIConnectionError: GeminiException LLM CONNECTION FAILED after baseline report"
+			exit 1
+			;;
+		vertex_ai/fallback-one | vertex_ai/fallback-two)
+			echo "litellm.APIConnectionError: GeminiException LLM CONNECTION FAILED: provider unavailable"
+			exit 1
+			;;
+		*)
+			echo "Error: pr-baseline-provider-exhausted unexpected model (${STRIX_LLM:-})" >&2
+			exit 35
+			;;
+		esac
+		;;
 	pr-critical-changed)
 		mkdir -p "$STRIX_REPORTS_DIR/fake-pr-changed/vulnerabilities"
 		cat >"$STRIX_REPORTS_DIR/fake-pr-changed/vulnerabilities/vuln-0001.md" <<'EOS'
@@ -6276,6 +6298,28 @@ run_filtered_gate_case_if_requested() {
 			"Materialized PR-head changed-file scope" \
 			"repository_dispatch"
 		;;
+	pr-baseline-provider-exhausted)
+		run_gate_case "pr-baseline-provider-exhausted" \
+			"vertex_ai/baseline-primary" \
+			"vertex_ai/fallback-one vertex_ai/fallback-two" \
+			"3" \
+			"Configured Vertex model and fallback models were unavailable after unchanged-file findings were excluded." \
+			"3" \
+			"vertex_ai/baseline-primary|vertex_ai/fallback-one|vertex_ai/fallback-two" \
+			"<unset>|<unset>|<unset>" \
+			"vertex_ai" \
+			"__DEFAULT__" \
+			"" \
+			"0" \
+			"CRITICAL" \
+			"0" \
+			"" \
+			"" \
+			"1200" \
+			"0" \
+			"pull_request" \
+			"sync-module-system/smart-crawling-biz/src/main/java/org/empasy/sync/modules/system/controller/SysPositionController.java"
+		;;
 	*)
 		record_failure "unknown STRIX_TEST_CASE_FILTER '${STRIX_TEST_CASE_FILTER:-}'"
 		;;
@@ -10909,6 +10953,27 @@ run_gate_case "pr-baseline-critical-unchanged" \
 	"pull_request" \
 	"sync-module-system/smart-crawling-biz/src/main/java/org/empasy/sync/modules/system/controller/SysPositionController.java"
 
+run_gate_case "pr-baseline-provider-exhausted" \
+	"vertex_ai/baseline-primary" \
+	"vertex_ai/fallback-one vertex_ai/fallback-two" \
+	"3" \
+	"Configured Vertex model and fallback models were unavailable after unchanged-file findings were excluded." \
+	"3" \
+	"vertex_ai/baseline-primary|vertex_ai/fallback-one|vertex_ai/fallback-two" \
+	"<unset>|<unset>|<unset>" \
+	"vertex_ai" \
+	"__DEFAULT__" \
+	"" \
+	"0" \
+	"CRITICAL" \
+	"0" \
+	"" \
+	"" \
+	"1200" \
+	"0" \
+	"pull_request" \
+	"sync-module-system/smart-crawling-biz/src/main/java/org/empasy/sync/modules/system/controller/SysPositionController.java"
+
 run_gate_case "pr-baseline-critical-absolute-target" \
 	"openai/gpt-4o-mini" \
 	"" \

tests/test_strix_nvidia_nim_not_found_fallback.py

@@ -243,6 +243,16 @@ def test_outer_workflow_never_neutralizes_reported_vulnerabilities(self) -> None
             )
         )
 
+    def test_outer_workflow_neutralizes_only_empty_caido_bootstrap_failure(self) -> None:
+        """Treat the exact scanner-local Caido outage as infrastructure only."""
+
+        caido_error = (
+            "RuntimeError: loginAsGuest failed after 10 attempts: curl exit 7: "
+            "curl: (7) Failed to connect to 127.0.0.1 port 48080\n"
+        )
+        self.assertTrue(_workflow_neutralizes(caido_error + "Vulnerabilities 0\n"))
+        self.assertFalse(_workflow_neutralizes(caido_error + "Vulnerabilities 1\n"))
+
     def test_workflow_neutralizes_only_nvidia_404_without_findings(self) -> None:
         """Retain the static fail-closed vulnerability evidence contract."""
 
@@ -256,6 +266,16 @@ def test_workflow_neutralizes_only_nvidia_404_without_findings(self) -> None:
             workflow,
         )
 
+    def test_workflow_accepts_trusted_unchanged_finding_provider_exhaustion(self) -> None:
+        """Keep gate-mapped baseline findings separate from provider exhaustion."""
+
+        workflow = STRIX_WORKFLOW.read_text(encoding="utf-8")
+        self.assertIn('if [ "$strix_rc" -eq 3 ]; then', workflow)
+        self.assertIn(
+            "Strix excluded all reported findings from the pull-request diff",
+            workflow,
+        )
+
 
 if __name__ == "__main__":
     unittest.main()