Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions .github/workflows/strix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -269,11 +269,21 @@ jobs:
echo "::error::Strix target repository must belong to ContextualWisdomLab."
exit 1
fi
is_private="$(gh api "repos/${TARGET_REPOSITORY}" --jq '.private')"
is_private=""
for target_visibility_attempt in 1 2 3 4 5 6; do
if is_private="$(gh api "repos/${TARGET_REPOSITORY}" --jq '.private')"; then
break
fi
is_private=""
if [ "$target_visibility_attempt" -lt 6 ]; then
echo "Repository visibility lookup failed (attempt ${target_visibility_attempt}/6), possibly a transient GitHub API rate limit; retrying after backoff." >&2
sleep "$(( target_visibility_attempt * 5 ))"
fi
Comment thread
seonghobae marked this conversation as resolved.
done
Comment thread
seonghobae marked this conversation as resolved.
case "$is_private" in
true | false) ;;
*)
echo "::error::Target repository visibility did not resolve to true or false."
echo "::error::Target repository visibility did not resolve to true or false after retries."
exit 1
;;
esac
Comment thread
seonghobae marked this conversation as resolved.
Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ Semantic Versioning where the repository publishes a release.

### Fixed

- Retried the Strix scan up to `STRIX_TRANSIENT_RETRY_PER_MODEL` times, same model, when the log shows the upstream strix-agent Caido sandbox bootstrap timing race (`loginAsGuest failed after N attempts` / `Failed to connect to 127.0.0.1 port <port>`; tracked upstream as usestrix/strix#1036, #1037, #1056). A slow CI runner can exceed strix-agent's fixed 10-attempt sandbox-login budget before its local intercepting proxy is reachable, even though the penetration test itself never started and no vulnerability evidence was produced or lost; the Docker image is already cached from the failed attempt, so a same-model retry is cheap and typically clears the one-off boot race. Not wired into cross-model fallback, since switching LLM models cannot change local sandbox container boot timing.
- Replaced nonexistent `job.workflow_repository` / `job.workflow_sha` / `job.workflow_ref` / `job.workflow_file_path` context references (actionlint: "property ... is not defined in object type") in `pr-review-fix-scheduler.yml`'s called-workflow source verification and `exact-artifact-sbom-attestation.yml`'s trusted-verifier checkout. Both always failed closed on the missing properties (ContextualWisdomLab/.github#1212) or, for the SBOM attestation checkout, silently resolved an empty repository/ref instead of the pinned trusted source (downstream `gh attestation verify --signer-repo`/`--signer-workflow`, using the separately hardcoded `SIGNER_REPOSITORY` constant rather than any workflow_ref, still failed closed on the resulting empty signer identity). `github.workflow_ref`/`github.workflow_sha` are real, documented properties, but for a `workflow_call` target they reflect the top-level *calling* workflow, not the reusable workflow's own file — a prefix match against the reusable workflow's own path can never succeed. `exact-artifact-sbom-attestation.yml`'s checkout now uses `github.workflow_sha` (correct today: it has no callers yet); `pr-review-fix-scheduler.yml`'s identity check instead validates `github.repository`, since every current caller uses a local, same-repo `uses: ./...` where caller and callee share one commit and `github.workflow_sha` is still the right pin. Tracked follow-up for the SBOM attestation checkout once a real (potentially cross-repo) caller exists: ContextualWisdomLab/.github#1228.
- Used the receiving repository's workflow token for same-repository scheduler
Actions inventory and read calls, while retaining the established mutation
Expand Down Expand Up @@ -80,6 +81,7 @@ Semantic Versioning where the repository publishes a release.
- Bind reusable scheduler implementation to the validated called-workflow repository, SHA, ref, and file path, and verify the checked-out commit before executing privileged scheduler logic.
- Removed the ambiguous central-repository schedule fallback that could scan `.github` instead of Clearfolio when no external variable was configured; the active product caller now names Clearfolio explicitly while the reusable engine retains caller and dispatch overrides.
- Corrected the conflict-ordering regression contract to select the conflict-specific snapshot and verification after the ordinary path adopted the same trusted helper.
- Retried the Strix target-repository visibility lookup up to six times with linear backoff before failing closed, matching the existing PR-head-fetch retry convention in the same workflow. A single transient `gh api` failure (observed as a shared GitHub App installation token hitting its hourly rate limit while dozens of org repositories run hourly review schedulers concurrently) previously failed the entire required Strix check immediately, blocking otherwise mergeable, fully reviewed pull requests fleet-wide with no code defect involved.

### Security

Expand Down
37 changes: 33 additions & 4 deletions scripts/ci/strix_quick_gate.sh
Comment thread
seonghobae marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -169,9 +169,11 @@ import sys
root = Path(sys.argv[1])
known_internal_warning = re.compile(
r"^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d+ WARNING "
r"[^ ]+ - strix\.core\.execution: agent [0-9a-f]+ produced "
r"non-lifecycle final output in non-interactive mode; forcing tool "
r"continuation \(\d+/\d+\): "
r"[^ ]+ - strix\.core\.execution: agent [0-9a-f]+ "
r"(?:"
r"produced non-lifecycle final output in non-interactive mode"
r"|ended a turn without a lifecycle tool call \(interactive=False\)"
r"); forcing tool continuation \(\d+/\d+\): "
)


Expand Down Expand Up @@ -2662,11 +2664,12 @@ is_nvidia_nim_not_found_error() {

## Determines whether the last strix failure is a transient error eligible
## for same-model retry (up to STRIX_TRANSIENT_RETRY_PER_MODEL times).
## Four error families qualify:
## Five error families qualify:
## - RateLimit / RESOURCE_EXHAUSTED / HTTP 429
## - litellm API connection failures with LLM-provider evidence
## - litellm service-unavailable / high-demand provider failures
## - MidStreamFallbackError (litellm mid-stream provider switch)
## - Caido bootstrap timing failures (guest login before the local proxy is up)
## Timeouts are infrastructure failures. In strict CI mode they fail closed;
## otherwise the caller may still move to fallback model evaluation.
is_transient_same_model_retry_error() {
Expand All @@ -2686,6 +2689,9 @@ is_transient_same_model_retry_error() {
if is_midstream_fallback_error; then
return 0
fi
if is_caido_bootstrap_timing_error; then
return 0
fi
Comment thread
seonghobae marked this conversation as resolved.
return 1
}

Expand Down Expand Up @@ -2747,6 +2753,8 @@ run_strix_with_transient_retry() {
retry_reason="LLM service unavailable"
elif is_midstream_fallback_error; then
retry_reason="midstream fallback"
elif is_caido_bootstrap_timing_error; then
retry_reason="Caido sandbox bootstrap timing"
fi
echo "Retrying model '$model' due to $retry_reason (attempt $((attempt + 1))/$max_attempts)." >&2
sleep "$STRIX_TRANSIENT_RETRY_BACKOFF_SECONDS"
Expand Down Expand Up @@ -2924,6 +2932,23 @@ is_midstream_fallback_error() {
return 1
}

## Detects the upstream strix-agent Caido sandbox bootstrap timing race
## (usestrix/strix#1036, #1037, #1056): the sandbox container runs a
## chown -R before starting caido-cli, and a slow CI runner can exceed
## Strix's fixed 10-attempt loginAsGuest budget before the proxy is
## reachable, even though the penetration test itself never started. This
## is local sandbox/container timing, not model-specific, so it is
## same-model-retry-eligible only; switching LLM models would not change
## how long the sandbox takes to boot.
is_caido_bootstrap_timing_error() {
if grep -Fq 'loginAsGuest failed after' "$STRIX_LOG" &&
grep -Eq 'Failed to connect to 127\.0\.0\.1 port [0-9]+' "$STRIX_LOG"; then
return 0
fi

return 1
}
Comment thread
seonghobae marked this conversation as resolved.
Comment thread
seonghobae marked this conversation as resolved.

# Narrower variant: LLM providers only, excluding HTTP transport libraries
# (httpx, httpcore, requests). Used for generic transport failures where
# library names alone are insufficient to prove the timeout/connection error
Expand Down Expand Up @@ -2976,6 +3001,10 @@ has_detected_infrastructure_error() {
return 0
fi

if is_caido_bootstrap_timing_error; then
return 0
fi

# Generic strix non-zero exit with known transport/connection errors
# that don't fall into the specific categories above.
# Use LLM_PROVIDER_ONLY_REGEX (not PROVIDER_CONTEXT_REGEX) to avoid
Expand Down
49 changes: 49 additions & 0 deletions scripts/ci/test_strix_quick_gate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4420,6 +4420,15 @@ EOS
echo "scan ok with sanitized internal Strix report notice"
exit 0
;;
report-known-internal-warning-variant-sanitized)
mkdir -p "$STRIX_REPORTS_DIR/fake-known-internal-warning-variant"
cat >"$STRIX_REPORTS_DIR/fake-known-internal-warning-variant/strix.log" <<'EOS'
2026-08-22 09:53:26.193 WARNING strix-pr-scope-example - strix.core.execution: agent 673f770f ended a turn without a lifecycle tool call (interactive=False); forcing tool continuation (1/500): <empty>
2026-06-18 13:10:44.089 INFO strix-pr-scope-example - strix.tools.finish.tool: finish_scan: completed scan with 0 vulnerability report(s)
EOS
echo "scan ok with sanitized internal Strix report notice variant"
exit 0
;;
report-unknown-warning-fails)
mkdir -p "$STRIX_REPORTS_DIR/fake-unknown-warning"
cat >"$STRIX_REPORTS_DIR/fake-unknown-warning/strix.log" <<'EOS'
Expand Down Expand Up @@ -5683,6 +5692,17 @@ PY
"scenario=$scenario does not rewrite logs through symlinked report directories"
fi

if [ "$scenario" = "report-known-internal-warning-variant-sanitized" ]; then
assert_file_not_contains \
"$repo_root_dir/strix_runs/fake-known-internal-warning-variant/strix.log" \
"ended a turn without a lifecycle tool call" \
"scenario=$scenario strips the newer-wording known internal Strix warning from published artifacts"
assert_file_contains \
"$repo_root_dir/strix_runs/fake-known-internal-warning-variant/strix.log" \
"finish_scan: completed scan with 0 vulnerability report(s)" \
"scenario=$scenario keeps non-warning Strix report evidence"
fi

if [ "$scenario" = "github-models-primary-ratelimit-fallback-success" ]; then
assert_file_contains \
"$output_log" \
Expand Down Expand Up @@ -9991,6 +10011,35 @@ run_gate_case "report-known-internal-warning-sanitized" \
"" \
"1"

run_gate_case "report-known-internal-warning-variant-sanitized" \
"vertex_ai/report-known-internal-warning-variant-sanitized" \
"" \
"0" \
"Strix run succeeded for model 'vertex_ai/report-known-internal-warning-variant-sanitized'" \
"1" \
"vertex_ai/report-known-internal-warning-variant-sanitized" \
"<unset>" \
"vertex_ai" \
"__DEFAULT__" \
"" \
"0" \
"CRITICAL" \
"0" \
"" \
"" \
"1200" \
"0" \
"" \
"" \
"" \
"" \
"" \
"" \
"" \
"__SAME_AS_FALLBACK_MODELS__" \
"" \
"1"

run_gate_case "report-unknown-warning-fails" \
"vertex_ai/report-unknown-warning-fails" \
"" \
Expand Down
142 changes: 142 additions & 0 deletions tests/test_strix_caido_bootstrap_timing_retry.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
"""Regression contract for the strix-agent Caido sandbox bootstrap timing race.

Upstream strix-agent (usestrix/strix#1036, #1037, #1056) runs a chown -R on
the sandbox container before starting caido-cli, and enforces a fixed
10-attempt loginAsGuest retry budget. A slow CI runner can exceed that
budget before the local proxy is reachable, even though the penetration
test itself never started and no security evidence was produced or lost.
This is local sandbox/container boot timing, not tied to any one LLM model,
so it must be retried same-model rather than treated as grounds to switch
models or as a genuine, non-backend scan failure.
"""

from __future__ import annotations

import re
import subprocess
import tempfile
import unittest
from pathlib import Path


REPOSITORY_ROOT = Path(__file__).resolve().parents[1]
STRIX_GATE = REPOSITORY_ROOT / "scripts" / "ci" / "strix_quick_gate.sh"

# The exact strings strix-agent's caido_bootstrap.py emits on this failure.
OBSERVED_LOG = (
"Error during penetration test: loginAsGuest failed after 10 attempts: "
"curl exit 7: curl: (7) Failed to connect to 127.0.0.1 port 48080 after "
"0 ms: Could not connect to server\n"
"Vulnerabilities 0\n"
)


def _function_block(source: str, function_name: str) -> str:
"""Return one top-level Bash function, including its closing brace."""

match = re.search(
rf"(?ms)^{re.escape(function_name)}\(\) \{{\n.*?^\}}\n",
source,
)
if match is None:
raise AssertionError(f"missing Bash function: {function_name}")
return match.group(0)


def _classifies_as_caido_bootstrap_timing(log_text: str) -> bool:
"""Execute the production classifier against a bounded synthetic log."""

gate_source = STRIX_GATE.read_text(encoding="utf-8")
function_source = _function_block(
gate_source,
"is_caido_bootstrap_timing_error",
)
with tempfile.TemporaryDirectory(prefix="strix-caido-timing-") as temp_dir:
log_path = Path(temp_dir) / "strix.log"
log_path.write_text(log_text, encoding="utf-8")
script = "\n".join(
(
"set -euo pipefail",
'STRIX_LOG="$1"',
function_source,
"is_caido_bootstrap_timing_error",
)
)
completed = subprocess.run(
["bash", "-c", script, "strix-classifier", str(log_path)],
check=False,
capture_output=True,
text=True,
)
if completed.returncode not in {0, 1}:
raise AssertionError(completed.stderr)
return completed.returncode == 0


class StrixCaidoBootstrapTimingRetryTests(unittest.TestCase):
"""Protect same-model retry for the upstream sandbox boot race."""

def test_observed_caido_login_failure_is_retryable(self) -> None:
"""Recognize the exact loginAsGuest failure observed in required CI."""

self.assertTrue(_classifies_as_caido_bootstrap_timing(OBSERVED_LOG))

def test_different_port_is_still_recognized(self) -> None:
"""Match the strix-agent message shape, not one hardcoded port."""

log = (
"loginAsGuest failed after 10 attempts: curl exit 7: curl: (7) "
"Failed to connect to 127.0.0.1 port 51234 after 0 ms: Could "
"not connect to server\n"
)
self.assertTrue(_classifies_as_caido_bootstrap_timing(log))

def test_unrelated_connection_refused_is_not_retryable(self) -> None:
"""Do not let an unrelated target-application connection error match."""

log = "requests.exceptions.ConnectionError: Failed to establish a new connection\n"
self.assertFalse(_classifies_as_caido_bootstrap_timing(log))

def test_login_failure_without_the_connect_evidence_is_not_retryable(self) -> None:
"""Require both the loginAsGuest phrase and the connect-failure line."""

log = "loginAsGuest failed after 10 attempts: unknown reason\n"
self.assertFalse(_classifies_as_caido_bootstrap_timing(log))

def test_wired_into_same_model_retry_and_infrastructure_not_cross_model(
self,
) -> None:
"""Retry the same model; do not treat this as a reason to switch models.

Switching LLM models cannot change how long the local sandbox
container takes to boot, so cross-model fallback (`is_model_retryable_error`)
must stay untouched by this classifier.
"""

gate_source = STRIX_GATE.read_text(encoding="utf-8")
infrastructure = _function_block(
gate_source,
"has_detected_infrastructure_error",
)
same_model_retry = _function_block(
gate_source,
"is_transient_same_model_retry_error",
)
cross_model_retry = _function_block(gate_source, "is_model_retryable_error")

self.assertIn("is_caido_bootstrap_timing_error", infrastructure)
self.assertIn("is_caido_bootstrap_timing_error", same_model_retry)
self.assertNotIn("is_caido_bootstrap_timing_error", cross_model_retry)

def test_retry_reason_is_logged_for_operators(self) -> None:
"""Keep the diagnostic retry-reason message in sync with the classifier."""

gate_source = STRIX_GATE.read_text(encoding="utf-8")
self.assertIn(
'retry_reason="Caido sandbox bootstrap timing"',
gate_source,
)


if __name__ == "__main__":
unittest.main()
Loading