Skip to content
Closed
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 52 additions & 11 deletions .github/workflows/strix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -863,6 +863,17 @@ jobs:
export "STRIX_PROCESS_${budget_suffix}_SECONDS=$process_budget_seconds"
export "STRIX_TOTAL_${budget_suffix}_SECONDS=5700"

# Recognized signals that the LLM backend was unavailable / starved.
# Defined before the gate loop so the bounded retry decision below
# can classify outcomes without duplicating the patterns later.
backend_unavailable_signal='RateLimitError|Too many requests\. For more on scraping GitHub|exceeded your current quota|insufficient_quota|billing details|"status"[[:space:]]*:[[:space:]]*"RESOURCE_EXHAUSTED"|tokens_limit_reached|Request body too large|Max size:[[:space:]]*[0-9]+[[:space:]]+tokens|Error code:[[:space:]]*413|LLM CONNECTION FAILED|Could not establish connection to the language model|LLM warm-up failed|Configured model and fallback models were unavailable|Configured Vertex model and fallback models were unavailable|emitted provider infrastructure or failure-signal output|before provider infrastructure failure|litellm(\.exceptions)?\.NotFoundError[^[:cntrl:]]*Nvidia_nimException[^[:cntrl:]]*Error code:[[:space:]]*404|Error during penetration test: loginAsGuest failed after [0-9]+ attempts: curl exit 7: curl: \(7\) Failed to connect to 127\.0\.0\.1 port 48080'
model_behavior_error_signal='(^|[^A-Za-z0-9_])(agents|pydantic_ai|strix)(\.[A-Za-z_][A-Za-z0-9_]*)*\.ModelBehaviorError([^A-Za-z0-9_]|$)'
# Any evidence that a vulnerability was actually reported. Its presence
# forces a hard failure so real findings are NEVER downgraded. Keep the
# severity branch anchored away from identifiers so environment lines
# such as STRIX_FAIL_ON_MIN_SEVERITY do not look like findings.
reported_vulnerability_signal='Vulnerabilities[[:space:]]+[1-9]|(^|[^A-Za-z0-9_])severity[[:space:]]*:'
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.

# Capture the gate exit code plus its console output. The gate returns
# exit 1 both for genuine blocking vulnerabilities AND for
# LLM-backend-unavailable outcomes (GitHub Models "Too many requests"
Expand All @@ -871,11 +882,50 @@ jobs:
# could not complete a scan. Provider failure is typed infrastructure
# evidence, but remains non-passing because no authoritative complete
# vulnerability result exists.
#
# A typed provider outage with no reported vulnerability finding is
# retried with bounded linear backoff inside this step so transient
# provider failures do not fail the required check on the first
# attempt. Genuine findings, configuration failures, and unexpected
# exit codes never retry; the deadline keeps every path inside the
# deterministic 120-minute job budget, and all-terminal outcomes
# remain fail-closed.
strix_run_log="$RUNNER_TEMP/strix_gate_console.log"
strix_rc=0
strix_gate_attempt=1
strix_gate_deadline=$(( SECONDS + 6000 ))
set +e
bash "$TRUSTED_STRIX_GATE" 2>&1 | tee "$strix_run_log"
strix_rc="${PIPESTATUS[0]}"
while : ; do
: > "$strix_run_log"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Earlier retry attempts' logs are discarded

: > "$strix_run_log" (strix.yml) truncates the console log each iteration, so only the final attempt reaches the uploaded artifact. Correct for classification, but debugging a multi-attempt outage loses the intermediate console output.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

bash "$TRUSTED_STRIX_GATE" 2>&1 | tee "$strix_run_log"
strix_rc="${PIPESTATUS[0]}"
if [ "$strix_rc" -eq 0 ]; then
break
fi
# Only exit-code 1 scan failures can be infrastructure outcomes.
if [ "$strix_rc" -ne 1 ]; then
break
fi
# A reported vulnerability is authoritative evidence: never retry
# and never risk downgrading it.
if grep -Eiq "$reported_vulnerability_signal" "$strix_run_log"; then
break
fi
# Retry only recognized provider-outage / model-behavior classes.
if ! grep -Eiq "$backend_unavailable_signal" "$strix_run_log" \
&& ! grep -Eq "$model_behavior_error_signal" "$strix_run_log"; then
break
Comment thread
coderabbitai[bot] marked this conversation as resolved.
fi
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.
remaining_seconds=$(( strix_gate_deadline - SECONDS ))
if [ "$strix_gate_attempt" -ge 3 ] || [ "$remaining_seconds" -lt 600 ]; then
echo "Provider-unavailable Strix attempt ${strix_gate_attempt} reached the bounded retry limit or the remaining job time budget (${remaining_seconds}s) is too small to retry; failing closed." >&2
break
fi
Comment on lines +929 to +933

@devin-ai-integration devin-ai-integration Bot Aug 25, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Retry can start with too little step budget

The retry gate permits a new attempt whenever remaining_seconds >= 600 (10 min), but a single attempt can run up to STRIX_TOTAL_TIMEOUT_SECONDS=5700 (~95 min, strix.yml:864). A retry started near the deadline can therefore be hard-killed mid-run by the step's timeout-minutes: 100 (strix.yml:808). The outcome stays fail-closed, so no false pass, but the 'deterministic' budget comment overstates this.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

backoff_seconds=$(( strix_gate_attempt * 90 ))
echo "Strix provider outage on attempt ${strix_gate_attempt}; retrying after ${backoff_seconds}s backoff." >&2
sleep "$backoff_seconds"
strix_gate_attempt=$(( strix_gate_attempt + 1 ))
done
Comment on lines +898 to +938

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Retry gating preserves fail-closed semantics

Retries occur only for exit code 1 with no vulnerability signal in the tail-scoped log and a matching outage/model-behavior signal; findings, exit 2, and unexpected codes break and propagate. Tail-scoping (strix.yml) mirrors the terminal classification, so an earlier exempted finding cannot mask a later outage and a real finding after the continuation marker still blocks. Attempts cap at 3 gate runs.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

set -e

if [ "$strix_rc" -eq 0 ]; then
Expand All @@ -889,15 +939,6 @@ jobs:
exit "$strix_rc"
fi

# Recognized signals that the LLM backend was unavailable / starved.
backend_unavailable_signal='RateLimitError|Too many requests\. For more on scraping GitHub|exceeded your current quota|insufficient_quota|billing details|"status"[[:space:]]*:[[:space:]]*"RESOURCE_EXHAUSTED"|tokens_limit_reached|Request body too large|Max size:[[:space:]]*[0-9]+[[:space:]]+tokens|Error code:[[:space:]]*413|LLM CONNECTION FAILED|Could not establish connection to the language model|LLM warm-up failed|Configured model and fallback models were unavailable|Configured Vertex model and fallback models were unavailable|emitted provider infrastructure or failure-signal output|before provider infrastructure failure|litellm(\.exceptions)?\.NotFoundError[^[:cntrl:]]*Nvidia_nimException[^[:cntrl:]]*Error code:[[:space:]]*404|Error during penetration test: loginAsGuest failed after [0-9]+ attempts: curl exit 7: curl: \(7\) Failed to connect to 127\.0\.0\.1 port 48080'
model_behavior_error_signal='(^|[^A-Za-z0-9_])(agents|pydantic_ai|strix)(\.[A-Za-z_][A-Za-z0-9_]*)*\.ModelBehaviorError([^A-Za-z0-9_]|$)'
# Any evidence that a vulnerability was actually reported. Its presence
# forces a hard failure so real findings are NEVER downgraded. Keep the
# severity branch anchored away from identifiers so environment lines
# such as STRIX_FAIL_ON_MIN_SEVERITY do not look like findings.
reported_vulnerability_signal='Vulnerabilities[[:space:]]+[1-9]|(^|[^A-Za-z0-9_])severity[[:space:]]*:'

# An earlier out-of-scope/below-threshold finding may already have
# been exempted by the trusted gate. Classify a later provider
# outage from the tail after the last continuation marker, but keep
Expand Down
Loading