diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 6310abcfe..67813bc61 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -126,6 +126,9 @@ sequenceDiagram invents a hypothesis, observed result, or verdict. - Sandbox helpers copy the workspace, drop secret environment values unless explicitly allowlisted by **name**, and run subprocesses with `shell=False`. + Web E2E readiness URLs are loopback-only; see + [`docs/doctoring/sandboxed-web-readiness-loopback-boundary.md`](docs/doctoring/sandboxed-web-readiness-loopback-boundary.md) + and [`docs/adr/0003-sandboxed-web-readiness-loopback-boundary.md`](docs/adr/0003-sandboxed-web-readiness-loopback-boundary.md). - Logs and review receipts redact credential shapes (tokens, bearer values, known provider prefixes). They do not mask operational PII that the control plane must process. @@ -173,3 +176,5 @@ resolver conflict. — product-specific psychometric repair heartbeat and scientific gates. - [`docs/doctoring/exact-artifact-sbom-attestation.md`](docs/doctoring/exact-artifact-sbom-attestation.md) — current increment's attestation decision and APA 7th citations. +- [`docs/doctoring/sandboxed-web-readiness-loopback-boundary.md`](docs/doctoring/sandboxed-web-readiness-loopback-boundary.md) + — loopback-only web E2E readiness polling and APA 7th citations. diff --git a/CHANGELOG.md b/CHANGELOG.md index 04f023e4a..1f183452c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -84,6 +84,12 @@ Semantic Versioning where the repository publishes a release. ### Fixed +- Web verification now checks services through local readiness addresses only. + Start the backend and frontend on this computer and use their local health + URLs when running the check. +- Review results now separate cosmetic notices from blocking failures. Open the + failure details and correct the requested issue before running the check + again. - Resolve Strix visibility from the trusted GitHub event for ordinary push, schedule, and pull-request runs, reserving API retries for cross-repository dispatches whose workflow token may not see the target repository. diff --git a/docs/adr/0003-sandboxed-web-readiness-loopback-boundary.md b/docs/adr/0003-sandboxed-web-readiness-loopback-boundary.md new file mode 100644 index 000000000..543270414 --- /dev/null +++ b/docs/adr/0003-sandboxed-web-readiness-loopback-boundary.md @@ -0,0 +1,9 @@ +# ADR-0003: Sandboxed web readiness loopback boundary + +- Status: accepted +- Date: 2026-08-25 +- Scope: ContextualWisdomLab/.github control-plane E2E sandbox +- Decision: Poll `--backend-ready-url` and `--frontend-ready-url` only after the URL is proven to be HTTP(S) loopback. Accept literal `localhost` or a standard-library loopback address, resolve `localhost` and require every answer to be loopback, unwrap IPv4-mapped IPv6, reject userinfo and missing hosts, and keep redirects disabled. +- Ownership: `.github` owns the sandbox helper. Product repositories keep pointing readiness at their own loopback services. +- Figma File ID: N/A. This repository has no customer UI. +- Consequence: A review run cannot use the sandbox poller as an SSRF trampoline to metadata services or public hosts. Operators fix a rejected URL by pointing it at `127.0.0.1` or `::1`. Papers live in `docs/doctoring/sandboxed-web-readiness-loopback-boundary.md`. diff --git a/docs/doctoring/sandboxed-web-readiness-loopback-boundary.md b/docs/doctoring/sandboxed-web-readiness-loopback-boundary.md new file mode 100644 index 000000000..65438e75d --- /dev/null +++ b/docs/doctoring/sandboxed-web-readiness-loopback-boundary.md @@ -0,0 +1,85 @@ +# Sandboxed web readiness loopback boundary + +## Decision + +`sandboxed_web_e2e.py` opens a readiness URL only after +`require_loopback_readiness_url` accepts it. The accepted destinations are +literal `localhost` (a trailing FQDN dot is stripped) or an address that +Python's standard-library `ipaddress` module classifies as loopback after +IPv4-mapped IPv6 addresses are unwrapped. Literal `localhost` is then +resolved; every A/AAAA answer must itself be loopback, so a poisoned hosts +file cannot smuggle a public address through the name allowlist. Redirects +remain disabled. + +This supports the complete IPv4 loopback block, including `127.0.0.2`, and +IPv6 `::1`. It rejects `0.0.0.0`, `::`, public hosts, `.localhost` +subdomains, cloud-metadata link-local addresses, missing hosts, and +userinfo-confused URLs such as `http://user@127.0.0.1/`. A mapped public +address such as `::ffff:8.8.8.8` cannot pass merely because it is IPv6. + +The boundary uses the standard library rather than a second address table. +It therefore follows the runtime's maintained special-purpose definitions and +keeps one fail-closed validation point before any network request. Do not add +individual non-loopback exceptions. + +This successor lands the same buyer-facing repair as ContextualWisdomLab/.github#1244 +on current `main` and keeps Strix classifier ownership out of the SSRF slice +(unlike ContextualWisdomLab/.github#1313). + +## Operator next action + +Point `--backend-ready-url` and `--frontend-ready-url` at the sandboxed +service on loopback. If readiness fails with `URL cannot target external +hostname`, replace the destination with `http://127.0.0.1:/...` or +`http://[::1]:/...` instead of opening the firewall or adding a +hostname exception. + +## Verification + +The regression exercises literal `localhost`, a trailing-dot `localhost.`, +`127.0.0.1`, another address in `127.0.0.0/8`, IPv6 `::1`, mapped loopback +`::ffff:127.0.0.1`, an unspecified address, a `.localhost` subdomain, a +public hostname, the common cloud metadata address, mapped public IPv6, +userinfo, a missing host, and poisoned localhost resolution (public A, +mapped public AAAA, empty answers, resolver errors, and non-IP answers). +The existing no-redirect test continues to prove that an allowed readiness +endpoint cannot redirect the poller across the boundary. + +```mermaid +flowchart TD + Url["Readiness URL"] + Scheme{"http or https?"} + Userinfo{"userinfo present?"} + Host{"loopback IP, or localhost whose every resolved answer is loopback?"} + Open["Poll with redirects disabled"] + Reject["Fail closed before any request"] + + Url --> Scheme + Scheme -->|"no"| Reject + Scheme -->|"yes"| Userinfo + Userinfo -->|"yes"| Reject + Userinfo -->|"no"| Host + Host -->|"no"| Reject + Host -->|"yes"| Open +``` + +## References + +Berners-Lee, T., Fielding, R., & Masinter, L. (2005). *Uniform Resource +Identifier (URI): Generic syntax* (RFC 3986). Internet Engineering Task +Force. https://doi.org/10.17487/RFC3986 + +Cotton, B., Vegoda, L., Bonica, R., & Haberman, B. (2013). *Special-purpose +IP address registries* (RFC 6890). Internet Engineering Task Force. +https://doi.org/10.17487/RFC6890 + +Internet Assigned Numbers Authority. (2026). *IANA IPv4 special-purpose +address registry*. Retrieved August 25, 2026, from +https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml + +OWASP Foundation. (n.d.). *Server-side request forgery prevention cheat +sheet*. Retrieved August 25, 2026, from +https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html + +Python Software Foundation. (2026). *ipaddress — IPv4/IPv6 manipulation +library*. https://docs.python.org/3/library/ipaddress.html diff --git a/scripts/ci/sandboxed_web_e2e.py b/scripts/ci/sandboxed_web_e2e.py index ae0c3105a..978113e69 100644 --- a/scripts/ci/sandboxed_web_e2e.py +++ b/scripts/ci/sandboxed_web_e2e.py @@ -3,16 +3,19 @@ from __future__ import annotations import argparse +import ipaddress import json import os import signal import shutil import shlex +import socket import subprocess import sys import tempfile import time import urllib.error +import urllib.parse import urllib.request from collections.abc import Sequence from dataclasses import dataclass @@ -115,12 +118,61 @@ def start_service(label: str, command: str, cwd: Path, env: dict[str, str], logs return Service(label=label, command=command, process=process, log_path=log_path) +def _require_loopback_ip_text(ip_text: str, hostname: str) -> None: + """Reject a literal or resolved address that is not loopback.""" + try: + address = ipaddress.ip_address(ip_text) + except ValueError as exc: + raise ValueError(f"URL cannot target external hostname: {hostname}") from exc + if address.version == 6 and address.ipv4_mapped is not None: + address = address.ipv4_mapped + if not address.is_loopback: + raise ValueError(f"URL cannot target external hostname: {hostname}") + + +def _require_resolved_loopback_hostname(hostname: str) -> None: + """Resolve a literal localhost name and require every answer to be loopback.""" + try: + results = socket.getaddrinfo(hostname, None) + except OSError as exc: + raise ValueError(f"URL cannot target unresolved hostname: {hostname}") from exc + if not results: + raise ValueError(f"URL cannot target unresolved hostname: {hostname}") + for result in results: + _require_loopback_ip_text(result[4][0], hostname) + + +def require_loopback_readiness_url(url: str) -> None: + """Reject a readiness URL that is not a local loopback HTTP(S) target. + + Operators should point ``--backend-ready-url`` and ``--frontend-ready-url`` + at the sandboxed service itself. Public hosts, cloud metadata addresses, + unspecified bind addresses, DNS names other than literal ``localhost``, + and userinfo-confused URLs are rejected before any request is opened. + Literal ``localhost`` is resolved and every answer must be loopback, so a + poisoned hosts file cannot smuggle a public A/AAAA record through the + name allowlist. IPv4-mapped IPv6 addresses are unwrapped and re-checked + so ``::ffff:8.8.8.8`` cannot bypass the loopback rule. + """ + parsed = urllib.parse.urlparse(url) + if parsed.scheme.lower() not in {"http", "https"}: + raise ValueError(f"URL must start with http:// or https://, got: {url}") + if parsed.username or parsed.password: + raise ValueError("URL cannot include userinfo") + hostname = (parsed.hostname or "").lower().rstrip(".") + if not hostname: + raise ValueError("URL must include a loopback hostname") + if hostname == "localhost": + _require_resolved_loopback_hostname(hostname) + return + _require_loopback_ip_text(hostname, hostname) + + def wait_for_url(url: str, timeout: int, service: Service) -> bool: """Poll a readiness URL until it responds or the service exits.""" if not url: return True - if not (url.startswith("http://") or url.startswith("https://")): - raise ValueError(f"URL must start with http:// or https://, got: {url}") + require_loopback_readiness_url(url) deadline = time.monotonic() + timeout opener = urllib.request.build_opener(NoRedirectHandler()) while time.monotonic() < deadline: diff --git a/scripts/ci/strix_quick_gate.sh b/scripts/ci/strix_quick_gate.sh index c9aa41545..cee3aa193 100755 --- a/scripts/ci/strix_quick_gate.sh +++ b/scripts/ci/strix_quick_gate.sh @@ -151,6 +151,118 @@ preserve_attempt_log() { fi } +# Write a classification copy of the raw console transcript. Never mutate +# $STRIX_LOG: publish_artifact_reports copies it to gate-last-attempt.log. +# Delete a cosmetic-only MODEL QUALITY WARNING box. If that same box also +# contains Fatal/Denied/Timeout or Provider WARNING, strip only the heading +# so the failure stays visible. A cross-box wildcard from the first ╭ to a +# later ╰ would erase a preceding Fatal/Denied box (fail-open). +sanitize_strix_console_log() { + local src="$1" + local dest="$2" + if [ -z "$src" ] || [ -z "$dest" ]; then + return 0 + fi + if [ ! -f "$src" ] || [ -L "$src" ]; then + # A missing source must not leave a prior attempt's classified copy active. + if [ -f "$dest" ] && [ ! -L "$dest" ]; then + rm -f -- "$dest" + fi + return 0 + fi + if [ -L "$dest" ]; then + return 0 + fi + python3 - "$src" "$dest" <<'PY' +from pathlib import Path +import re +import sys + +src = Path(sys.argv[1]) +dest = Path(sys.argv[2]) +model_quality_heading = re.compile(r"│[ \t]*MODEL QUALITY WARNING[ \t]*│") +ansi_csi = re.compile(r"\x1b\[[0-?]*[ -/]*[@-~]") +known_scanner_warning = re.compile( + r"^(?:│ MODEL QUALITY WARNING\s+│|" + r"Warning: You are sending unauthenticated requests to the HF Hub\.)" +) +known_internal_warning = re.compile( + r"^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d+ WARNING " + r"[^ ]+ - strix\.core\.execution: agent [0-9a-f]+ " + r"(?:" + r"produced non-lifecycle final output in non-interactive mode" + r"|ended a turn without a lifecycle tool call \(interactive=False\)" + r"); forcing tool continuation \(\d+/\d+\): " +) +other_failure = re.compile( + r"(?:^|[^A-Za-z])(?:Fatal|Denied|Timeout)(?:[^A-Za-z]|$)" + r"|Provider WARNING" + r"|RateLimitError|Nvidia_nimException|LLM CONNECTION FAILED" + r"|APIConnectionError|Too Many Requests", + re.I, +) + + +def strip_heading_lines(text: str) -> str: + kept: list[str] = [] + for line in text.splitlines(keepends=True): + if model_quality_heading.search(ansi_csi.sub("", line)): + continue + kept.append(line) + return "".join(kept) + + +def strip_model_quality_warning_boxes(text: str) -> str: + pieces: list[str] = [] + index = 0 + length = len(text) + while index < length: + start = text.find("╭", index) + if start < 0: + pieces.append(text[index:]) + break + pieces.append(text[index:start]) + close = text.find("╰", start) + if close < 0: + pieces.append(text[start:]) + break + line_end = text.find("\n", close) + if line_end < 0: + box = text[start:] + index = length + else: + box = text[start : line_end + 1] + index = line_end + 1 + if "MODEL QUALITY WARNING" in box: + if other_failure.search(box): + pieces.append(strip_heading_lines(box)) + continue + pieces.append(box) + return "".join(pieces) + + +try: + text = src.read_text(encoding="utf-8") +except UnicodeDecodeError: + # Preserve byte-level evidence so ASCII failure markers in an otherwise + # malformed transcript still reach the fail-closed grep classifiers. This + # also replaces any previous attempt's classified copy without mutating + # the raw source. + dest.write_bytes(src.read_bytes()) + raise SystemExit(0) +text = strip_model_quality_warning_boxes(text) +text = "".join( + line + for line in text.splitlines(keepends=True) + if not ( + known_scanner_warning.match(ansi_csi.sub("", line)) + or known_internal_warning.match(ansi_csi.sub("", line)) + ) +) +dest.write_text(text, encoding="utf-8") +PY +} + sanitize_known_strix_report_warnings() { local report_root for report_root in "$@"; do @@ -178,6 +290,55 @@ known_scanner_warning = re.compile( ) +model_quality_heading = re.compile(r"│[ \t]*MODEL QUALITY WARNING[ \t]*│") +ansi_csi = re.compile(r"\x1b\[[0-?]*[ -/]*[@-~]") +other_failure = re.compile( + r"(?:^|[^A-Za-z])(?:Fatal|Denied|Timeout)(?:[^A-Za-z]|$)" + r"|Provider WARNING" + r"|RateLimitError|Nvidia_nimException|LLM CONNECTION FAILED" + r"|APIConnectionError|Too Many Requests", + re.I, +) + + +def strip_heading_lines(text: str) -> str: + kept = [] + for line in text.splitlines(keepends=True): + if model_quality_heading.search(ansi_csi.sub("", line)): + continue + kept.append(line) + return "".join(kept) + + +def strip_model_quality_warning_boxes(text: str) -> str: + pieces = [] + index = 0 + length = len(text) + while index < length: + start = text.find("╭", index) + if start < 0: + pieces.append(text[index:]) + break + pieces.append(text[index:start]) + close = text.find("╰", start) + if close < 0: + pieces.append(text[start:]) + break + line_end = text.find("\n", close) + if line_end < 0: + box = text[start:] + index = length + else: + box = text[start : line_end + 1] + index = line_end + 1 + if "MODEL QUALITY WARNING" in box: + if other_failure.search(box): + pieces.append(strip_heading_lines(box)) + continue + pieces.append(box) + return "".join(pieces) + + def iter_report_logs(root: Path): if root.is_file() and root.suffix == ".log": yield root @@ -198,17 +359,20 @@ def iter_report_logs(root: Path): for log_path in iter_report_logs(root): try: - lines = log_path.read_text(encoding="utf-8").splitlines(keepends=True) + original = log_path.read_text(encoding="utf-8") except UnicodeDecodeError: continue + text = strip_model_quality_warning_boxes(original) + lines = text.splitlines(keepends=True) filtered = [ line for line in lines if not known_internal_warning.match(line) and not known_scanner_warning.match(line) ] - if filtered != lines: - log_path.write_text("".join(filtered), encoding="utf-8") + text = "".join(filtered) + if text != original: + log_path.write_text(text, encoding="utf-8") PY done } @@ -2832,15 +2996,21 @@ PY fi fi preserve_attempt_log "$model" "$rc" + local classified_log="${STRIX_LOG}.classified" + sanitize_strix_console_log "$STRIX_LOG" "$classified_log" + local inspect_log="$STRIX_LOG" + if [ -f "$classified_log" ] && [ ! -L "$classified_log" ]; then + inspect_log="$classified_log" + fi - sanitize_known_strix_report_warnings "$STRIX_LOG" "$ACTIVE_REPORTS_DIR" "${resolved_target_path%/}/strix_runs" + sanitize_known_strix_report_warnings "$ACTIVE_REPORTS_DIR" "${resolved_target_path%/}/strix_runs" local report_failure_signal=0 if has_strix_report_failure_signal "$ACTIVE_REPORTS_DIR" "${resolved_target_path%/}/strix_runs"; then report_failure_signal=1 echo "Strix report artifacts emitted warning/fatal/denied/timeout output; failing closed." | tee -a "$STRIX_LOG" >&2 fi - if [ "$report_failure_signal" -eq 1 ] || has_detected_infrastructure_error; then + if [ "$report_failure_signal" -eq 1 ] || STRIX_LOG="$inspect_log" has_detected_infrastructure_error; then INFRA_ERROR_DETECTED=1 if [ "$rc" -eq 0 ] && provider_signal_fail_closed_enabled; then echo "Strix run emitted provider infrastructure or failure-signal output; failing closed." >&2 diff --git a/scripts/ci/test_strix_quick_gate.sh b/scripts/ci/test_strix_quick_gate.sh index 3d3449dae..4b5df0967 100755 --- a/scripts/ci/test_strix_quick_gate.sh +++ b/scripts/ci/test_strix_quick_gate.sh @@ -331,6 +331,9 @@ assert_strix_workflow_pr_trigger_hardened() { assert_file_contains "$GATE_SCRIPT" '[[ "$normalized_changed_file" == scripts/ci/test_*.sh || "$normalized_changed_file" == scripts/ci/*_test.sh ]]' "strix gate excludes large CI test harness scripts from model scan input" assert_file_contains "$GATE_SCRIPT" "Materialized PR-head changed-file scope for Strix scan" "strix gate avoids copying the full PR head tree into privileged scan targets by default" assert_file_contains "$GATE_SCRIPT" "sanitize_known_strix_report_warnings" "strix gate sanitizes only known internal Strix report warnings" + assert_file_contains "$GATE_SCRIPT" "known_internal_warning = re.compile" "strix gate sanitizes known internal console warnings" + assert_file_contains "$GATE_SCRIPT" "dest.write_bytes(src.read_bytes())" "strix gate preserves byte-level console evidence after decode errors" + assert_file_contains "$GATE_SCRIPT" "A missing source must not leave a prior attempt's classified copy active" "strix gate removes stale classified console copies" assert_file_contains "$GATE_SCRIPT" 'MODEL QUALITY WARNING' "strix gate accepts the scanner's informational fallback-model banner" assert_file_contains "$GATE_SCRIPT" 'unauthenticated requests to the HF Hub' "strix gate accepts the scanner dependency's non-fatal download warning" assert_file_not_contains "$GATE_SCRIPT" 'known_scanner_warning = re.compile(r".*Warn' "strix gate does not broadly suppress warning-class evidence" @@ -4534,6 +4537,14 @@ EOS echo "Warning: provider response included incomplete scan state" exit 0 ;; + console-known-internal-warning-sanitized) + cat <<'EOS' +2026-06-18 13:08:05.986 WARNING strix-pr-scope-example - strix.core.execution: agent a9fb4033 produced non-lifecycle final output in non-interactive mode; forcing tool continuation (1/500): internal console coordination note +2026-08-22 09:53:26.193 WARNING strix-pr-scope-example - strix.core.execution: agent 673f770f ended a turn without a lifecycle tool call (interactive=False); forcing tool continuation (1/500): +scan completed with 0 vulnerability report(s) +EOS + exit 0 + ;; provider-denied-success-signal) echo "Denied: provider credentials were rejected" exit 0 @@ -4598,6 +4609,119 @@ EOS echo "scan ok but unknown report warning remains" exit 0 ;; + console-model-quality-warning-banner-sanitized) + # Cosmetic startup banner on a non-frontier model. Classification + # must ignore only that box so a clean 0-finding scan succeeds. + cat <<'EOS' +╭─ STRIX ──────────────────────────────────────────────────────────────────────╮ +│ │ +│ MODEL QUALITY WARNING │ +│ │ +│ 'vertex_ai/console-model-quality-warning-banner-sanitized' is not a │ +│ recommended frontier model for Strix. │ +│ │ +│ You can continue, but weaker models may miss vulnerabilities or produce │ +│ lower-quality findings. │ +│ │ +╰──────────────────────────────────────────────────────────────────────────────╯ +╭─ STRIX ──────────────────────────────────────────────────────────────────────╮ +│ │ +│ Penetration test completed │ +│ │ +│ Vulnerabilities 0 │ +│ │ +╰──────────────────────────────────────────────────────────────────────────────╯ +EOS + exit 0 + ;; + console-model-quality-warning-preserves-prior-fatal-box) + # A genuine Fatal box must survive when it precedes the cosmetic + # banner. A cross-box regex would delete both and fail open. + cat <<'EOS' +╭─ STRIX ──────────────────────────────────────────────────────────────────────╮ +│ Fatal: LLM CONNECTION FAILED after 3 retries │ +╰──────────────────────────────────────────────────────────────────────────────╯ +╭─ STRIX ──────────────────────────────────────────────────────────────────────╮ +│ │ +│ MODEL QUALITY WARNING │ +│ │ +│ 'vertex_ai/console-model-quality-warning-preserves-prior-fatal-box' is not │ +│ a recommended frontier model for Strix. │ +│ │ +╰──────────────────────────────────────────────────────────────────────────────╯ +Penetration test completed +Vulnerabilities 0 (No exploitable vulnerabilities detected) +EOS + exit 0 + ;; + report-model-quality-warning-banner-sanitized) + mkdir -p "$STRIX_REPORTS_DIR/fake-model-quality-banner" + cat >"$STRIX_REPORTS_DIR/fake-model-quality-banner/strix.log" <<'EOS' +╭─ STRIX ──────────────────────────────────────────────────────────────────────╮ +│ MODEL QUALITY WARNING │ +│ 'vertex_ai/report-model-quality-warning-banner-sanitized' is not a │ +│ recommended frontier model for Strix. │ +╰──────────────────────────────────────────────────────────────────────────────╯ +2026-06-18 13:10:44.089 INFO strix-pr-scope-example - strix.tools.finish.tool: finish_scan: completed scan with 0 vulnerability report(s) +EOS + echo "scan ok with sanitized model-quality report banner" + exit 0 + ;; + report-model-quality-warning-preserves-prior-fatal-box) + mkdir -p "$STRIX_REPORTS_DIR/fake-model-quality-fatal" + cat >"$STRIX_REPORTS_DIR/fake-model-quality-fatal/strix.log" <<'EOS' +╭─ STRIX ──────────────────────────────────────────────────────────────────────╮ +│ Fatal: LLM CONNECTION FAILED after 3 retries │ +╰──────────────────────────────────────────────────────────────────────────────╯ +╭─ STRIX ──────────────────────────────────────────────────────────────────────╮ +│ MODEL QUALITY WARNING │ +│ 'vertex_ai/report-model-quality-warning-preserves-prior-fatal-box' is not │ +│ a recommended frontier model for Strix. │ +╰──────────────────────────────────────────────────────────────────────────────╯ +2026-06-18 13:10:44.089 INFO strix-pr-scope-example - strix.tools.finish.tool: finish_scan: completed scan with 0 vulnerability report(s) +EOS + echo "scan ok but prior fatal report box must remain" + exit 0 + ;; + console-model-quality-warning-preserves-same-box-failure) + # A whole-box delete would drop this Fatal with the heading + # and fail open. Strip only the cosmetic heading line. + cat <<'EOS' +╭─ STRIX ──────────────────────────────────────────────────────────────────────╮ +│ MODEL QUALITY WARNING │ +│ Fatal: LLM CONNECTION FAILED after 3 retries │ +╰──────────────────────────────────────────────────────────────────────────────╯ +Penetration test completed +Vulnerabilities 0 (No exploitable vulnerabilities detected) +EOS + exit 0 + ;; + report-model-quality-warning-preserves-same-box-failure) + mkdir -p "$STRIX_REPORTS_DIR/fake-model-quality-same-box-failure" + cat >"$STRIX_REPORTS_DIR/fake-model-quality-same-box-failure/strix.log" <<'EOS' +╭─ STRIX ──────────────────────────────────────────────────────────────────────╮ +│ MODEL QUALITY WARNING │ +│ Provider WARNING: report evidence is incomplete │ +╰──────────────────────────────────────────────────────────────────────────────╯ +EOS + echo "scan returned zero findings with incomplete evidence in the banner box" + exit 0 + ;; + console-model-quality-warning-preserves-same-box-ratelimit) + # RateLimitError is retryable. Keeping it in the classified copy + # therefore tries the two default Vertex fallbacks. Deleting the + # whole box would hide the token, inspect would pass, and the + # gate would stop after one successful call. + cat <<'EOS' +╭─ STRIX ──────────────────────────────────────────────────────────────────────╮ +│ MODEL QUALITY WARNING │ +│ RateLimitError: Too Many Requests │ +╰──────────────────────────────────────────────────────────────────────────────╯ +Penetration test completed +Vulnerabilities 0 (No exploitable vulnerabilities detected) +EOS + exit 0 + ;; bare-timeout-with-provider-marker) # Emit bare "Connection timed out" alongside a provider marker so # is_timeout_error() matches the Tier 3 branch gated on @@ -5896,6 +6020,88 @@ PY "scenario=$scenario does not rewrite logs through symlinked report directories" fi + if [ "$scenario" = "console-model-quality-warning-banner-sanitized" ]; then + assert_file_contains \ + "$repo_root_dir/strix_runs/gate-last-attempt.log" \ + "MODEL QUALITY WARNING" \ + "scenario=$scenario publishes the raw console including the cosmetic banner" + local attempt_log="" + attempt_log="$(find "$repo_root_dir/strix_runs/gate-attempts" -type f -name '*.log' -print -quit 2>/dev/null || true)" + if [ -z "$attempt_log" ]; then + record_failure "scenario=$scenario should preserve a raw per-attempt log" + else + assert_file_contains \ + "$attempt_log" \ + "MODEL QUALITY WARNING" \ + "scenario=$scenario keeps raw per-attempt logs" + fi + fi + + if [ "$scenario" = "console-known-internal-warning-sanitized" ]; then + assert_file_contains \ + "$repo_root_dir/strix_runs/gate-last-attempt.log" \ + "produced non-lifecycle final output" \ + "scenario=$scenario preserves the raw non-lifecycle console warning" + assert_file_contains \ + "$repo_root_dir/strix_runs/gate-last-attempt.log" \ + "ended a turn without a lifecycle tool call" \ + "scenario=$scenario preserves the raw lifecycle console warning" + fi + + if [ "$scenario" = "console-model-quality-warning-preserves-prior-fatal-box" ]; then + assert_file_contains \ + "$repo_root_dir/strix_runs/gate-last-attempt.log" \ + "Fatal: LLM CONNECTION FAILED after 3 retries" \ + "scenario=$scenario keeps the raw Fatal box in the published last-attempt log" + fi + + if [ "$scenario" = "report-model-quality-warning-banner-sanitized" ]; then + assert_file_not_contains \ + "$repo_root_dir/strix_runs/fake-model-quality-banner/strix.log" \ + "MODEL QUALITY WARNING" \ + "scenario=$scenario strips only the cosmetic report banner" + assert_file_contains \ + "$repo_root_dir/strix_runs/fake-model-quality-banner/strix.log" \ + "finish_scan: completed scan with 0 vulnerability report(s)" \ + "scenario=$scenario keeps non-banner report evidence" + fi + + if [ "$scenario" = "report-model-quality-warning-preserves-prior-fatal-box" ]; then + assert_file_contains \ + "$repo_root_dir/strix_runs/fake-model-quality-fatal/strix.log" \ + "Fatal: LLM CONNECTION FAILED after 3 retries" \ + "scenario=$scenario keeps a preceding Fatal report box" + assert_file_not_contains \ + "$repo_root_dir/strix_runs/fake-model-quality-fatal/strix.log" \ + "MODEL QUALITY WARNING" \ + "scenario=$scenario still strips the cosmetic report banner" + fi + + if [ "$scenario" = "console-model-quality-warning-preserves-same-box-failure" ]; then + assert_file_contains \ + "$repo_root_dir/strix_runs/gate-last-attempt.log" \ + "Fatal: LLM CONNECTION FAILED after 3 retries" \ + "scenario=$scenario keeps the raw same-box Fatal line in last-attempt" + fi + + if [ "$scenario" = "report-model-quality-warning-preserves-same-box-failure" ]; then + assert_file_contains \ + "$repo_root_dir/strix_runs/fake-model-quality-same-box-failure/strix.log" \ + "Provider WARNING: report evidence is incomplete" \ + "scenario=$scenario keeps same-box provider warning evidence" + assert_file_not_contains \ + "$repo_root_dir/strix_runs/fake-model-quality-same-box-failure/strix.log" \ + "MODEL QUALITY WARNING" \ + "scenario=$scenario still strips only the cosmetic heading" + fi + + if [ "$scenario" = "console-model-quality-warning-preserves-same-box-ratelimit" ]; then + assert_file_contains \ + "$repo_root_dir/strix_runs/gate-last-attempt.log" \ + "RateLimitError: Too Many Requests" \ + "scenario=$scenario keeps the raw same-box rate-limit line in last-attempt" + fi + if [ "$scenario" = "report-known-internal-warning-variant-sanitized" ]; then assert_file_not_contains \ "$repo_root_dir/strix_runs/fake-known-internal-warning-variant/strix.log" \ @@ -6446,6 +6652,16 @@ run_filtered_gate_case_if_requested() { "vertex_ai/excluded-dir-primary" \ "" ;; + console-known-internal-warning-sanitized) + run_gate_case "$STRIX_TEST_CASE_FILTER" \ + "vertex_ai/console-known-internal-warning-sanitized" \ + "" \ + "0" \ + "Strix run succeeded for model 'vertex_ai/console-known-internal-warning-sanitized'" \ + "1" \ + "vertex_ai/console-known-internal-warning-sanitized" \ + "" + ;; pull-request-target-changed-backend-context) run_pull_request_target_changed_backend_context_scope_case ;; @@ -6459,6 +6675,76 @@ run_filtered_gate_case_if_requested() { "vertex_ai/report-known-internal-warning-sanitized" \ "" ;; + console-model-quality-warning-banner-sanitized) + run_gate_case "$STRIX_TEST_CASE_FILTER" \ + "vertex_ai/console-model-quality-warning-banner-sanitized" \ + "" \ + "0" \ + "Strix run succeeded for model 'vertex_ai/console-model-quality-warning-banner-sanitized'" \ + "1" \ + "vertex_ai/console-model-quality-warning-banner-sanitized" \ + "" + ;; + console-model-quality-warning-preserves-prior-fatal-box) + run_gate_case "$STRIX_TEST_CASE_FILTER" \ + "vertex_ai/console-model-quality-warning-preserves-prior-fatal-box" \ + "" \ + "1" \ + "Strix run emitted provider infrastructure or failure-signal output; failing closed." \ + "1" \ + "vertex_ai/console-model-quality-warning-preserves-prior-fatal-box" \ + "" + ;; + report-model-quality-warning-banner-sanitized) + run_gate_case "$STRIX_TEST_CASE_FILTER" \ + "vertex_ai/report-model-quality-warning-banner-sanitized" \ + "" \ + "0" \ + "Strix run succeeded for model 'vertex_ai/report-model-quality-warning-banner-sanitized'" \ + "1" \ + "vertex_ai/report-model-quality-warning-banner-sanitized" \ + "" + ;; + report-model-quality-warning-preserves-prior-fatal-box) + run_gate_case "$STRIX_TEST_CASE_FILTER" \ + "vertex_ai/report-model-quality-warning-preserves-prior-fatal-box" \ + "" \ + "1" \ + "Strix report artifacts emitted warning/fatal/denied/timeout output; failing closed." \ + "1" \ + "vertex_ai/report-model-quality-warning-preserves-prior-fatal-box" \ + "" + ;; + console-model-quality-warning-preserves-same-box-failure) + run_gate_case "$STRIX_TEST_CASE_FILTER" \ + "vertex_ai/console-model-quality-warning-preserves-same-box-failure" \ + "" \ + "1" \ + "Strix run emitted provider infrastructure or failure-signal output; failing closed." \ + "1" \ + "vertex_ai/console-model-quality-warning-preserves-same-box-failure" \ + "" + ;; + report-model-quality-warning-preserves-same-box-failure) + run_gate_case "$STRIX_TEST_CASE_FILTER" \ + "vertex_ai/report-model-quality-warning-preserves-same-box-failure" \ + "" \ + "1" \ + "Strix report artifacts emitted warning/fatal/denied/timeout output; failing closed." \ + "1" \ + "vertex_ai/report-model-quality-warning-preserves-same-box-failure" \ + "" + ;; + console-model-quality-warning-preserves-same-box-ratelimit) + run_gate_case "$STRIX_TEST_CASE_FILTER" \ + "vertex_ai/console-model-quality-warning-preserves-same-box-ratelimit" \ + "vertex_ai/gemini-2.5-pro vertex_ai/gemini-2.5-flash" \ + "1" \ + "Strix run emitted provider infrastructure or failure-signal output; failing closed." \ + "3" \ + "vertex_ai/console-model-quality-warning-preserves-same-box-ratelimit|vertex_ai/gemini-2.5-pro|vertex_ai/gemini-2.5-flash" \ + "||" + ;; provider-fatal-success-signal | provider-warning-success-signal) run_gate_case "$STRIX_TEST_CASE_FILTER" \ "vertex_ai/$STRIX_TEST_CASE_FILTER" \ @@ -10574,6 +10860,15 @@ run_gate_case "provider-warning-success-signal" \ "" \ "1" +run_gate_case "console-known-internal-warning-sanitized" \ + "vertex_ai/console-known-internal-warning-sanitized" \ + "" \ + "0" \ + "Strix run succeeded for model 'vertex_ai/console-known-internal-warning-sanitized'" \ + "1" \ + "vertex_ai/console-known-internal-warning-sanitized" \ + "" + run_gate_case "provider-report-rate-limit-fallback-success" \ "vertex_ai/report-rate-limit-primary" \ "vertex_ai/fallback-one vertex_ai/fallback-two" \ @@ -10613,6 +10908,69 @@ run_gate_case "report-known-internal-warning-sanitized" \ "" \ "1" +run_gate_case "console-model-quality-warning-banner-sanitized" \ + "vertex_ai/console-model-quality-warning-banner-sanitized" \ + "" \ + "0" \ + "Strix run succeeded for model 'vertex_ai/console-model-quality-warning-banner-sanitized'" \ + "1" \ + "vertex_ai/console-model-quality-warning-banner-sanitized" \ + "" + +run_gate_case "console-model-quality-warning-preserves-prior-fatal-box" \ + "vertex_ai/console-model-quality-warning-preserves-prior-fatal-box" \ + "" \ + "1" \ + "Strix run emitted provider infrastructure or failure-signal output; failing closed." \ + "1" \ + "vertex_ai/console-model-quality-warning-preserves-prior-fatal-box" \ + "" + +run_gate_case "report-model-quality-warning-banner-sanitized" \ + "vertex_ai/report-model-quality-warning-banner-sanitized" \ + "" \ + "0" \ + "Strix run succeeded for model 'vertex_ai/report-model-quality-warning-banner-sanitized'" \ + "1" \ + "vertex_ai/report-model-quality-warning-banner-sanitized" \ + "" + +run_gate_case "report-model-quality-warning-preserves-prior-fatal-box" \ + "vertex_ai/report-model-quality-warning-preserves-prior-fatal-box" \ + "" \ + "1" \ + "Strix report artifacts emitted warning/fatal/denied/timeout output; failing closed." \ + "1" \ + "vertex_ai/report-model-quality-warning-preserves-prior-fatal-box" \ + "" + +run_gate_case "console-model-quality-warning-preserves-same-box-failure" \ + "vertex_ai/console-model-quality-warning-preserves-same-box-failure" \ + "" \ + "1" \ + "Strix run emitted provider infrastructure or failure-signal output; failing closed." \ + "1" \ + "vertex_ai/console-model-quality-warning-preserves-same-box-failure" \ + "" + +run_gate_case "report-model-quality-warning-preserves-same-box-failure" \ + "vertex_ai/report-model-quality-warning-preserves-same-box-failure" \ + "" \ + "1" \ + "Strix report artifacts emitted warning/fatal/denied/timeout output; failing closed." \ + "1" \ + "vertex_ai/report-model-quality-warning-preserves-same-box-failure" \ + "" + +run_gate_case "console-model-quality-warning-preserves-same-box-ratelimit" \ + "vertex_ai/console-model-quality-warning-preserves-same-box-ratelimit" \ + "vertex_ai/gemini-2.5-pro vertex_ai/gemini-2.5-flash" \ + "1" \ + "Strix run emitted provider infrastructure or failure-signal output; failing closed." \ + "3" \ + "vertex_ai/console-model-quality-warning-preserves-same-box-ratelimit|vertex_ai/gemini-2.5-pro|vertex_ai/gemini-2.5-flash" \ + "||" + run_gate_case "report-known-internal-warning-variant-sanitized" \ "vertex_ai/report-known-internal-warning-variant-sanitized" \ "" \ diff --git a/tests/test_sandboxed_web_e2e.py b/tests/test_sandboxed_web_e2e.py index 6e092c293..39339a316 100644 --- a/tests/test_sandboxed_web_e2e.py +++ b/tests/test_sandboxed_web_e2e.py @@ -105,13 +105,19 @@ def test_sandboxed_web_e2e_runs_services_and_does_not_mutate_source(tmp_path, ca def test_wait_helpers_and_service_cleanup_edges(monkeypatch, tmp_path): - """Small helper branches handle empty URLs, exited services, and hard cleanup.""" + """Small helper branches handle empty URLs, loopback readiness, and hard cleanup.""" exited = subprocess.Popen([sys.executable, "-c", ""], text=True) exited.wait(timeout=5) exited_service = sandboxed_web_e2e.Service("done", "true", exited, tmp_path / "missing.log") assert sandboxed_web_e2e.wait_for_url("", 1, exited_service) is True assert sandboxed_web_e2e.wait_for_url("http://127.0.0.1:1/", 1, exited_service) is False + assert sandboxed_web_e2e.wait_for_url("http://localhost:1/", 1, exited_service) is False + assert sandboxed_web_e2e.wait_for_url("http://localhost./health", 1, exited_service) is False + assert sandboxed_web_e2e.wait_for_url("http://127.0.0.2:1/", 1, exited_service) is False + assert sandboxed_web_e2e.wait_for_url("http://[::1]:1/health", 1, exited_service) is False + assert sandboxed_web_e2e.wait_for_url("HTTP://[::ffff:127.0.0.1]:1/", 1, exited_service) is False + assert sandboxed_web_e2e.wait_for_url("https://127.0.0.1:1/", 1, exited_service) is False with pytest.raises(ValueError, match="URL must start with http:// or https://"): sandboxed_web_e2e.wait_for_url("file:///etc/passwd", 1, exited_service) sandboxed_web_e2e.stop_service(exited_service) @@ -228,6 +234,83 @@ def open(self, url, timeout): assert sandboxed_web_e2e.tail_text(log_path).splitlines()[0] == "line-10" +def test_wait_for_url_rejects_non_loopback_and_confused_deputy_targets(tmp_path): + """Readiness polling must fail closed on public, metadata, and userinfo targets.""" + exited = subprocess.Popen([sys.executable, "-c", ""], text=True) + exited.wait(timeout=5) + exited_service = sandboxed_web_e2e.Service("done", "true", exited, tmp_path / "missing.log") + + with pytest.raises(ValueError, match="URL cannot target external hostname: example\\.com"): + sandboxed_web_e2e.wait_for_url("http://example.com/health", 1, exited_service) + with pytest.raises(ValueError, match="URL cannot target external hostname: app\\.localhost"): + sandboxed_web_e2e.wait_for_url("http://app.localhost:8000/health", 1, exited_service) + with pytest.raises(ValueError, match="URL cannot target external hostname: 169\\.254\\.169\\.254"): + sandboxed_web_e2e.wait_for_url("http://169.254.169.254/latest/meta-data/", 1, exited_service) + with pytest.raises(ValueError, match="URL cannot target external hostname: 0\\.0\\.0\\.0"): + sandboxed_web_e2e.wait_for_url("http://0.0.0.0:8000/health", 1, exited_service) + with pytest.raises(ValueError, match="URL cannot target external hostname: ::"): + sandboxed_web_e2e.wait_for_url("http://[::]/", 1, exited_service) + with pytest.raises(ValueError, match="URL cannot target external hostname: ::ffff:8\\.8\\.8\\.8"): + sandboxed_web_e2e.wait_for_url("http://[::ffff:8.8.8.8]/", 1, exited_service) + with pytest.raises(ValueError, match="URL cannot include userinfo"): + sandboxed_web_e2e.wait_for_url("http://user@127.0.0.1/", 1, exited_service) + with pytest.raises(ValueError, match="URL cannot include userinfo"): + sandboxed_web_e2e.wait_for_url("http://:pass@127.0.0.1/", 1, exited_service) + with pytest.raises(ValueError, match="URL must include a loopback hostname"): + sandboxed_web_e2e.wait_for_url("http:///health", 1, exited_service) + sandboxed_web_e2e.stop_service(exited_service) + + +def test_localhost_resolution_must_stay_loopback(monkeypatch, tmp_path): + """Literal localhost is allowed only when every resolved address is loopback.""" + exited = subprocess.Popen([sys.executable, "-c", ""], text=True) + exited.wait(timeout=5) + exited_service = sandboxed_web_e2e.Service("done", "true", exited, tmp_path / "missing.log") + + monkeypatch.setattr( + sandboxed_web_e2e.socket, + "getaddrinfo", + lambda host, port: [(0, 0, 0, "", ("8.8.8.8", 0))], + ) + with pytest.raises(ValueError, match="URL cannot target external hostname: localhost"): + sandboxed_web_e2e.wait_for_url("http://localhost:1/", 1, exited_service) + + monkeypatch.setattr(sandboxed_web_e2e.socket, "getaddrinfo", lambda host, port: []) + with pytest.raises(ValueError, match="URL cannot target unresolved hostname: localhost"): + sandboxed_web_e2e.wait_for_url("http://localhost:1/", 1, exited_service) + + def _unresolved(host, port): + raise socket.gaierror("name not known") + + monkeypatch.setattr(sandboxed_web_e2e.socket, "getaddrinfo", _unresolved) + with pytest.raises(ValueError, match="URL cannot target unresolved hostname: localhost"): + sandboxed_web_e2e.wait_for_url("http://localhost:1/", 1, exited_service) + + monkeypatch.setattr( + sandboxed_web_e2e.socket, + "getaddrinfo", + lambda host, port: [(0, 0, 0, "", ("not-an-ip", 0))], + ) + with pytest.raises(ValueError, match="URL cannot target external hostname: localhost"): + sandboxed_web_e2e.wait_for_url("http://localhost:1/", 1, exited_service) + + monkeypatch.setattr( + sandboxed_web_e2e.socket, + "getaddrinfo", + lambda host, port: [(0, 0, 0, "", ("::ffff:8.8.8.8", 0))], + ) + with pytest.raises(ValueError, match="URL cannot target external hostname: localhost"): + sandboxed_web_e2e.wait_for_url("http://localhost:1/", 1, exited_service) + + monkeypatch.setattr( + sandboxed_web_e2e.socket, + "getaddrinfo", + lambda host, port: [(0, 0, 0, "", ("::ffff:127.0.0.1", 0))], + ) + assert sandboxed_web_e2e.wait_for_url("http://localhost:1/", 1, exited_service) is False + sandboxed_web_e2e.stop_service(exited_service) + + def test_no_redirect_handler_raises_httperror_without_following(): """Readiness checks must raise HTTPError on redirects to prevent attacker-controlled internal URLs.""" import urllib.error