Skip to content

fix(test): make CI tests resilient to sandboxed/offline environments - #2179

Closed
zmanian wants to merge 1 commit into
stagingfrom
fix/ci-test-failures-dns-sandbox
Closed

zmanian wants to merge 1 commit into
stagingfrom
fix/ci-test-failures-dns-sandbox

Conversation

@zmanian

@zmanian zmanian commented Apr 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • config/helpers: Add 10s DNS resolution timeout to validate_base_url to prevent indefinite hangs; guard invalid_tld_resolves_locally() with channel timeout
  • config/llm: Only validate explicitly-configured URLs (env vars/DB settings), not hardcoded registry defaults — prevents DNS failures for known-good URLs like api.openai.com in offline environments; use localhost URLs in openai_compatible model resolution tests
  • cli/doctor: Accept DNS failure as valid outcome in offline doctor tests
  • channels/webhook_server: Fix bind test for root/container environments by falling back to already-occupied port when privileged port bind succeeds
  • tools/mcp/auth: Skip validate_url_safe HTTPS test when DNS is unavailable
  • tunnel/custom: Check response status code in health_check (is_ok_and(|r| r.status().is_success())), not just successful send — a proxy may return non-2xx for unreachable targets

Root cause: Commit 315c4cf8 added validate_base_url calls that perform DNS resolution at config construction time for ALL URLs including hardcoded defaults. In sandboxed/offline CI environments, DNS for external hostnames fails, breaking 10 tests.

Test plan

  • All 4339 lib tests pass (0 failed, 3 ignored)
  • cargo clippy --all --benches --tests --examples --all-features — zero warnings
  • cargo fmt --check — clean
  • Verified openai_codex_rejects_ssrf_api_url still properly rejects malicious user-configured URLs
  • Verified SSRF protection is preserved for all explicitly-configured URLs

https://claude.ai/code/session_01LyzwS5oHA68ARhDuXqazpn

Several tests were failing in CI due to DNS resolution requirements:

- config/helpers: Add 10s DNS timeout to validate_base_url to prevent
  indefinite hangs. Guard invalid_tld_resolves_locally() with timeout.
- config/llm: Only validate explicitly-configured URLs (env vars/DB
  settings), not hardcoded defaults. Prevents DNS failures for known-good
  registry URLs like api.openai.com in offline environments.
- cli/doctor: Accept DNS failure as valid outcome in offline tests.
- channels/webhook_server: Fix bind test for root/container environments
  by falling back to already-occupied port when privileged port succeeds.
- tools/mcp/auth: Skip validate_url_safe HTTPS test when DNS unavailable.
- tunnel/custom: Check response status in health_check, not just
  successful send (proxy may return non-2xx for unreachable targets).

[skip-regression-check]

https://claude.ai/code/session_01LyzwS5oHA68ARhDuXqazpn
@github-actions github-actions Bot added scope: channel/cli TUI / CLI channel scope: tool/mcp MCP client size: M 50-199 changed lines risk: medium Business logic, config, or moderate-risk modules contributor: core 20+ merged PRs labels Apr 9, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b9a2bb6d14

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/config/helpers.rs
Comment on lines +348 to +352
std::thread::spawn(move || {
let result = (h.as_str(), port)
.to_socket_addrs()
.map(|addrs| addrs.map(|a| a.ip()).collect::<Vec<_>>());
let _ = tx.send(result);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid leaking resolver threads on DNS timeout

Spawning a detached OS thread for every hostname lookup here means a timed-out recv_timeout returns to the caller while the worker thread can remain blocked in to_socket_addrs() indefinitely; repeated validations (for example through async LLM endpoint checks that call validate_operator_base_url) can accumulate stuck threads and eventually exhaust process resources. This regresses from bounded blocking to an unbounded thread-growth failure mode whenever DNS hangs rather than fails fast.

Useful? React with 👍 / 👎.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request significantly improves the application's robustness and reliability, particularly in sandboxed or offline environments where DNS resolution can be problematic. Key changes include implementing timeouts for blocking DNS resolution calls, conditionally validating URLs only when they are explicitly configured (to avoid spurious failures for known-good defaults), and adapting various test cases to gracefully handle or skip when external DNS resolution is unavailable. Additionally, the health check for custom tunnels has been made more robust by requiring a successful HTTP status code, not just a successful request.

zmanian pushed a commit that referenced this pull request Apr 9, 2026
Six categories of test failures in sandboxed CI:

1. DNS resolution hangs: Add 10s timeout to validate_base_url DNS lookups
   and test helpers (helpers.rs) to prevent indefinite blocking in
   restricted-DNS environments.

2. Hardcoded URL validation: Skip DNS-based SSRF validation for
   hardcoded default URLs (nearai, openai codex) that don't need it,
   only validate explicitly-configured URLs (llm.rs).

3. Privileged port binding: Handle root/container environments where
   port 1 bind succeeds by falling back to already-occupied port
   assertion (webhook_server.rs).

4. DNS-dependent assertions: Accept DNS resolution errors alongside
   expected skip/pass messages in doctor tests (doctor.rs).

5. External DNS in tests: Skip test_validate_url_safe_https when DNS
   is unavailable (auth.rs), use localhost URLs in test fixtures (llm.rs).

6. Health check accuracy: Use is_ok_and to check response status, not
   just connection success (custom.rs).

Supersedes #2133, #2151, #2179 (rebased onto current staging).

https://claude.ai/code/session_01NPAmzdyUAoxRvbQHBEAfhw
Comment thread src/config/llm.rs
};
let nearai_base_url_explicit = nearai_override
.and_then(|o| o.base_url.clone())
.or_else(|| optional_env("NEARAI_BASE_URL").ok().flatten());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium: .ok().flatten() silently swallows env-var parse errors

nearai_base_url_explicit uses .ok().flatten() on optional_env("NEARAI_BASE_URL"), silently discarding ConfigError. If optional_env fails, nearai_base_url_explicit becomes None, and validate_base_url is skipped. The error surfaces on the next optional_env("NEARAI_BASE_URL")? call, so behavior is correct in practice, but .ok().flatten() is fragile — a refactor removing the second call would silently lose the error.

Suggested fix: Use optional_env("NEARAI_BASE_URL")? for the explicit check too, or extract the result into a shared let-binding.

Comment thread src/config/helpers.rs
// or broken resolvers).
let host_owned = host.to_string();
let resolve = move || -> std::io::Result<Vec<IpAddr>> {
use std::sync::mpsc;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium: DNS timeout thread leaks ~8MB stack if resolution hangs

When the 10s DNS timeout fires, the spawned std::thread continues running indefinitely (blocked in to_socket_addrs()). In pathological DNS environments, repeated config-load attempts could accumulate zombie threads.

Acceptable for config-time (bounded number of calls at startup), but should be documented.

Suggested fix: Document as a known limitation. Consider std::thread::Builder::new().name("dns-timeout".into()).spawn(...) for debuggability.

Comment thread src/config/llm.rs
// settings). Registry-provided defaults are hardcoded known-good
// URLs that don't need DNS-based SSRF validation — validating them
// causes spurious failures in offline / sandboxed environments.
if explicit_base_url.is_some() && !base_url.is_empty() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium: Skipping SSRF validation for registry defaults assumes trusted registry

Skipping validate_base_url for registry-default URLs is safe only if the registry content is trusted. If ProviderRegistry is ever loaded from untrusted sources (user-editable config, remote fetches), an attacker could inject a malicious default_base_url that bypasses SSRF validation entirely.

Suggested fix: Add a comment documenting the trust assumption: "Registry defaults are compile-time constants from src/llm/providers/*.json — if registry loading changes to accept external sources, this must be revisited."

@serrrfirat

Copy link
Copy Markdown
Collaborator

🏗️ Paranoid Architect Review — PR #2179

Verdict: ✅ APPROVE with medium fixes

Summary

Severity Count
Medium 3

Assessment

Well-motivated fix for CI failures in sandboxed/offline environments. The core approach — skipping DNS validation for hardcoded default URLs — is correct. The DNS timeout addition is good defense-in-depth. The custom.rs tunnel health check fix is a genuine bug fix.

Three medium concerns:

  1. .ok().flatten() pattern silently swallows parse errors (fragile, could mask SSRF bypass)
  2. DNS timeout threads leak stack memory if resolution hangs indefinitely
  3. Registry default trust assumption should be documented

Note: This appears to be a duplicate of #2183. Consider closing one.

🤖 Generated with Claude Code

zmanian pushed a commit that referenced this pull request Apr 9, 2026
Six categories of test failures in sandboxed CI:

1. DNS resolution hangs: Add 10s timeout to validate_base_url DNS lookups
   and test helpers (helpers.rs) to prevent indefinite blocking in
   restricted-DNS environments.

2. Hardcoded URL validation: Skip DNS-based SSRF validation for
   hardcoded default URLs (nearai, openai codex) that don't need it,
   only validate explicitly-configured URLs (llm.rs).

3. Privileged port binding: Handle root/container environments where
   port 1 bind succeeds by falling back to already-occupied port
   assertion (webhook_server.rs).

4. DNS-dependent assertions: Accept DNS resolution errors alongside
   expected skip/pass messages in doctor tests (doctor.rs).

5. External DNS in tests: Skip test_validate_url_safe_https when DNS
   is unavailable (auth.rs), use localhost URLs in test fixtures (llm.rs).

6. Health check accuracy: Use is_ok_and to check response status, not
   just connection success (custom.rs).

Supersedes #2133, #2151, #2179 (rebased onto current staging).

https://claude.ai/code/session_01NPAmzdyUAoxRvbQHBEAfhw
@zmanian

zmanian commented Apr 12, 2026

Copy link
Copy Markdown
Collaborator Author

Closing: superseded by #2080 (production NearAI URL gating) + #2257 (test-layer hardening). These two PRs together cover this fix more cleanly.

@zmanian zmanian closed this Apr 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor: core 20+ merged PRs risk: medium Business logic, config, or moderate-risk modules scope: channel/cli TUI / CLI channel scope: tool/mcp MCP client size: M 50-199 changed lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants