fix(sandbox): restore sandbox DNS resolution for web tools (fixes #626) - #1062
Merged
Conversation
…NVIDIA#626) The sandbox runs in an isolated network namespace (10.200.0.0/24) where OpenShell's iptables rules reject all non-proxy traffic including UDP. This causes getaddrinfo EAI_AGAIN for every outbound DNS lookup, breaking web_fetch, web_search, and all Node.js HTTP tools. Fix (three steps in new scripts/setup-dns-proxy.sh): 1. Run a Python UDP DNS forwarder on the pod-side veth gateway (10.200.0.1:53), forwarding to the real CoreDNS pod IP 2. Add an iptables rule in the sandbox namespace allowing UDP to the gateway on port 53 (the only non-proxy firewall exception) 3. Update the sandbox's /etc/resolv.conf to point to 10.200.0.1 Also broadens the CoreDNS patch (fix-coredns.sh / shouldPatchCoredns) to run on all Docker-based runtimes, not just Colima — k3s-inside-Docker has broken DNS forwarding on Linux hosts with systemd-resolved too. Inspired by PR NVIDIA#732's DNS forwarder approach. Key differences: - Uses kubectl exec instead of nsenter (fixes PR NVIDIA#732's launch bug) - Handles the sandbox iptables constraint (all UDP blocked) - Resolves systemd-resolved upstreams via resolvectl before falling back to 8.8.8.8 - Uses grep -F for fixed-string sandbox name matching Tested on DGX Spark (ARM64): fresh destroy + onboard, getent hosts and node dns.lookup both resolve from inside the sandbox.
setup-dns-proxy.sh now verifies all three DNS bridge layers from inside the sandbox namespace post-deployment: resolv.conf target, iptables UDP rule, and actual getent hosts resolution. Reports [PASS]/[FAIL] per check so failures are immediately visible in onboard logs.
Contributor
📝 WalkthroughWalkthroughThe changes expand CoreDNS patching and DNS forwarding capabilities beyond Colima to all identified container runtimes, introduce a new DNS proxy setup script for isolated sandbox network namespaces, refactor Docker host detection logic, and add corresponding test coverage for the new functionality and modified behaviors. Changes
Sequence DiagramsequenceDiagram
actor User
participant Gateway as Gateway Container
participant K8s as Kubernetes API
participant Core as CoreDNS Pod
participant Target as Target Sandbox Pod
participant NS as Sandbox Netns
User->>Gateway: scripts/setup-dns-proxy.sh (gateway, sandbox)
Gateway->>K8s: kubectl get endpoints kube-dns
K8s-->>Gateway: CoreDNS Pod IP (e.g., 10.200.0.2)
Gateway->>K8s: kubectl get pod (target sandbox)
K8s-->>Gateway: Sandbox pod details
Gateway->>Target: Deploy Python DNS forwarder
Target->>Target: Start dns-proxy.py (listen on 10.200.0.1:53)
Target-->>Gateway: Forwarder PID logged
Gateway->>NS: Insert iptables OUTPUT rule (ACCEPT UDP/53)
Gateway->>Target: Overwrite /etc/resolv.conf
Gateway->>NS: getent hosts github.com (verify DNS)
NS-->>Gateway: Resolution result
Gateway-->>User: Verification pass/fail report
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (2)
scripts/setup-dns-proxy.sh (2)
122-135: Consider adding basic error logging in the DNS forwarder.The
except Exception: passblock silently swallows all forwarding errors, which can make debugging DNS issues difficult. Consider logging failures to/tmp/dns-proxy.logfor observability.♻️ Suggested improvement
def forward(data, addr): try: f = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) f.settimeout(5) f.sendto(data, UPSTREAM) r, _ = f.recvfrom(4096) sock.sendto(r, addr) f.close() - except Exception: - pass + except Exception as e: + try: + with open('/tmp/dns-proxy.log', 'a') as log: + log.write('forward error: {} -> {}\n'.format(addr, e)) + except: + pass🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@scripts/setup-dns-proxy.sh` around lines 122 - 135, The forward function currently swallows all exceptions (except Exception: pass) which hides forwarding failures; update forward to catch exceptions, log the error with context (include exception message and stack trace) to /tmp/dns-proxy.log (append mode), and ensure the upstream socket f is closed in a finally block; reference function name forward and variable UPSTREAM and sock so you add logging around the f.sendto/f.recvfrom and sock.sendto calls and write timestamped messages to /tmp/dns-proxy.log for observability.
133-135: Unbounded thread creation under high DNS load.Each incoming DNS query spawns a new daemon thread with no concurrency limit. Under heavy DNS load, this could exhaust resources. For a sandbox environment with limited DNS traffic, this is likely acceptable, but worth noting.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@scripts/setup-dns-proxy.sh` around lines 133 - 135, The loop currently spawns an unbounded daemon thread per incoming packet (sock.recvfrom -> threading.Thread(target=forward, args=(d, a), daemon=True).start()), which can exhaust resources under load; change to a bounded worker pool by using a ThreadPoolExecutor or a fixed set of worker threads reading from a Queue and submit tasks to it (or protect creation with a Semaphore) so only a limited number of concurrent forward(...) calls run; update the loop to submit incoming (d, a) to the pool/queue and ensure forward is compatible with being called by pooled workers.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@scripts/setup-dns-proxy.sh`:
- Around line 122-135: The forward function currently swallows all exceptions
(except Exception: pass) which hides forwarding failures; update forward to
catch exceptions, log the error with context (include exception message and
stack trace) to /tmp/dns-proxy.log (append mode), and ensure the upstream socket
f is closed in a finally block; reference function name forward and variable
UPSTREAM and sock so you add logging around the f.sendto/f.recvfrom and
sock.sendto calls and write timestamped messages to /tmp/dns-proxy.log for
observability.
- Around line 133-135: The loop currently spawns an unbounded daemon thread per
incoming packet (sock.recvfrom -> threading.Thread(target=forward, args=(d, a),
daemon=True).start()), which can exhaust resources under load; change to a
bounded worker pool by using a ThreadPoolExecutor or a fixed set of worker
threads reading from a Queue and submit tasks to it (or protect creation with a
Semaphore) so only a limited number of concurrent forward(...) calls run; update
the loop to submit incoming (d, a) to the pool/queue and ensure forward is
compatible with being called by pooled workers.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: a4be866d-0ab1-4580-9dc4-2f5e14380254
📒 Files selected for processing (7)
bin/lib/onboard.jsbin/lib/platform.jsscripts/fix-coredns.shscripts/setup-dns-proxy.shscripts/setup.shtest/dns-proxy.test.jstest/platform.test.js
kjw3
approved these changes
Mar 29, 2026
Contributor
|
I reran this end to end and the result looks good. Validation I ran:
What I saw during onboard:
What I verified inside the fresh sandbox after onboard:
I also did one extra check with Node So from my side this looks like the right fix for
|
2 tasks
laitingsheng
pushed a commit
that referenced
this pull request
Apr 2, 2026
… (#1062) * fix(sandbox): add DNS forwarder so web_fetch resolves hostnames (fixes #626) The sandbox runs in an isolated network namespace (10.200.0.0/24) where OpenShell's iptables rules reject all non-proxy traffic including UDP. This causes getaddrinfo EAI_AGAIN for every outbound DNS lookup, breaking web_fetch, web_search, and all Node.js HTTP tools. Fix (three steps in new scripts/setup-dns-proxy.sh): 1. Run a Python UDP DNS forwarder on the pod-side veth gateway (10.200.0.1:53), forwarding to the real CoreDNS pod IP 2. Add an iptables rule in the sandbox namespace allowing UDP to the gateway on port 53 (the only non-proxy firewall exception) 3. Update the sandbox's /etc/resolv.conf to point to 10.200.0.1 Also broadens the CoreDNS patch (fix-coredns.sh / shouldPatchCoredns) to run on all Docker-based runtimes, not just Colima — k3s-inside-Docker has broken DNS forwarding on Linux hosts with systemd-resolved too. Inspired by PR #732's DNS forwarder approach. Key differences: - Uses kubectl exec instead of nsenter (fixes PR #732's launch bug) - Handles the sandbox iptables constraint (all UDP blocked) - Resolves systemd-resolved upstreams via resolvectl before falling back to 8.8.8.8 - Uses grep -F for fixed-string sandbox name matching Tested on DGX Spark (ARM64): fresh destroy + onboard, getent hosts and node dns.lookup both resolve from inside the sandbox. * fix(sandbox): add runtime DNS verification after setup setup-dns-proxy.sh now verifies all three DNS bridge layers from inside the sandbox namespace post-deployment: resolv.conf target, iptables UDP rule, and actual getent hosts resolution. Reports [PASS]/[FAIL] per check so failures are immediately visible in onboard logs.
lakamsani
pushed a commit
to lakamsani/NemoClaw
that referenced
this pull request
Apr 4, 2026
…DIA#626) (NVIDIA#1062) * fix(sandbox): add DNS forwarder so web_fetch resolves hostnames (fixes NVIDIA#626) The sandbox runs in an isolated network namespace (10.200.0.0/24) where OpenShell's iptables rules reject all non-proxy traffic including UDP. This causes getaddrinfo EAI_AGAIN for every outbound DNS lookup, breaking web_fetch, web_search, and all Node.js HTTP tools. Fix (three steps in new scripts/setup-dns-proxy.sh): 1. Run a Python UDP DNS forwarder on the pod-side veth gateway (10.200.0.1:53), forwarding to the real CoreDNS pod IP 2. Add an iptables rule in the sandbox namespace allowing UDP to the gateway on port 53 (the only non-proxy firewall exception) 3. Update the sandbox's /etc/resolv.conf to point to 10.200.0.1 Also broadens the CoreDNS patch (fix-coredns.sh / shouldPatchCoredns) to run on all Docker-based runtimes, not just Colima — k3s-inside-Docker has broken DNS forwarding on Linux hosts with systemd-resolved too. Inspired by PR NVIDIA#732's DNS forwarder approach. Key differences: - Uses kubectl exec instead of nsenter (fixes PR NVIDIA#732's launch bug) - Handles the sandbox iptables constraint (all UDP blocked) - Resolves systemd-resolved upstreams via resolvectl before falling back to 8.8.8.8 - Uses grep -F for fixed-string sandbox name matching Tested on DGX Spark (ARM64): fresh destroy + onboard, getent hosts and node dns.lookup both resolve from inside the sandbox. * fix(sandbox): add runtime DNS verification after setup setup-dns-proxy.sh now verifies all three DNS bridge layers from inside the sandbox namespace post-deployment: resolv.conf target, iptables UDP rule, and actual getent hosts resolution. Reports [PASS]/[FAIL] per check so failures are immediately visible in onboard logs.
gemini2026
pushed a commit
to gemini2026/NemoClaw
that referenced
this pull request
Apr 14, 2026
…DIA#626) (NVIDIA#1062) * fix(sandbox): add DNS forwarder so web_fetch resolves hostnames (fixes NVIDIA#626) The sandbox runs in an isolated network namespace (10.200.0.0/24) where OpenShell's iptables rules reject all non-proxy traffic including UDP. This causes getaddrinfo EAI_AGAIN for every outbound DNS lookup, breaking web_fetch, web_search, and all Node.js HTTP tools. Fix (three steps in new scripts/setup-dns-proxy.sh): 1. Run a Python UDP DNS forwarder on the pod-side veth gateway (10.200.0.1:53), forwarding to the real CoreDNS pod IP 2. Add an iptables rule in the sandbox namespace allowing UDP to the gateway on port 53 (the only non-proxy firewall exception) 3. Update the sandbox's /etc/resolv.conf to point to 10.200.0.1 Also broadens the CoreDNS patch (fix-coredns.sh / shouldPatchCoredns) to run on all Docker-based runtimes, not just Colima — k3s-inside-Docker has broken DNS forwarding on Linux hosts with systemd-resolved too. Inspired by PR NVIDIA#732's DNS forwarder approach. Key differences: - Uses kubectl exec instead of nsenter (fixes PR NVIDIA#732's launch bug) - Handles the sandbox iptables constraint (all UDP blocked) - Resolves systemd-resolved upstreams via resolvectl before falling back to 8.8.8.8 - Uses grep -F for fixed-string sandbox name matching Tested on DGX Spark (ARM64): fresh destroy + onboard, getent hosts and node dns.lookup both resolve from inside the sandbox. * fix(sandbox): add runtime DNS verification after setup setup-dns-proxy.sh now verifies all three DNS bridge layers from inside the sandbox namespace post-deployment: resolv.conf target, iptables UDP rule, and actual getent hosts resolution. Reports [PASS]/[FAIL] per check so failures are immediately visible in onboard logs.
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The sandbox network namespace has no working DNS —
getaddrinforeturnsEAI_AGAINfor every public hostname. This breaksweb_fetch,web_search, and any Node.js tool that resolves hostnames before connecting.Root cause: Two independent problems block DNS from the sandbox:
resolv.confpoints to CoreDNS ClusterIP (10.43.0.10), which is only routable in the pod namespace — not the sandbox's nested namespace (10.200.0.0/24)Fix: A three-step DNS bridge deployed automatically during onboard:
10.200.0.1:53), forwarding to the CoreDNS pod IP10.200.0.1:53(the only non-proxy firewall exception)resolv.confpointing to10.200.0.1instead of the unreachable10.43.0.10Additionally broadens the CoreDNS upstream patch to all Docker-based runtimes (not just Colima), since k3s-inside-Docker has broken DNS forwarding on all platforms.
DNS setup is applied automatically during onboard/setup and currently remains best-effort: failures are logged clearly but do not block sandbox creation.
Credit
The DNS forwarder approach originated in #732 by @jestyr27. The core idea — run a lightweight Python UDP forwarder in the pod to relay DNS queries from the sandbox — was correct and materially guided this fix.
The part that needed more iteration was the deployment mechanism (
nsentervskubectl exec) and the discovery that the sandbox iptables reject all UDP to non-loopback addresses, which meant the forwarder address and firewall rules had to be coordinated together. This PR is a clean rewrite addressing all 6 CodeRabbit issues from #732 plus the iptables constraint it didn't account for.Changes
scripts/setup-dns-proxy.shtest/dns-proxy.test.jsbin/lib/onboard.jssetup-dns-proxy.shafter sandbox creationbin/lib/platform.jsshouldPatchCoredns()→ true for all runtimes (not just Colima)scripts/fix-coredns.shscripts/setup.shtest/platform.test.jsshouldPatchCorednsTesting
Unit tests:
npm test— 616 passed, 2 skipped (DGX Spark ARM64)make check— lint, format, shellcheck all greenBuilt-in runtime verification:
setup-dns-proxy.shnow verifies all three layers post-deployment from inside the sandbox namespace: (1)resolv.confpoints to the veth gateway, (2) the UDP DNSiptablesrule is present (iptables -C OUTPUT), and (3) actual DNS resolution works (getent hosts github.com). Output shows[PASS]/[FAIL]per check.End-to-end (fresh destroy + onboard on both platforms):
10.200.0.1:53 → 10.42.0.610.200.0.1:53 → 10.42.0.2getent hosts google.comgetent hosts techcrunch.comdns.lookup()getaddrinfo()web_search(Perplexity)web_fetchFirewall proof (
iptables -S OUTPUTinside sandbox namespace)DGX Spark (ARM64):
Brev VM (x86_64):
The UDP DNS rule (
-A OUTPUT -d 10.200.0.1/32 -p udp --dport 53 -j ACCEPT) is inserted at position 1 bysetup-dns-proxy.sh, appearing before the blanket UDPREJECT. Without this rule, all DNS queries are dropped.Related
getaddrinfo EAI_AGAIN api.search.brave.com#414 — SameEAI_AGAINsymptom on Windows/WSL (may also benefit)web_fetchSSRF proxy fix (pending)Summary by CodeRabbit
Bug Fixes
New Features
Tests