improvement(interception): keep tunnels serving with prime-tunnel by kcoopermiller · Pull Request #3041 · PrimeIntellect-ai/verifiers

kcoopermiller · 2026-11-02T18:01:17Z

Problem

A tunnel can look connected while requests through its URL fail. The current watchdog only checks that status, so it never restarts frpc, and the elastic pool keeps sending new rollouts to the broken server.

What this PR changes

Tunnels are checked through their URL. _watch is removed. PrimeTunnel.expose now runs Tunnel.keep_serving() from prime-tunnel 0.1.17, which restarts frpc under the same URL when the URL fails but the local service still answers.
Servers get a health route. Interception servers and tool servers answer GET /vf/health with ok, without auth. keep_serving probes this route.
Interception servers whose tunnel is gone are retired. When a tunnel's registration can no longer be used, the server's rollouts fail with TunnelError and it refuses new ones. The elastic pool starts a replacement and stops the old server once its rollouts have left. A static pool uses its remaining servers, and fails the rollout if none are left.
Restarts show up in run logs. TunnelLogHandler now listens to prime_tunnel instead of prime_tunnel.frpc.

Behaviour to review

A long network failure can restart a working frpc. keep_serving restarts frpc if the URL check fails twice, 60 s apart, while the local service answers. That drops requests in flight, even if the failure is only on the host's path to the URL.
A deleted or expired tunnel that's still serving stays in use. It's retired only after it stops serving and a restart is rejected.

Note

Medium Risk
Changes remote-eval critical path (tunnel keepalive, server retirement, and pool routing); mis-handled tunnel loss or aggressive frpc restarts could fail or disrupt in-flight rollouts.

Overview
Replaces the in-process tunnel watchdog (periodic status() checks and manual frpc restarts) with prime-tunnel 0.1.17’s keep_serving(HEALTH_PATH), so liveness is judged by whether requests through the public URL reach the local service. Minimum dependency is bumped to prime-tunnel>=0.1.17.

Interception and MCP servers now expose unauthenticated GET /vf/health (HEALTH_PATH) returning ok for those probes. PrimeTunnel wires Tunnel.on_gone when registration is permanently lost; InterceptionServer marks itself unhealthy, sets active rollouts’ TunnelError, releases sessions, and blocks new registrations. Pools skip unhealthy servers—elastic spins up replacements and shielded _free_retired stops old listeners once load is zero; static raises TunnelError when every server’s tunnel is gone. Tunnel SDK logs are forwarded from the prime_tunnel logger.

Unit tests drop the old watchdog matrix and add coverage for keep-serving lifecycle, gone callbacks, the health route, and pool retirement.

^{Reviewed by Cursor Bugbot for commit b60e598. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Keep prime tunnels serving via SDK `keep_serving` and fail rollouts on tunnel loss

Replaces the polling/restart watchdog in PrimeTunnel.expose with the prime-tunnel SDK's keep_serving using HEALTH_PATH; permanent loss (TunnelGoneError) fires the new on_gone callback. Cleanup cancels the monitor task and stops the client via a worker thread. Bumps prime-tunnel minimum to 0.1.17 (pyproject.toml)
InterceptionServer now starts healthy, serves an ok health route at HEALTH_PATH, and on tunnel loss marks itself unhealthy, fails registered sessions with TunnelError, and refuses new registrations (server.py)
StaticInterceptionPool.acquire selects only healthy servers and raises TunnelError when none exist; ElasticInterceptionPool stops and removes retired zero-load servers, and skips unhealthy or full servers when creating new ones (pool.py)
Tunnel log handler attaches to the prime_tunnel logger and strips a leading "Tunnel " prefix (prime.py). The MCP server also serves the health route (server.py)
Tests: watchdog status/restart cases removed; new tests cover keep-serving lifecycle, gone handling, health route, and pool retirement. Risk: StaticInterceptionPool.acquire now raises TunnelError where it previously could return an unhealthy server; the removed _keep_serving restart loop means tunnels are no longer restarted in-process

Changes since #3041 opened

Changed the value of the HEALTH_PATH constant in the verifiers.v1.interception.tunnel.base module from /__verifiers/healthz to /vf/health. [c81e4f0]
Changed the TunnelError message raised in StaticInterceptionPool.acquire when no healthy interception servers remain. [6e2dbf9]
Updated the docstring of the TunnelError exception class. [6e2dbf9]

^{Macroscope summarized b60e598.}

The watchdog only reads the tunnel service's status, so a tunnel that stays connected without serving is never restarted, and the elastic pool keeps assigning rollouts to it. prime-tunnel 0.1.16 has Tunnel.keep_serving(), which probes a health route through the URL, so the watchdog goes. Interception servers and tool servers both answer the health route. A tunnel that can never work again now retires its server: its rollouts fail with TunnelError and the pools stop choosing it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kcoopermiller · 2026-11-02T18:03:59Z

@codex review

chatgpt-codex-connector · 2026-11-02T18:04:18Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-03T00:55:31.120387Z	`10ffdd0`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector · 2026-11-02T18:05:52Z

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 94fd920ac4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

cursor

Stale Bugbot comment from a previous run.

macroscopeapp · 2026-11-02T18:12:35Z

Approvability

Verdict: Approved at b60e598

Macroscope's review found this PR approvable — Tunnel recovery now uses end-to-end health checks, and permanently lost interception tunnels retire cleanly instead of accepting more rollouts. Health endpoints expose only a constant response, while replacement and cleanup remain scoped to affected runtime servers.

Notes:

Diff unchanged. Approvability was decided on eligibility alone.

^{You can add or adjust custom eligibility rules. Learn more.}

A retired server kept its listener and client pool until the pool stopped, so tunnel failures in a long run would pile them up. Stop one when the pool next picks a server, once its rollouts have left. A rollout still on it is using its client. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kcoopermiller · 2026-11-02T18:38:34Z

@codex review

chatgpt-codex-connector · 2026-11-02T18:40:08Z

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: dd2bf28426

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit dd2bf28. Configure here.}

…cancelled A rollout cancelled while its acquire was stopping a retired server could interrupt the listener cleanup. The server's exit stack is already drained by then, so the pool's own teardown wouldn't close the listener either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kcoopermiller · 2026-11-02T19:01:20Z

@codex review

chatgpt-codex-connector · 2026-11-02T19:03:32Z

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 0cce892a56

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

keep_serving now waits 60 s between its two probes, so a short failure of the check on the host doesn't restart a working frpc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…els-serving Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kcoopermiller · 2026-11-03T00:53:47Z

@codex review

chatgpt-codex-connector · 2026-11-03T00:55:29Z

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 10ffdd0035

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…els-serving Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cursor Bot reviewed Nov 2, 2026

View reviewed changes

Comment thread verifiers/v1/interception/pool.py Resolved

cursor Bot reviewed Nov 2, 2026

View reviewed changes

Comment thread verifiers/v1/interception/pool.py Resolved

macroscopeapp Bot previously approved these changes Nov 2, 2026

View reviewed changes

chore(deps): require prime-tunnel 0.1.17

a05ff44

keep_serving now waits 60 s between its two probes, so a short failure of the check on the host doesn't restart a working frpc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kcoopermiller dismissed macroscopeapp[bot]’s stale review via a05ff44 November 2, 2026 20:32

macroscopeapp Bot previously approved these changes Nov 2, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into improvement/keep-tunn…

7aef005

…els-serving Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kcoopermiller dismissed macroscopeapp[bot]’s stale review via 7aef005 November 2, 2026 22:38

Merge remote-tracking branch 'origin/main' into improvement/keep-tunn…

10ffdd0

…els-serving Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

mikasenghaas reviewed Nov 3, 2026

View reviewed changes

Comment thread verifiers/v1/interception/tunnel/base.py Resolved

Comment thread verifiers/v1/interception/pool.py Resolved

kcoopermiller and others added 3 commits November 2, 2026 17:18

improvement(interception): serve the health route at /vf/health

c81e4f0

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

improvement(interception): explain why a static pool has no servers left

6e2dbf9

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into improvement/keep-tunn…

b60e598

…els-serving Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

mikasenghaas approved these changes Nov 3, 2026

View reviewed changes

macroscopeapp Bot approved these changes Nov 3, 2026

View reviewed changes

kcoopermiller merged commit d2c3fcb into main Nov 3, 2026
64 of 65 checks passed

kcoopermiller deleted the improvement/keep-tunnels-serving branch November 3, 2026 01:39

kcoopermiller mentioned this pull request Nov 3, 2026

chore(deps): bump verifiers to 29623317 for self-healing tunnels PrimeIntellect-ai/prime-rl#4374

Open

mikasenghaas mentioned this pull request Nov 5, 2026

feat: run a relay between the harness and the interception server #3074

Open

Conversation

kcoopermiller commented Nov 2, 2026 • edited by macroscopeapp Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

What this PR changes

Behaviour to review

Keep prime tunnels serving via SDK keep_serving and fail rollouts on tunnel loss

Changes since #3041 opened

Uh oh!

kcoopermiller commented Nov 2, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot commented Nov 2, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

macroscopeapp Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approvability

Uh oh!

kcoopermiller commented Nov 2, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 2, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kcoopermiller commented Nov 2, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 2, 2026

Uh oh!

kcoopermiller commented Nov 3, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kcoopermiller commented Nov 2, 2026 •

edited by macroscopeapp Bot

Loading

Keep prime tunnels serving via SDK `keep_serving` and fail rollouts on tunnel loss

chatgpt-codex-connector Bot commented Nov 2, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

macroscopeapp Bot commented Nov 2, 2026 •

edited

Loading