fix(agent): stop re-probing endpoints that blackhole TCP connects (salvage #71282) - #77847
Merged
kshitijk4poor merged 1 commit intoAug 3, 2026
Merged
Conversation
Salvage of NousResearch#71282 (Fixes NousResearch#71281): a routable-but-dead endpoint (corp LAN address while off-VPN) blackholes TCP SYNs, so every probe in the model-metadata waterfall waits out its full connect timeout — 20+ seconds of stall per startup across detect_local_server_type, fetch_endpoint_model_metadata, and the per-model probes. A module-level blackhole cache keyed on host:port is populated when any probe observes a ConnectTimeout (httpx or requests; read timeouts deliberately excluded — an accepted connection is not a blackhole) and consulted at the top of each guarded function. 30s TTL: long enough to collapse one startup burst, short enough that VPN recovery is picked up without a restart. Guard ordering: blackhole check -> disk L2 -> HTTP waterfall, and a blackholed leg aborts the remaining legs instead of letting each stall in turn. Squash of the PR's two real commits (the branch's merge commits made it un-rebase-merge-able; content verified identical via merge-tree).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Salvages #71282 by @rlaope — the PR's two real commits squashed with authorship preserved. Fixes #71281.
Context — what this fixes, for whom
Anyone with a configured endpoint that's routable but dead (the classic case: a corp-LAN model server address while off-VPN): TCP SYNs blackhole — no SYN-ACK, no RST, no ICMP — so every probe in the model-metadata waterfall waits out its FULL connect timeout. Across
detect_local_server_type(4 legs),fetch_endpoint_model_metadata(2 candidates), and the per-model probes, one dead host stalls startup 20+ seconds.The fix: a module-level blackhole cache keyed host:port, populated when any probe observes a ConnectTimeout (read timeouts deliberately excluded — an accepted connection is not a blackhole), consulted at the top of each guarded function, 30s TTL so VPN recovery is picked up without a restart. A blackholed leg also aborts the remaining waterfall legs instead of letting each stall in turn.
Why a salvage rather than arming the original
The original is content-perfect and nearly current (1 behind, MERGEABLE/CLEAN) — but its branch carries two merge commits, and GitHub refuses rebase-merge on branches with merges ("This branch can't be rebased"). Squash-picked the two real commits (5c5abc1 + 3fc03dd); content verified identical via merge-tree.
Review notes (dossier findings, all resolved by the author's own rebase)
_ensure_requests()preserved in fetch_endpoint_model_metadataHERMES_ENDPOINT_BLACKHOLE_TTLenv var in their second commit (config-policy compliant)Verification
tests/agent/test_endpoint_blackhole.py: 22 passed (cache TTL, host:port keying, per-function guards, refused-vs-blackholed negatives)Closes #71282 (superseded by this salvage — original author credited via squash authorship).