Skip to content

fix(network): default to IPv4-first DNS ordering to avoid dead-IPv6-route timeouts - #71373

Closed
toprakeker wants to merge 1 commit into
NousResearch:mainfrom
toprakeker:fix/ipv6-fallback-timeout-71215
Closed

fix(network): default to IPv4-first DNS ordering to avoid dead-IPv6-route timeouts#71373
toprakeker wants to merge 1 commit into
NousResearch:mainfrom
toprakeker:fix/ipv6-fallback-timeout-71215

Conversation

@toprakeker

Copy link
Copy Markdown
Contributor

Fixes #71215.

Problem

Dual-stack hosts whose providers publish AAAA records that route nowhere pay a 10–30 s IPv6 connect timeout on every API call before falling back to IPv4. Python's socket.getaddrinfo() tries IPv6 first per the default address ordering. The existing workaround (network.force_ipv4: true) fully disables IPv6, which is too aggressive for hosts that genuinely need it for some destinations.

Approach

Add a lighter-weight network.ipv4_first option (on by default) that monkey-patches socket.getaddrinfo to sort results IPv4-first instead of filtering IPv6 out entirely. Both A and AAAA records remain available — IPv4 is just tried first. If no A record exists (pure-IPv6 host), the original ordering is preserved, so IPv6 connectivity is retained.

This is applied alongside (and before) the existing apply_ipv4_preference at all three entrypoints:

  • gateway/run.py (gateway bootstrap)
  • cron/scheduler.py (cron scheduler init)
  • hermes_cli/main.py (early CLI bootstrap)

Changes

  • hermes_constants.py: new apply_ipv6_fallback_ordering(enabled: bool = True) matching the style of apply_ipv4_preference (same guard-against-double-patch pattern, same _hermes_*_patched attribute convention).
  • hermes_cli/config.py: add "ipv4_first": True to DEFAULT_CONFIG["network"] alongside force_ipv4.
  • gateway/run.py, cron/scheduler.py, hermes_cli/main.py: wire apply_ipv6_fallback_ordering into the same blocks that call apply_ipv4_preference. ipv4_first defaults to True, so the fix is active out of the box.
  • tests/test_ipv4_first.py: new test file mirroring test_ipv4_preference.py — covers no-op when disabled, patching, double-patch safety, IPv4-first sorting of AF_UNSPEC results, explicit-family pass-through, IPv6-only preservation, and config default.

Relationship to #52538

PR #52538 (open) proposes auto-detection of dead IPv6 routes (probing + caching). This PR is a smaller-scope, immediately-shippable complement: it changes the default DNS ordering so IPv4 is tried first, which eliminates the timeout for the common case without the complexity of a detection/cache layer. The two approaches compose — if #52538 lands, ipv4_first can become a fallback for when auto-detection is inconclusive, or be demoted to opt-in.

Testing

pytest tests/test_ipv4_first.py tests/test_ipv4_preference.py -v14 passed.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cli CLI entry point, hermes_cli/, setup wizard comp/gateway Gateway runner, session dispatch, delivery comp/cron Cron scheduler and job management area/config Config system, migrations, profiles needs-decision Awaiting maintainer decision before any implementation sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Jul 25, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related to #52538 and #71215: both address dead-IPv6-route delays, but this patch globally defaults to IPv4-first while #52538 probes for a dead route before enabling an IPv4-only fallback. This is a policy/mechanism decision, not a duplicate.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the focused transport-level fix. Current main still has only the opt-in force_ipv4 path: apply_ipv4_preference() is a no-op unless forced (hermes_constants.py:1205-1206), and gateway bootstrap calls it only for network.force_ipv4 (gateway/run.py:2122-2127).

Problems

  • The CLI default is skipped when config.yaml does not exist. The proposed flag assignment remains under _cfg_path.exists() (hermes_cli/main.py:711), so a configless profile never reaches the new IPv4-first call.
  • Current main moved DEFAULT_CONFIG into hermes_cli/config_defaults.py (hermes_cli/config.py:935; network defaults at config_defaults.py:2298-2304), so the config-default hunk needs relocation.
  • The config reference currently documents only force_ipv4 (website/docs/user-guide/configuration.md:2265-2274).

Suggested changes

  • Apply the default outside the config-exists guard while preserving explicit ipv4_first: false, with a configless-profile regression test.
  • Move the default to config_defaults.py and document the new option and its interaction with force_ipv4.

This is an automated hermes-sweeper review.

Comment thread hermes_cli/main.py Outdated
_FORCE_IPV4_EARLY = True
# ipv4_first defaults to True — apply the lighter-weight IPv4-first
# DNS ordering unless explicitly disabled.
if _early_net_cfg.get("ipv4_first", True):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assignment is still inside the surrounding _cfg_path.exists() guard. A new/configless profile therefore leaves _IPV4_FIRST_EARLY false and never applies the claimed default; initialize the default outside that guard and override it only when raw config explicitly disables it.

@toprakeker

Copy link
Copy Markdown
Contributor Author

Thanks — I addressed the review points in the PR branch.\n\nChanges:\n- CLI now applies the default network.ipv4_first: true even when config.yaml does not exist\n- explicit network.ipv4_first: false still disables the lighter-weight ordering\n- docs now cover ipv4_first and explain its relationship to force_ipv4\n- added CLI startup regressions for configless profile and explicit disable\n\nValidation run locally:\n\n\npython -m pytest tests/test_ipv4_first.py tests/test_ipv4_first_cli_startup.py tests/test_ipv4_preference.py -q -o 'addopts='\n# 16 passed\n\n\nNote: I also verified the current-main rebase locally and moved the default to hermes_cli/config_defaults.py there; pushing the rebased branch is blocked by this token lacking GitHub's workflow scope because upstream main contains workflow changes. The branch update here keeps the PR pushable while preserving the requested behavior/test/docs fixes.

@toprakeker
toprakeker force-pushed the fix/ipv6-fallback-timeout-71215 branch from e96abdf to 8dd3bee Compare July 30, 2026 11:15
@teknium1 teknium1 added the sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit label Jul 30, 2026
@toprakeker
toprakeker requested a review from teknium1 July 30, 2026 12:31
@toprakeker

Copy link
Copy Markdown
Contributor Author

Closing this older pre-contributor branch. Network/bootstrap architecture has changed since this was opened, and the fork CI still requires manual approval. I will re-check the underlying IPv6 timeout issue against current main before proposing any fresh fix.

@toprakeker toprakeker closed this Aug 9, 2026
@toprakeker
toprakeker deleted the fix/ipv6-fallback-timeout-71215 branch August 9, 2026 18:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery needs-decision Awaiting maintainer decision before any implementation P2 Medium — degraded but workaround exists sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Token Plan endpoint IPv6 unreachable — every API call adds 10-30s timeout on dual-stack hosts

3 participants