Skip to content

Add bounded direct path repair - #846

Merged
i386 merged 4 commits into
mainfrom
codex/direct-path-repair
Jun 13, 2026
Merged

Add bounded direct path repair#846
i386 merged 4 commits into
mainfrom
codex/direct-path-repair

Conversation

@i386

@i386 i386 commented Jun 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a targeted mesh-level direct-path repair flow for peers whose selected iroh path has fallen back to relay or is unknown even though a direct UDP candidate is available.

The repair path uses a dedicated mesh stream byte (0x0e) and a DirectPathRequest frame. It does not use STREAM_SUBPROTOCOL, does not gossip, and only asks one peer per maintenance tick to reverse-dial the requester’s current advertised endpoint address.

Why

In private/mDNS lab meshes, one side may need the other side to initiate the UDP attempt for iroh to converge on a direct path. The previous behavior could sit on relay/unknown path state even when a usable LAN candidate existed. This keeps the fix close to mesh networking and iroh connection management without adding broad discovery chatter.

This follows the iroh layering model: iroh authenticates the endpoint reached by a dial, while mesh-llm constrains which peer-advertised candidates this mesh-level repair path is allowed to hand to iroh.

Flow Diagram

sequenceDiagram
    autonumber
    participant A as Node A maintenance loop
    participant Conn as Existing mesh QUIC connection
    participant B as Node B mesh dispatcher
    participant Iroh as iroh endpoint

    Note over A,B: Existing admitted mesh connection is already alive
    A->>A: Observe selected path = relay/unknown<br/>and peer has direct UDP candidate
    A->>A: Apply grace period, one-at-a-time gate,<br/>sender cooldown, inflight suppression
    A->>Conn: Open bi stream with byte 0x0e
    Conn->>B: DirectPathRequest{requester_id, gen, EndpointAddr}
    B->>B: Validate generation, requester id,<br/>admitted peer, known direct candidate, receiver cooldown
    B->>Iroh: Dial A using only previously advertised direct candidates
    Iroh-->>B: New QUIC connection if endpoint identity verifies
    B->>B: Install connection, dispatch streams,<br/>initiate gossip
Loading

Security / Layering Note

This is only reachable from an already-admitted mesh peer over an existing QUIC connection. iroh still verifies the remote endpoint identity during the reverse dial; a candidate cannot produce an accepted connection unless the remote proves it owns the requested endpoint id.

mesh-llm still has to authorize candidate eligibility before handing addresses to iroh. Otherwise this repair frame could become a new low-rate UDP egress primitive where an admitted peer asks us to try socket candidates that did not come through normal mesh membership. To keep the repair path aligned with iroh’s design, the receiver intersects the requested EndpointAddr with that peer’s already-known membership address and keeps only previously advertised direct IP candidates. Unknown candidates and relay candidates in the request are ignored before any dial attempt or receiver cooldown is recorded.

Details

  • Adds bounded direct-path maintenance with grace period, one-at-a-time planning, sender cooldown, receiver cooldown, and request timeout.
  • Adds DirectPathRequest to the mesh protocol surface and validates generation/requester identity.
  • Wires STREAM_DIRECT_PATH_REQUEST = 0x0e into the mesh stream dispatcher.
  • In LAN-only/mDNS mode, strips public/STUN candidates when a bind IP is selected so advertised addresses point peers at the intended lab interface.
  • Adds focused tests for LAN-only candidate filtering, direct-path maintenance throttling, and rejecting unadvertised direct candidates in repair requests.

Validation

  • cargo fmt --all
  • git diff --check
  • cargo test -p mesh-llm-host-runtime endpoint_addr_filter --lib
  • cargo test -p mesh-llm-host-runtime direct_path_maintenance --lib
  • cargo test -p mesh-llm-host-runtime direct_path --lib
  • cargo test -p mesh-llm-protocol control_plane_messages_constants_are_stable --lib
  • cargo check -p mesh-llm
  • cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings
  • cargo clippy -p mesh-llm --all-targets -- -D warnings
  • cargo clippy -p mesh-llm-protocol --all-targets -- -D warnings
  • just build

Summary by CodeRabbit

  • New Features

    • Automatic direct peer-to-peer path maintenance to discover and establish direct connections, reducing relay use when possible.
  • Bug Fixes / Reliability

    • Improved request validation and cooldown handling to avoid stale or malformed direct-path requests and reduce unnecessary retries.
  • Documentation

    • Clarified stream multiplexing behavior and transport activation details.
  • Tests

    • Added unit tests covering maintenance planning, request cooldowns, and candidate filtering.
  • Chores

    • Added protocol support and configuration hooks for direct-path management.

@coderabbitai

coderabbitai Bot commented Jun 13, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f5c1698a-d60d-4729-a5f8-fbd7ea599392

📥 Commits

Reviewing files that changed from the base of the PR and between d053842 and 0afc72c.

📒 Files selected for processing (6)
  • crates/mesh-llm-host-runtime/src/mesh/direct_path.rs
  • crates/mesh-llm-host-runtime/src/mesh/mod.rs
  • crates/mesh-llm-host-runtime/src/mesh/tests.rs
  • crates/mesh-llm-host-runtime/src/mesh/tests/direct_path.rs
  • crates/mesh-llm-host-runtime/src/protocol/mod.rs
  • crates/mesh-llm-protocol/src/protocol/mod.rs
🚧 Files skipped from review as they are similar to previous changes (3)
  • crates/mesh-llm-host-runtime/src/protocol/mod.rs
  • crates/mesh-llm-protocol/src/protocol/mod.rs
  • crates/mesh-llm-host-runtime/src/mesh/direct_path.rs

📝 Walkthrough

Walkthrough

Adds DirectPathRequest protocol and a periodic direct-path maintenance controller that selects peers for reverse-dial repair, mesh wiring/state for cooldown and filtering, receiver-side validation and reverse-dial installation, and unit tests covering planning, cooldown, and candidate filtering.

Changes

Direct Path Maintenance Feature

Layer / File(s) Summary
Protocol message, stream ID, and frame validation
crates/mesh-llm-protocol/proto/node.proto, crates/mesh-llm-protocol/src/proto/node.rs, crates/mesh-llm-protocol/src/protocol/mod.rs, crates/mesh-llm-host-runtime/src/protocol/mod.rs
Adds DirectPathRequest proto message and STREAM_DIRECT_PATH_REQUEST (0x0e). Extends ControlFrameError with MissingDirectPathAddress and implements validation enforcing protocol generation, 32-byte requester_id, and non-empty serialized_addr.
Mesh state, filtering, initialization, and dispatch wiring
crates/mesh-llm-host-runtime/src/mesh/mod.rs
Updates docs, refactors filter_endpoint_addr_for_bind_ip(preserve_public_ipv4_candidates), adds Node.relay_policy and MeshState.direct_path_request_last_at, initializes them in start/test paths, updates advertisement callers, wires STREAM_DIRECT_PATH_REQUEST dispatch, and registers direct_path module.
Maintenance controller and repair gating
crates/mesh-llm-host-runtime/src/mesh/direct_path.rs (lines 1–165)
Defines maintenance cadence/grace/cooldown constants, DirectPathMaintenanceController, peer health types, plan_request logic, repair reason derivation, and helper to filter requested addresses to previously advertised direct candidates.
Periodic maintenance task and lifecycle start
crates/mesh-llm-host-runtime/src/mesh/direct_path.rs (lines 166–266), crates/mesh-llm-host-runtime/src/mesh/heartbeat.rs, crates/mesh-llm-host-runtime/src/runtime/mod.rs
Adds Node::start_direct_path_maintenance() periodic task that samples observations, uses the controller to plan a single repair per tick, and sends DirectPathRequest over a bidirectional stream. Starts maintenance during node startup flows and prunes stale direct_path_request_last_at entries in heartbeat GC.
Request framing, receiver entrypoint, and reverse-dial flow
crates/mesh-llm-host-runtime/src/mesh/direct_path.rs (lines 267–452)
Implements framing/send utilities, spawn_direct_path_request_stream handler that decodes/validates requests (including QUIC peer id match), filters requested address to previously advertised direct candidates, enforces per-peer receiver cooldown, performs reverse-dial with timeout, and on success installs the direct-path connection and triggers follow-up gossip; failures/timeouts are debug-logged.
Direct-path and endpoint filtering tests
crates/mesh-llm-host-runtime/src/mesh/tests.rs, crates/mesh-llm-host-runtime/src/mesh/tests/direct_path.rs
Adds tests and test wiring: bind-IP filtering behavior, LAN-only candidate stripping, test node relay_policy/state initialization, maintenance planning gating (grace/candidate), inflight/cooldown suppression, and advertised-candidate filtering assertions.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Suggested reviewers

  • ndizazzo
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 34.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title accurately summarizes the main change: adding a bounded direct path repair mechanism for peers to recover from relay/unknown path selection when direct UDP candidates are available.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/direct-path-repair

Comment @coderabbitai help to get the list of available commands and usage tips.

@i386 i386 changed the title [codex] Add bounded direct path repair Add bounded direct path repair Jun 13, 2026
@i386
i386 force-pushed the codex/direct-path-repair branch from e3cd7e0 to 08bb730 Compare June 13, 2026 07:50
@i386
i386 marked this pull request as ready for review June 13, 2026 07:53
@github-actions
github-actions Bot requested a review from ndizazzo June 13, 2026 07:53
@i386
i386 requested a review from michaelneale June 13, 2026 07:56

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (1)
crates/mesh-llm-host-runtime/src/protocol/mod.rs (1)

280-292: ⚡ Quick win

Use a dedicated validation error for missing direct-path address.

Line 290 currently maps an empty serialized_addr to InvalidEndpointId { got: 0 }, which misreports the fault and makes logs harder to triage. Use a direct-path-specific error (and keep it aligned with the protocol crate’s validator behavior).

Suggested direction
-        if self.serialized_addr.is_empty() {
-            return Err(ControlFrameError::InvalidEndpointId { got: 0 });
-        }
+        if self.serialized_addr.is_empty() {
+            return Err(ControlFrameError::MissingDirectPathAddress);
+        }

(Plus add MissingDirectPathAddress to ControlFrameError and its Display impl.)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/mesh-llm-host-runtime/src/protocol/mod.rs` around lines 280 - 292, The
validate_frame implementation for crate::proto::node::DirectPathRequest
currently returns ControlFrameError::InvalidEndpointId when serialized_addr is
empty; change this to return a new, dedicated variant
ControlFrameError::MissingDirectPathAddress (add that variant to the
ControlFrameError enum and update its Display implementation) and update
validate_frame to return MissingDirectPathAddress when
serialized_addr.is_empty() so the error accurately reports a missing direct-path
address and stays aligned with the protocol crate's validator behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/mesh-llm-host-runtime/src/mesh/direct_path.rs`:
- Around line 204-210: The mapping that builds DirectPathObservation currently
iterates over state.connections without ensuring the peer is an admitted mesh
peer; update the closure that produces DirectPathObservation to first check the
peer admission flag (e.g., only proceed when peer.is_admitted or peer.admitted
== true depending on your Peer struct) and skip non-admitted peers so requests
are only planned for admitted peers; keep use of
heartbeat::selected_path_snapshot(conn) and
endpoint_addr_has_direct_candidate(&peer.addr) unchanged for admitted peers.

In `@crates/mesh-llm-host-runtime/src/mesh/mod.rs`:
- Around line 4528-4532: The call to filter_endpoint_addr_for_bind_ip currently
passes self.relay_policy.uses_relay(), which causes
RelayPolicy::ExplicitlyDisabled to be treated like relay-enabled and drops
public candidates; change those call sites (where
filter_endpoint_addr_for_bind_ip is invoked, including the occurrences around
the current context and the similar site ~4670-4674) to instead gate on
raw-STUN/public-discovery mode: use an existing uses_raw_stun() method or add a
dedicated helper (e.g., preserve_public_ipv4_candidates) on RelayPolicy and pass
its boolean result to filter_endpoint_addr_for_bind_ip so ExplicitlyDisabled
preserves public IPv4 candidates. Ensure the change is applied to every call
site mentioned.
- Line 3892: The helper new_test_node_from_endpoint constructs an Endpoint with
RelayMode::Disabled but assigns RelayPolicy::DefaultPublic, causing mismatched
behavior for relay/invite/token tests; update the helper so the Endpoint's
relay_policy matches the intended RelayMode (e.g., set relay_policy to
RelayPolicy::Disabled) or add a parameter to new_test_node_from_endpoint to
accept the desired RelayPolicy and thread it through when building the Endpoint;
ensure the change uses the same symbols (new_test_node_from_endpoint, Endpoint,
RelayMode::Disabled, RelayPolicy::DefaultPublic/Disabled) so tests that exercise
invite-token or advertisement filtering take the correct branch.

In `@crates/mesh-llm-host-runtime/src/mesh/tests.rs`:
- Around line 3158-3280: These three direct-path tests
(direct_path_maintenance_requires_candidate_and_grace_period,
direct_path_maintenance_cooldown_and_inflight_suppress_requests,
direct_path_request_keeps_only_previously_advertised_direct_candidates) should
be moved out of the large tests.rs into a new semantically named test module
(e.g., mod direct_path) so the direct-path responsibility is isolated; create
the new test module file, paste the three tests there, add a mod declaration in
the original tests module to include it, and update imports/visibility so the
tests can see DirectPathMaintenanceController, DirectPathObservation,
RelayPathSnapshot, SelectedPathKind, DIRECT_PATH_REPAIR_GRACE_SECS,
DIRECT_PATH_REPAIR_COOLDOWN_SECS,
endpoint_addr_with_previously_advertised_direct_candidates,
make_test_endpoint_id and TransportAddr/EndpointAddr types (adjust to pub(crate)
or re-export helpers if needed) so the tests compile and cargo test passes.

---

Nitpick comments:
In `@crates/mesh-llm-host-runtime/src/protocol/mod.rs`:
- Around line 280-292: The validate_frame implementation for
crate::proto::node::DirectPathRequest currently returns
ControlFrameError::InvalidEndpointId when serialized_addr is empty; change this
to return a new, dedicated variant ControlFrameError::MissingDirectPathAddress
(add that variant to the ControlFrameError enum and update its Display
implementation) and update validate_frame to return MissingDirectPathAddress
when serialized_addr.is_empty() so the error accurately reports a missing
direct-path address and stays aligned with the protocol crate's validator
behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 54b06bef-7ae1-464c-a2ff-7ef87b65f527

📥 Commits

Reviewing files that changed from the base of the PR and between 39cd7d2 and 08bb730.

📒 Files selected for processing (9)
  • crates/mesh-llm-host-runtime/src/mesh/direct_path.rs
  • crates/mesh-llm-host-runtime/src/mesh/heartbeat.rs
  • crates/mesh-llm-host-runtime/src/mesh/mod.rs
  • crates/mesh-llm-host-runtime/src/mesh/tests.rs
  • crates/mesh-llm-host-runtime/src/protocol/mod.rs
  • crates/mesh-llm-host-runtime/src/runtime/mod.rs
  • crates/mesh-llm-protocol/proto/node.proto
  • crates/mesh-llm-protocol/src/proto/node.rs
  • crates/mesh-llm-protocol/src/protocol/mod.rs

Comment thread crates/mesh-llm-host-runtime/src/mesh/direct_path.rs
Comment thread crates/mesh-llm-host-runtime/src/mesh/mod.rs Outdated
Comment thread crates/mesh-llm-host-runtime/src/mesh/mod.rs
Comment thread crates/mesh-llm-host-runtime/src/mesh/tests.rs Outdated
@michaelneale

Copy link
Copy Markdown
Collaborator

Wow. Going to try this out.

@i386
i386 force-pushed the codex/direct-path-repair branch from 08bb730 to 4431fcf Compare June 13, 2026 09:45
@i386
i386 force-pushed the codex/direct-path-repair branch from 4431fcf to d053842 Compare June 13, 2026 10:18
@i386
i386 merged commit d3b8b2f into main Jun 13, 2026
25 checks passed
@i386
i386 deleted the codex/direct-path-repair branch June 13, 2026 23:46
michaelneale added a commit that referenced this pull request Jun 14, 2026
* origin/main:
  Add bounded direct path repair (#846)
  Fix skippy smoke PR gate (#850)
  Stabilize skippy smoke chain startup (#849)

# Conflicts:
#	crates/mesh-llm-host-runtime/src/mesh/direct_path.rs
#	crates/mesh-llm-host-runtime/src/mesh/mod.rs
#	crates/mesh-llm-host-runtime/src/protocol/mod.rs
#	crates/mesh-llm-protocol/src/protocol/mod.rs
michaelneale added a commit that referenced this pull request Jun 14, 2026
* origin/main: (29 commits)
  MoA: don't let small-model consensus pre-empt a still-running large model (#837)
  fix(console): render thinking traces as markdown
  Add bounded direct path repair (#846)
  Fix skippy smoke PR gate (#850)
  Stabilize skippy smoke chain startup (#849)
  fix(ci): switch back to auto-assign workflow
  fix(website): polish longform visual explainer (#843)
  fix: gemma thinking
  Carry GLM llama MTP patches (#840)
  Refresh llama.cpp canary patch queue (#839)
  Add transport-aware Skippy stage ordering (#814)
  Share Skippy stage wire byte accounting (#818)
  Report Skippy artifact cold-start costs (#815)
  fix: debug output capturing for TUI / panics (#827)
  fix(hero): visual corrections for iPhone SE size devices (#838)
  Add Skippy stage role metadata (#816)
  Add Skippy request cache epoch telemetry (#817)
  Consolidate agent skills and fix stale docs (Windows deploy, repo map, design docs) (#836)
  feature(version): normalize version markers for different build types (#831)
  fix(website): fix visual regressions (#835)
  ...

# Conflicts:
#	AGENTS.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants