Skip to content

fix: improve acknowledged unicast retry reliability - #11320

Merged
vidplace7 merged 4 commits into
meshtastic:developfrom
RCGV1:codex/unicast-retry-5-3
Aug 13, 2026
Merged

fix: improve acknowledged unicast retry reliability#11320
vidplace7 merged 4 commits into
meshtastic:developfrom
RCGV1:codex/unicast-retry-5-3

Conversation

@RCGV1

@RCGV1 RCGV1 commented Aug 1, 2026

Copy link
Copy Markdown
Member

Summary

  • Raise acknowledged unicast delivery from 3 to 5 total attempts at the originating node.
  • Give directed intermediate-hop delivery 3 total attempts while retaining the final flood fallback.
  • Keep reliable broadcast delivery at its existing 3-attempt budget.
  • Make pending-packet cancellation relative to each record's initial budget.
  • Prevent directed retries from registering and replacing their own pending entry.

Why

This is a scoped follow-up to #9862 and GUVWAF's retry-policy discussion.

Increasing the intermediate-hop budget exposed a latent lifetime bug: a scheduled directed retry re-entered NextHopRouter::send(), which replaced the same pending map entry being iterated. That invalidated the active record and could crash when the retry scheduler subsequently updated it. The new helper still refreshes the selected next hop but skips nested retransmission registration for an already-tracked retry.

The change is local routing behavior only. It does not modify protobufs, configuration, packet format, or client behavior.

Validation

  • PATH="$HOME/.platformio/penv/bin:/opt/homebrew/opt/findutils/libexec/gnubin:$PATH" ./bin/run-tests.sh -e native-macos --quiet: GREEN, 43/43 suites
  • Focused next-hop routing suite: 38/38
  • Focused packet-signing/reliable-routing suite: 73/73
  • trunk fmt: clean across all five modified files
  • Embedded builds:
    • pio run -e rak4631: passed
    • pio run -e heltec-v3: passed
    • pio run -e tbeam-s3-core: passed
  • Hardware soak using candidate firmware on a Muzi Base (nRF52840) and LilyGo T-Beam S3:
    • 50/50 acknowledged unicasts delivered
    • 25/25 in each direction
    • Median observed delivery latency approximately 5.15 seconds; maximum 9.18 seconds
    • Node IDs, keys, region, and channel were preserved across state-preserving update flashes

Meshtasticator evidence

An out-of-tree paired campaign ran 10,000 unique scenarios with 38,932 identical offered unicasts under each policy:

  • Aggregate delivery: 14.8798% -> 15.4577% (+0.578 percentage points)
  • Paired mean delivery improvement: +0.599 pp, bootstrap 95% CI +0.423 to +0.772 pp
  • Direct scenarios: +1.20 pp
  • Relay scenarios: +0.42 pp
  • Dense scenarios: +0.15 pp, with the confidence interval crossing zero
  • Retry exhaustion: 20,178 -> 14,847
  • Cost: source attempts/message 2.19 -> 3.04, collisions +4.9%, airtime per delivered packet +24.3%, and p95 latency +3.17 seconds

The campaign used a local adaptation of Meshtasticator's discrete-event model to represent both source and intermediate retry budgets while keeping offered workload independent of retry timing. Its model tests were 30 passed with one hard-coded sequence-count golden mismatch (180 expected versus 200) caused by the intentional asynchronous workload generation.

Hardware limitation

The available radios were co-located, so the bench could not suppress direct reception and force a genuine RF-isolated three-node route. Originating unicast behavior has physical hardware coverage; intermediate-hop behavior has native regression and simulator coverage but still needs a separated or attenuated three-node hardware topology for full RF validation.

🤝 Attestations

  • I have tested that my proposed changes behave as described.
  • I have tested that my proposed changes do not cause any obvious regressions on the following devices:
    • Heltec (Lora32) V3
    • LilyGo T-Deck
    • LilyGo T-Beam
    • RAK WisBlock 4631
    • Seeed Studio T-1000E tracker card
    • Other: Muzi Base (nRF52840)

Summary by CodeRabbit

  • New Features

    • Improved reliable message delivery with separate retry limits for unicast and broadcast messages.
    • Reliable unicast messages now support up to five total attempts.
    • Reliable broadcast messages support up to three attempts.
    • Enhanced next-hop routing preserves routes during retries and performs final flooding when needed.
  • Bug Fixes

    • Prevented duplicate retransmission tracking when retrying existing packets.
    • Improved cancellation and timing of pending retransmissions.
  • Tests

    • Added coverage for retry budgets, route preservation, cancellation, and fallback flooding.

@coderabbitai

coderabbitai Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: d67dbf9b-c253-4820-b198-5fec72bd5850

📥 Commits

Reviewing files that changed from the base of the PR and between 0dc3d2f and 19e2299.

📒 Files selected for processing (1)
  • test/test_nexthop_routing/test_main.cpp
🚧 Files skipped from review as they are similar to previous changes (1)
  • test/test_nexthop_routing/test_main.cpp

📝 Walkthrough

Walkthrough

The change separates initial and remaining retransmission counts, adds non-tracking sends for existing retries, and defines distinct reliable broadcast and unicast attempt budgets. Tests cover cancellation, next-hop preservation, final flooding, and total attempts.

Changes

Retransmission tracking and attempt limits

Layer / File(s) Summary
Retry contracts and attempt budgets
src/mesh/NextHopRouter.h, src/mesh/ReliableRouter.cpp
PendingPacket records its initial retransmission count. Retry constants define three intermediate retransmissions, three reliable broadcast attempts, and five reliable unicast attempts.
Tracked next-hop retransmission flow
src/mesh/NextHopRouter.cpp, src/mesh/NextHopRouter.h
sendWithNextHop can skip retransmission tracking for existing retries. Cancellation uses the recorded initial count, and final retries preserve packet handling while allowing flooding.
Retransmission behavior validation
test/test_nexthop_routing/test_main.cpp, test/test_packet_signing/test_main.cpp
Tests cover pending-record cancellation, retry timing, directed next hops, final flooding, and three-versus-five total attempts.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Mergeability Score: ⚪ Minimal · up to 19e22

The PR adjusts localized unicast retry behavior and pending-packet cancellation, with reported tests and embedded builds passing; no actionable merge-blocking risk remains.

Sequence Diagram(s)

sequenceDiagram
  participant ReliableRouter
  participant NextHopRouter
  participant RadioInterface
  ReliableRouter->>NextHopRouter: Select broadcast or unicast attempt budget
  NextHopRouter->>RadioInterface: Send packet with retransmission tracking
  NextHopRouter->>NextHopRouter: Schedule pending retry
  NextHopRouter->>RadioInterface: Resend without creating another pending record
  NextHopRouter->>RadioInterface: Flood on the final retry when no relay occurs
Loading

Suggested reviewers: thebentern

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: improved reliability for acknowledged unicast retries.
Description check ✅ Passed The description covers the change, rationale, validation, hardware limitation, and required attestations in sufficient detail.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

⚡ Try this PR in the Web Flasher

Note

Building this pull request… the flash button, badges and supported-board
list will appear here automatically once CI finishes.

@RCGV1

RCGV1 commented Aug 1, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/mesh/NextHopRouter.h (1)

115-120: 🩺 Stability & Availability | 🔵 Trivial

Verify airtime/duty-cycle impact of the increased retry budgets.

NUM_INTERMEDIATE_RETX goes from 2 to 3, and NUM_RELIABLE_UNICAST_ATTEMPTS adds a fifth attempt for originating unicast sends. Each additional attempt increases airtime and duty-cycle consumption on directed routes. This effect is stronger in duty-cycle-limited regions.

The PR description states that hardware testing did not cover a genuinely RF-isolated three-node intermediate-hop topology. Confirm mesh-wide airtime and duty-cycle behavior under this new budget with a real multi-hop RF test before this change reaches production, since simulator and soak-test results alone may not surface duty-cycle interactions specific to that topology.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/mesh/NextHopRouter.h` around lines 115 - 120, Validate the increased
retry budgets represented by NUM_INTERMEDIATE_RETX and
NUM_RELIABLE_UNICAST_ATTEMPTS with a real RF test using an isolated three-node
intermediate-hop topology, measuring mesh-wide airtime and duty-cycle behavior,
especially in duty-cycle-limited regions. Do not rely solely on simulator or
soak-test results; confirm the observed impact before production release.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/mesh/NextHopRouter.h`:
- Around line 115-120: Validate the increased retry budgets represented by
NUM_INTERMEDIATE_RETX and NUM_RELIABLE_UNICAST_ATTEMPTS with a real RF test
using an isolated three-node intermediate-hop topology, measuring mesh-wide
airtime and duty-cycle behavior, especially in duty-cycle-limited regions. Do
not rely solely on simulator or soak-test results; confirm the observed impact
before production release.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 5c4abfe8-31f6-45b4-b506-3e846ec1a8d2

📥 Commits

Reviewing files that changed from the base of the PR and between d0e9d02 and cadece7.

📒 Files selected for processing (5)
  • src/mesh/NextHopRouter.cpp
  • src/mesh/NextHopRouter.h
  • src/mesh/ReliableRouter.cpp
  • test/test_nexthop_routing/test_main.cpp
  • test/test_packet_signing/test_main.cpp

@RCGV1
RCGV1 marked this pull request as ready for review August 10, 2026 06:39
@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@RCGV1

RCGV1 commented Aug 10, 2026

Copy link
Copy Markdown
Member Author

@GUVWAF

@RCGV1

RCGV1 commented Aug 10, 2026

Copy link
Copy Markdown
Member Author

@h3lix1

@RCGV1

RCGV1 commented Aug 10, 2026

Copy link
Copy Markdown
Member Author

CI is blocked only by the required PR-label gate. I do not have permission to apply labels on meshtastic/firmware; could a maintainer add the bugfix label? No code change is needed for this check.

@thebentern thebentern added the enhancement New feature or request label Aug 10, 2026
@caveman99 caveman99 added bugfix Pull request that fixes bugs and removed enhancement New feature or request labels Aug 10, 2026 — with Claude
@h3lix1

h3lix1 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Will need to validate how this interacts with NEXTHOP_EARLY_FLOOD_ON_UNVERIFIED

Recovery via flooding takes twice as long: ~15 s → ~30 s on idle LongFast, ~29 s → ~58 s saturated.

If we flood earlier instead of later, this could save a lot of time per message.

@RCGV1

RCGV1 commented Aug 12, 2026

Copy link
Copy Markdown
Member Author

@h3lix1 I also have an idea to make hops through next hop not cost anything so there would be a bigger benefit to next hopping

@RCGV1

RCGV1 commented Aug 13, 2026

Copy link
Copy Markdown
Member Author

Validated the interaction with NEXTHOP_EARLY_FLOOD_ON_UNVERIFIED at the routing layer and added regression coverage in 19e2299.

  • gate off: an unverified route keeps the first retry directed and floods the final retry
  • gate on: an unverified route floods one retry earlier
  • gate on with a fresh, zero-failure route-health record: the retry remains directed

The focused test_nexthop_routing suite passes 46/46 in both builds (native-macos, with and without -DNEXTHOP_EARLY_FLOOD_ON_UNVERIFIED=1). The same commit also repairs a missing brace and test-order lifetime regression introduced by the latest develop merge; before the repair the merged suite compiled incorrectly/formatted badly and then dereferenced a replaced capture interface.

@thebentern
thebentern added this pull request to the merge queue Aug 13, 2026
@vidplace7
vidplace7 removed this pull request from the merge queue due to a manual request Aug 13, 2026
@vidplace7
vidplace7 merged commit a400143 into meshtastic:develop Aug 13, 2026
61 of 63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Pull request that fixes bugs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants