feat: Attempt to make DMs more reliable by flood routing the first hop by default

[h3lix1]

Summary

We (baymesh) have heard many complaints about DMs not making their destiantions as expected. Looking at the different paths through the code, there is a wasFallback path that is supposed to flood route packets if the relay fails to send the packet. There are a couple problems with this though.

1. The first hop depends on the next_hop router to relay. If another node in the vicinity hears the packet as well (say, a router on a hill) it will make it into the PacketHistory of that router. When the next_hop router gets around to sending the packet, the router has already seen the packet, and will silently drop it.
2. If next_hop decides to flood the packet, nodes will silently drop it.
3. This is the same pretty much the issue every hop along the way.

This was created with the assistance of claude and codex to find my bugs and basically re-write everything I did with something better, sadly.

This PR is to implement a few changes

1. The first unicast hop now sends with next_hop = 0 so nearby nodes hear and can relay it
2. off-path directed copies still enter PacketHistory, but observer-only records keep directed retries deduped while still allowing a later next_hop = 0 flood fallback
3. destination nodes still insert repeated copies into PacketHistory so the phone/apps do not see the same DM twice
4. after the first hop, relays can still resume next_hop routing from their own local view

For the visual types...

Before

flowchart LR
    S[Source] -->|DM with next_hop=A| A[Directed relay A]
    S -->|same packet overheard| B[Off-path node B]
    B --> H1[PacketHistory stores sender+id+next_hop=A]
    A --> X[Route fails farther downstream]
    A -->|fallback retry with next_hop=0| B
    B --> H2[seenRecently=true]
    H2 --> H3[wasFallback often false]
    H3 --> D[Drop as duplicate]

Loading

Note:

ROUTER roles do not rebroadcast this traffic by default unless it matches next_hop.

After

flowchart LR
    S[Source] -->|first unicast hop with next_hop=0| N1[Neighbor 1]
    S -->|first unicast hop with next_hop=0| N2[Neighbor 2]
    N1 -->|first relay stays flood| M[Mesh]
    N2 -->|first relay stays flood| M
    M -->|later relay computes local next_hop| R[Downstream relay]
    R -->|directed copy overheard off-path| O[Off-path observer]
    O --> P[Store observer-only PacketHistory record]
    R -->|same packet reaches destination| D[Destination]
    D --> Q[Normal PacketHistory dedup still applies]
    R -->|if route breaks later, fallback next_hop=0 marks wasFallback on that observer record| F[Fallback flood path]

Loading

Mixed-Version Deployment

This change is compatible with older firmware. It does not change packet format or protobuf layout; it only changes local routing decisions. It requires most nodes to upgrade to take advantage of the patch.

• New sender -> old relays: safe. The new sender emits the first unicast hop with next_hop=0, which old nodes already understand as normal flood behavior. Old relays will usually keep the packet in flood-style forwarding instead of resuming directed routing, so this path is compatible but more airtime-heavy.
• Old sender -> new relays: partial improvement. The old sender still chooses a directed first hop, but new off-path observers now dedupe directed retries without blocking a later fallback flood. This improves recovery even before the whole mesh is updated.
• New sender -> new relays: intended behavior. The first hop is flood-heard, nearby relays can pick it up, and later hops can resume next_hop from each relay's local view.
• Mixed observers on the same path: any older off-path observer can still cache the directed copy and may suppress a later fallback copy locally. The benefit therefore increases as more overhearing relays are updated.

A partial rollout is safe and still helps, but the strongest fallback recovery appears once most relays near the affected route are on the new behavior.

Remaining Risk

This patch does not address the separate issue in where the original sender may stop retrying too early after overhearing the first relay of its own unicast packet.

Testing

• On-device validation of the first-hop flood bootstrap and later-hop directed routing
• pio test -e native -f test_packet_history

🤝 Attestations

• I have tested that my proposed changes behave as described.
• I have tested that my proposed changes do not cause any obvious regressions on the following devices:
  • Heltec (Lora32) V3
  • LilyGo T-Deck
  • LilyGo T-Beam
  • RAK WisBlock 4631
  • Seeed Studio T-1000E tracker card
  • Other (please specify below)
    Heltec V4

This will likely require alphanauts to test this one on a grander scale than I can.

[NomDeTom]

I like this idea. I did wonder if the traceroute packets were getting eaten by hop-crocs on the way to my t1000000e.

[GUVWAF]

First of all, I think we should not underestimate the impact of always using flooding for the 0-hop. For the initial hop, it's important to point the packet at least towards the right "direction" as nodes in the other direction likely don't have next-hops set for the destination and it keeps on flooding in a direction it never needs to be. That's why we also still have e.g. #7770.

Next I'm not sure I follow why "wasFallback is often false". I checked the difference between your observerOnlyDirectedRecord and the original condition for setting wasFallBack. The original condition only has wasRelayer(p->relay_node, *found) and !wasRelayer(found->next_hop, *found) more:

• The former there checks whether it was the current relayer (A in this case) that first sent it with a specific next-hop, which is indeed the case.
• The latter checks whether the intended next-hop did not yet relay. In the "Before" scenario it says "Route fails farther downstream", which either means B did not hear the next-hop of A relay and hence the condition is true, or it means its next-hop did relay, and that next-hop is now responsible for falling back to flooding.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[h3lix1]

The specific failure mode I’m pointing at is not that fallback never works, but that it can be missed by off-path observers.

If the source sends a DM with next_hop = A, an off-path node B can overhear that directed copy and insert it into PacketHistory. Later, if A gives up and retries with next_hop = 0, B already sees the same sender + id as a duplicate. At that point, whether B helps depends on wasFallback.

The current wasFallback check is fairly strict. It wants proof not just that the packet used to be directed and is now flooded, but also that the current relay already appeared in history and the intended next hop did not. That works if B heard enough of the packets, but an off-path observer may only have heard the original directed copy and then the later flood retry. In that case it does not have enough history to prove fallback, so wasFallback stays false and the retry can be dropped as an ordinary duplicate.

I agree that “wasFallback is often false” was misstaed on my part. The better statement is that "wasFallback can be false for off-path observers that missed the relay-progress packet needed to satisfy the conditions"

That is the PacketHistory side of the change.. keep directed observer records for normal dedupe, but if the stored record shows we were only an observer and a later copy arrives with next_hop = 0, treat that as valid fallback recovery instead of requiring the observer to have witnessed every intermediate relay step.

On the first-hop flooding point, I agree that always flooding the first hop has a real airtime and directionality cost. My intent there was only a one-hop bootstrap so at least one alternate relay hears the packet when the sender’s cached first next_hop is stale or wrong (for example, when a node is moving). I’m not claiming that part is free, and if needed I’m happy to separate that discussion from the PacketHistory fix, because the duplicate/fallback issue stands on its own. It's putting a lot of trust in that first hop to do the right thing, and directionality can still happen after the first hop.

sequenceDiagram
    participant S as Source
    participant A as Directed relay A
    participant B as Off-path observer B
    participant D as Destination

    S->>A: Packet X with next_hop=A
    S-->>B: B overhears Packet X
    Note over B: PacketHistory stores sender=S, id=X, next_hop=A

    A-xD: Directed path fails farther downstream

    A->>D: Retry Packet X with next_hop=0
    A-->>B: B overhears flood retry

    Note over B: Same sender + id already seen
    Note over B: Old predicate may not have enough relay history
    Note over B: wasFallback = false
    B-xD: Drop as duplicate

Loading

[nullrouten0]

Almost none of the traffic on Baymesh is DM, because DM stopped working many months ago. The mesh uses channels (public) 99% of the time (300-500 messages a day, give or take), ... which is also only 1.8% of our total mesh traffic. (98.2% is position from stationary nodes, telemetry, and nodeID for 1050 nodes trying to fit into DB's much smaller)... but my point is not to gripe... my point is to state that 99+% of our traffic is flood anyhow. Relay field has not provided any value here for a few reasons. 1. Paths change constantly (thats just mesh being normal) .. some users have 8-12 direct next hops that each only work 20% of the time, and its somewhat random who is lucky to hear you and pass your packet on... but in aggregate this works (83%-93% success on avg), and we have handfuls of nodes on each hash (many many hash collisions... lending to packets going multiple directions at once anyhow). In summary, Group messages are in the vicinity of 90% reliable and DM's are <50% reliable, on the same mesh to the same people.

[GUVWAF]

The current wasFallback check is fairly strict.

This is true, but it has to be as it needs to filter out normal duplicate rebroadcasts. I don’t see how observerOnlyDirectedRecord checks for this. Let’s say we have this scenario:

Node 3 overhears the packet from 0 with a next-hop set, and 1 doesn’t have a next-hop set for 2, so it sets NO_NEXT_HOP_PREFERENCE. According to 3, this is now a fallback to flooding.

In my opinion the scenario where B did hear the original rebroadcast, did not hear the directed relay by A, but did hear the fallback to flooding by A is rather rare. But even in this case, the packet starts flooding from A again, so we still have the redundancy of any other node in range of A that can jump in.

Almost none of the traffic on Baymesh is DM, because DM stopped working many months ago.

This is the first report I hear where "DM stopped working". How well did it work before next-hop routing was introduced?
And do your numbers mean the broadcast reaches on average 90% of all nodes in the mesh within the hop limit, and DMs reach the intended recipient in <50% of the cases? Or do you include the acknowledgment of the DM also?
It really depends on how you count it, because obviously reaching your 0-hop neighbours is much easier than reaching a specific node 3 hops away.

98.2% is position from stationary nodes, telemetry, and nodeID

To be honest, this sounds more like it's the source of problem. Yes, using next-hop routing does mean we have less redundancy, but the upside is it generates less traffic. However, if there is already a lot of broadcast traffic, then I can see how next-hop routing doesn't really help and can make things worse.

Don't get me wrong: I would love to make DMs more reliable, I'm just not sure going back to a more flooding-based approach is the way to go. We have other knobs to tune, e.g. the number of retransmissions (on intermediate hops).

[nullrouten0]

DM has a couple issues causing people not to use it. Very common to see encrypted send failed (even when nodes are favoriting each other and are not aware of any reason that any keys would be wrong), or silent failure (no ack seen, no cause known). In the meantime the same people can be chatting with each other in the public channel fine.

A little off topic for this thread, but we've been chasing nodeID, position, and Telemetry for over a year. Setting timers low-frequency/high-delay doesn't help. Nodes hear ID and respond with theirs. Nodes receive position and respond with theirs. People send their battery stats to 1000 other nodes when nobody cares... we've suggested different hop settings per type and that would at least cool this off a bit. Node DB's are so small that 1000 nodes wont come close to fitting, so nodes forget who they learned one hour ago, and re-introduce themselves. All of this is obvious when humans are sleeping , not even moving (no positions triggered by smart)... no chats.. no button pressing or intentional reboots... yet the mesh hums along at 40% ch Util all night long. Things should be biased more toward manually-requested, not offered automatically constantly IMHO. Telemetry defaulting off in 2.7.i_forget ... made a tiny dent, I was hopeful for a larger impact.... I imagine the 25% utilization throttle was already being exercised hard, and the reduction has yet to come under that ceiling. Most of the noise seems to be nodes responding to other nodes on their own, not even timer based... to a large extent. Harping on everyone to set very high timers on all this has yielded very little.

Back on topic for conclusion... DM used to work more often, and its hard to discern when exactly it broke (I know there were very specific bugs with reciprocal hop counts too! (which is bad here because things happen "1 hop getting here but 5 hops getting back" quite often... ).... but the casual opinion is that DM works a minority of the time while group public chat works MOST of the time.

[shalberd]

Node DB's are so small that 1000 nodes wont come close to fitting, so nodes forget who they learned one hour ago, and re-introduce themselves.

I think this is a big factor ... most of our mountain solar nodes are nrf52 based ... very limited node DB. Maybe we need to switch to using node hardware with more space for nodeDB entries? e.g. using larger solar panels and going for e.g. ESP32 boards. Even those do not have enough node DB capacity, I think, for 800+ nodes ...
Maybe the best way would be to create a better, more hardware- and memory-hungry, algo for Meshtastic v3 and then update the mountain node hardware along with a radically different algo, I don't know ...

[h3lix1]

@GUVWAF fair point about observerOnlyDirectedRecord path. After thinking through it more carefully, you're right that an off-path observer can't distinguish a fallback retry from a downstream relay that simply has no cached route and sends with NO_NEXT_HOP_PREFERENCE. There's nothing to differentiate them, so relaxing the check creates false positives (like your diagram).

I removed the observerOnlyDirectedRecord logic and restored the original strict wasFallback check and updated the test.

The first-hop flood is likely the thing that will help the most. It prevents the bad PacketHistory state from being created in the first place. When the sender floods the first hop, all nearby relays get a flood copy (stored with next_hop=0), so there's no directed-vs-flood mismatch to cause dedup issues later.

You're right that broadcast congestion is likely a contributing factor on BayMesh. The first-hop flood does add some airtime on the first hop, but I think there is a risk if the directed first hop goes to a node that can't deliver, and the flood fallback gets silently dropped by every off-path node that overheard the original.

Calude worked through the code paths and found an issue due to packet history, and here is what it had to say...

When Source sends with next_hop=A, an off-path node B skips rebroadcast (perhapsRebroadcast at NextHopRouter.cpp:155 requires next_hop to match) but still stores the packet in PacketHistory. Critically, B's record has an empty relayed_by array because B never relayed (PacketHistory.cpp:73-77 only populates relayed_by when relay_node == ourRelayID). Later, when the directed path fails and doRetransmissions (NextHopRouter.cpp:325-327) sends the last retry as a flood with next_hop=0, B hears it but wasSeenRecently returns seenRecently=true. The wasFallback check (PacketHistory.cpp:103-111) requires wasRelayer(p->relay_node, *found) — which fails because B's relayed_by is empty. So shouldFilterReceived (NextHopRouter.cpp:64) silently drops the packet as a duplicate. The DM is dead at that point — B was potentially the only alternative path, but will not relay it.

I think flooding the first hop is the best way to give the packet the best chance of making it the "first mile", and the increase in traffic is relatively localized. Things go back to using next_hop for subsequent hops.

[NomDeTom]

I think flooding the first hop is the best way to give the packet the best chance of making it the "first mile", and the increase in traffic is relatively localized. Things go back to using next_hop for subsequent hops.

Does this have a risk of the DM disappearing off in the opposite direction as well, with all NO_NEXT_HOP_PREFERENCE set?

There's nothing to differentiate them

Does this need adding to the list of things for 3.0 packet structure? Do you have some suggestions for useful flags for next-hop DMs?

[h3lix1]

I think flooding the first hop is the best way to give the packet the best chance of making it the "first mile", and the increase in traffic is relatively localized. Things go back to using next_hop for subsequent hops.

Does this have a risk of the DM disappearing off in the opposite direction as well, with all NO_NEXT_HOP_PREFERENCE set?

It's possible if none of the nodes along the way if they don't have useful routes cached. I think it just mostly means there is a risk of DMs flooding if there is no usable route defined. It's not the worst case scenario, since it likely means the user will still get their message, which ultimately meets the goal.. just less efficiently.

There is a period of time where all DMs will be flooded until most nodes are upgraded, since NO_NEXT_HOP_PREFERENCE is set and will not be unset after the first hop. I don't have a good solution for this. This might already be happening a lot since flooding is the fallback method, but it is difficult to tell without installing debug code.

There's nothing to differentiate them

Does this need adding to the list of things for 3.0 packet structure? Do you have some suggestions for useful flags for next-hop DMs?

IMHO, in 3.0 the packet (and routing) structure should be re-built so these types of workarounds and patches are no longer necessary. The next iteration should focus on reliability of packets using pragmatic means instead of today's safety-in-numbers approach.

[h3lix1]

@GUVWAF Working through other scenarios that might cause problems. The more obvious one is the last byte collision seems to happen a lot more than I originally anticipated..

https://gist.github.com/h3lix1/4a994b88d3e3992a397c5ce3d0996b9f

I thought 256 bits would mean there would be 5.5 collisions per last byte with 1400 nodes, but the distribution is very off with up to 18 nodes sharing the same last byte.

The problem is the distribution is not even. There are more nodes with the last byte that is divisible by 4.

IDs ending in 0, 4, 8, or c account for 743 / 1400 = 53.1% of nodes. If random, those four endings should account for only 25%.

Using a plain simulation, >=18 nodes would only happen 0.4% of the time, but for baymesh it's happening 3%.

Nodes that always have a last byte that is divisible by four are HELTEC_V3, HELTEC_V4, TBEAM etc..

[GUVWAF]

Thanks for the analysis. That's indeed quite unfortunate regarding the last byte of ESP32 nodes, as changing it will be a breaking change. However, such collisions only happen for direct neighbors, which would more be in the range of ten's of nodes instead of 1400.

If it happens though, it means multiple nodes will try to rebroadcast, which makes it closer to flooding. I don't think there's much of an issue with that given that the chance is still low, and the chance that more than 2 nodes try rebroadcasting is even lower. It can also happen that a node concludes its next-hop correctly relays, while it was actually a different node. Also here I don't think the impact is that big. If the "wrong" next-hop has a next-hop set for the destination, it can go via that (likely less efficient) route, and else it would use flooding from that hop on.

Regarding this PR in general: I'm still not sure using flooding on the first hop is a good way to go around issues caused by packet loss. I'm more inclined to just use more retransmissions for DMs, e.g. 5 on the initial hop and 3 on intermediate ones (instead of 3 and 2 like it is now). This way we fix packet loss on the link level instead of via redundancy with routing.

[h3lix1]

@GUVWAF that is fair, although the general consensus is that channel flood routing works better. Some have resorted to setting up private channels to send messages reliably.

Hopefully this will get better one way or another.

[NomDeTom]

I'd say adding some retry logic on the sender side might alleviate the perceived unreliability - we have acks on acks, but perhaps more retries on implicit acks would help?

[GUVWAF]

I'd say adding some retry logic on the sender side might alleviate the perceived unreliability - we have acks on acks, but perhaps more retries on implicit acks would help?

To me this sounds perfectly reasonable, and is a rather easy change. For example, only for DMs (and when we have a next-hop set?) make the initial retransmissions 5, and retransmissions on intermediate hops 3. We briefly had the automatic extra retries in the Android app, which was reverted (meshtastic/Meshtastic-Android#4124), because it lead to duplicate packets, and the timing of the retries is unpredictable for a client app. Handling it in the firmware is hence much better.

It will result into more time before someone sees the "max. retransmission reached" in case it really failed, though, but even that is variable now if it is receiving other packets in between, for example.

[NomDeTom]

To me this sounds perfectly reasonable, and is a rather easy change.

I think we should look into this after @thebentern gets chance to extend the nodeDB for NRF devices, so that any gains are not lost as simple floods in the middle of a megamesh.

[thebentern]

To me this sounds perfectly reasonable, and is a rather easy change.

I think we should look into this after @thebentern gets chance to extend the nodeDB for NRF devices, so that any gains are not lost as simple floods in the middle of a megamesh.

Sounds good. Can we capture that in an issue?

[h3lix1]

@NomDeTom

I think we should look into this after @thebentern gets chance to extend the nodeDB for NRF devices, so that any gains are not lost as simple floods in the middle of a megamesh.

Simple floods in the middle of a megamesh is used for almost every other type of packet, and those work. The only ones that seem to have the most issues are literally the least sent packets.

DMs have suffered greatly. People are unable to remote admin their nodes. Traceroutes are failing that should be working. The packets that work really well? The ones telling everyone about the battery life of their node.

[h3lix1]

#11320 landed in develop last week and implements @GUVWAF's suggestion..

One new issue I found while preparing the rebase. The hopsAway == 0 gate in perhapsRebroadcast (NextHopRouter.cpp:201-209 on this branch) does more than end the bootstrap flood. When an intermediate relay's final retry falls back to flooding (next_hop = 0, set at NextHopRouter.cpp:402 and :424), downstream copies arrive with hopsAway >= 1, so upgraded relays convert that fallback flood back into a directed send.

That seemed like it would be the largest win here.. the ability to revert back to next_hop in future hops.

This PR is probably stale enough to close for now.