fix(NextHopRouter): don't use the pending record after a send re-enters the router - #11475
fix(NextHopRouter): don't use the pending record after a send re-enters the router#11475h3lix1 wants to merge 6 commits into
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
⚡ Try this PR in the Web FlasherNote Building this pull request… the flash button, badges and supported-board |
7958777 to
d6a360c
Compare
…rs the router doRetransmissions() holds a reference into the pending map across calls that can delete the very entry it is iterating, then keeps using it. The re-entry is synchronous. Both sendAckNak() on the MAX_RETRANSMIT path and the sends below it reach Router::send(), whose duty-cycle and encode-failure paths call abortSendAndNak() -> sendLocal() -> deliverLocal(). deliverLocal() only defers the loopback when handleDepth > 0, and doRetransmissions() runs from runOnce() rather than from inside handleReceived(), so handleDepth is 0 and the delivery happens inline on this stack. The NAK is addressed to us with request_id set to the packet being retried, so ReliableRouter::sniffReceived() calls stopRetransmission() for exactly this key, which erases the map node and releases p.packet. Control then returns and the loop writes --p.numRetransmissions, hands &p to setNextTx() which dereferences p.packet, and reads p.nextTxMsec for the sleep calculation. On the MAX_RETRANSMIT path it also reads it->first off the freed node to pass to stopRetransmission(). Copy the key by value up front and re-look the record up after the sends instead of reusing the reference. Where the lookup comes back empty the record is gone and there is nothing left to reschedule. The duty-cycle abort needs a region with dutyCycle < 100, so EU_433, EU_868, EU_866, EU_N_868, TH and UA_433. The encode-failure abort has no such precondition: the pending copy is the decoded packet, so every retry re-encodes and a channel or PKI state change between attempts reaches the same path.
MockRoutingModule records sendAckNak() and stops, so nothing in this suite exercised the loopback that makes the lifetime bug reachable. Add a routing module that re-enters the router the way the real path does, delivering a self-addressed NAK into ReliableRouter::sniffReceived() while doRetransmissions() is still working on the record. Seeded with one attempt the record sits at numRetransmissions == 0, so the first due pass takes the MAX_RETRANSMIT branch and the loopback deletes the entry before control returns. Worth knowing how this test behaves: revert the fix and it still passes under plain native-macos, because reading a freed unordered_map node is undefined rather than reliably fatal. It is the sanitizer that makes it visible, so run it under native-macos-debug or the coverage environment, both of which carry -fsanitize=address.
d6a360c to
6d0f264
Compare
doRetransmissions()holds a reference into thependingmap across calls that can delete the entryit is iterating, then keeps using it.
The re-entry is synchronous, which is the part that makes this reachable rather than theoretical.
Both
sendAckNak()on theMAX_RETRANSMITpath and the sends below it reachRouter::send(), whoseduty-cycle and encode-failure paths call
abortSendAndNak()->sendLocal()->deliverLocal().deliverLocal()only defers the loopback whenhandleDepth > 0, anddoRetransmissions()runs fromrunOnce()rather than from insidehandleReceived(), sohandleDepthis 0 and the deliveryhappens inline on this stack. The NAK is addressed to us with
request_idset to the packet beingretried, so
ReliableRouter::sniffReceived()callsstopRetransmission()for exactly this key, whicherases the map node and releases
p.packet.Control then returns into the loop, which writes
--p.numRetransmissions, hands&ptosetNextTx()which dereferencesp.packet, and readsp.nextTxMsecfor the sleep calculation. Onthe
MAX_RETRANSMITpath it also readsit->firstoff the freed node to pass tostopRetransmission().The fix copies the key by value up front and re-looks the record up after the sends instead of
reusing the reference. Where the lookup comes back empty the record is gone and there is nothing left
to reschedule.
When it fires
The duty-cycle abort needs a region with
dutyCycle < 100, so EU_433, EU_868, EU_866, EU_N_868, THand UA_433. US, ANZ and JP never take it.
The encode-failure abort has no such precondition. The pending copy is the decoded packet, since
ReliableRouter::sendcopies beforeRouter::sendencrypts, so every retry re-encodes and a channelor PKI state change between attempts reaches the same path.
About the test
MockRoutingModulein this suite recordssendAckNak()and stops, so nothing here exercised theloopback at all. The second commit adds a routing module that re-enters the router the way the real
path does. Seeded with one attempt the record sits at
numRetransmissions == 0, so the first duepass takes the
MAX_RETRANSMITbranch and the loopback deletes the entry before control returns.Be aware of what this test does and does not catch on its own. Revert the fix and it still passes
under plain
native-macos, because reading a freedunordered_mapnode is undefined rather thanreliably fatal. It needs an allocator that actually enforces the lifetime.
Reproduced
native-macos-debugcarries-fsanitize=addressbut does not start on macOS 15: it spins inside__asan::AsanInitInternal()->InitializeShadowMemory()->MemoryRangeIsAvailable()->get_dyld_hdr()->dyld_shared_cache_iterate_text_swift, which callsmalloc, which ASAN hasalready intercepted while still initialising. It never reaches
main.macOS Guard Malloc works where ASAN does not, and it unmaps freed pages, so the read faults instead
of silently succeeding:
test_reliableAckStopsNormalPendingTransmissionpasses and on entry totest_retransmissionSurvivesLoopbackErasingItsOwnRecord.47 Tests 0 Failures 0 Ignored.So the lifetime bug is observed, not just argued from the source. That recipe may be worth knowing
generally for this repo, since it gives macOS contributors use-after-free detection that the ASAN
environments cannot currently provide.
Hardware
Rebased onto current develop, so this now sits on top of #11320 and #11502. Flashed to two nodes, a
Heltec V4 and a Seeed XIAO ESP32-S3, both running
2.8.0.6d0f264from this branch, and driventhrough the path the bug is on.
The interesting case is a reliable unicast whose retry ladder runs out, because that is what makes
doRetransmissions()callsendAckNak(MAX_RETRANSMIT), which delivers inline and re-enters therouter to erase the very record the loop is holding. To force it I disabled
lora.tx_enabledon theXIAO so it would receive but never ACK.
want_ack:Received an ACK.Received a NAK, error reason: MAX_RETRANSMIT. Repeated three more times, same result each time.Received an ACKagain, mesh recovered with no intervention.rebootCountwas 13 on the Heltec and 10 on the XIAO before and after the whole sequence, sonothing panicked or reset.
I then flashed unfixed develop to the Heltec and ran the identical sequence, so as not to claim more
than the bench shows. It behaves the same: four
MAX_RETRANSMITNAKs,rebootCountunchanged.So the hardware run demonstrates no regression, and nothing more. It does not discriminate between
fixed and unfixed, which is the expected outcome for a read of a freed heap block that nothing has
reallocated yet. The Guard Malloc run above is the part that actually pins the bug.
Testing: builds clean on
heltec-v4andseeed-xiao-s3, 148/148 acrosstest_nexthop_routing,test_packet_signingandtest_mesh_module.Note on rebasing: #11291 rewrites the
if (p.nextTxMsec <= now)line this patch sits next to andadds a CI check that rejects raw
millis()comparisons. If that lands first the fix needs rebasingonto
Throttle::deadlinePassedAt, and the shape of it does not change. Worth also knowing that#11235 edits
setNextTx()'s prologue, so a fix that changed that function's signature would collide.This one deliberately stays inside
doRetransmissions().🤖 Generated with Claude Code
🤝 Attestations
exercised, and the same sequence run on unfixed develop as a control