sys/net/gnrc/tx_sync: new module

[maribu]

Contribution description

The new gnrc_tx_sync module allows users of the GNRC network stack to synchronize with the actual transmission of outgoing packets. This is directly integrated into gnrc_sock. Hence, if gnrc_tx_sync is used, calls to e.g. sock_udp_send() will block until the network stack has processed the message.

Use cases:

1. Prevent packet drop when sending at high rate
   • If the application is sending faster than the stack can handle, the message queues will overflow and outgoing packets are lost
2. Passing auxiliary data about the transmission back the stack
   • When e.g. the number of required retransmissions, the transmission time stamp, etc. should be made available to a user of an UDP sock, a synchronization mechanism is needed
3. Simpler error reporting without footguns
   • The current approach of using core/msg for passing up error messages is difficult to use if other message come in. Currently, gnrc_sock is busy-waiting and fetching messages from the message queue until the number of expected status reports is received. It will enqueue all non-status-report messages again at the end of the queue. This has multiple issues:
     • Busy waiting is especially in lower power scenarios with time slotted MAC protocols harmful, as the CPU will remain active and consume power even though the it could sleep until the TX slot is reached
     • The status reports from the network stack are send to the user thread blocking. If the message queue of the user thread is full, the network stack would block until the user stack can fetch the messages. If another higher priority thread would start sending a message, it would busy wait for its status reports to completely come in. Hence, the first thread doesn't get CPU time to fetch messages and unblock the network stack. As a result, the system would lock up completely.
   • Just adding the error/status code to the gnrc_tx_sync_t would preallocate and reserve memory for the error reporting. That way gnrc_sock does not need to search through the message queue for status reports and the network stack does not need to block for the user thread fetching it.

Testing procedure

1. Run the test provided as tests/gnrc_tx_sync as is. It should pass.
2. Run the test again, but comment out USEMODULE += gnrc_tx_sync in the test's Makefile. The test should fail now.

State

• Integrate into link layer fragmentation. (I think only 6LoWPAN has this, right?) But that shouldn't be too hard to add. Just split of the TX sync snip before creating the fragments, send out all fragments as before, and finally release the TX sync snip.
• Get drivers/net: Add netdev_driver_t::confirm_send #14660 merged, so that this can not only sync with GNRC but also with the actual transmission of the hardware.

The first should be handled before merging, the second could be a follow up.

Issues/PRs references

None

[miri64]

Some initial thoughts.

[maribu]

Rebased on master, as #15839 is now merged. I guess it is time to also add synchronization for the link-layer fragmentation use case, so that the WIP label can be dropped.

[maribu]

let's reorder the commits, so that each commit can be individually compiled to ease git bisect.

[maribu]

So, vera++ should now also be happy :-)

[maribu]

@jia200x suggested that it is a bad idea to bundle gnrc_tx_sync with throttling 6LoWPAN fragmentation, as e.g. multiple 6LoWPAN interfaces (e.g. when using `at86rf215) could be present.

I can split this out and just split the tx_sync snip from the IPv6 packet and instead attach it to the last fragment. This way, gnrc_tx_sync would still sync with the transmission of the whole packet (including all fragments), but it would never blow the 6LoWPAN thread.

Generally, I do believe that throttling 6LoWPAN fragmentation to match the speed of the transceiver is beneficial. (I also agree that just blocking in the 6LoWPAN thread is a quick and dirty hack, rather than a clean solution.) But I fully agree that bundling throttling of fragmentation with gnrc_tx_sync is a bad idea.

I will look into this asap.

[benpicco]

@jia200x suggested that it is a bad idea to bundle gnrc_tx_sync with throttling 6LoWPAN fragmentation, as e.g. multiple 6LoWPAN interfaces (e.g. when using `at86rf215) could be present.

I still think this would be a small price to pay for getting actually working 6LoWPAN fragmentation with all drivers. (And the possibility to remove the hacks that were added to the drivers to make 6LoWPAN fragmentation work)

[maribu]

@jia200x suggested that it is a bad idea to bundle gnrc_tx_sync with throttling 6LoWPAN fragmentation, as e.g. multiple 6LoWPAN interfaces (e.g. when using `at86rf215) could be present.

I still think this would be a small price to pay for getting actually working 6LoWPAN fragmentation with all drivers. (And the possibility to remove the hacks that were added to the drivers to make 6LoWPAN fragmentation work)

Agreed. But there is no harm in allowing the user to decide on this. E.g. one could instead use gnrc_netif_pktq to pump out fragments on multiple transceivers at high rate, if faced with use case for this.

(And we could still add (say) gnrc_sixlowpan_throttle to Makefile.default for boards with a single 6LoWPAN transceiver, if we want to provide users with a reasonable default.)

[jia200x]

I still think this would be a small price to pay for getting actually working 6LoWPAN fragmentation with all drivers. (And the possibility to remove the hacks that were added to the drivers to make 6LoWPAN fragmentation work)

well, wouldn't 6LoWPAN work with all ieee802154_radio_hal based drivers? And which hack are you referring to?

[benpicco]

And which hack are you referring to?

I had to add _block_while_busy() in at86rf215, also nrf24l01p_ng can't handle large packets with many fragments.

[jia200x]

I had to add _block_while_busy() in at86rf215, also nrf24l01p_ng can't handle large packets with many fragments.

Ah yes, but you don't need to add that if you use the Radio HAL (the send function never blocks). That's how it's implemented in the CC2538 and NRF802154

[maribu]

Both nrf24l01p_ng and cc110x are using 6LoWPAN, but are not 802.15.4. So porting them to the 802.15.4 radio HAL is not really an option.

The hack in the cc110x is particularly nasty, as the cc110x can send frames of 255 bytes (including some header not supplied by the hardware), but only has a 64 byte FIFO. So the driver provides its own mechanism to call netdev_driver_t::isr() during TX, while it is blocking until TX completes. It would be super nice to get rid of that.

[jia200x]

Both nrf24l01p_ng and cc110x are using 6LoWPAN, but are not 802.15.4. So porting them to the 802.15.4 radio HAL is not really an option.

The hack in the cc110x is particularly nasty, as the cc110x can send frames of 255 bytes (including some header not supplied by the hardware), but only has a 64 byte FIFO. So the driver provides its own mechanism to call netdev_driver_t::isr() during TX, while it is blocking until TX completes. It would be super nice to get rid of that.

I remember you proposed to create a "generic network device" in the RDM. Although there were some discussions there, maybe this use case can be a concrete example (let's say a generic radio + some technology specific extensions).

[maribu]

I remember you proposed to create a "generic network device" in the RDM. Although there were some discussions there, maybe this use case can be a concrete example (let's say a generic radio + some technology specific extensions).

I'm still all in for this :-) In the meantime, allowing users to throttle the 6LoWPAN thread should be a quick fix. But anyway, I dropped this from this PR, so that this could be a separate PR and a separate (pseudo)module. We could document that we plan removing this module right away, when a better solution is merged.

[maribu]

@miri64: I adapted the test to now use a simulated IEEE 802.15.4 or a simulated Ethernet devices and dropped the additional netdev type.

[miri64]

From my side this is ready to get merged, so please squash. Let's see if Murdock finds any issues.

[maribu]

All green :-)

[miri64]

Then take this green as well :-)! ACK!

[maribu]

Thanks! :-)

[miri64]

From #15694 (comment)

[…]

And a good invitation to provide generalized mock devices and/or interfaces for certain L2 protocols in the future so one does not have to re-implement them all the time. ^^

See #16020