WebSocket cuts off buffers over 528 bytes

[timleg002]

Tried sending more than 528 bytes, len is never more than 528. The code is similar to the example code provided:

ws.onEvent([](AsyncWebSocket *server, AsyncWebSocketClient *client, AwsEventType type, void *arg, uint8_t *data, size_t len) {
    (void)len;

    if (type == WS_EVT_CONNECT) {
      client->setCloseClientOnQueueFull(false);
      client->ping();
    } else if (type == WS_EVT_DATA) {
      AwsFrameInfo *info = (AwsFrameInfo *)arg;
      if (info->final && info->index == 0 && info->len == len) {
        if (info->opcode == WS_BINARY) {
          Serial.println(len); // never more than 528
          memcpy(myBuffer, data, len);
        }
      }
    }
  });

What could cause this?

[mathieucarbou]

I cannot reproduce with our Websocket example:

> base64 -i /dev/urandom | head -c 1024 | websocat ws://192.168.4.1/ws

in the serial console:

ws connect
ws text: 2Wt/RoQUYHhb+g0/CLAxFIuhUlYB7meN35esb/XQynVhcg+vMXCb9GhnOdScZt3nfI+5MOchghyWIplF1a0tDl/MRTxkxsoz3KPUcPP+bZxDITxkOq7C20PqHBwiUpoyWYsVlMDcBxlNtZZV2MH33g05bT2BKcMnvh/TPBEpDympUzOXpNipfJAKOt06iYea2rIN8Vrb4W+3nlj9tqvNcJSaK5nb1DwpTy21ZcFVvDFGjomeh3rwxyOUzJQBK4F/gbmEvgth1desoON1uM/fYv87yFKp8LDHXVDD2BCK5oBDvvTyca4+5o2BLYCurgAZgcXaa9OBh8YcG6GRxDrA3Jc1HvJFWfTIrfH4V6N+IeYcjWKTMGkLQQb3xz56FJX8xXIkMaMTWhVOvOiF/S7JnpcWyEV3RsEsVait7c3QN9uX/I8rQyy3wJbQZ+GSE3jGFBHNCsLEhR5EoPp3kR1Va7WNMa7sMkTTXG8DfSJFZOPGzH1l9kQAQ3FAwV7+VWHKY88TneAAwUFzogCX1VkOHCD2lFz5dnyyuiRRy/eB/OpMSiJdTV/FM1xbhfVytjjCX4etEiMWCxJXCQfpNyzJNvF0zQhNF0rHFn4HpuK4Lus5l0LbNIL5FdG0kRwXQb8eYlbSCpSCbHrYTx3bH+S/ElhV/yOos2M7RU4LUXW+0Sw3fR55pi+5Dxc19x2lYhBu8vBwztVeA4yjl1IagouxnF9F77AeVR0rlkBblNw5PauZiiXGWuHeB+DfDGi63X/7ifsZpNvvG+OhvzYkBEdmoB4qmQRc0B2m1Zcxeayl6eCyXJLosxKVxrMoO7kJsu0OGv9/pTBNjpO82ELkEeZD9qbvl/eXUmdHbrFzOb9Sz98kbtUDn6wTvI8gsAm5mbaaBx4aEqByPOMcHbpdXvk65QePuwGiRcfXcY8Un4maSeV3SjhiOECisfPVIc/qGPUySqI1gYRDVRO69vs7w/lWc09tG6F5nR0uHDn0eMq60ax9zk+eak3RpyRg080GVwT2
ws pong
ws disconnect

[timleg002]

I cannot reproduce with our Websocket example:

> base64 -i /dev/urandom | head -c 1024 | websocat ws://192.168.4.1/ws

in the serial console:

ws connect

ws text: 2Wt/RoQUYHhb+g0/CLAxFIuhUlYB7meN35esb/XQynVhcg+vMXCb9GhnOdScZt3nfI+5MOchghyWIplF1a0tDl/MRTxkxsoz3KPUcPP+bZxDITxkOq7C20PqHBwiUpoyWYsVlMDcBxlNtZZV2MH33g05bT2BKcMnvh/TPBEpDympUzOXpNipfJAKOt06iYea2rIN8Vrb4W+3nlj9tqvNcJSaK5nb1DwpTy21ZcFVvDFGjomeh3rwxyOUzJQBK4F/gbmEvgth1desoON1uM/fYv87yFKp8LDHXVDD2BCK5oBDvvTyca4+5o2BLYCurgAZgcXaa9OBh8YcG6GRxDrA3Jc1HvJFWfTIrfH4V6N+IeYcjWKTMGkLQQb3xz56FJX8xXIkMaMTWhVOvOiF/S7JnpcWyEV3RsEsVait7c3QN9uX/I8rQyy3wJbQZ+GSE3jGFBHNCsLEhR5EoPp3kR1Va7WNMa7sMkTTXG8DfSJFZOPGzH1l9kQAQ3FAwV7+VWHKY88TneAAwUFzogCX1VkOHCD2lFz5dnyyuiRRy/eB/OpMSiJdTV/FM1xbhfVytjjCX4etEiMWCxJXCQfpNyzJNvF0zQhNF0rHFn4HpuK4Lus5l0LbNIL5FdG0kRwXQb8eYlbSCpSCbHrYTx3bH+S/ElhV/yOos2M7RU4LUXW+0Sw3fR55pi+5Dxc19x2lYhBu8vBwztVeA4yjl1IagouxnF9F77AeVR0rlkBblNw5PauZiiXGWuHeB+DfDGi63X/7ifsZpNvvG+OhvzYkBEdmoB4qmQRc0B2m1Zcxeayl6eCyXJLosxKVxrMoO7kJsu0OGv9/pTBNjpO82ELkEeZD9qbvl/eXUmdHbrFzOb9Sz98kbtUDn6wTvI8gsAm5mbaaBx4aEqByPOMcHbpdXvk65QePuwGiRcfXcY8Un4maSeV3SjhiOECisfPVIc/qGPUySqI1gYRDVRO69vs7w/lWc09tG6F5nR0uHDn0eMq60ax9zk+eak3RpyRg080GVwT2

ws pong

ws disconnect

I believe you're sending text data, not binary. When using JS APIs, only data sent as an Uint8Array (or similar, TypedArray, ArrayBuffer, DataView etc.) seem to be affected (because then the opcode is WS_BINARY).

[mathieucarbou]

@timleg002 : I wanted to say that maybe you do not need the check if (info->final && info->index == 0 && info->len == len) {
Also, yes I tested with text but that's not important. What is important is to test with a tool which is not linked to your app code. You can reproduce with your app code. To prove there is an issue, you would need to reproduce without your app code, in a sort of MRE (minimal reproductible example).

I give you an example of code:

    } else if (type == WS_EVT_DATA) {
      AwsFrameInfo *info = (AwsFrameInfo *)arg;
      Serial.printf("index: %" PRIu64 ", len: %" PRIu64 ", final: %" PRIu8 ", opcode: %" PRIu8 "\n", info->index, info->len, info->final, info->opcode);

With text:

> base64 -i /dev/urandom | head -c 1024 | websocat  ws://192.168.4.1/ws

outputs:

ws connect
index: 0, len: 1024, final: 1, opcode: 1
ws text: 
[...]
ws pong
ws pong

with binary:

base64 -i /dev/urandom | head -c 1024 | websocat -b ws://192.168.4.1/ws

outputs:

ws connect
index: 0, len: 1024, final: 1, opcode: 2
ws disconnect

See the opcode change to 2 (binary), and it is receiving 1024b.

So there is nothing wrong in the library and that's this kind of test I would have liked you to have done, in order to move out of scope your application code and directly hit the websocket endpoint.

So I suppose there is probably something going on in your app instead.

[timleg002]

The check was provided in the example code and I didn't know the significance of it - hopefully you could clear it up. I will try using websocat or similar, when I'll have access to it.

The application code is basically:

const ws = new WebSocket("/ws");
ws.send(new Uint8Array(1024));

In the DevTools network inspector panel, I see that the message is being sent in its original length, but on the device the received length is still never more than 528. I haven't thought about removing the check, as it didn't seem likely to me that it's sending two messages, one being cut-off and one having the full length.

[willmmiles]

If one or more of the devices in question is an ESP8266, the LwIP TCP stack defaults to a MSS (maximum segment size) of 536. This means it sends and receives no more than 536 payload bytes per packet. I don't know much about the WebSocket protocol specifically, but add a little framing overhead and 528 is a reasonable maximum payload to expect in a single call.

The onEvent() callback is called for every data packet handed up by LwIP. You should be able to tell you've received only partial data via comparing len to info->len -- info->len carries the original event size, while len is the number of bytes received in this packet. If len < info->len, you're responsible for doing your own buffering and assembly.

That said: if you are using an ESP8266, and you want to receive >536 bytes in one packet, you can configure LwIP with a larger MSS, at the cost of needing more free RAM for basically all network operations. See the ESP8266 Arduino Core docs on the different LwIP configuration options, and the PlatformIO docs on how to select one.

[timleg002]

If one or more of the devices in question is an ESP8266, the LwIP TCP stack defaults to a MSS (maximum segment size) of 536. This means it sends and receives no more than 536 payload bytes per packet. I don't know much about the WebSocket protocol specifically, but add a little framing overhead and 528 is a reasonable maximum payload to expect in a single call.

Thank you - this is exactly an answer I expected. I forgot to mention I was using an ESP8266, Mathieu probably used an ESP32 which probably uses a higher MSS by default.

The onEvent() callback is called for every data packet handed up by LwIP. You should be able to tell you've received only partial data via comparing len to info->len -- info->len carries the original event size, while len is the number of bytes received in this packet. If len < info->len, you're responsible for doing your own buffering and assembly.

That's why I was asking about relevany of the check, thank you for clearing this up.

By the way, I think this should be mentioned somewhere in the documentation, so users would avoid avoidable debugging.

[mathieucarbou]

Thank you - this is exactly an answer I expected. I forgot to mention I was using an ESP8266, Mathieu probably used an ESP32 which probably uses a higher MSS by default.

Exactly! And I didn't know and would have never imagined that ESP8266 had a lower MSS also!

[mathieucarbou]

By the way, I think this should be mentioned somewhere in the documentation, so users would avoid avoidable debugging.

If you think the example in the wiki is not self-describing enough, please do not hesitate to edit the wiki to improve the doc 👍