Example of an ArduinoJson request and response

[butaikis]

Hello. Please make an example of efficiently reading and sending large json data using the ArdinoJson 7 library. At the moment, I have implemented it this way.

Processing a request to the server:

WebServer.on(
      "/api", HTTP_POST, [this](AsyncWebServerRequest *request)
      {
      },
      NULL, [this](AsyncWebServerRequest *request, uint8_t *data, size_t len, size_t index, size_t total)
      {
        
        if (CORE.busy != t_core::t_busy::FREE)
        {
          API_Busy(t_core::t_busy::HTTP, request);
          return;
        }
        debug("HTTP Req /api");
        CORE.busy = t_core::t_busy::HTTP;
        //read to buffer
        HTTP_SERVER_JSON_BUFFER.concat(data, len);


        debug("len: " + (String)len + " index: " + (String)index + " total: " + (String)total);

        if (index + len == total)
        {
          request->client()->setRxTimeout(10);
          TMR.http = millis();
          HTTP_Request = request;
          DeserializationError derr;
          if (derr = deserializeJson(INPUT_JSON, HTTP_SERVER_JSON_BUFFER))
          {
            // debug("Deserealiz ERR! " + (String ) derr.c_str());
            API_Send_Answer("error");
          }
          else
          {
            // debug("Deserealiz OK!");
            API_Read_JSON();
          }
        }
      });

Response processing:

//check size
 if (measureJson(OUT_JSON) > 61440)
 {
   API_Send_Answer("error_large_json_out");
   return;
 }
 
 AsyncResponseStream  *response = request->beginResponseStream("application/json", 61440);
 serializeJson(OUT_JSON, *response);
 request->send(response);

[mathieucarbou]

Moved to a discussion.

[mathieucarbou]

Hello,
The way you did it by collecting the data in a buffer is ok.
There is currently no Printer implementation accessible that would allow to stream directly ArduinoJson memory object to the network, or read the incoming data directly into a ArduinoJson object because internally it does not work in a stream way by by sending or receiving packets by sequences.
Some people have tries to work around that you can find some discussion in my forks.

In my app using ESP-DASH, I have Json payloads that can reach up to 8k to I go through websocket, split them in batches and use message pack +binary transport.

If your client can support message pack, I added support for that in ESPAsyncWS http endpoints too. It works like the json API but with message pack. It would reduce your payload size by 30-60% depending.

[mathieucarbou]

The discussion is inside here but do not remember where. Two people have posted code examples how they handle that:
https://github.com/mathieucarbou/ESPAsyncWebServer/discussions

[butaikis]

I don't use websockets. As far as I understand from your message, it is impossible to make receiving/sending json more efficient than it is now?

[mathieucarbou]

yes, that's right.

there is no way to make ArduinoJson directly serialize or deserialize from the "network" / socket.

but I do not understand also the use case, when you say big json documents. Are they document that you are building (JsonDocument) and that are staying staying on heap ? Or they are files that you want to serve and upload.

These are completely different use cases.

If you are talking about a JsonDocument, and you are building a huge JsonDocument in the heap, just know that the limitation will comes from there first (ArduinoJson), not ESPAsyncWS, because the JsonDocument takes a LOT of space on heap, far more than what would be the serialized length.

And same for deserialisation.

So in order to upload a json document that you want to deserialise and put on heap, you need:

1. to accumulate the doc in a buffer => heap buffer length == document size
2. to make ArduinJson deserialise into a JsonDocument on heap => will take more than the document size

So you need to have at least free heap which is about 2.5 times the incoming document size.

ArduinoJson provides some filtering mechanism to handle large documents:

https://arduinojson.org/v7/how-to/deserialize-a-very-large-document/

But just know that the limitations are not set by ESPASyncWS: any WS on ESP32 would have the same.

Limitations comes from:

1. MCU - not a lot of haap
2. The need to have a buffer to read whats coming from the network and use it to build a JsonDoc (so before deleting the buffer, there is a point in time where you can have 2.5 times the doc size used in heap)

[butaikis]

Thanks for the tips) at the moment I have no difficulties using the ArduinoJson library and large documents (<100KB). I don't keep json in the heap on a permanent basis, only when making API calls. It is executed and the json response is returned, then JsonDocument (clear()) is reset.
I just wanted to clarify if there is a more efficient method, because I noticed that the EspAsyncWebServer library has started to be updated and this is encouraging)

[mathieucarbou]

yeah... no there is no alternative sadly.
In many languages there have json libraries that work by streaming and calling callbacks (or visitors) when they see a new element, but ArduinoJson does not do that....
You would need a lib that is able to read the buffer, and on each json item call a method.
if you are only interested in some fields, probably the filter mechanism can help.