Client Chunked encoding does not receive MG_EV_HTTP_REPLY

[dlizewski]

I am using mongoose 6.16 as a web client.

If I perform a GET and consume the data in MG_EV_HTTP_REPLY, I will get the appropriate amount of MG_EV_HTTP_CHUNK callbacks followed by a MG_EV_HTTP_REPLY as expected in a timely manor (a few milliseconds) . This works fine and as expected.

If I instead try to consume the data in the MG_EV_HTTP_CHUNK and set nc->flags |= MG_F_DELETE_CHUNK, I still receive all the data but I don't get a MG_EV_HTTP_REPLY callback until much much later. All of the MG_EV_HTTP_CHUNK come within a few milliseconds but the MG_EV_HTTP_REPLY doesnt come for another 5 seconds after.

Additionally, there seem to be no examples using MG_F_DELETE_CHUNK.
I need to consume the data in chunks because the data I am receiving is too large to be stored in a single buffer.

This may be related to #971

[jimellert]

Hello Sergey,
In addition to a server, I too am using the client feature of Mongoose. But I am also running into the problem of mongoose mishandling the receipt of a chunked transfer encoding response. The problem can be reproduced by hitting this url:
https://jigsaw.w3.org/HTTP/ChunkedScript
I am using ssl and my platform is both windows and linux.

The error is that chunks are not reassembled correctly - chunk lengths appear in the result buffer. Also, the terminating chunk is not recognized, so the connection is not closed, and MG_EV_HTTP_REPLY is not called.
Jim
[email protected]

[jimellert]

Related to above comment on chunked transfer encoding:
My issue appears using version 6.15-2 from microsoft vcpkg.
Jim

[JanValiska]

Same ISSUE here. Sometimes chunks contains data that shouldn't be part of the body data. I encountered that the first attempt to download larger file by chunks succseeded but second attempt failed. Reconstructed data contained data from previous call. Strange. Voting for reproduce and fix.

[cpq]

Please use updated API, see https://github.com/cesanta/mongoose/tree/master/examples/http-streaming-client

[psykokwak-com]

Had the same problem in 6.18.
I cannot update to 7.xx because of the broken API.

[cpq]

@psykokwak-com which exactly broken API are you talking about?

[psykokwak-com]

I tried to update my code to 7.xx and I got lot of errors "symbol undefined" and others. I have no time to check carrefully.
I found a way to patch my code to handle chunks response correctly.

[jimellert]

Sergey, I too patched an old version of mongoose to get myself rolling. If you'd like, I can send you my patched version and the tests I used to verify.

[cpq]

Not really interested in the patched 6.x version.
I'd be happy to help with the 7.x version though, if enough information is provided.

[psykokwak-com]

@jimellert , I am interested in your patch in version 6.xx

[jimellert]

@psykokwak-com , sorry for the late response.
Attached are the .c and .h files that I am using.
mongoose-old.zip
Sergey, I do not have the time to integrate the latest version right now. I will let you know when I get there.

[garag-lib]

hello, the problem is in mg_http_handler, MG_EV_RECV condition.

If we have any chunks... The condition mg_http_create_proto_data reset the pointer to the information needed for concat all chunks to body.

this is a small example of how it is solved

before:

if (req_len > 0) {

  /* New request - new proto data */
  pd = mg_http_create_proto_data(nc);
  pd->rcvd = io->len;

}

after:

if (req_len > 0 &&
    (s = mg_get_http_header(hm, "Transfer-Encoding")) != NULL &&
    mg_vcasecmp(s, "chunked") == 0) {

  if (req_len > 0 && nc->proto_data == NULL) {
      /* New request - new proto data */
      pd = mg_http_create_proto_data(nc);
      pd->rcvd = io->len;
  }

  mg_handle_chunked(nc, hm, io->buf + req_len, io->len - req_len);

} else if (req_len > 0) {

  /* New request - new proto data */
  pd = mg_http_create_proto_data(nc);
  pd->rcvd = io->len;

}