http.POST() limit of payload size?

[TheAustrian]

Hi!

I'm trying to send data as an HTTPClient to a server with http.POST() and it works great but the payload gets cut off around the 3000 character mark. Does this have something to do with "application/x-www-form-urlencoded"? It shouldn't, iirc, cause POST shouldn't have a size limit?

HTTPClient http;
http.begin("http://example.com/weather/espdata.php");
http.addHeader("Content-Type", "application/x-www-form-urlencoded");
http.POST(textToSend);
http.end();

I just wanted to know if there is a limit on payload size so I can rule out some other issue with my server. The String I'm sending gets put out normally in the serial monitor, so that shouldn't be the problem.

Thanks!

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[Humancell]

Bump!

I'm seeing this same issue ... when my payload crossed the 3k mark I'm now getting a http.POST() return of -3 ... HTTPC_ERROR_SEND_PAYLOAD_FAILED ... but can't find any info on why this is, and what exactly failed?

Is there a buffer that needs to be adjusted, or?

[chaeplin]

I think payload size > one packet size --> err.
https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp#L376-L380

[Humancell]

Ok ... this is what I wondered. I actually created a workaround that I'm using successfully, just in case anyone else requires a similar fix. To do it I had to use WifiClient:

#include <ESP8266HTTPClient.h>

String jsonPayload = "{ my big giant 3.5k byte json payload ....}";
int contentLength = jsonPayload.length();

  // create the request and headers
  String request = "POST " + String(endpointURL) + " HTTP/1.1\r\n" +
    "Host: " + String(endpointHost) + "\r\n" + 
    "Accept: application/json" + "\r\n" + 
    "Content-Type: application/json\r\n" +
    "Content-Length: " + String(contentLength) + "\r\n\r\n";

  // Use WiFiClient class to create TCP connections
  WiFiClient client;

  connectResult = client.connect(endpointHost, atoi(endpointPort));

  // This will send the request and headers to the server
  client.print(request);

  // now we need to chunk the payload into 1000 byte chunks
  int cIndex;
  for (cIndex = 0; cIndex < contentLength; cIndex = cIndex + 1000) {
    client.print(jsonPayload.substring(cIndex, cIndex+1000));
    //Serial.print(jsonPayload.substring(cIndex, cIndex+1000));
  }
  client.print(jsonPayload.substring(cIndex));
  //Serial.println(jsonPayload.substring(cIndex));

I have a little more response checking and other stuff ... but you get the idea. Using this I am now sending pretty big payloads, and it seems to work fine. The 1000 bytes was just an arbitrary guess ... if I have time I might try to adjust this to fine tune the operation.

Obviously there are limitations in this approach as I can't do SSL easily (or can I?) ... but maybe something like this ought to be rolled into the handling of the payload on the call to http.POST in the HTTPClient?

[chaeplin]

@Humancell use WiFiClientSecure.h for ssl.
https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/examples/HTTPSRequest/HTTPSRequest.ino

Chunk can be large to 1460.

[Humancell]

Thanks! Nice bonus! I'll make my changes. :-)

[espmut]

I just ran into this issue as well. I believe the limit is just under 2000 Bytes. I hope there is a work around or solution for this. I'd rather use HTTPClient.

[melvinpmathew]

Is there a work around for this issue? I need to send 8000bytes on POST>

[devyte]

@TheAustrian @Humancell is this issue still valid with latest git?

[mcanyucel]

I am trying to upload a csv file with size of around 200 kB and using the method int HTTPClient::sendRequest(const char * type, Stream * stream, size_t size) and get the error HTTPC_ERROR_SEND_PAYLOAD_FAILED. Smaller size files work perfectly so I am assuming there is an unwritten file size limit or kind of a bug. This is mentioned in few stackoverflow questions as well but they have no answer (here.
Any new information, suggestions, progress, workaround?

[espmut]

For me, I broke up the post into multiple posts.

…

On Mon, Jan 29, 2018 at 6:54 AM, Mustafa Can Yücel ***@***.*** > wrote: I am trying to upload a csv file with size of around 200 kB and using the method int HTTPClient::sendRequest(const char * type, Stream * stream, size_t size) and get the error HTTPC_ERROR_SEND_PAYLOAD_FAILED. Smaller size files work perfectly so I am assuming there is an unwritten file size limit or kind of a bug. This is mentioned in few stackoverflow questions as well but they have no answer (here <https://stackoverflow.com/questions/41388417/send-payload-error-in-http-post-arduino-esp8266> . Any new information, suggestions, progress, workaround? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1872 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AbYZfq_JO_OP1ezEystbN2ttXG2DpOgyks5tPc2ngaJpZM4IBZhy> .

-- Machine Utilization Tracker www.mutcloud.com 602.524.0192

[Silberlachs]

I see an issue when receiving rss feeds from a news portal over http GET (get string to be clear)

One RSS page with kind of high priority news gets transmitted normally and the esp can send it to a lcd display.

But whenever I try to get the "normal" news feeds, the esp just doesn't do anything at all.
Maybe it restarted?

The normal rss feed has 14527 Characters, the high priority one has 2676 Characters...

I've gone with the high priority one for now and I can see why nearly 15k characters is a bit overkill for this little thing, but still it's an issue....

[mcanyucel]

@espmut How did you manage to divide the file? I used another approach that reads the file into a WiFiClient using MTU_Size chunks but it floods everywhere with stack errors :(

[d-a-v]

I could reproduce the error.
Can you retry sending big data without chunking into pieces, with DebugLevel HTTP_CLIENT (in Tools menu, with Serial for DebugPort) ?
I get:

[HTTP-Client][sendRequest] short write, asked for xxxx but got yyyy retry...

2MB data are working flawlessly with all debug options activated (they slow down processing a lot).
Bug is located, fix will come hopefully.

Full sketch:

#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>

#define ssid "open"
#define password ""

void setup()
{
  Serial.begin(115200);
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");

  // Print the IP address
  Serial.println(WiFi.localIP());

}

class FakeStream: public Stream
{
  protected:
    size_t fakesize;

  public:
    FakeStream (size_t size): fakesize(size) { }

    virtual int available() {
      return fakesize;
    }
    virtual int read() {
      if (fakesize) {
        --fakesize;
        return 0x5a;
      }
      return -1;
    }
    virtual int peek() {
      return 0x5a;
    }
    virtual void flush() {
      fakesize = 0;
    };
    virtual size_t write (uint8_t data) {
      (void)data;
      return 1;
    }
    size_t size () {
      return fakesize;
    }
    const char* name () {
      return "5a";
    }
};


void loop()
{
  // put your main code here, to run repeatedly:
  HTTPClient http;
  http.begin("http://10.43.1.254:80/test/upload.php");

  int contentLength = 2000000;

  http.addHeader("Content-Type", "application/octet-stream");
  http.addHeader("Content-Length", String(contentLength));

  Serial.println("a----------------------------------------------");
  FakeStream s(contentLength);
  int httpCode = http.sendRequest("POST", &s, contentLength);
  Serial.println(httpCode);
  Serial.println("b----------------------------------------------");

  while (true)delay(1000);
}

The upload.php (got in some other issue):

<?php
file_put_contents('/var/www/html/test/test.txt', file_get_contents('php://input'));
?>