no putRecords for batch writting

[shiroh]

Hi,
I found it is only putRecord but no putRecords for batch operation. I have just do the load test today, when the write reach 200/sec, response will be 400 and show exceed rate limitation which may tell the requests send to fast according to amazon's document.
Did I miss something in the code?

[mhart]

No you haven't missed anything in the code – there is no batch writing. This may be an option to add, but there would need to be some configuration around it – how many messages to batch, how often to send the batch, etc – as it could result in unflushed/lost messages.

It wouldn't necessarily stop you getting 400 messages though – the rate limitation from Amazon is applied on a per shard basis so it doesn't matter if you're batching or not – basically if you want to increase throughput then you need to create more shards.

Batching would reduce the number of client requests made – so if that's your bottleneck, then it would help there, but it sounds to me like you're getting throttled by the server.

[shiroh]

ProvisionedThroughputExceededException
The request rate is too high, or the requested data is too large for the available throughput. Reduce the frequency or size of your requests. For more information, see Error Retries and Exponential Backoff in AWS in the AWS General Reference.

above is the explanation of 400 in amazon's doc
I guess the _request rate_ here is the http request but not how many writes.
I have tested against a 10 shards stream, but only managed to send around 200 request / sec before I got 400.

So I am wondering whether a _putRecords_ could be a workaround. I will try to add the function into the lib to see if it works.