Add a rate limiter to jaeger-ingester

[vprithvi]

Requirement - what kind of business use case are you trying to solve?

Limit the rate of writes to the storage backend when using jaeger-ingester.

When there is a failure of the storage backed, the ingester can consume and write to the backing store at rates higher than the backing store can handle, resulting in a failure in storage nodes, leading to more chaos. We've seen this happen with Cassandra, and is probably an issue with any storage engine that doesn't gracefully degrade.

Problem - what in Jaeger blocks you from solving the requirement?

none

Proposal - what do you suggest to solve the problem or improve the existing situation?

Adding a configurable rate limiter to the Kafka reader will enable operators to set a safe throughput target.

Any open questions to address

What would be a sensible default value?

[yurishkuro]

What would be a sensible default value?

Default could be no rate limiting (backwards-compatible).

[isaachier]

I'm wondering if this makes more sense as an ingester-specific option or as a general storage writer option?

OK never mind. Seems to affect reader, not writer.

[isaachier]

So should we revisit my crossed out comment above:

I'm wondering if this makes more sense as an ingester-specific option or as a general storage writer option?