Add a batching queue implementation.

[erikjohnston]

This will be used during for the rewrite of the KeyRing.

[clokep]

Could this cause a Deferred to have both callback and errback called on it if e.g. the second deferred.callback causes an error to be raised?

(I wonder if we should be guarding with if not deferred.called?)

[erikjohnston]

I don't think deferred.callback should ever explode, but may as well add the guard.

[clokep]

I would hope not. Another option would be to move the callback calls to an else so that only calling the processing function is in the try clause?

[erikjohnston]

I think I prefer it as it is, as then we really should always call either .errback or .callback for each?

[clokep]

Right, isn't that what my suggestion guarantees though?

try:
    result = processing_callback(values)
except Exception as e:
    for d in deferreds:
        d.errback(e)
else:
    for d in deferreds:
        d.callback(result)

[erikjohnston]

But if d.callback never throws then that's the same as

try:
    result = processing_callback(values)

    for d in deferreds:
        d.callback(result)
except Exception as e:
    for d in deferreds:
        d.errback(e)

?

[clokep]

Yes. 😄

If it does throw it has the benefit of causing a stack trace and letting us know about it.

[clokep]

OK, you can ignore me. It can't throw (for the reason I thought it could):

https://github.com/twisted/twisted/blob/cae365213868e8226a055f6b5e4d2c0c89c9ab3b/src/twisted/internet/defer.py#L851-L871

[clokep]

A fun trick you can do is values, deferreds = zip(*next_values), doesn't really save anything here though unfortunately since you have to iterate all the Deferreds anyway.

[clokep]

This seems to have some failing tests too?

[erikjohnston]

This seems to have some failing tests too?

Serves me right for not running the tests after adding the metrics!

[clokep]

LGTM!

[clokep]

Suggested change

	given key and call the `self.process_batch_callback` with the values.
	given key and call the `self._process_batch_callback` with the values.