Add replay utility. #155
Add replay utility. #155
Conversation
| burst: bool, optional | ||
| If True, emit as fast as possible, ignoring timing. False by default. | ||
| """ | ||
| DOCUMENTS_WITHOUT_A_TIME = {'datum', 'datum_page', 'resource'} |
There was a problem hiding this comment.
Looks like a list. Why the curly braces?
There was a problem hiding this comment.
Micro-optimization to use a set instead of a list. It make the init a bit slower, makes x in DOCUMENTS_WITHOUT_A_TIME a bit faster. Removes the [n] access because there are no meaningful semantics in the order of these values.
|
Hmm, maybe we should take a time window to batch stuff into, an N to batch into, or just always break the pages up into individual events? |
|
We have code in databroker that joins events across streams, subdividing pages only as necessary to ensure monotonic time. I think it would take more than trivial effort to factor it out, but that seems like the right path here. We want that code to be available widely. |
This ports a utility that we used for the FXI experiment from my gist to event-model.
The test shows that this does not work with
event_page. Need some thought.