Rely on noMoreOperators for dynamic filters collection

[raunaqmorarka]

Relying on noMoreOperators from DynamicFilterSourceOperatorFactory
for detecting completion of dynamic filter collection will simplify
the implementation for fault tolerant execution where the collection
may take place in a source stage.

[sopel39]

Can it increase DF collection latency?

[raunaqmorarka]

Can it increase DF collection latency?

Normally OperatorFactory#noMoreOperators should fire before or around when the last operator finishes. Do you see a reason to suspect impact to latency ? I can run the usual benchmarks to confirm if needed.

[sopel39]

Normally OperatorFactory#noMoreOperators should fire before or around when the last operator finishes

I think we can bail-out faster before noMoreOperators is called. Idk, maybe it's not an issue. Haven't looked at the code for a while

[raunaqmorarka]

Normally OperatorFactory#noMoreOperators should fire before or around when the last operator finishes

I think we can bail-out faster before noMoreOperators is called. Idk, maybe it's not an issue. Haven't looked at the code for a while

I've run TPC benchmarks and confirmed no regressions. Let me know if you have any input about the code or any problematic scenarios in mind.

[sopel39]

Let me know if you have any input about the code or any problematic scenarios in mind.

Nope. Just wanted to confirm latency is not harmed

[arhimondr]

nit: DynamicFilterConsumer, or maybe simply drop this interface and override the method you need for testing?

[colebow]

Do we want a release notes entry for this?

[raunaqmorarka]

Do we want a release notes entry for this?

This is just an internal refactoring, no user facing change