Avoid blocking on a split source in fault tolerant task scheduling

[raunaqmorarka]

It's possible for the probe side table scan of a join to block
producing splits until the build side collects dynamic filters.
In such a scenario, we need to avoid waiting on the blocked splits
synchronously and allow other stages to schedule.

Description

Is this change a fix, improvement, new feature, refactoring, or other?

fix

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

core query engine

How would you describe this change to a non-technical end user or system administrator?

Related issues, pull requests, and links

Documentation

( x) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

( x) No release notes entries required.
( ) Release notes entries required with the following suggested text:

[arhimondr]

The implementation looks good to me.

I think we should probably improve our test cases (Sorry for such a late afterthought, I should've thought about it earlier and comment).

When the components (TaskSource / FaultTolerantScheduler) were simply waiting for futures to resolve there wasn't any difference between a synchronous and asynchronous SplitSource, but now there is. Thus I wonder if it makes sense to add additional test cases that verify correctness when the futures returned (by either a TaskSource or a SplitSource) are no immediately "done").

[losipiuk]

LGTM - one comment