Add Delta dynamic filtering tests

[findinpath]

Description

Test showcasing the dynamic filtering functionality in the Delta Lake connector

Is this change a fix, improvement, new feature, refactoring, or other?

Test

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

Delta Lake connector test

How would you describe this change to a non-technical end user or system administrator?

Related issues, pull requests, and links

Documentation

(x) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

(x) No release notes entries required.
( ) Release notes entries required with the following suggested text:

[findepi]

i renamed the PR, please rename the commit accordingly.

[raunaqmorarka]

Can we add a test for delta lake that extends BaseDynamicPartitionPruningTest similar to the way it's done for Hive and Iceberg ?

[raunaqmorarka]

We end up needing this and the large table on build side as we don't have a way to make delta lake split manager wait on DF (similar to dynamic-filtering.wait-timeout in hive and iceberg). It would be great to have that as it would simplify the testing and make it less prone to being flaky.

[findinpath]

Compared to Iceberg, DeltaLakeSplitSource produces the splits asynchronously. Does the dynamicFilteringWaitTimeoutMillis concept still apply in this case?

[raunaqmorarka]

Yes, HiveSplitSource produces splits asynchronously as well.

[findinpath]

Created #11600 follow up issue

[raunaqmorarka]

Can we use searchScanFilterAndProjectOperatorStats from AbstractTestQueryFramework instead ?

[findinpath]

I tried now using searchScanFilterAndProjectOperatorStats. However, in case of dealing with the unfilteredStats it seems that there is no corresponding TableScanNode for the table lineitem.

[raunaqmorarka]

It's not working for the unfilteredStats case because searchScanFilterAndProjectOperatorStats always looks for a filter on top of the scan. We can keep the existing code here.

[findinpath]

Can we add a test for delta lake that extends BaseDynamicPartitionPruningTest similar to the way it's done for Hive and Iceberg ?

I will try adding such a test within this PR.

[raunaqmorarka]

This can be parametrised as
@Test(timeOut = 60_000, dataProvider = "joinDistributionTypes")

@DataProvider
    public Object[][] joinDistributionTypes()
    {
        return Stream.of(JoinDistributionType.values())
                .collect(toDataProvider());
    }