Improve IS NULL pushdown for ClickHouse complex expression

[ssheikin]

Description

Additional context and related issues

• More predicate pushdown in ClickHouse connector #7100
• Running branch for Clickhouse pushdowns #23460
• Improve Clickhouse pushdown. Attempt 1 #22921
• Test predicates pushdown to ClickHouse FLOAT type #23496
• Unconditionally pushdown varchar predicate to Clickhouse #23516
• Improve Clickhouse varchar equalities pushdown #23455
• Support Clickhouse varchar comparison pushdown #23558
• Improve ClickHouse varchar IN pushdown for complex expression #23581
• Improve ClickHouse LIKE pushdown for complex expression #23591
• Improve IS NULL pushdown for ClickHouse complex expression #23459

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
() Release notes are required. Please propose a release note for me.
(x ) Release notes are required, with the following suggested text:

# Section
* Improve IS NULL pushdown for ClickHouse complex expression. ({issue}`23459`)

[ssheikin]

@Praveen2112 @hashhar @ebyhr please review.

[krvikash]

Overall lgtm.

[krvikash]

What we are verifying here actually? It is verifying for only first test case that inputLiteral should not be null.
It seems that before adding the round trip test case, we should be knowing that 1st test case should not have NULL as input literal.

[krvikash]

After going through the code, I got the intention of why we needed SPECIAL_COLUMNS. May be NON_NULL_COLUMNS makes more sense here. WDYT?

[ssheikin]

I had TestClickHouseConnectorTest#testTextualPredicatePushdown in mind, where SPECIAL_COLUMNS are

                        unsupported_1 Point,
                        unsupported_2 Point,
                        some_column String,

[Praveen2112]

What if we could handle this special columns as part of this framework ?

[ssheikin]

I think it's possible however I'm not sure how to make it look better that it is right now, because the type of the special columns may differ for different cases:

                .addTestCase("String", "'z'", VARCHAR, "CAST('z' AS varchar)") // special, non null column
                .addTestCase("String", "'z'", VARBINARY, "CAST('z' AS varbinary)") // special, non null column

[Praveen2112]

Can we inject queryAssertions instead of query runner ?

[ssheikin]

It conforms to the api of SqlDataTypeTest if we later want to unify these assertions.

[ssheikin]

Also not sure if it gives benefits.

[Praveen2112]

What if we could handle this special columns as part of this framework ?

[ssheikin]

@Praveen2112 please approve and merge.

[Praveen2112]

Instead of asserting them twice like once with all columns AND ? Can we assert them each column at a time ? Like

            for (int column = SPECIAL_COLUMNS; column < testCases.size(); column++) {
                String queryWithSingleColumnPredicate = "SELECT " + firstColumnName + " FROM " + temporaryRelation.getName() + " WHERE " + getPredicate(column, isNull) + withConnectorExpression;
                assertPushdown(expectPushdown,
                        assertResult(isNull ? Optional.of(firstCase.expectedLiteral()) : Optional.empty(),
                                assertThat(queryAssertions.query(session, queryWithSingleColumnPredicate))));
            }

This would allow us to identify the specific columns which would cause failure in the future

[ssheikin]

Not sure if I follow your suggestion. We already asserting each column at a time, in case of failure.

[Praveen2112]

Instead of handling twice can we try asserting only once for each column so that it would be easier to debug in case of failures

[ssheikin]

The first check for all columns is N times faster than the second for single columns. So it's an improvement for sunny-day scenario.
In case of failure there is overhead of 1 additional check, performing N additional checks for single columns.
This was copied from SqlDataTypeTest.
Please let me know if you want to simplify code.

[Praveen2112]

Makes sense !! Let's have it like this.

[Praveen2112]

If we could inline IS NULL and IS NOT NULL and the assertion could be inlined right ? Ideally for the test data - we for IS NULL we get a value and for IS NOT NULL we get an empty value - Do we need this dedicated method for each assertion.

[ssheikin]

Not sure if I follow. I'm ok with any solution, just to merge this PR.

[ssheikin]

@Praveen2112 do you think we are good to merge this PR?
It has a nice score of modified lines:

[Praveen2112]

LGTM

[ssheikin]

@Praveen2112 ptal

[Praveen2112]

Thanks for working on this

[Praveen2112]

@ssheikin Can we update the PR description.