Support SQL MERGE in the Trino engine and Hive and Kudu connectors

[djsagain]

This PR consists of three commits that add support for SQL MERGE in the Trino engine, the Hive connector and the Kudu connector. The implementation is structured so that most of the work happens in the Trino engine, so adding support in a connector is pretty simple.

The SQL MERGE implementation allows update of all columns, including partition or bucket columns, and the Trino engine performs redistribution to ensure that the updated rows end up on the appropriate nodes.

The Trino engine commit introduces an enum RowChangeParadigm, which characterizes how a connector modifies rows. Hive uses and Iceberg will use the DELETE_ROW_AND_INSERT_ROW paradigm, since they represent an updated row as a deleted row and an inserted row. Kudu uses the CHANGE_ONLY_UPDATED_COLUMNS paradigm.

Each paradigm corresponds to an implementation of the RowChangeProcessor interface. After this PR is merged, the intent is to retrofit SQL UPDATE to use the same RowChangeParadigm/Processor mechanism.

Extensive documentation on the internal MERGE architecture can be found in the developer doc supporting-merge.rst.

This is a big, complicated PR, and requires close review. @kasiafi, I'm hoping can find time to do your usual excellent job of identifying what doesn't make sense and what needs to be improved.

For #7708

[findepi]

(not a review)

[findepi]

the connector may need to do a prolonged IO operation.
if it does it synchronously, the query may be non-cancellable

add CompletableFuture<?> isBlocked() and use it in MergeOperator.isBlocked

[djsagain]

As side conversations confirmed, this is correctly handled by the fact that AbstractRowChangeOperator and ScanFilterAndProjectOperator both talk to the pageSource.

[findepi]

Can you point me where AbstractRowChangeOperator can return blocked?
i.e. It does not feel correct to push more updates into an updatable page source that declares "i am blocked", does it?

(also, if something required side conversations to confirm, it needs to be documented)

[mosabua]

General question.. how do you plan to document this in the connectors. Specifically what do you document in the connectors where it is supported (Kudu and Hive) and what do we document in all others?

I suggest that we add a Limitations section to all other connectors and add MERGER there .. and build that up to be more complete over time. It might also make sense to have a generic paragraph for readonly connectors. Should we do this in a separate PR or add it here? Either works for me as long as we get both merged for the same release..

wdyt @electrum ?

[djsagain]

I suggest that we add a Limitations section to all other connectors and add ...

This is a repeated suggestion from @mosabua, and there is certainly a problem to be resolved - - how can you tell if a connector supports DELETE or UPDATE or MERGE? There seem to be three choices:

• Centralize that knowledge in the documentation for those SQL operations, e.g., in merge.rst, say: "Only the Hive and Kudu connectors now support MERGE."
• Provide a single table of connectors X operations that shows the which operations are supported for each connectors.
• Add a table in the limitations section in each connector showing the status of each operation, as suggested by @mosabua.

I'm not sure which alternative is best, but the current situation, where you can only tell if a connector supports an operation by trying it and seeing it fail doesn't seem ideal.

[kasiafi]

I've only gone through a part of the execution, and took a look at the StatementAnalyzer.
I had difficulties understanding the page transformations.
One thing I'm concerned about is synthesizing AST in the StatementAnalyzer. That logic should probably be moved to the Planner.

[kasiafi]

The finalQuery should not be created in the Analyzer. This belongs to the Planner. cc @martint

[djsagain]

Let's get @martint's take on it.

[djsagain]

I had difficulties understanding the page transformations.

Did you take a look at https://github.com/djsstarburst/trino/blob/david.stryker/support-sql-merge/docs/src/main/sphinx/develop/supporting-merge.rst? Can you suggest where additional documentation would make it clearer?

One thing I'm concerned about is synthesizing AST in the StatementAnalyzer. That logic should probably be moved to the Planner.

That's what I started out doing. What I found was that it seemed to require re-inventing all the analysis that went on in the StatementAnalysis phase. Moreover, the result seemed fragile. The implementation of MERGE really does need a RIGHT JOIN and a SELECT CASE, and creating them in the planning phase ends up copying a ton of code.

I'd like to get @martint's thoughts on this question.

[findepi]

What Failure is here?

[djsagain]

There is a log line at the end that implies the problem:

                         `log.info("Verifying MERGE on Hive fails - - and it shouldn't");`

But it needs a real comment. Here is the comment I added:

    /**
     * This test demonstrates a failure of Hive to verify the result of a MERGE operation,
     * specifically, Hive fails to recognize the delete_delta file written by the MERGE.  I
     * captured the HDFS delta and delete_delta files and verified that they are correct.
     * I used Wireshark to capture the traffic between Trino and the Hive metastore during
     * the MERGE, and it was all as expected.  I tried to vary the test to understand the
     * issue, but almost any change I made to the test caused Hive to correctly verify the
     * MERGE.
     * TODO: Determine what is causing the Hive verification failure 
     */

[findepi]

Thanks for pointing this out. This looks like a bug somewhere and requires an action item. Can you please make sure this gets properly addressed?

[findepi]

Do you have a test that would update the partition key (purchase here)?

[djsagain]

Several of the tests update partition and bucket columns. For example, testMergeSimpleSelectPartitioned updates the address partition column at: ccc584f#diff-ee4c5c10f12500119c780c8c1ba6b88df410f13b0282bd69e315fe32c5938ffaR1336

However, I would love to have suggestions for more tests to write. I'm always happy to write more tests, but lack imagination as to what would be good ones 😞

[djsagain]

This PR is closed in favor of the improved implementation in #7933