Add table_supports_delta_delete property and support delta delete function for Raptor table

[kewang1024]

== RELEASE NOTES ==

Raptor Changes

• Add delta delete

[rongrong]

Does the feature exist already? The PR didn't seem to suggest that. Logically the config to enable the feature should be after the introduction of the feature. If we really want to merge this before the actual feature is available, at least there should not be any release notes.

[kewang1024]

Does the feature exist already? The PR didn't seem to suggest that. Logically the config to enable the feature should be after the introduction of the feature. If we really want to merge this before the actual feature is available, at least there should not be any release notes.

So we had an offline discussion that we will be creating a PR to release this feature, and we will have multiple commits. Every commit will be reviewed but not merged. We will be merging the PR after all the commits are finished. @jessesleeping

[jessesleeping]

The first commit LGTM. Some comments other than the inline ones:

• Change the commit message to be more specific. We not only added a column to the metadata, but also added a new table property to gate the feature.

[kewang1024]

All issues addressed, thanks for the review😄! Will start next commit.

[jessesleeping]

Reviewed "Add deltaDeleteEnabled in RaptorSplit". Let's make deltaDeleteEnabled an non-optional member in RaptorSplit.

[jessesleeping]

Reviewed " Add delta_shard_uuid in index table".

Make sure the code refactoring in CREATE table SQL construction doesn't break the previous behavior.

Also, maybe add a comment explaining why every newly created index table will have a delta_shard_uuid column.

[jessesleeping]

Reviewed "Add deltaShardUuid to Raptor split". The current code LGTM but it's incomplete right now as we are lacking the commit logic, which is fine :). We will get there soon.

• We might want to add more tests to RaptorSplitManager once we figure out how to commit delta to index table.

[kewang1024]

Reviewed "Add deltaShardUuid to Raptor split". The current code LGTM but it's incomplete right now as we are lacking the commit logic, which is fine :). We will get there soon.

• We might want to add more tests to RaptorSplitManager once we figure out how to commit delta to index table.

I agree, will add more tests once the whole logic is through

[jessesleeping]

Reviewed all none test code.

[highker]

Some early comments. Haven't started compactor or rewriter parts yet

[highker]

more comments

[highker]

This new OrcReader method seems to be used quite often; can we make it a helper in a util?

[highker]

• Use else for the following branch
• remove return
• move clearRollback(); to the end

[linux-foundation-easycla]

The committers are authorized under a signed CLA.

• ✅ Ke (2923151cf3bc2b938d223c5378acf24c1f8eef86, 3ce349de0a4ea68b662e3c642c57a68543a45eaa, 5d09ecdbd9ec6f2640d90dc9124f4f399ebca9d4, b3dc2a8a47bf69064f68a66fc84795968e3b1f73, 9bdff076b7c5b4b856b1b9df75d2edcb2cedb3cb, ee87c133ed6b393b6c89e46472abd5ecb86057d7, 59b1c88d70e09737f8bd47ca903bf945e04ae6f9, afdb6ac8eb3133ff88df02a2c61e5d79593a2593, 5b5cd4c091c34d3c38519c9f63dae5300c00bfd8, 80818550110ec314f07af7636501fda034493a2d, 5c341611c08db84f51e234d07245004bca832d16)

[kewang1024]

All the issues addressed except for the delta format.

In order to make review easier, most of the newly added code is in new commits, except some bugs and rebase conflicts are merged into old commits.
Currently we already have too many commits and as @jessesleeping suggested, we will be merging them as one or a few bigger commits later. Therefore for current review, pay more attention to code logic instead of commits info

@jessesleeping will commit the data verification logic, so it's not included here yet.

[highker]

"Enable create table with table_supports_delta_delete properties" minor comments

• The commit message body is too wide. Keep each line within 72 characters.
• I would suggest sending out different PRs for each commit. The PR is too large to merge at once.

[highker]

We don't need a config to flag the table property. Just make it false.

[kewang1024]

We don't need a config to flag the table property. Just make it false.

According to the early discussion, we want to have the ability to control the default behavior of all the table newly created, instead of making it a manual operation.

Besides, this feature is a good to have, then maybe we don't need to remove it

[highker]

We don't need this config. Remove

[highker]

not necessary, remove.

[highker]

"Add tableSupportsDeltaDelete and deltaShardUuid in RaptorSplit": Let's make sure the commit message body is within 72 chars

[highker]

We don't need this flag if we can use Optional.empty for deltaShardUuid

[kewang1024]

We don't need this flag if we can use Optional.empty for deltaShardUuid

According to the discussion with Jiexi, the design is that we have a table property indicating if this table supports delta_delete. But in the future, we could have the concept of delta_delete as session property.

Thus having these two parameters give us the flexibility and compatibility to future features.

That's why I didn't fix it for last comment, sorry that I didn't explain last time.

[highker]

remove

[highker]

remove

[highker]

move boolean tableSupportsDeltaDelete to the end

[kewang1024]

Same reason as above.

[highker]

The following three fields are confusing. I assume we only need a BitSet to denote what rows to delete and a flag tableSupportsDeltaDelete to denote what kind of delete we wanna apply?

[highker]

private

[kewang1024]

As @highker suggested, this PR is too large, thus splitting to three PRs.
Will link those three PRs to this one