API: Assign the right field ids when merging schema, #5394

[karuppayya]

Starting with schema:

{A: 1, B: 2, C:3}

And then a user drops C, resulting in:

{A: 1, B: 2} Highest Field ID = 3

Then do a mergeSchema with

{A, B, D}

This will end up assigning (highestId({A:1, B:2}) + 1) to D (3)
The table would then have schemas :

Schemas {
{A: 1, B: 2, C: 3}
{A: 1, B: 2}
{A: 1, B :2, D: 3}
}

[rdblue]

@karuppayya, can you add a description to this PR that documents the problem and how this fixes it?

[rdblue]

This looks correct to me.

[RussellSpitzer]

@rdblue Just to note this is still actually incorrect as we are using the table.currentSchema.highestId and not table.allschemas.highestId. But that fix is going to take a bit more refactoring. I originally guessed the issue was related to dropped fields when this got reported to us.

Imagine I have
{A: 1, B: 2, C:3}

And then a user drops C so I have
{A: 1, B: 2} Highest Field ID = 3

Then we do a mergeSchema with
{A, B, D}

I will end up assigning (highestId({A:1, B:2}) + 1) to D (3)
The table would then have schemas :

Schemas {
{A: 1, B: 2, C: 3}
{A: 1, B: 2}
{A: 1, B :2, D: 3}
}

I still think we should merge this fix first as it corrects monotonically changing tables

[RussellSpitzer]

Known Test Failure, https://github.com/apache/iceberg/pull/5366/files#diff-7883582b4a4aa07e2c25d94e55959d23cfc2ad8e2721b5c1881a2cd2a5205a44

[rdblue]

Re-running failed CI jobs

[rdblue]

Merged. Thanks, @karuppayya! I've also marked this for inclusion in the next patch release.