Skip to content

[HUDI-2509] OverwriteNonDefaultsWithLatestAvroPayload doesn`t work when upsert data with some null value column #3761

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nsivabalan merged 6 commits into from
Nov 3, 2021
Merged

[HUDI-2509] OverwriteNonDefaultsWithLatestAvroPayload doesn`t work when upsert data with some null value column #3761

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
cf20d97
Update OverwriteWithLatestAvroPayload.java
peanut-chenzhong Oct 8, 2021
ee35aaa
Merge branch 'master' of github.com:apache/hudi
Oct 26, 2021
55482c7
[HUDI-2509] OverwriteNonDefaultsWithLatestAvroPayload doesn`t work wh…
Oct 26, 2021
6ccb8e4
[HUDI-2509] OverwriteNonDefaultsWithLatestAvroPayload doesn`t work wh…
peanut-chenzhong Nov 2, 2021
fcd8c5f
Merge branch 'apache:master' into master
peanut-chenzhong Nov 2, 2021
aa24de9
[HUDI-2509] OverwriteNonDefaultsWithLatestAvroPayload doesn`t work wh…
peanut-chenzhong Nov 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithLatestAvroPayload.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@

package org.apache.hudi.common.model;

import org.apache.avro.JsonProperties;
import org.apache.hudi.avro.HoodieAvroUtils;
import org.apache.hudi.common.util.Option;

Expand Down Expand Up @@ -99,6 +100,9 @@ protected boolean isDeleteRecord(GenericRecord genericRecord) {
* Return true if value equals defaultValue otherwise false.
*/
public Boolean overwriteField(Object value, Object defaultValue) {
if (JsonProperties.NULL_VALUE.equals(defaultValue)) {
return value == null;
}
return Objects.equals(value, defaultValue);
}
}
31 changes: 31 additions & 0 deletions ...test/java/org/apache/hudi/common/model/TestOverwriteNonDefaultsWithLatestAvroPayload.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@

package org.apache.hudi.common.model;

import org.apache.avro.JsonProperties;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericRecord;
Expand Down Expand Up @@ -126,4 +127,34 @@ public void testDeletedRecord() throws IOException {
assertEquals(payload1.combineAndGetUpdateValue(delRecord1, schema).get(), record2);
assertFalse(payload2.combineAndGetUpdateValue(record1, schema).isPresent());
}

@Test
Copy link
Member

@codope codope Nov 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a checkstyle violation. Please add an empty line separator between methods.

Copy link
Contributor Author

@peanut-chenzhong peanut-chenzhong Nov 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@codope done

public void testNullColumn() throws IOException {
Schema avroSchema = Schema.createRecord(Arrays.asList(
new Schema.Field("id", Schema.createUnion(Schema.create(Schema.Type.STRING), Schema.create(Schema.Type.NULL)), "", JsonProperties.NULL_VALUE),
new Schema.Field("name", Schema.createUnion(Schema.create(Schema.Type.STRING), Schema.create(Schema.Type.NULL)), "", JsonProperties.NULL_VALUE),
new Schema.Field("age", Schema.createUnion(Schema.create(Schema.Type.STRING), Schema.create(Schema.Type.NULL)), "", JsonProperties.NULL_VALUE),
new Schema.Field("job", Schema.createUnion(Schema.create(Schema.Type.STRING), Schema.create(Schema.Type.NULL)), "", JsonProperties.NULL_VALUE)
));
GenericRecord record1 = new GenericData.Record(avroSchema);
record1.put("id", "1");
record1.put("name", "aa");
record1.put("age", "1");
record1.put("job", "1");

GenericRecord record2 = new GenericData.Record(avroSchema);
record2.put("id", "1");
record2.put("name", "bb");
record2.put("age", "2");
record2.put("job", null);

GenericRecord record3 = new GenericData.Record(avroSchema);
record3.put("id", "1");
record3.put("name", "bb");
record3.put("age", "2");
record3.put("job", "1");

OverwriteNonDefaultsWithLatestAvroPayload payload2 = new OverwriteNonDefaultsWithLatestAvroPayload(record2, 1);
assertEquals(payload2.combineAndGetUpdateValue(record1, avroSchema).get(), record3);
}
}