Skip to content

[HUDI-4828] Fix the extraction of record keys which may be cut out #6650

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yihua merged 2 commits into from
Sep 17, 2022
Merged

[HUDI-4828] Fix the extraction of record keys which may be cut out #6650

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
102f14f
[minor] following HUDI-4828, fix the extraction for record keys conta…
y0908105023 Sep 10, 2022
28ec2e9
Add a unit test
yihua Sep 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ public static String getPartitionPathFromGenericRecord(GenericRecord genericReco
public static String[] extractRecordKeys(String recordKey) {
String[] fieldKV = recordKey.split(",");
return Arrays.stream(fieldKV).map(kv -> {
final String[] kvArray = kv.split(":");
final String[] kvArray = kv.split(":", 2);
Copy link
Contributor

@yihua yihua Sep 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this is needed when the value contains a colon (:). Could you add a unit test for that?

Copy link
Contributor Author

@y0908105023 y0908105023 Sep 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

Copy link
Contributor

@yihua yihua Sep 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I was referring to the same case and asking for adding a unit test in the repo. I added a unit for this PR. Will land it once CI passes.

if (kvArray.length == 1) {
return kvArray[0];
} else if (kvArray[1].equals(NULL_RECORDKEY_PLACEHOLDER)) {
Expand Down
13 changes: 8 additions & 5 deletions hudi-client/hudi-client-common/src/test/java/org/apache/hudi/keygen/TestKeyGenUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,16 +27,19 @@ public class TestKeyGenUtils {
public void testExtractRecordKeys() {
// test complex key form: field1:val1,field2:val2,...
String[] s1 = KeyGenUtils.extractRecordKeys("id:1");
Assertions.assertArrayEquals(new String[]{"1"}, s1);
Assertions.assertArrayEquals(new String[] {"1"}, s1);

String[] s2 = KeyGenUtils.extractRecordKeys("id:1,id:2");
Assertions.assertArrayEquals(new String[]{"1", "2"}, s2);
Assertions.assertArrayEquals(new String[] {"1", "2"}, s2);

String[] s3 = KeyGenUtils.extractRecordKeys("id:1,id2:__null__,id3:__empty__");
Assertions.assertArrayEquals(new String[]{"1", null, ""}, s3);
Assertions.assertArrayEquals(new String[] {"1", null, ""}, s3);

String[] s4 = KeyGenUtils.extractRecordKeys("id:ab:cd,id2:ef");
Assertions.assertArrayEquals(new String[] {"ab:cd", "ef"}, s4);

// test simple key form: val1
String[] s4 = KeyGenUtils.extractRecordKeys("1");
Assertions.assertArrayEquals(new String[]{"1"}, s4);
String[] s5 = KeyGenUtils.extractRecordKeys("1");
Assertions.assertArrayEquals(new String[] {"1"}, s5);
}
}