Skip to content

[HUDI-1383] Modify hive partition synchronization #2262

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
linshan-ma wants to merge 24 commits into from
Closed

[HUDI-1383] Modify hive partition synchronization #2262

Changes from 23 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
9990f95
[HUDI-1064]Delete extra rows from the code
linshan-ma Jul 7, 2020
48e79da
[HUDI-1064]remove .get
linshan-ma Jul 7, 2020
8fd3b6c
[HUDI-1064]Trim hoodie table name
linshan-ma Jul 7, 2020
b2ef0fd
Merge remote-tracking branch 'origin/HUDI-1064' into HUDI-1064
linshan-ma Jul 7, 2020
c0302a1
[HUDI-1064]remove .get()
linshan-ma Jul 7, 2020
ed585fc
[HUDI-1064]To solve the java.util.NoSuchElementException: None.get ex…
linshan-ma Jul 7, 2020
8b3e6b5
[HUDI-1064]compiler suceess
linshan-ma Jul 7, 2020
7b0601d
Sync from upstream
linshan-ma Jul 14, 2020
d0bbff7
Merge remote-tracking branch 'upstream/master'
linshan-ma Jul 15, 2020
a10f9fd
Merge remote-tracking branch 'upstream/master'
linshan-ma Jul 22, 2020
9a82e96
Merge remote-tracking branch 'upstream/master'
linshan-ma Jul 28, 2020
a19ca75
Merge remote-tracking branch 'upstream/master'
linshan-ma Jul 29, 2020
d86ddab
Merge remote-tracking branch 'upstream/master'
linshan-ma Aug 6, 2020
d06d94b
Merge remote-tracking branch 'upstream/master'
linshan-ma Aug 7, 2020
d0b9ba2
Merge remote-tracking branch 'upstream/master'
linshan-ma Aug 10, 2020
2999cb8
Merge remote-tracking branch 'upstream/master'
linshan-ma Aug 18, 2020
e6fc3fa
Merge remote-tracking branch 'upstream/master'
linshan-ma Sep 1, 2020
9519226
Merge remote-tracking branch 'upstream/master'
linshan-ma Sep 10, 2020
4656129
Merge remote-tracking branch 'upstream/master'
linshan-ma Sep 27, 2020
7b28e17
Merge remote-tracking branch 'upstream/master'
linshan-ma Nov 11, 2020
48de20d
Merge remote-tracking branch 'upstream/master'
linshan-ma Nov 19, 2020
80b1a4c
Merge remote-tracking branch 'upstream/master'
linshan-ma Nov 19, 2020
c9917ff
[HUDI-1383]Incorrect partitions getting hive synced,Modify hive parti…
linshan-ma Nov 19, 2020
c1e36b2
Adding a test to ensure fix works
nsivabalan Jan 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,6 @@ List<PartitionEvent> getPartitionEvents(List<Partition> tablePartitions, List<St
Map<String, String> paths = new HashMap<>();
for (Partition tablePartition : tablePartitions) {
List<String> hivePartitionValues = tablePartition.getValues();
Collections.sort(hivePartitionValues);
String fullTablePartitionPath =
Path.getPathWithoutSchemeAndAuthority(new Path(tablePartition.getSd().getLocation())).toUri().getPath();
paths.put(String.join(", ", hivePartitionValues), fullTablePartitionPath);
Expand All @@ -206,7 +205,6 @@ List<PartitionEvent> getPartitionEvents(List<Partition> tablePartitions, List<St
String fullStoragePartitionPath = Path.getPathWithoutSchemeAndAuthority(storagePartitionPath).toUri().getPath();
// Check if the partition values or if hdfs path is the same
List<String> storagePartitionValues = partitionValueExtractor.extractPartitionValuesInPath(storagePartition);
Collections.sort(storagePartitionValues);
if (!storagePartitionValues.isEmpty()) {
String storageValue = String.join(", ", storagePartitionValues);
if (!paths.containsKey(storageValue)) {
Expand Down