Skip to content

Correct release notes entry in 377 #12022

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
findepi merged 1 commit into from
Apr 20, 2022
Merged

Correct release notes entry in 377 #12022

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/src/main/sphinx/release/release-377.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,10 @@

## Hive connector

* Add support for partition names that include special characters. ({issue}`11719`)
Copy link
Copy Markdown
Member Author

@mosabua mosabua Apr 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This linked was linked from the one we now add and we thought this is fixed.

* Add support for `date` type partition names with timestamp formatting. ({issue}`11873`)
* Improve performance of queries that use Glue metadata. ({issue}`11869`)
* Fix `sync_partition_metadata` procedure when partition names differ from
Copy link
Copy Markdown
Member Author

@mosabua mosabua Apr 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wording is still not ideal .. e.g. what was happening before .. did it just fail or corrupt something?

Copy link
Copy Markdown
Contributor

@findinpath findinpath Apr 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sync_partition_metadata can cope with partition names which differ from the partition paths on the file system

Copy link
Copy Markdown
Member Author

@mosabua mosabua Apr 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thats not a full sentence .. also still missing what this did before

E.g.

Prevent failures of ...

Copy link
Copy Markdown
Contributor

@findinpath findinpath Apr 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix the comparison mechanism for the partition names between the metastore and the file system in the system.sync_partition_metadata stored procedure.

Does this fit better?

Copy link
Copy Markdown
Member Author

@mosabua mosabua Apr 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well... still not sure what happened before.. did data get corrupted or did the procedure just fail or something else?

Copy link
Copy Markdown
Contributor

@findinpath findinpath Apr 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to the fact that the comparison mechanism between the partition names stored in HMS and partition paths on the file system/ object storage was incorrect, when trying to sync the partition metadata the second time for a table, the user kept hitting rather puzzling exceptions.

When using 'FULL' mode after the first successful 'FULL' mode run:

dropping a partition added in the same transaction is not supported: testschema.test_table_external [2021-04-25, 20:00:13, 12345]

When using 'ADD' mode after the first successful 'FULL' mode run :

One or more partitions already exist for table 'testschema.test_table_external'

This was due to the fact that partition paths corresponding to existing partition names in HMS were falsely marked to be added/removed from HMS, reason why HMS threw the exceptions mentioned earlier.

I hope this comment gives a better context of the problem.

Copy link
Copy Markdown
Member Author

@mosabua mosabua Apr 20, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay .. thats rather tricky. Do you have any suggestions on how to make a useful release notes entry out of this @martint ?

Maybe just

Prevent failures of the sync_partition_metadata system procedure for inconsistent partition name values stored in Hive metastore and used in the file system.

partition paths on the file system. ({issue}`11864`)

## Iceberg connector

Expand Down