Fix an issue while accessing Symlink tables#25307
Conversation
Introduction of WrapperJobConf and CopyOnFirstWriteConfiguration lead to issues while accessing Symlink tables since the actual configuration object is wrapped inside a configuration object
7244684 to
81a6edf
Compare
agrawalreetika
left a comment
There was a problem hiding this comment.
Is there any specific scenario when config will create an issue? Since there is an existing symlink Tests here https://github.com/prestodb/presto/blob/master/presto-product-tests/src/main/java/com/facebook/presto/tests/hive/TestSymlinkTableListCaching.java#L72 which is using catalog with config resource in catalog and works fine.
|
One example is included in the linked issue in the description. This problem occurs only when a query relies on Hadoop configurations that have been modified by Presto. The integration tests run on HDFS and Hadoop and that probably works just fine with the default Hadoop Configuration object. |
|
Any way to unit test this? |
Yeah, there aren't any existing tests to catch these edge cases. To catch this issue, we need to use a filesystem other than HDFS, something like S3 or GCS, which are supported by Presto out of the box. I will look to either add an integration test using MinIO (which simulates S3) or explore adding a unit test specifically to verify the Configuration object being used in that part of the code. I will add it in a follow-up PR as discussed, Thanks. |
Description
Introduction of WrapperJobConf and CopyOnFirstWriteConfiguration lead to issues while accessing Symlink tables since the actual configuration object is wrapped inside a configuration object
Motivation and Context
Fixes #25306
Test Plan
Ran manual tests and was able to access the Symlink tables on the S3 filesystem.
Contributor checklist
Release Notes
Please follow release notes guidelines and fill in the release notes below.