Skip to content

Add Hive S3 tests for comma and pipe delimited CSVs#18786

Merged
pettyjamesm merged 1 commit intoprestodb:masterfrom
dnanuti:additional-s3-tests
Jan 10, 2023
Merged

Add Hive S3 tests for comma and pipe delimited CSVs#18786
pettyjamesm merged 1 commit intoprestodb:masterfrom
dnanuti:additional-s3-tests

Conversation

@dnanuti
Copy link

@dnanuti dnanuti commented Dec 8, 2022

Test plan
Locally ran tests:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.fs.HadoopExtendedFileSystemCache (file:/Users/dnnanuti/.m2/repository/com/facebook/presto/presto-hive-common/0.279-SNAPSHOT/presto-hive-common-0.279-SNAPSHOT.jar) to field java.lang.reflect.Field.modifiers
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.fs.HadoopExtendedFileSystemCache
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2022-12-08T11:22:56.901-0600 WARNING Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2022-12-08T11:22:57.069-0600 INFO Successfully loaded & initialized native-bzip2 library system-native
2022-12-08T11:22:57.077-0600 INFO Successfully loaded & initialized native-zlib library
2022-12-08T11:22:59.853-0600 INFO io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2022-12-08T11:23:01.081-0600 WARNING NoSuchMethodException was thrown when disabling normalizeUri. This indicates you are using an old version (< 4.5.8) of Apache http client. It is recommended to use http client version >= 4.5.9 to avoid the breaking change introduced in apache client 4.5.7 and the latency in exception handling. See https://github.com/aws/aws-sdk-java/issues/1919 for more information
2022-12-08T11:23:02.610-0600 INFO Got brand-new decompressor [.bz2]
2022-12-08T11:23:02.914-0600 INFO Got brand-new decompressor [.gz]
2022-12-08T11:23:02.963-0600 INFO Got brand-new decompressor [.lz4]
2022-12-08T11:23:03.654-0600 INFO mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2022-12-08T11:23:03.659-0600 INFO mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
2022-12-08T11:23:19.978-0600 INFO mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
2022-12-08T11:23:20.160-0600 INFO Got brand-new compressor [.gz]
2022-12-08T11:23:39.428-0600 INFO Got brand-new decompressor [.gz]
2022-12-08T11:23:39.429-0600 INFO Got brand-new decompressor [.gz]
2022-12-08T11:23:39.429-0600 INFO Got brand-new decompressor [.gz]
2022-12-08T11:23:45.967-0600 INFO Test com.facebook.presto.hive.TestHiveFileSystemS3SelectPushdown::testTableCreation took 42.96s
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.375 s - in TestSuite
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:22 min
[INFO] Finished at: 2022-12-08T17:23:48Z
[INFO] ------------------------------------------------------------------------

== NO RELEASE NOTE ==

Rebased on master to include #18779 changes.

Implement method for table filtering in utility class.
Add extra tests for S3 Select pushdown for Hive connector.
@dnanuti dnanuti force-pushed the additional-s3-tests branch from f391dde to 1eeb3f1 Compare January 3, 2023 17:13
@dnanuti dnanuti changed the title WIP: Add Hive S3 tests for comma and pipe delimited CSVs Add Hive S3 tests for comma and pipe delimited CSVs Jan 3, 2023
Copy link
Contributor

@pettyjamesm pettyjamesm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for adding the additional tests!

@pettyjamesm pettyjamesm merged commit 2a7dd51 into prestodb:master Jan 10, 2023
@pettyjamesm
Copy link
Contributor

Merged, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants