Skip to content

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Jan 17, 2024

What changes were proposed in this pull request?

This PR aims to add ORC compression tests for hive module OrcFileFormat.

Why are the changes needed?

public enum CompressionKind {
  NONE, ZLIB, SNAPPY, LZO, LZ4, ZSTD
}

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs.

Was this patch authored or co-authored using generative AI tooling?

No.

@github-actions github-actions bot added the SQL label Jan 17, 2024
@dongjoon-hyun
Copy link
Member Author

cc @MaxGekk and @HyukjinKwon

[info] HiveOrcSourceSuite:
[info] - SPARK-46742: Read and write with NONE compressions (1 second, 313 milliseconds)
[info] - SPARK-46742: Read and write with SNAPPY compressions (562 milliseconds)
[info] - SPARK-46742: Read and write with ZLIB compressions (564 milliseconds)
[info] - SPARK-46742: Read and write with LZ4 compressions (576 milliseconds)
[info] - SPARK-46742: Read and write with LZO compressions (528 milliseconds)
[info] - SPARK-46742: Read and write with ZSTD compressions (509 milliseconds)
[info] Run completed in 10 seconds, 179 milliseconds.
[info] Total number of tests run: 6
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 6, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 27 s, completed Jan 16, 2024, 8:19:47 PM

@dongjoon-hyun
Copy link
Member Author

Thank you, @HyukjinKwon .
Merged to master.

@dongjoon-hyun dongjoon-hyun deleted the SPARK-46742 branch January 29, 2024 07:54
szehon-ho pushed a commit to szehon-ho/spark that referenced this pull request Feb 7, 2024
… `OrcFileFormat`

### What changes were proposed in this pull request?

This PR aims to add ORC compression tests for `hive` module `OrcFileFormat`.

### Why are the changes needed?

- https://github.com/apache/orc/blob/branch-1.9/java/core/src/java/org/apache/orc/CompressionKind.java
```java
public enum CompressionKind {
  NONE, ZLIB, SNAPPY, LZO, LZ4, ZSTD
}
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#44765 from dongjoon-hyun/SPARK-46742.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants