[SPARK-34029][SQL][TESTS] Add OrcEncryptionSuite and FakeKeyProvider #31065

dongjoon-hyun · 2021-01-06T12:02:04Z

What changes were proposed in this pull request?

This PR aims to add a basis for columnar encryption test framework by add OrcEncryptionSuite and FakeKeyProvider.

Please note that we will improve more in both Apache Spark and Apache ORC in Apache Spark 3.2.0 timeframe.

Why are the changes needed?

Apache ORC 1.6 supports columnar encryption.

Does this PR introduce any user-facing change?

No. This is for a test case.

How was this patch tested?

Pass the newly added test suite.

dongjoon-hyun · 2021-01-06T12:07:34Z

Could you review this, @HyukjinKwon and @maropu ?

sql/core/src/test/java/test/org/apache/spark/sql/execution/datasources/orc/FakeKeyProvider.java

maropu

Minor comments and it looks okay otherwise (if the tests pass).

sql/core/src/test/java/test/org/apache/spark/sql/execution/datasources/orc/FakeKeyProvider.java

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcEncryptionSuite.scala

dongjoon-hyun · 2021-01-06T13:16:38Z

Although only the last commit is the change, the PR is rebased to the master. Sorry about that.

dongjoon-hyun · 2021-01-06T13:21:41Z

This is a test-only patch and tested like the following.

$ build/sbt "sql/testOnly *.OrcEncryptionSuite"
[info] OrcEncryptionSuite:
[info] - Write and read an encrypted file (3 seconds, 918 milliseconds)
[info] - Write and read an encrypted table (786 milliseconds)
...
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 2, Failed 0, Errors 0, Passed 2
[success] Total time: 77 s (01:17), completed Jan 6, 2021 5:19:14 AM

dongjoon-hyun · 2021-01-06T13:23:24Z

Thank you for review and approval, @maropu !

SparkQA · 2021-01-06T13:35:01Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38324/

SparkQA · 2021-01-06T14:10:50Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38324/

SparkQA · 2021-01-06T14:15:04Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38331/

SparkQA · 2021-01-06T14:45:12Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38331/

SparkQA · 2021-01-06T15:22:02Z

Test build #133736 has finished for PR 31065 at commit 079998d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2021-01-06T16:56:27Z

I'm looking at the Jenkins issue. It seems to fail due to environmental issue.

SparkQA · 2021-01-06T16:58:50Z

Test build #133743 has finished for PR 31065 at commit ff1f7d3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2021-01-06T16:58:58Z

Retest this please

SparkQA · 2021-01-06T18:05:54Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38340/

SparkQA · 2021-01-06T18:37:41Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38340/

SparkQA · 2021-01-06T20:00:07Z

Test build #133752 has finished for PR 31065 at commit ff1f7d3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2021-01-06T20:59:24Z

The current Jenkins passed Java/Scala/Python UTs already.

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133754/testReport

Merged to master.

SparkQA · 2021-01-06T21:08:03Z

Test build #133754 has finished for PR 31065 at commit 070d3fd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2021-01-07T01:13:42Z

Jenkins passed, but GitHub Action seems to hit another issue. I'll take a look.

HyukjinKwon · 2021-01-07T04:56:14Z

+1 LGTM!

HyukjinKwon · 2021-01-07T05:30:54Z

@dongjoon-hyun seems like it's still flaky (?). Can you take a quick look for https://github.com/apache/spark/pull/31068/checks?check_run_id=1660627329? I saw this in a PR not related to ORC change.

dongjoon-hyun · 2021-01-07T05:31:48Z

Yes, I'm working on here.

[SPARK-34029][SQL][TESTS][FOLLOWUP][test-maven] Move OrcEncryptionSuite to hive module #31076

dongjoon-hyun · 2021-01-07T05:32:48Z

The Hadoop class is not loaded correctly in sql/core module and we cannot add test dependency.

The failure happens when we build first and run test separately. This is the similar one we hit before. I'm thinking about moving the test from sql/core to sql/hive

HyukjinKwon · 2021-01-07T05:34:37Z

Thanks for working on it!

dongjoon-hyun · 2021-01-07T07:43:47Z

Since the test suite was not isolated correctly, this is reverted via 194edc8

dongjoon-hyun · 2021-01-07T07:44:24Z

According to the current observation, I'll make a new PR and make it pass GitHub Action and Jenkins Maven.

### What changes were proposed in this pull request? This is a retry of #31065 . Last time, the newly add test cases passed in Jenkins and individually, but it's reverted because they fail when `GitHub Action` runs with `SERIAL_SBT_TESTS=1`. In this PR, `SecurityTest` tag is used to isolate `KeyProvider`. This PR aims to add a basis for columnar encryption test framework by add `OrcEncryptionSuite` and `FakeKeyProvider`. Please note that we will improve more in both Apache Spark and Apache ORC in Apache Spark 3.2.0 timeframe. ### Why are the changes needed? Apache ORC 1.6 supports columnar encryption. ### Does this PR introduce _any_ user-facing change? No. This is for a test case. ### How was this patch tested? Pass the newly added test suite. Closes #31603 from dongjoon-hyun/SPARK-34486-RETRY. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

dongjoon-hyun added 3 commits January 6, 2021 04:17

[SPARK-34029][SQL][TESTS] Add OrcEncryptionSuite and FakeKeyProvider

f825502

Adjust import order

56ff451

Add reference

acc5c38

dongjoon-hyun commented Jan 6, 2021

View reviewed changes

sql/core/src/test/java/test/org/apache/spark/sql/execution/datasources/orc/FakeKeyProvider.java Outdated Show resolved Hide resolved

maropu approved these changes Jan 6, 2021

View reviewed changes

Address comments

ff1f7d3

Use a clean JVM

070d3fd

github-actions bot added BUILD SQL labels Jan 6, 2021

dongjoon-hyun closed this in 8bb70bf Jan 6, 2021

dongjoon-hyun deleted the SPARK-34029 branch January 6, 2021 21:00

dongjoon-hyun mentioned this pull request Feb 20, 2021

[SPARK-34029][SQL][TESTS] Add OrcEncryptionSuite and FakeKeyProvider #31603

Closed

[SPARK-34029][SQL][TESTS] Add OrcEncryptionSuite and FakeKeyProvider #31065

[SPARK-34029][SQL][TESTS] Add OrcEncryptionSuite and FakeKeyProvider #31065

Uh oh!

Conversation

dongjoon-hyun commented Jan 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun commented Jan 6, 2021

Uh oh!

Uh oh!

maropu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dongjoon-hyun commented Jan 6, 2021

Uh oh!

dongjoon-hyun commented Jan 6, 2021

Uh oh!

dongjoon-hyun commented Jan 6, 2021

Uh oh!

SparkQA commented Jan 6, 2021

Uh oh!

SparkQA commented Jan 6, 2021

Uh oh!

SparkQA commented Jan 6, 2021

Uh oh!

SparkQA commented Jan 6, 2021

Uh oh!

SparkQA commented Jan 6, 2021

Uh oh!

dongjoon-hyun commented Jan 6, 2021

Uh oh!

SparkQA commented Jan 6, 2021

Uh oh!

dongjoon-hyun commented Jan 6, 2021

Uh oh!

SparkQA commented Jan 6, 2021

Uh oh!

SparkQA commented Jan 6, 2021

Uh oh!

SparkQA commented Jan 6, 2021

Uh oh!

dongjoon-hyun commented Jan 6, 2021

Uh oh!

SparkQA commented Jan 6, 2021

Uh oh!

dongjoon-hyun commented Jan 7, 2021

Uh oh!

HyukjinKwon commented Jan 7, 2021

Uh oh!

HyukjinKwon commented Jan 7, 2021

Uh oh!

dongjoon-hyun commented Jan 7, 2021

Uh oh!

dongjoon-hyun commented Jan 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Jan 7, 2021

Uh oh!

dongjoon-hyun commented Jan 7, 2021

Uh oh!

dongjoon-hyun commented Jan 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dongjoon-hyun commented Jan 6, 2021 •

edited

Loading

maropu left a comment •

edited

Loading

dongjoon-hyun commented Jan 7, 2021 •

edited

Loading