[SPARK-33483][INFRA][TESTS] Fix rat exclusion patterns and add a LICENSE #30418

dongjoon-hyun · 2020-11-19T03:15:32Z

What changes were proposed in this pull request?

This PR fixes the RAT exclusion rule which was originated from SPARK-1144 (Apache Spark 1.0)

Why are the changes needed?

This prevents the situation like #30415.

Currently, it missed catalog directory due to .log rule.

$ dev/check-license
Could not find Apache license headers in the following files:
 !????? /Users/dongjoon/APACHE/spark-merge/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/MetadataColumn.java
 !????? /Users/dongjoon/APACHE/spark-merge/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsMetadataColumns.java

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CI with the new rule.

dongjoon-hyun · 2020-11-19T03:20:54Z

cc @rdblue , @HyukjinKwon , @viirya

dongjoon-hyun · 2020-11-19T03:24:26Z

cc @ScrapCodes for SPARK-1144, too

viirya · 2020-11-19T03:35:53Z

dev/.rat-excludes

 .*txt
 .*json
 .*data


Does it also apply to above .txt, .json, .data?

To be safe, yes. Those are also looks suspicious.

sunchao · 2020-11-19T03:48:10Z

nice find @dongjoon-hyun !

SparkQA · 2020-11-19T04:00:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35916/

dongjoon-hyun · 2020-11-19T04:02:36Z

Thanks. @viirya , @HyukjinKwon , @sunchao .
I updated more according to @viirya 's comment.

dongjoon-hyun · 2020-11-19T04:04:18Z

sql/hive/src/test/resources/data/scripts/test_transform.py

+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#


I created a JIRA because I touch this file inevitabily.

For this change, we need to wait for the full test result of GitHub Action.

dongjoon-hyun · 2020-11-19T04:07:41Z

dev/.rat-excludes

+.*\.avsc
+.*\.txt
+.*\.json
+.*\.data


This hides the directory data and metadata.

dongjoon-hyun · 2020-11-19T04:07:52Z

dev/.rat-excludes

+.*\.txt
+.*\.json
+.*\.data
+.*\.log


This hides catalog directory.

HyukjinKwon · 2020-11-19T04:20:29Z

dev/.rat-excludes

+.*\.res
+flights_tiny.txt.1
+over1k
+over10k


I see. I checked these are .txt

HyukjinKwon · 2020-11-19T04:20:34Z

dev/.rat-excludes

+flights_tiny.txt.1
+over1k
+over10k
+exported_table/*


SparkQA · 2020-11-19T04:33:07Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35916/

SparkQA · 2020-11-19T04:51:18Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35920/

ScrapCodes · 2020-11-19T04:53:01Z

@dongjoon-hyun Thanks for fixing this. My bad, this bug have existed for years.

SparkQA · 2020-11-19T05:23:04Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35920/

SparkQA · 2020-11-19T05:33:02Z

Test build #131314 has finished for PR 30418 at commit b7aa6f9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-11-19T06:18:29Z

retest this please

SparkQA · 2020-11-19T06:49:31Z

Test build #131316 has finished for PR 30418 at commit 45658f0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-19T07:16:11Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35927/

SparkQA · 2020-11-19T07:43:01Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35927/

dongjoon-hyun · 2020-11-19T07:58:43Z

Thank you, @HyukjinKwon , @viirya , @ScrapCodes , @sunchao .
Merged to master.

SparkQA · 2020-11-19T08:05:02Z

Test build #131323 has finished for PR 30418 at commit 45658f0.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

[MINOR][INFRA] Fix rat exclusion not to exclude catalog

1d35ad4

dongjoon-hyun mentioned this pull request Nov 19, 2020

[SPARK-31255][SQL] Add SupportsMetadataColumns to DSv2 #28027

Closed

viirya approved these changes Nov 19, 2020

View reviewed changes

viirya reviewed Nov 19, 2020

View reviewed changes

HyukjinKwon approved these changes Nov 19, 2020

View reviewed changes

Fix more

45658f0

github-actions bot added the SQL label Nov 19, 2020

dongjoon-hyun changed the title ~~[MINOR][INFRA] Fix rat exclusion not to exclude catalog~~ [MINOR][INFRA][TESTS] Fix rat exclusion not to exclude catalog Nov 19, 2020

dongjoon-hyun commented Nov 19, 2020

View reviewed changes

dongjoon-hyun changed the title ~~[MINOR][INFRA][TESTS] Fix rat exclusion not to exclude catalog~~ [SPARK-33483][INFRA][TESTS] Fix rat exclusion not to exclude catalog Nov 19, 2020

dongjoon-hyun changed the title ~~[SPARK-33483][INFRA][TESTS] Fix rat exclusion not to exclude catalog~~ [SPARK-33483][INFRA][TESTS] Fix rat exclusion patterns and add a LICENSE Nov 19, 2020

dongjoon-hyun commented Nov 19, 2020

View reviewed changes

dev/.rat-excludes

.*\.avsc

.*\.txt

.*\.json

.*\.data

Copy link

Member Author

dongjoon-hyun Nov 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hides the directory data and metadata.

dongjoon-hyun commented Nov 19, 2020

View reviewed changes

HyukjinKwon reviewed Nov 19, 2020

View reviewed changes

dev/.rat-excludes

.*\.res

flights_tiny.txt.1

over1k

over10k

Copy link

Member

HyukjinKwon Nov 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I checked these are .txt

HyukjinKwon reviewed Nov 19, 2020

View reviewed changes

dev/.rat-excludes

flights_tiny.txt.1

over1k

over10k

exported_table/*

Copy link

Member

HyukjinKwon Nov 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is .q

HyukjinKwon approved these changes Nov 19, 2020

View reviewed changes

viirya approved these changes Nov 19, 2020

View reviewed changes

ScrapCodes approved these changes Nov 19, 2020

View reviewed changes

dongjoon-hyun closed this in d5e7bd0 Nov 19, 2020

[SPARK-33483][INFRA][TESTS] Fix rat exclusion patterns and add a LICENSE #30418

[SPARK-33483][INFRA][TESTS] Fix rat exclusion patterns and add a LICENSE #30418

Uh oh!

Conversation

dongjoon-hyun commented Nov 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun commented Nov 19, 2020

Uh oh!

dongjoon-hyun commented Nov 19, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sunchao commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

dongjoon-hyun commented Nov 19, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

ScrapCodes commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

HyukjinKwon commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

dongjoon-hyun commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

dongjoon-hyun commented Nov 19, 2020 •

edited

Loading