Skip to content

Conversation

@taklwu
Copy link
Contributor

@taklwu taklwu commented Feb 16, 2022

  • fix NOTICE.aggregate generation

please see the other discussion on #41 , and I think #41 should be closed and continue the discussion here.

- fix NOTICE.aggregate generation
@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 2m 35s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-0 ⚠️ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+0 🆗 mvndep 0m 24s Maven dependency ordering for branch
+1 💚 mvninstall 1m 48s master passed
+1 💚 compile 1m 1s master passed
+1 💚 javadoc 1m 6s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 7s Maven dependency ordering for patch
+1 💚 mvninstall 0m 59s the patch passed
+1 💚 compile 1m 1s the patch passed
+1 💚 javac 1m 1s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 2s The patch has no ill-formed XML file.
+1 💚 javadoc 0m 55s the patch passed
_ Other Tests _
+1 💚 unit 9m 45s root in the patch passed.
20m 49s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/1/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR #93
Optional Tests dupname javac javadoc unit xml compile
uname Linux 9084a66a50d4 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 GNU/Linux
Build tool hb_maven
Personality dev-support/jenkins/hbase-personality.sh
git revision master / 2bfc5f1
Default Java Oracle Corporation-1.8.0_282-b08
Test Results https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/1/testReport/
Max. process+thread count 916 (vs. ulimit of 12500)
modules C: hbase-connectors-assembly . U: .
Console output https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/1/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@busbey busbey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have examples of the generated LICENSE/NOTIC files after this change?

I'll build locally in the mean time to get them.

pom.xml Outdated
<java.min.version>${compileSource}</java.min.version>
<!-- override on command line to have generated LICENSE files include
diagnostic info for verifying notice requirements -->
<build.year>2022</build.year>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we set this already via the buildnumber-maven-plugin. is that not working?

Copy link
Contributor Author

@taklwu taklwu Feb 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

during my test, I don't know why it's not working, here is the output https://issues.apache.org/jira/secure/attachment/13040127/NOTICE.aggregate-no-build-year (I did check the hbase code, seems very similar just not working)

Apache HBase Connectors - Assembly
Copyright 2018-${copyright-end-year} The Apache Software Foundation

let me continue to check it.

Copy link
Contributor Author

@taklwu taklwu Feb 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't the reason -.- although I tried to use -Dbuild.year=2022 and it worked.... I will revert the change for this build.year first. let me know if you find any thing. my goal is to have the dependency included in the NOTICE file correctly.

In addition, I'm wondered if we need to add a generated NOTICE for modules of spark e.g. hbase-spark?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition, I'm wondered if we need to add a generated NOTICE for modules of spark e.g. hbase-spark

every artifact we create needs to have a proper LICENSE and NOTICE file. for the most part that should happen automatically for us unless we incorporate third party works in either the source or binary of an artifact.

sorry for the lag, let me look again at what the current artifacts look like with this change in place.

@taklwu taklwu requested a review from busbey February 17, 2022 02:29
@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 41s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-0 ⚠️ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+0 🆗 mvndep 0m 25s Maven dependency ordering for branch
+1 💚 mvninstall 2m 0s master passed
+1 💚 compile 1m 18s master passed
+1 💚 javadoc 1m 19s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 9s Maven dependency ordering for patch
+1 💚 mvninstall 1m 6s the patch passed
+1 💚 compile 1m 20s the patch passed
+1 💚 javac 1m 20s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 2s The patch has no ill-formed XML file.
+1 💚 javadoc 1m 23s the patch passed
_ Other Tests _
+1 💚 unit 9m 33s root in the patch passed.
21m 31s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/2/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR #93
Optional Tests dupname javac javadoc unit xml compile
uname Linux 645239bb5a3d 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 GNU/Linux
Build tool hb_maven
Personality dev-support/jenkins/hbase-personality.sh
git revision master / 2bfc5f1
Default Java Oracle Corporation-1.8.0_282-b08
Test Results https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/2/testReport/
Max. process+thread count 912 (vs. ulimit of 12500)
modules C: hbase-connectors-assembly . U: .
Console output https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/2/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 38s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-0 ⚠️ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+0 🆗 mvndep 0m 25s Maven dependency ordering for branch
+1 💚 mvninstall 1m 42s master passed
+1 💚 compile 1m 0s master passed
+1 💚 javadoc 0m 52s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 5s Maven dependency ordering for patch
+1 💚 mvninstall 0m 48s the patch passed
+1 💚 compile 0m 53s the patch passed
+1 💚 javac 0m 53s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 1s The patch has no ill-formed XML file.
+1 💚 javadoc 0m 49s the patch passed
_ Other Tests _
+1 💚 unit 8m 35s root in the patch passed.
17m 43s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/3/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR #93
Optional Tests dupname javac javadoc unit xml compile
uname Linux 49fc9edceba8 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 GNU/Linux
Build tool hb_maven
Personality dev-support/jenkins/hbase-personality.sh
git revision master / 2bfc5f1
Default Java Oracle Corporation-1.8.0_282-b08
Test Results https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/3/testReport/
Max. process+thread count 920 (vs. ulimit of 12500)
modules C: hbase-connectors-assembly . U: .
Console output https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/3/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@busbey
Copy link
Contributor

busbey commented Feb 23, 2022

Here's what I see when verifying things.

Starting with the branch for this PR:

% git clean -xdf
...
% mvn package verify
...
% cd hbase-connectors-assembly/target
% tar xzf hbase-connectors-1.0.1-SNAPSHOT-bin.tar.gz
% ls hbase-connectors-1.0.1-SNAPSHOT/lib | grep -v -E "^hbase-.*\.jar" | wc -l # confirm there are third party jars
      68
% cat hbase-connectors-1.0.1-SNAPSHOT/NOTICE.txt 
Apache HBase - Connectors
Copyright 2019 The Apache Software Foundation
% vim hbase-connectors-1.0.1-SNAPSHOT/LICENSE.txt  # it's just the ALv2 license text

Copy link
Contributor

@busbey busbey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this currently solves the issue with the generated assembly tarball

@taklwu
Copy link
Contributor Author

taklwu commented Feb 24, 2022

sorry, I'm learning toward this JIRA as well, what you mean here is that we need a NOTICE.txt/NOTICE after unzipping the tar ball.

basically, I knew that the NOTICE file were generated at target/NOTICE.aggregate that was not correctly generated before this change. Let me come back on it and see how I can rename or move this target/NOTICE.aggregate to NOTICE.

but for the LICENSE file, I may need another look.

@taklwu
Copy link
Contributor Author

taklwu commented Feb 24, 2022

yup, you're right, I missed that file copy in the assembly tar, I checked it but please double check as well

% pwd
/Users/taklwu/workspace/hbase-connectors/hbase-connectors-assembly/target/hbase-connectors-1.0.1-SNAPSHOT

% ls -l
total 224
-rw-r--r--   1 taklwu  staff  12119 Jan 22  2020 CHANGELOG.md
-rw-r--r--   1 taklwu  staff      0 Jan 22  2020 LEGAL
-rw-r--r--   1 taklwu  staff  67420 Jan 22  2020 LICENSE.txt
-rw-r--r--   1 taklwu  staff  19709 Jan 22  2020 NOTICE.txt
-rw-r--r--   1 taklwu  staff   1006 Jan 22  2020 README.md
-rw-r--r--   1 taklwu  staff   6845 Jan 22  2020 RELEASENOTES.md
drwxr-xr-x   5 taklwu  staff    160 Feb 24 14:28 bin
drwxr-xr-x   3 taklwu  staff     96 Jan 22  2020 conf
drwxr-xr-x  89 taklwu  staff   2848 Feb 24 14:28 lib

% diff NOTICE.txt ../NOTICE.aggregate # no difference 

@taklwu taklwu requested a review from busbey February 24, 2022 22:32
@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 2m 1s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-0 ⚠️ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+0 🆗 mvndep 0m 24s Maven dependency ordering for branch
+1 💚 mvninstall 1m 23s master passed
+1 💚 compile 0m 52s master passed
+1 💚 javadoc 0m 51s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 6s Maven dependency ordering for patch
+1 💚 mvninstall 0m 48s the patch passed
+1 💚 compile 0m 52s the patch passed
+1 💚 javac 0m 52s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 2s The patch has no ill-formed XML file.
+1 💚 javadoc 0m 50s the patch passed
_ Other Tests _
+1 💚 unit 8m 33s root in the patch passed.
17m 34s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/4/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR #93
Optional Tests dupname javac javadoc unit xml compile
uname Linux 5c6f28987365 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 GNU/Linux
Build tool hb_maven
Personality dev-support/jenkins/hbase-personality.sh
git revision master / 2bfc5f1
Default Java Oracle Corporation-1.8.0_282-b08
Test Results https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/4/testReport/
Max. process+thread count 920 (vs. ulimit of 12500)
modules C: hbase-connectors-assembly . U: .
Console output https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/4/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@busbey
Copy link
Contributor

busbey commented Feb 26, 2022

we're missing the part of the main project tooling that checks for problems with the generated license:

(base) sbusbey@seans-mbp hbase-connectors-1.0.1-SNAPSHOT % grep -B 2 "^ERROR:" LICENSE.txt
This product includes Streaming API for XML licensed under the GNU General Public Library.

ERROR: Please check ^^^^^^^^^^^^ this License for acceptability here:

stax-api is dual licensed GPL and CDDL. We'll also need to specify that we're picking CDDL.

The generated NOTICE is missing the build year:

(base) sbusbey@seans-mbp hbase-connectors-1.0.1-SNAPSHOT % head -n 3 NOTICE.txt 

Apache HBase Connectors - Assembly
Copyright 2018-${copyright-end-year} The Apache Software Foundation

It looks like this is because we need to activate the buildnumber-maven-plugin in the hbase-connectors-assembly module. we only define the version/configs in plugin management on the top level pom; we never actually tell the plugin to run.

@taklwu
Copy link
Contributor Author

taklwu commented Mar 1, 2022

for 2/ about the copyright-end-year has been fixed, but for 1/ of LICENSE.txt, this package does not have a supplemental-models.xml and it's inherited from hbase-resource-bundle.

> main project tooling that checks for problems with the generated license

or what does generated license mean here? (I'm checking how the LICENSE file is being generated at this point)

I'm wondered if you have any suggestion here, my current thought is to copy the supplemental-models.xml from HBase repo, but I'm checking with you at the same time before I find the solution.

nvm, I found the solution....

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 39s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-0 ⚠️ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+0 🆗 mvndep 0m 27s Maven dependency ordering for branch
+1 💚 mvninstall 1m 38s master passed
+1 💚 compile 1m 1s master passed
+1 💚 javadoc 1m 7s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 6s Maven dependency ordering for patch
+1 💚 mvninstall 1m 2s the patch passed
+1 💚 compile 1m 9s the patch passed
+1 💚 javac 1m 9s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 5s The patch has no ill-formed XML file.
+1 💚 javadoc 0m 58s the patch passed
_ Other Tests _
+1 💚 unit 8m 58s root in the patch passed.
19m 15s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/5/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR #93
Optional Tests dupname javac javadoc unit xml compile
uname Linux 2f401baf0e5e 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 GNU/Linux
Build tool hb_maven
Personality dev-support/jenkins/hbase-personality.sh
git revision master / 2bfc5f1
Default Java Oracle Corporation-1.8.0_282-b08
Test Results https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/5/testReport/
Max. process+thread count 916 (vs. ulimit of 12500)
modules C: hbase-connectors-assembly . U: .
Console output https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/5/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@taklwu
Copy link
Contributor Author

taklwu commented Mar 1, 2022

@busbey I was out last weekend, sorry for the delay, but can you help to check again, I should have fixed what you commented.

taklwu@MacBook-Pro hbase-connectors-1.0.1-SNAPSHOT % grep -B 2 "^ERROR:" LICENSE.txt
taklwu@MacBook-Pro hbase-connectors-1.0.1-SNAPSHOT % head -n 3 NOTICE.txt

Apache HBase Connectors - Assembly
Copyright 2018-2022 The Apache Software Foundation

Copy link
Contributor

@busbey busbey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The generated files look correct. But we still need to make sure the build will fail if we add some new dependency that is under a category-x license.

this enforcer check is what does it in the main project: https://github.com/apache/hbase/blob/rel/3.0.0-alpha-2/pom.xml#L1316

@taklwu
Copy link
Contributor Author

taklwu commented Mar 2, 2022

add the goal of check-aggregate-license, and if we remove the local supplemental-models.xml , failed as expected.

[INFO] --- maven-enforcer-plugin:3.0.0:enforce (check-aggregate-license) @ hbase-connectors-assembly ---
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.EvaluateBeanshell failed with message:
License errors detected, for more detail find ERROR in
                    /Users/taklwu/workspace/hbase-connectors/hbase-connectors-assembly/target/maven-shared-archive-resources/META-INF/LICENSE
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  6.543 s
[INFO] Finished at: 2022-03-01T18:11:45-08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0:enforce (check-aggregate-license) on project hbase-connectors-assembly: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 14s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-0 ⚠️ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+0 🆗 mvndep 0m 21s Maven dependency ordering for branch
+1 💚 mvninstall 1m 22s master passed
+1 💚 compile 0m 50s master passed
+1 💚 javadoc 0m 50s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 6s Maven dependency ordering for patch
+1 💚 mvninstall 0m 50s the patch passed
+1 💚 compile 0m 52s the patch passed
+1 💚 javac 0m 52s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 2s The patch has no ill-formed XML file.
+1 💚 javadoc 0m 48s the patch passed
_ Other Tests _
+1 💚 unit 8m 36s root in the patch passed.
16m 44s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/6/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR #93
Optional Tests dupname javac javadoc unit xml compile
uname Linux 68400d687a37 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 GNU/Linux
Build tool hb_maven
Personality dev-support/jenkins/hbase-personality.sh
git revision master / 2bfc5f1
Default Java Oracle Corporation-1.8.0_282-b08
Test Results https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/6/testReport/
Max. process+thread count 917 (vs. ulimit of 12500)
modules C: hbase-connectors-assembly . U: .
Console output https://ci-hbase.apache.org/job/HBase-Connectors-PreCommit/job/PR-93/6/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@taklwu taklwu requested a review from busbey March 2, 2022 06:42
@taklwu
Copy link
Contributor Author

taklwu commented Mar 7, 2022

checking again, @busbey do you see if there is any other problem ?

@taklwu taklwu merged commit 036729d into apache:master Mar 7, 2022
@taklwu taklwu deleted the HBASE-22338 branch March 7, 2022 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants