[RFC] [Licensing] Address ASF Feedback for 1.9.0.rc7 release

[josephevans]

Problem statement

Based on the feedback from ASF on our vote to release MXNet v1.9.0.rc7, we still have a few licensing issues that caused a -1 vote, which prevents us from moving forward with the release.

Based on the RC7 voting thread, the following licensing issues are still outstanding:

• Files incorrectly have ASF headers. These files are all copyrighted by NumPy developers and released under the 3-clause BSD license. We should not have added the ASF header and kept the original license headers intact, and mention this by including it in the "3-clause BSD license" section in LICENSE.

  • src/operator/numpy/np_einsum_op.cc
  • src/operator/numpy/np_einsum_op-inl.h
  • src/operator/numpy/np_einsum_path_op-inl.h

• Some files are not correctly listed in LICENSE. For example, these files contain a "Copyright Microsoft" line but are released under ASF-2 or MIT license. These need to be properly mentioned in LICENSE:

  • src/operator/contrib/deformable_psroi_pooling.cu
  • src/operator/contrib/deformable_psroi_pooling-inl.h
  • src/operator/contrib/deformable_psroi_pooling.cc
  • src/operator/contrib/multi_proposal-inl.h
  • src/operator/contrib/multi_proposal.cc
  • src/operator/contrib/multi_proposal.cu
  • src/operator/contrib/psroi_pooling.cc
  • src/operator/contrib/psroi_pooling.cu
  • Additional files in the same situation as above (identified by running skywalking-eyes):
  • src/operator/contrib/psroi_pooling-inl.h
  • src/operator/contrib/deformable_convolution-inl.h
  • src/operator/contrib/deformable_convolution.cc
  • src/operator/contrib/deformable_convolution.cu

As ASF noted, not all files were checked and there could be others that are missed. In order to better identify licensing issues in the future, we should extend the use of automated tools (such as skywalking-eyes).

• Update our CI pipelines to utilize the latest Apache tool for license checks (skywalking-eyes)
• Update our automated tools to detect files that have multiple license headers
• Create/extend current tool(s) to ensure files are correctly listed in the correct section in LICENSE #20723
• Properly document 3rd-party files/symlinks in include directory in LICENSE

References

• RC7 Voting Thread: https://lists.apache.org/thread.html/r5789f5d93716d9d710b855775e71e4892fd37e41338c70c0855958ad%40%3Cgeneral.incubator.apache.org%3E

[josephevans]

I've created a PR (#20598) to address the first three items and the last item (and checked as complete.) Please review with any comments/suggestions/concerns. This PR also adds a Github workflow (thanks to @barry-jin for help) to use Apache's latest license checking tool, skywalking-eyes.

[josephevans]

I manually checked all source files in src/operator to ensure they are either:

1. Licensed under ASF-2.0 license with standard recognizable header. If they do not contain the standard header, they should be listed under the ASF-2.0 section in LICENSE
2. Licensed under some other compatible license, but clearly mentioned in LICENSE under the compatible license section (ie BSD 2/3-clause, MIT, etc.)

Any help manually auditing the rest of the source tree would be greatly appreciated.

[barry-jin]

I have manually checked all the header files in include/onednn(mkldnn), include/dlpack, include/dmlc, include/mshadow and found one more issue:

• include/dmlc/concurrentqueue.h and include/dmlc/blockingconcurrentqueue.h are under the Simplified BSD license (2-clause BSD license), which are not correctly listed in LICENSE.

[josephevans]

I'm currently having issues getting #20626 to pass CI because of MKLDNN-related tests - opened an issue (#20643) to track.

[josephevans]

I looked into the license_header.py script, and it actually will detect when a file has multiple licenses and throw an error. We may need to expand the list of strings it recognizes licenses with here:

https://github.com/apache/incubator-mxnet/blob/v1.9.x/tools/license_header.py#L62-L63

[josephevans]

At this point, we have manually reviewed all the source files in src to ensure they are properly ASF-2.0 licensed, or licensed under a difference license and mentioned in LICENSE. We also added another CI pipeline to run the skywalking-eyes license checker and identified a few minor license typos (such as extra characters accidentally added to license headers.)

Since we don't believe there are any additional outstanding license issues, I think we should move forward with the v1.9.0 release. Does anyone have comments or issues?

[szha]

Thanks Joe and Zhenghui and others who helped on the efforts in thoroughly addressing the license issues. Also thanks to @kezhenxu94 for sharing the great license checker tool from the skywalking community. I have a question from reviewing the LICENSE file from the master branch:

• For the files under "Caffe Licensing Model", I checked the first file in that section and found the license to be under BSD-2 with Caffe's own copyright model. Since "Caffe Licensing Model" isn't listed in the Apache's license categories, should we call it "BSD-2 with Caffe Copyright Notice and Disclaimer" instead?
• I saw that we still have several sections in LICENSE where they have dual licenses (e.g. files under "Apache-2.0 license + 3-clause BSD license" are adapted from ONNX and NumPy code). From the past votes we know we want to avoid them. What's preventing us from just using original licenses which are permissible? For the numpy files, they appear to be the first item in the checklist of this issue, and their content was updated. Why then are we still listing them in LICENSE under two licenses?

[josephevans]

Thanks for the feedback @szha. I addressed the two issues you list above in #20709. I also audited the LICENSE file in v1.x branch and ensured everything listed is in the proper section.

After reading through https://infra.apache.org/licensing-howto.html, I also wanted to ensure that we comply to licensing terms for binary distributions - we need to include any copyright clauses and provide the associated license text in any binary distribution of MXNet. In order to achieve this, I created a number of LICENSE.*.txt files that includes the required text. These files should automatically get included in pip wheels (see https://github.com/apache/incubator-mxnet/blob/v1.x/tools/pip/setup.py#L94-L95) to fulfill terms. This is how the TVM project includes license text, although MXNet has many more files under different licenses.

I would appreciate a review or comments on this approach.

[josephevans]

Created a PR for the v1.9.x branch: #20722

[josephevans]

Once #20722 is merged, I would like to close this issue and move forward with 1.9.0.rc8. I've converted the single outstanding task of updating the license tool to a new item for tracking (#20723).

[szha]

Thanks Joe & community for continuing to push this along. Given that this release process has drawn on quite a while, would it be possible to pull in PR #20137 [1]? This goes along with #20107 [2] that was part of 1.9.0.rc7 [3] for deconvolution support. I would be happy to do the backporting. Thanks! Sam [1] #20137 [2] #20107 [3] https://github.com/apache/incubator-mxnet/releases/tag/1.9.0.rc7 On 11/2/21, 7:12 PM, "Joe Evans" ***@***.***> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Once #20722 is merged, I would like to close this issue and move forward with 1.9.0.rc8. I've converted the single outstanding task of updating the license tool to a new item for tracking (#20723).

…

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #20616 (comment)

[josephevans]

Merged #20726 into v1.9.x, we should be ready for another release candidate. Closing this issue.