Update maven detections #3447

AyanSinhaMahapatra · 2023-07-05T14:30:54Z

Update maven package and license detections:

Fix MANIFEST.MF parsing
Add package assembly for both pom.xml and MANIFEST.MF
implement functionality to do package specific license detection
run license detection on whole extracted license statement for maven
Add rules for license detection fixes
Fix detection for pom.properties file
Fix --todo plugin for package detection
Fix false positive case for bare word rules
Fix issue of unknown license detection in package manifest

Tasks

Reviewed contribution guidelines
PR is descriptively titled 📑 and links the original issue above 🔗
Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
Run tests locally to check for errors.
Commits are in uniquely-named feature branch and has no merge conflicts 📁
Looked for possible updates in documentation and added updates if applicable
Updated CHANGELOG.rst

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Add a new config variable for datafile handlers, which if enabled will run a package-ecosystem specific implementation of license detection. Added a function to implement this for maven which only keeps the relevant information and runs license detection on the whole license statement and not on the respective values/attributes one-by-one. Enabled this for maven to improve license detection there. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

src/packagedcode/maven.py

Adds a fix for maven packacge assembly, combining MANIFEST.MF and pom.xml files into a single package, instead of multiple top level packages. Also fixes pom.properties file parsing. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

JonoYang

@AyanSinhaMahapatra Looks good, I just highlighted some small issues.

src/packagedcode/jar_manifest.py

tests/summarycode/data/classify/with_package_data.expected.json

pombredanne

Please see some nits for your consideration

src/licensedcode/data/rules/bsd-3-clause-no-nuclear-warranty_and_jj2000_2.RULE

src/licensedcode/data/rules/json_17.RULE

src/licensedcode/data/rules/lgpl-2.1_and_apache-2.0_1.RULE

src/licensedcode/data/rules/lgpl-2.1_and_apache-2.0_2.RULE

src/licensedcode/detection.py

pombredanne · 2023-07-12T21:26:28Z

src/packagedcode/jar_manifest.py

+    if comment or doc_url:
+        package["extra_data"] = {}
+        if comment:
+            package["extra_data"]['notes'] = comment


we need to find a better way than using extra_data.

See #3447 (comment). We might want to add fields to the PackageData class then. And we might handle this separately, AFAIK this data is not really used anywhere and this is just a fix to make sure MANIFEST parsing returns everything else, and not crash.

pombredanne · 2023-07-12T21:26:55Z

src/packagedcode/jar_manifest.py

@@ -278,7 +248,7 @@ def dget(s):
    package['homepage_url'] = dget('Implementation-URL') or dget('Implementation-Url')

    # Bundle-DocURL: http://logging.apache.org/log4j/1.2
-    package['documentation_url'] = dget('Bundle-DocURL')
+    doc_url = dget('Bundle-DocURL')


I get what you are doing, but we are losing data here.

We are returning this package as-is to be used for creating a PackageData class, see https://github.com/nexB/scancode-toolkit/blob/update-maven-detections/src/packagedcode/maven.py#L138 with models.PackageData(**manifest,). We don't have a documentation_url or a notes field in PackageData, and this is why the Manifest parsing used to fail and not return anything. I'm just ensuring we return whatever we can, instead of crashing and not returning anything. And since I'm having this in extra_data, we're not really losing any data IMHO

pombredanne · 2023-07-12T21:28:24Z

src/packagedcode/jar_manifest.py

    return package


+def parse_scm_connection(scm_connection):


@TG1999 ping ... is this related or relevant to your purldb work?

pombredanne · 2023-07-12T21:29:16Z

src/packagedcode/jar_manifest.py

+    if package_type in ['maven', 'jar']:
+        return JavaJarManifestHandler.datasource_id
+    elif package_type == 'osgi':
+        return JavaOSGiManifestHandler.datasource_id


What if a maven package also has OSGi thing?s

We can improve the parsing and handle more cases allright, but let's do this separately through another issue, and if you have an example of that case please provide the same. But this was also a fix, and I'm only making sure this works without crashing here.

src/packagedcode/licensing.py

AyanSinhaMahapatra · 2023-07-13T09:35:52Z

@pombredanne @JonoYang addressed all your comments, thanks!

src/licensedcode/data/rules/bsd-3-clause-no-nuclear-warranty_12.RULE

src/licensedcode/data/rules/jj2000_1.RULE

pombredanne

LGTM with a few extra nitrs that you can fix or not as you like before merging. Thanks!

src/packagedcode/licensing.py

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

* Add rules for license detection fixes * Fix detection for pom.properties file * Fix --todo plugin for package detection * Fix false positive case for bare word rules * Fix issue of unknown license detection in package manifest Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Adds additional license rules to take care of maven package license detection adjustments. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra force-pushed the update-maven-detections branch from bc8c56d to 7f96760 Compare July 6, 2023 13:40

AyanSinhaMahapatra added 2 commits July 6, 2023 19:17

Fix parsing of MANIFEST.MF files

d5e811b

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra force-pushed the update-maven-detections branch from 7f96760 to ac04984 Compare July 6, 2023 13:48

AyanSinhaMahapatra requested review from pombredanne and JonoYang July 7, 2023 08:17

pombredanne reviewed Jul 7, 2023

View reviewed changes

src/packagedcode/maven.py Outdated Show resolved Hide resolved

Fix maven package assembly

f117a76

Adds a fix for maven packacge assembly, combining MANIFEST.MF and pom.xml files into a single package, instead of multiple top level packages. Also fixes pom.properties file parsing. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra requested a review from pombredanne July 11, 2023 15:21

AyanSinhaMahapatra force-pushed the update-maven-detections branch from c23072c to 4456975 Compare July 11, 2023 15:48

AyanSinhaMahapatra mentioned this pull request Jul 11, 2023

Update scancode-toolkit to v32.0.6 aboutcode-org/purldb#142

Merged

JonoYang requested changes Jul 11, 2023

View reviewed changes

src/packagedcode/jar_manifest.py Outdated Show resolved Hide resolved

src/packagedcode/jar_manifest.py Outdated Show resolved Hide resolved

src/packagedcode/jar_manifest.py Outdated Show resolved Hide resolved

AyanSinhaMahapatra force-pushed the update-maven-detections branch 2 times, most recently from 503f7d8 to 0165bdf Compare July 12, 2023 11:14

AyanSinhaMahapatra requested a review from JonoYang July 12, 2023 11:14

AyanSinhaMahapatra commented Jul 12, 2023

View reviewed changes

tests/summarycode/data/classify/with_package_data.expected.json Outdated Show resolved Hide resolved

AyanSinhaMahapatra force-pushed the update-maven-detections branch from dd51a69 to 76ef47f Compare July 12, 2023 16:12

pombredanne requested changes Jul 12, 2023

View reviewed changes

AyanSinhaMahapatra force-pushed the update-maven-detections branch from 76ef47f to 8d8cbe3 Compare July 13, 2023 09:25

pombredanne reviewed Jul 13, 2023

View reviewed changes

src/licensedcode/data/rules/bsd-3-clause-no-nuclear-warranty_12.RULE Show resolved Hide resolved

pombredanne reviewed Jul 13, 2023

View reviewed changes

src/licensedcode/data/rules/jj2000_1.RULE Show resolved Hide resolved

pombredanne approved these changes Jul 13, 2023

View reviewed changes

src/packagedcode/licensing.py Outdated Show resolved Hide resolved

src/packagedcode/licensing.py Outdated Show resolved Hide resolved

AyanSinhaMahapatra added 3 commits July 13, 2023 20:12

Refactor and address feedback

1af074f

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Add more license rules

e3a1c90

Adds additional license rules to take care of maven package license detection adjustments. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra force-pushed the update-maven-detections branch from dc8f545 to e3a1c90 Compare July 13, 2023 14:44

AyanSinhaMahapatra merged commit f55754a into develop Jul 13, 2023

AyanSinhaMahapatra deleted the update-maven-detections branch July 13, 2023 15:16

AyanSinhaMahapatra mentioned this pull request Aug 1, 2023

Improve "scan_package" or "scan_codebase" pipelines to improve the analysis of a single Java JAR aboutcode-org/scancode.io#652

Open

AyanSinhaMahapatra mentioned this pull request Oct 19, 2023

SCTK: Validate that we can extract all valuable information from a JAR manifest #3333

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update maven detections #3447

Update maven detections #3447

AyanSinhaMahapatra commented Jul 5, 2023 •

edited

Loading

JonoYang left a comment

pombredanne left a comment

pombredanne Jul 12, 2023

AyanSinhaMahapatra Jul 13, 2023

pombredanne Jul 12, 2023

AyanSinhaMahapatra Jul 13, 2023

pombredanne Jul 12, 2023

pombredanne Jul 12, 2023

AyanSinhaMahapatra Jul 13, 2023

AyanSinhaMahapatra commented Jul 13, 2023

pombredanne left a comment

Update maven detections #3447

Update maven detections #3447

Conversation

AyanSinhaMahapatra commented Jul 5, 2023 • edited Loading

Tasks

JonoYang left a comment

Choose a reason for hiding this comment

pombredanne left a comment

Choose a reason for hiding this comment

pombredanne Jul 12, 2023

Choose a reason for hiding this comment

AyanSinhaMahapatra Jul 13, 2023

Choose a reason for hiding this comment

pombredanne Jul 12, 2023

Choose a reason for hiding this comment

AyanSinhaMahapatra Jul 13, 2023

Choose a reason for hiding this comment

pombredanne Jul 12, 2023

Choose a reason for hiding this comment

pombredanne Jul 12, 2023

Choose a reason for hiding this comment

AyanSinhaMahapatra Jul 13, 2023

Choose a reason for hiding this comment

AyanSinhaMahapatra commented Jul 13, 2023

pombredanne left a comment

Choose a reason for hiding this comment

AyanSinhaMahapatra commented Jul 5, 2023 •

edited

Loading