Add support for SPDX 2.3 #164

kzantow · 2022-10-06T23:24:35Z

This PR adds support for SPDX 2.3 with the following changes:

SPDX 2.3 data model
Add SPDX 2.3 variants for utility functions
Add support for all formats to export and import SPDX 2.3

Fixes: #156

NOTE: this PR also includes a change to fix the references to the SPDX spec in the v2_2 model. This can be removed, but I've done this so it's possible to run a diff between the models and see a more meaningful set of changes, e.g. run:

diff v2_2 v2_3 --ignore-matching-lines='package v2_.'

Results

diff --ignore-matching-lines='package v2_.' v2_2/annotation.go v2_3/annotation.go
7c7
< // Annotation is an Annotation section of an SPDX Document for version 2.2 of the spec.
---
> // Annotation is an Annotation section of an SPDX Document for version 2.3 of the spec.
24c24
<       AnnotationSPDXIdentifier common.DocElementID `json:"-"`
---
>       AnnotationSPDXIdentifier common.DocElementID `json:"-" yaml:"-"`
diff --ignore-matching-lines='package v2_.' v2_2/creation_info.go v2_3/creation_info.go
8c8
< // SPDX Document for version 2.2 of the spec.
---
> // SPDX Document for version 2.3 of the spec.
diff --ignore-matching-lines='package v2_.' v2_2/document.go v2_3/document.go
9c9
< // as defined in section 6.6 for version 2.2 of the spec.
---
> // as defined in section 6.6 for version 2.3 of the spec.
23,24c23,24
< // Document is an SPDX Document for version 2.2 of the spec.
< // See https://spdx.github.io/spdx-spec/v2-draft/ (DRAFT)
---
> // Document is an SPDX Document for version 2.3 of the spec.
> // See https://spdx.github.io/spdx-spec/v2.3/document-creation-information
26c26
<       // 6.1: SPDX Version; should be in the format "SPDX-2.2"
---
>       // 6.1: SPDX Version; should be in the format "SPDX-2.3"
56,61c56,61
<       Packages      []*Package      `json:"packages"`
<       Files         []*File         `json:"files"`
<       OtherLicenses []*OtherLicense `json:"hasExtractedLicensingInfos"`
<       Relationships []*Relationship `json:"relationships"`
<       Annotations   []*Annotation   `json:"annotations"`
<       Snippets      []Snippet       `json:"snippets"`
---
>       Packages      []*Package      `json:"packages,omitempty"`
>       Files         []*File         `json:"files,omitempty"`
>       OtherLicenses []*OtherLicense `json:"hasExtractedLicensingInfos,omitempty"`
>       Relationships []*Relationship `json:"relationships,omitempty"`
>       Annotations   []*Annotation   `json:"annotations,omitempty"`
>       Snippets      []Snippet       `json:"snippets,omitempty"`
64c64
<       Reviews []*Review
---
>       Reviews []*Review `json:"-" yaml:"-"`
diff --ignore-matching-lines='package v2_.' v2_2/file.go v2_3/file.go
7c7
< // File is a File section of an SPDX Document for version 2.2 of the spec.
---
> // File is a File section of an SPDX Document for version 2.3 of the spec.
21c21
<       // 8.4: File Checksum: may have keys for SHA1, SHA256 and/or MD5
---
>       // 8.4: File Checksum: may have keys for SHA1, SHA256, MD5, SHA3-256, SHA3-384, SHA3-512, BLAKE2b-256, BLAKE2b-384, BLAKE2b-512, BLAKE3, ADLER32
26,27c26,27
<       // Cardinality: mandatory, one
<       LicenseConcluded string `json:"licenseConcluded"`
---
>       // Cardinality: optional, one
>       LicenseConcluded string `json:"licenseConcluded,omitempty"`
30,31c30,31
<       // Cardinality: mandatory, one or many
<       LicenseInfoInFiles []string `json:"licenseInfoInFiles"`
---
>       // Cardinality: optional, one or many
>       LicenseInfoInFiles []string `json:"licenseInfoInFiles,omitempty"`
44c44
<       ArtifactOfProjects []*ArtifactOfProject `json:"-"`
---
>       ArtifactOfProjects []*ArtifactOfProject `json:"artifactOfs,omitempty"`
65c65
<       FileDependencies []string `json:"-"`
---
>       FileDependencies []string `json:"fileDependencies,omitempty"`
71c71
<       Snippets map[common.ElementID]*Snippet `json:"-"`
---
>       Snippets map[common.ElementID]*Snippet `json:"-" yaml:"-"`
77c77,79
< // a Package, as defined in sections 8.9-8.11 in version 2.2 of the spec.
---
> // a Package, as defined in sections 8.9-8.11 in version 2.3 of the spec.
> // NOTE: the JSON schema does not define the structure of this object:
> // https://github.com/spdx/spdx-spec/blob/development/v2.3.1/schemas/spdx-schema.json#L480
83c85
<       Name string
---
>       Name string `json:"name"`
88c90
<       HomePage string
---
>       HomePage string `json:"homePage"`
93c95
<       URI string
---
>       URI string `json:"URI"`
diff --ignore-matching-lines='package v2_.' v2_2/other_license.go v2_3/other_license.go
6c6
< // SPDX Document for version 2.2 of the spec.
---
> // SPDX Document for version 2.3 of the spec.
diff --ignore-matching-lines='package v2_.' v2_2/package.go v2_3/package.go
7c7
< // Package is a Package section of an SPDX Document for version 2.2 of the spec.
---
> // Package is a Package section of an SPDX Document for version 2.3 of the spec.
12c12
<       IsUnpackaged bool
---
>       IsUnpackaged bool `json:"-" yaml:"-"`
48c48
<       IsFilesAnalyzedTagPresent bool
---
>       IsFilesAnalyzedTagPresent bool `json:"-" yaml:"-"`
51c51,52
<       PackageVerificationCode common.PackageVerificationCode `json:"packageVerificationCode"`
---
>       // Cardinality: if FilesAnalyzed == true must be present, if FilesAnalyzed == false must be omitted
>       PackageVerificationCode *common.PackageVerificationCode `json:"packageVerificationCode,omitempty"`
53c54
<       // 7.10: Package Checksum: may have keys for SHA1, SHA256 and/or MD5
---
>       // 7.10: Package Checksum: may have keys for SHA1, SHA256, MD5, SHA3-256, SHA3-384, SHA3-512, BLAKE2b-256, BLAKE2b-384, BLAKE2b-512, BLAKE3, ADLER32
55c56
<       PackageChecksums []common.Checksum `json:"checksums"`
---
>       PackageChecksums []common.Checksum `json:"checksums,omitempty"`
66,67c67,68
<       // Cardinality: mandatory, one
<       PackageLicenseConcluded string `json:"licenseConcluded"`
---
>       // Cardinality: optional, one
>       PackageLicenseConcluded string `json:"licenseConcluded,omitempty"`
70c71
<       // Cardinality: mandatory, one or many if filesAnalyzed is true / omitted;
---
>       // Cardinality: optional, one or many if filesAnalyzed is true / omitted;
72c73
<       PackageLicenseInfoFromFiles []string `json:"licenseInfoFromFiles"`
---
>       PackageLicenseInfoFromFiles []string `json:"licenseInfoFromFiles,omitempty"`
75,76c76,77
<       // Cardinality: mandatory, one
<       PackageLicenseDeclared string `json:"licenseDeclared"`
---
>       // Cardinality: optional, one
>       PackageLicenseDeclared string `json:"licenseDeclared,omitempty"`
109a111,127
>       // 7.24: Primary Package Purpose
>       // Cardinality: optional, one or many
>       // Allowed values: APPLICATION, FRAMEWORK, LIBRARY, CONTAINER, OPERATING-SYSTEM, DEVICE, FIRMWARE, SOURCE, ARCHIVE, FILE, INSTALL, OTHER
>       PrimaryPackagePurpose string `json:"primaryPackagePurpose,omitempty"`
> 
>       // 7.25: Release Date: YYYY-MM-DDThh:mm:ssZ
>       // Cardinality: optional, one
>       ReleaseDate string `json:"releaseDate,omitempty"`
> 
>       // 7.26: Build Date: YYYY-MM-DDThh:mm:ssZ
>       // Cardinality: optional, one
>       BuiltDate string `json:"builtDate,omitempty"`
> 
>       // 7.27: Valid Until Date: YYYY-MM-DDThh:mm:ssZ
>       // Cardinality: optional, one
>       ValidUntilDate string `json:"validUntilDate,omitempty"`
> 
111c129
<       Files []*File
---
>       Files []*File `json:"files,omitempty"`
113c131
<       Annotations []Annotation `json:"annotations"`
---
>       Annotations []Annotation `json:"annotations,omitempty"`
117c135
< // about a Package, as defined in section 7.21 in version 2.2 of the spec.
---
> // about a Package, as defined in section 7.21 in version 2.3 of the spec.
diff --ignore-matching-lines='package v2_.' v2_2/relationship.go v2_3/relationship.go
8c8
< // version 2.2 of the spec.
---
> // version 2.3 of the spec.
diff --ignore-matching-lines='package v2_.' v2_2/review.go v2_3/review.go
5c5
< // Review is a Review section of an SPDX Document for version 2.2 of the spec.
---
> // Review is a Review section of an SPDX Document for version 2.3 of the spec.
diff --ignore-matching-lines='package v2_.' v2_2/snippet.go v2_3/snippet.go
7c7
< // Snippet is a Snippet section of an SPDX Document for version 2.2 of the spec.
---
> // Snippet is a Snippet section of an SPDX Document for version 2.3 of the spec.
22,23c22,23
<       // Cardinality: mandatory, one
<       SnippetLicenseConcluded string `json:"licenseConcluded"`
---
>       // Cardinality: optional, one
>       SnippetLicenseConcluded string `json:"licenseConcluded,omitempty"`
47c47
<       SnippetAttributionTexts []string `json:"-"`
---
>       SnippetAttributionTexts []string `json:"-" yaml:"-"`

Signed-off-by: Keith Zantow <[email protected]>

kzantow · 2022-10-10T16:45:40Z

NOTE: I would squash all of these commits into a feat: Support SPDX 2.3 type of commit. I may have mislabeled a few commits as chore updates.

Signed-off-by: Keith Zantow <[email protected]>

kzantow · 2022-10-19T21:22:17Z

Ping @lumjjb @swinslow (sorry for the ping)... is this something that we could get in reasonably soon?

lumjjb · 2022-10-19T22:30:30Z

Sorry i missed this (kubecon prep crunch). Let me review this this weekend and hope we can merge it in the next few weeks!

lumjjb

Thanks so much @kzantow for this effort! It was a lot of code wrangling!! Have a couple minor nits on adding ENUMs and should be good to merge, this will provide a good basis of using the struct, and then additional validation we can add on later!

builder/builder2v3/build_creation_info_test.go

lumjjb · 2022-10-21T18:53:41Z

examples/sample-docs/json/SPDXJSONExample-v2.3.spdx.json

@@ -0,0 +1,406 @@
+{
+  "spdxVersion": "SPDX-2.3",


TODO: open a separate follow-up issue to have another test case which contains more 2.3 specific fields.

NOTE: there are all the additional fields added to at least one package: https://github.com/spdx/tools-golang/pull/164/files/d326e1063b8265703fae4677e7fd63eb97e2df28#diff-f21e5fa3e6e7c061fc6553ed162414f5a0394e27bae30f647627d0e1a7199a11R151-R154

lumjjb · 2022-10-21T19:00:07Z

spdx/v2_3/file.go

+
+	// 8.4: File Checksum: may have keys for SHA1, SHA256, MD5, SHA3-256, SHA3-384, SHA3-512, BLAKE2b-256, BLAKE2b-384, BLAKE2b-512, BLAKE3, ADLER32
+	// Cardinality: mandatory, one SHA1, others may be optionally provided
+	Checksums []common.Checksum `json:"checksums"`


I think we should add the new algorithms to the common folder as well as consts.

Added hash algorithms (SHA3-256, SHA3-384, SHA3-512, BLAKE2b-256, BLAKE2b-384, BLAKE2b-512, BLAKE3, ADLER32 ) to the set recognized by 7.10 (Package Checksum field) and 8.4 (File checksum field)

lumjjb · 2022-10-21T19:04:03Z

spdx/v2_3/relationship.go

+	// Relationship is type from 11.1.1
+	RefA         common.DocElementID `json:"spdxElementId"`
+	RefB         common.DocElementID `json:"relatedSpdxElement"`
+	Relationship string              `json:"relationshipType"`


Add 2 new relationship consts to spdx/common/external.go

Update Clause 11 to add the new relationship types: REQUIREMENT_DESCRIPTION_FOR and SPECIFICATION_FOR.

In addition, since its touching the same files, adding the following consts:

Update Annex F ( External Repository Identifiers ) to expand security references to include advisory, fix, URL, SWID. Expand persistent identifiers to include gitoid.

spdx/common/external.go already has those constants: https://github.com/spdx/tools-golang/blob/main/spdx/common/external.go#L64-L65

The other fields are here: https://github.com/spdx/tools-golang/blob/main/spdx/common/external.go#L10-L13

lumjjb · 2022-10-21T19:06:42Z

spdxlib/described_elements_test.go

+	// set up document and one package, but no relationships
+	// b/c only one package
+	doc := &v2_3.Document{
+		SPDXVersion:    "SPDX-2.2",


SPDX-2.2 -> SPDX-2.3

Couple of occurrences

lumjjb · 2022-10-21T19:08:58Z

spdxlib/documents_test.go

+func Test2_3InvalidDocumentFailsValidation(t *testing.T) {
+	// set up document and some packages and relationships
+	doc := &v2_3.Document{
+		SPDXVersion:    "SPDX-2.1",


is the old version intended for failure here?

This was copied from the 2.2 version, but doesn't actually do much to test the failure condition; so I've updated it.

lumjjb · 2022-10-21T19:11:00Z

tvloader/parser2v3/parse_file.go

+		switch common.ChecksumAlgorithm(subkey) {
+		case common.SHA1, common.SHA256, common.MD5:
+			algorithm := common.ChecksumAlgorithm(subkey)
+			parser.file.Checksums = append(parser.file.Checksums, common.Checksum{Algorithm: algorithm, Value: subvalue})


add case for newly defined unhandled checksums and create an issue as a TODO to add support for new checksums

lumjjb · 2022-10-21T19:13:56Z

tvloader/parser2v3/parse_other_license.go

+	"github.com/spdx/tools-golang/spdx/v2_3"
+)
+
+func (parser *tvParser2_3) parsePairFromOtherLicense2_3(tag string, value string) error {


@swinslow i'm not as familiar with the license change assertion of XML format in 2.3.. From my knowledge its just additional validation and parsing required (which we should add to as part of #159)? Thoughts here?

Signed-off-by: Keith Zantow <[email protected]>

kzantow · 2022-10-24T17:30:15Z

Thank you so much for the review @lumjjb! I think I've made all the changes requested -- please let me know if I've missed anything!

lumjjb

LGTM!

kzantow mentioned this pull request Oct 6, 2022

SPDX 2.3 Support #156

Closed

kzantow added 4 commits October 6, 2022 19:34

chore: Fix section numbers in 2.2 data model

993e491

Signed-off-by: Keith Zantow <[email protected]>

chore: Add v2.3 data model and JSON support

6fda811

Signed-off-by: Keith Zantow <[email protected]>

chore: Add v2.3 data model to reporter

53dc87d

Signed-off-by: Keith Zantow <[email protected]>

chore: Add v2.3 data model to licensediff

11771ee

Signed-off-by: Keith Zantow <[email protected]>

kzantow force-pushed the feat/spdx-2.3-support branch from a445943 to 11771ee Compare October 6, 2022 23:36

kzantow added 8 commits October 7, 2022 12:08

chore: Update v2.3 JSON test

74a5f1d

Signed-off-by: Keith Zantow <[email protected]>

chore: Add YAML support for v2.3 data model

edca4e8

Signed-off-by: Keith Zantow <[email protected]>

chore: missed YAML writer for v2.3 data model

066a798

Signed-off-by: Keith Zantow <[email protected]>

chore: add tag value reader and parser for v2.3

be2eb82

Signed-off-by: Keith Zantow <[email protected]>

chore: add builder and verification functions for 2.3

b57c493

Signed-off-by: Keith Zantow <[email protected]>

chore: add idsercher functions for 2.3

ab3e717

Signed-off-by: Keith Zantow <[email protected]>

chore: support for 2.3 features to rdfloader

0b2bd4b

Signed-off-by: Keith Zantow <[email protected]>

chore: correct RDF 2.2 example

7779493

Signed-off-by: Keith Zantow <[email protected]>

kzantow marked this pull request as ready for review October 10, 2022 16:37

This was referenced Oct 10, 2022

Refactoring based on versions #165

Closed

chore: refactor project structure kzantow-anchore/tools-golang#1

Closed

chore: update a few missed functions for 2.3

d326e10

Signed-off-by: Keith Zantow <[email protected]>

lumjjb reviewed Oct 21, 2022

View reviewed changes

kzantow added 4 commits October 24, 2022 13:03

chore: add common hash algorithms

6ec5391

Signed-off-by: Keith Zantow <[email protected]>

chore: fix a few 2.2 references

aaabc92

Signed-off-by: Keith Zantow <[email protected]>

chore: fix a 2.1 reference

55fcfed

Signed-off-by: Keith Zantow <[email protected]>

chore: add remaining 2.3 checksum algorithms to tvloader/parse_file

bc62528

Signed-off-by: Keith Zantow <[email protected]>

kzantow mentioned this pull request Oct 26, 2022

Support SPDX 2.3 anchore/syft#1292

Closed

lumjjb approved these changes Oct 31, 2022

View reviewed changes

lumjjb merged commit e20ab88 into spdx:main Oct 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for SPDX 2.3 #164

Add support for SPDX 2.3 #164

kzantow commented Oct 6, 2022 •

edited

Loading

kzantow commented Oct 10, 2022 •

edited

Loading

kzantow commented Oct 19, 2022 •

edited

Loading

lumjjb commented Oct 19, 2022

lumjjb left a comment

lumjjb Oct 21, 2022

kzantow Oct 24, 2022 •

edited

Loading

lumjjb Oct 21, 2022

lumjjb Oct 21, 2022

lumjjb Oct 21, 2022

kzantow Oct 24, 2022

lumjjb Oct 21, 2022

lumjjb Oct 21, 2022

kzantow Oct 24, 2022

lumjjb Oct 21, 2022

lumjjb Oct 21, 2022

kzantow commented Oct 24, 2022

lumjjb left a comment

Add support for SPDX 2.3 #164

Add support for SPDX 2.3 #164

Conversation

kzantow commented Oct 6, 2022 • edited Loading

kzantow commented Oct 10, 2022 • edited Loading

kzantow commented Oct 19, 2022 • edited Loading

lumjjb commented Oct 19, 2022

lumjjb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kzantow Oct 24, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kzantow commented Oct 24, 2022

lumjjb left a comment

Choose a reason for hiding this comment

kzantow commented Oct 6, 2022 •

edited

Loading

kzantow commented Oct 10, 2022 •

edited

Loading

kzantow commented Oct 19, 2022 •

edited

Loading

kzantow Oct 24, 2022 •

edited

Loading