bug: spdx checksum empty array; allow syft to generate SHA1 for spdx-tag-value documents by spiffcs · Pull Request #1404 · anchore/syft

spiffcs · 2022-12-14T16:08:11Z

Summary

Syft currently generates SPDX documents that do not contain SHA1 digests for the files by default.

For a user to obtain a document that meets the current specification the following has to be true:

app.FileMetadata.Cataloger.Enabled
app.FileMetadata.Digests must add sha1 to the list of desired digests (currently defaults to sha256)

Here is a link to the reference that shows this requirement:
https://spdx.github.io/spdx-spec/v2.3/file-information/#84-file-checksum-field

How to test ---

I've provided a docker image that wraps the spdx-java-tools: caphill4/java-spdx-tools

Three images are supplied by #1401

alpine:latest
debian:latest
photon:3.0

You can follow these steps to see the newly valid spdx documents. Make sure to replace <IMAGE> with one of the above.

SYFT_FILE_METADATA_CATALOGER_ENABLED=true SYFT_FILE_METADATA_DIGESTS=sha1 go run cmd/syft/main.go <IMAGE> -o spdx > test.spdx

docker run -v (pwd)/test.spdx:/test.spdx caphill4/java-spdx-tools Verify /test.spdx

Context

Running the following command syft -o spdx-json alpine:latest > spdx.json generates an SPDX json document.

Using https://github.com/spdx/tools-python a user can run:
pyspdxtools_parser --file spdx.spdx

This results in a python error:
TypeError: 'NoneType' object is not iterable

The fix in this PR updates the checksum array from null ==> []

After an empty array is replaced the following errors are seen:

alpine:latest: /usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-616ac3bc.rsa.pub: At least one file checksum algorithm must be SHA1
alpine:latest: /usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-616adfeb.rsa.pub: At least one file checksum algorithm must be SHA1
alpine:latest: /usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-616ae350.rsa.pub: At least one file checksum algorithm must be SHA1
alpine:latest: /usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-616db30d.rsa.pub: At least one file checksum algorithm must be SHA1
alpine:latest: /usr/share/udhcpc/default.script: At least one file checksum algorithm must be SHA1
Errors while parsing:  True

Given that a user has enabled the file digest cataloger:
SYFT_FILE_METADATA_CATALOGER_ENABLED=true SYFT_FILE_METADATA_DIGESTS=sha1

This PR adds the ability to generate a valid SPDX document by enhancing the package entity with a PackageVerificationCode.

The PackageVerificationCode is generated by obtaining all of the Contains relationships for a package and performing the following algorithm:

verificationcode = 0
filelist = templist = ""
for all files in the package {
    if file is an "excludes" file, skip it /* exclude SPDX analysis file(s) */

        append templist with "SHA1(file)/n"
    }
sort templist in ascending order by SHA1 value
filelist = templist with "/n"s removed. /* ordered sequence of SHA1 values with no separators */
verificationcode = SHA1(filelist)

https://spdx.github.io/spdx-spec/v2.3/package-information/#79-package-verification-code-field

Note:

We use the Contains relationship here since this is the value the SPDX validator will key off of if a package verification code is missing.

Example:
Validator warns: At least one file checksum algorithm must be SHA1 for package x files

After providing a SHA1 digest for the file:

Validator warns: Found analyzed files for package x when analyzedFiles is set to false

After settting analyzedFiles to true:

Validator warns: Missing Package Verification Code for package x

Note

Empty-size checks were removed from the digest cataloger in order to produce a valid document where empty files like /lib/apk/db/lock were not being summed.

If this is incorrect or needs to be reworked so we're not missing some edge case regarding the removed tar file note I'm happy to refactor.