Exception when creating cyclonedx output

[jkwasny]

There is an exception when generating cyclonedx output and external references are merged. The error is:
seen = set(x)
TypeError: unhashable type: 'CycloneDxExternalRef'

It is not reproduceable using sample files but I've managed to slightly modify json output from scanning the samples and got the same error. Json file is attached, please try it with --cyclonedx and --from-json options.
I've run it on Windows 10 using python 3.8.10.

Thank you,
Janusz

scancode-samples-mod.txt

[AyanSinhaMahapatra]

Thanks for the report, I could reproduce from this JSON file you attached, and looking into this.

[MochonDeVuelta]

I have the same problem, when I try to generate a cyclonedx or cyclonedx-xml I get the following error message:

ERROR: failed to run output plugin: cyclonedx:

Traceback (most recent call last):
File "scancode-toolkit-v31.1.0\venv\lib\site-packages\scancode\cli.py", line 1077, in run_codebase_plugins
plugin.process_codebase(codebase, **kwargs)
File "scancode-toolkit-v31.1.0\venv\lib\site-packages\formattedcode\output_cyclonedx.py", line 794, in process_codebase
bom = CycloneDxBom.from_codebase(codebase)
File "scancode-toolkit-v31.1.0\venv\lib\site-packages\formattedcode\output_cyclonedx.py", line 681, in from_codebase
components = list(CycloneDxComponent.from_packages(packages))
File "scancode-toolkit-v31.1.0\venv\lib\site-packages\formattedcode\output_cyclonedx.py", line 339, in from_packages
base_component.merge(other_component)
File "scancode-toolkit-v31.1.0\venv\lib\site-packages\formattedcode\output_cyclonedx.py", line 375, in merge
merge_lists(self.externalReferences, other.externalReferences)
File "scancode-toolkit-v31.1.0\venv\lib\site-packages\formattedcode\output_cyclonedx.py", line 431, in merge_lists
seen = set(x)
TypeError: unhashable type: 'CycloneDxExternalRef'

Using scancode scancode-toolkit-v31.1.1 and phyton 3.8.10

[tgodau]

Hi there,

got a similar issue with the cyclonedx-output when scanning a very large package...

Used version:

ScanCode version: 31.0.1
ScanCode Output Format version: 2.0.0
SPDX License list version: 3.17

Download, prepare package and start scanning:

> wget https://download.qt.io/official_releases/qt/5.15/5.15.7/submodules/qtwebengine-everywhere-opensource-src-5.15.7.zip
> unzip qtwebengine-everywhere-opensource-src-5.15.7.zip
> mv qtwebengine-everywhere-src-5.15.7/ qtwebengine-5.15.7/
> scancode -n 2 -clip qtwebengine-5.15.7/ --json-pp qtwebengine-5.15.7.json --cyclonedx qtwebengine-5.15.7_cdxSBOM.json

Gives the following error message:

:
:
ERROR: failed to run output plugin: cyclonedx:
Traceback (most recent call last):
  File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/scancode/cli.py", line 1077, in run_codebase_plugins
    plugin.process_codebase(codebase, **kwargs)
  File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/formattedcode/output_cyclonedx.py", line 794, in process_codebas
e
    bom = CycloneDxBom.from_codebase(codebase)
  File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/formattedcode/output_cyclonedx.py", line 681, in from_codebase
    components = list(CycloneDxComponent.from_packages(packages))
  File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/formattedcode/output_cyclonedx.py", line 339, in from_packages
    base_component.merge(other_component)
  File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/formattedcode/output_cyclonedx.py", line 370, in merge
    merge_lists(self.licenses, other.licenses)
  File "/home/scanner/venv3.8-scancode/lib/python3.8/site-packages/formattedcode/output_cyclonedx.py", line 431, in merge_lists
    seen = set(x)
TypeError: unhashable type: 'CycloneDxLicenseExpression'

Scanning done.
Some files failed to scan properly:
Path: qtwebengine-5.15.7/src/3rdparty/chromium/third_party/icu/common/icudtb.dat
Path: qtwebengine-5.15.7/src/3rdparty/chromium/third_party/icu/common/icudtl.dat
Path: qtwebengine-5.15.7/src/3rdparty/chromium/third_party/icu/ios/icudtl.dat
Summary:        info, licenses, copyrights, packages with 2 process(es)
Errors count:   3
Scan Speed:     2.45 files/sec. 27.69 KB/sec.
Initial counts: 183592 resource(s): 167376 file(s) and 16216 directorie(s)
Final counts:   183592 resource(s): 167376 file(s) and 16216 directorie(s) for 1.80 GB
Timings:
  scan_start: 2022-11-21T121128.360738
  scan_end:   2022-11-22T072026.947766
  setup_scan:licenses: 2.22s
  setup: 2.22s
  inventory: 185.09s
  scan:packages: 180.89s
  scan: 68326.72s
  output:json-pp: 123.34s
  output: 123.37s
  total: 69087.68s
Removing temporary files...done.

Maybe this helps...

Regards, Thorsten

[pombredanne]

@tgodau Thanks for the report. I am planning this for the next milestone

[AyanSinhaMahapatra]

Oops closed by mistake!

@tgodau @jkwasny @MochonDeVuelta could you test the fix? this is in latest develop (and will be released soon!).

[AyanSinhaMahapatra]

@tgodau @jkwasny @MochonDeVuelta Gentle ping, did you get to try this out?

[pombredanne]

@AyanSinhaMahapatra if you have a test that verifies that this is fixed, let's close this as fixed.

[AyanSinhaMahapatra]

Yes, closing this.

[tgodau]

@AyanSinhaMahapatra
Maybe you could use the four lines from my description to download, prepare and scan the specific mentioned Qt package where this error appeared...

[AyanSinhaMahapatra]

@tgodau we have added this failure as a test in scancode already, see this commit: 553e624.

Apart from that I've also run the full scan for you, which ran successfully. See attached file below: (The JSON was too big ~500mb to attach)

scancode -n 12 -clip qtwebengine-5.15.7/ --json-pp qtwebengine-5.15.7.json --cyclonedx qtwebengine-5.15.7_cdxSBOM.json --classify --summary --license-text
Setup plugins...
Collect file inventory...
Scan files for: info, packages, licenses, copyrights with 12 process(es)...
[####################] 334752
/home/ayansinha/nexB/write_access/scancode-toolkit/venv/lib/python3.8/site-packages/fingerprints/cleanup.py:54: ICUWarning: Install 'pyicu' for better text transliteration.
  text = ascii_text(text)
Scanning done.
Summary:        info, packages, licenses, copyrights with 12 process(es)
Errors count:   0
Scan Speed:     7.98 files/sec. 90.23 KB/sec.
Initial counts: 183592 resource(s): 167376 file(s) and 16216 directorie(s)
Final counts:   183592 resource(s): 167376 file(s) and 16216 directorie(s) for 1.80 GB
Timings:
  scan_start: 2023-01-18T131259.695160
  scan_end:   2023-01-18T191854.259676
  setup_scan:licenses: 1.40s
  setup: 1.40s
  inventory: 55.32s
  pre-scan:classify: 76.43s
  pre-scan: 76.43s
  scan:packages: 77.72s
  scan:licenses: 8586.14s
  scan: 20964.43s
  post-scan:summary: 714.78s
  post-scan:license-references: 26.71s
  post-scan: 741.50s
  output:json-pp: 56.46s
  output: 56.47s
  total: 22022.89s

qtwebengine-5.15.7_cdxSBOM.json.txt