when uploading a BOM via API will fail

[taicomjp]

The defect may already be reported! Please search for the defect before creating one.

Current Behavior:

generate bom.json from gradle ( https://plugins.gradle.org/plugin/org.cyclonedx.bom )
if I upload by using web site, it will work, but api will fail
(1.5.x will work, but 1.6.x and 1.7.x not)
same bom file, upload from website will work, but api fail

Steps to Reproduce:

1.generate bom.json from gradle
plugins {
id "org.cyclonedx.bom" version "1.7.1"
}
2.jenkins upload to DT server (from API)

Expected Behavior:

There should not be an error when uploading the BOM,

Environment:

• Dependency-Track Version: 4.5.1
• Distribution: Docker
• BOM Format & Version: CycloneDX 1.4 JSON
• Database Server: MSSQL
• Browser: na

Additional Details:

(e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, gitter, etc)

[nscuro]

Can you share the BOM you uploaded, so we can reproduce? Of course, please redact any internal or confidential information beforehand.

Also, did you check the logs of the Dependency-Track API server? Any related errors?

[taicomjp]

bom.zip

[syalioune]

FYI, the BOM are uploaded just fine through API in my test DT instance.

[gelexgaray]

I can reproduce it on dotnet version of CycloneDX

Until version 2.3.0 of CycloneDX dotnet tool the API works OK (format 1.3)
From version 2.4.1 and up of CycloneDX (format 1.4) the API server gives a 200 (OK) response, but it doesn't process the results.

The following warning is written to the log:

2022-12-02T14:01:53.654590002Z 2022-12-02 14:01:53,654 WARN [BomUploadProcessingTask] The BOM uploaded is not in a supported format. Supported formats include CycloneDX XML and JSON

It works ok when uploading the same file through the UI. I attach a failing file
bom_Viroo.Monitor.Application.xml.zip

We are running containerized version of Dependency Track with separated UI and API and latest tag:

• dependencytrack/apiserver --> Fails to process BOM format 1.4
• dependencytrack/frontend --> processes correctly the same file

[nscuro]

@gelexgaray What version of the API server are you running? The behavior you're describing strongly suggests you're using an outdated version that does not support CycloneDX 1.4.

[gelexgaray]

Both containers are deployed to a kubernetes cluster using "latest". I will pin both containers to the latest version tag and check it again

[gelexgaray]

Repro with latest versions of the containers:

• dependencytrack/frontend:4.6.1
• dependencytrack/apiserver:4.6.3

CycloneDX tool for dotnet was 2.7.0

Logs from API Server:
2022-12-02T17:30:39.726435563Z 2022-12-02 17:30:39,726 WARN [BomUploadProcessingTask] The BOM uploaded is not in a supported format. Supported formats include CycloneDX XML and JSON

[colinfyfe]

TL;DR: the issue affecting @gelexgaray might be resolved in Dependency Track 4.8 but generating a JSON BOM with the dotnet tooling instead of XML (or stripping the byte order mark from the XML) might work in the meantime. That doesn't help with tiacomjp's original gradle issue unfortunately as that referred to a JSON-based BOM

EDIT: Just after posting this, I searched for the error message which took me on a quest to find the source generating the error, then to looksLikeCycloneDX() which in turn lead me to an azure-pipelines-dependency-track issue that ultimately links back to a Dependency Track issue reported as #2312. Looks like this might be the same problem so possibly fixed in 4.8?

--

I've managed to reproduce the issue @gelexgaray reported using dotnet-cyclonedx when uploading an XML BOM but generating a JSON BOM and uploading via the API worked without issue. The XML BOM also worked without issue when uploaded via the frontend.

We normally use the Jenkins plugin and Maven plugin to handle BOM uploads for our Java projects but we're now adding Dependency Track to our .net builds managed by TeamCity using Powershell to generate the BOM and upload it. The script is WIP - I've not sorted out a version numbering strategy yet and there's no error handling but the script below generates the "BOM uploaded is not in a support format" error in the apiserver logs:

function Generate-Bom {
    param(
        [Parameter(Mandatory=$true)][String]$ProjectDir
    )

    $slnPath = Get-ChildItem -Path $ProjectDir -Filter *.sln | Select -First 1
    dotnet CycloneDX $slnPath --disable-github-licenses --disable-package-restore --out $ProjectDir
}


function Publish-Bom {
    param(
        [Parameter(Mandatory=$true)][String]$BomPath
    )

    [Net.ServicePointManager]::SecurityProtocol =[Net.SecurityProtocolType]::Tls12

    $bomContentsBase64 = [convert]::ToBase64String((Get-Content -Path $BomPath -Encoding byte))
    
    Invoke-RestMethod -Method Put -Uri "%DepTrack.URL%/api/v1/bom" -Headers @{
        'X-API-Key' = '%DepTrack.ApiKey%'
    } -ContentType "application/json" -Body ([PSCustomObject]@{
        projectName = '%system.teamcity.projectName%'
        projectVersion = 'CI'
        autoCreate = 'true'
        bom = $bomContentsBase64
    } | ConvertTo-Json)
}


$WorkingDir = "%teamcity.build.checkoutDir%"
Set-Location $WorkingDir
Generate-Bom -ProjectDir $WorkingDir
Publish-Bom -BomPath "$WorkingDir\bom.xml"

Changing the CycloneDX call to:
dotnet CycloneDX $slnPath --disable-github-licenses --disable-package-restore --json --out $PATH

... and the publish call to:
Publish-Bom "$PATH\bom.json"

... results in a successful upload and BOM import.

We were also using frontend:4.6.1 and apiserver:4.6.3 but bumped both to 4.7.0 with the same result.

[Mario-Hofstaetter]

This issue is STILL happening in the latest versions with XML, getting this in api server docker logs:

dtrack-apiserver_1  | 2023-05-10 11:21:58,028 WARN [BomUploadProcessingTask] The BOM uploaded is not in a supported format. Supported formats include CycloneDX XML and JSON

Uploading the same .XML file via WEB UI is successful, BOM Format CycloneDX 1.4 is then shown.
Running

• dependencytrack/apiserver:4.8.0
• and dotnet-cyclonedx 2.7.0 since today.

Using https://github.com/jenkinsci/dependency-track-plugin latest version 4.3.1 to upload the BOM.
Even worse, the jenkins plugins reports SUCCESS, but the BOM is just rejected ?!

13:22:02 [DependencyTrack] Hochladen von Artefakt in Dependency-Track - https://XXXXXXX:8080
13:22:02 [DependencyTrack] Das Artefakt wurde erfolgreich hochgeladen. Sie können nun https://XXXXXXX/projects/ aufrufen, um die Ergebnisse anzusehen.
13:22:02 Finished: SUCCESS

I'm not using the synchronous upload mode but imho the api should return an error if the uploaded file is deemed "invalid"?

I have now too switched to JSON format and API upload has been successful again 😕

[JCH2k]

Same problem here using an XML-BOM 1.5 generated with dotnet CycloneDX.
Using Dependency-Track v4.10.1 and dotnet-cyclonedx 3.0.5.
Uploading via the UI works fine, via the API it fails with the same message.
Using --json works around the problem.

That means it is NOT fixed with 4.8 😐

I agree that there are 2 problems

• There should be an HTTP code 4xx
• Uploading via UI or via API should not make any difference

[nscuro]

v4.11.0 will ship some related improvements. We will validate uploaded BOMs against the CycloneDX schema synchronously, and return a HTTP 400 response when said validation failed:

• Validate uploaded BOMs against CycloneDX schema #3522
• Handle BOM validation errors frontend#762

In contrast to the looksLikeCycloneDX method mentioned by @colinfyfe, the new logic does not inspect the first X bytes of the uploaded file, but instead "streams" through the actual JSON / XML to verify whether it's a CycloneDX BOM in supported format and spec version.

It will thus not matter whether the file is prefixed with byte order marks, comments, or anything else, as long as the contents are valid JSON or XML.

@JCH2k Uploading via UI or via API should not make any difference

The frontend is just a client to the REST API. If there's a difference between the frontend and other clients, it's likely a client-side issue rather than a server-side one.

[mtsfoni]

We still encounter this issue, but with the new validation setting, get a 400 Error:

22:12:25  [DependencyTrack] Publishing artifact to Dependency-Track - [Address}
22:12:25  [DependencyTrack] Invalid payload submitted to server
22:12:25  [DependencyTrack] {"status":400,"title":"The uploaded BOM is invalid","detail":"BOM is neither valid JSON nor XML"}

(The BOM is XML)

When uploading manually, it works fine - the dotnet library also attests the sbom to be valid.

We recently switched to the Jenkins Plugin for uploading, which I now suspect to be the culprit in some way.

[msymons]

I have been encountered the error using CycloneDX module for .NET v3.0.8.0 and Dependency-Track v4.11.5

DT is configured for "BOM Validation" and "BOM Processing V2"

issue-1988-demo.xml

[nscuro]

Still unable to reproduce. The XML BOMs generated by the .NET tool contain byte-order-marks, but we already have handling in place for those.

I tried uploading the BOMs provided in this thread via both multipart, and base64-encoded JSON field, and for none of these methods I'm seeing BOMs getting rejected.

It seems like the Jenkins plugin may indeed be the problem.

[msymons]

@mtsfoni, @JCH2k (and others)... can anyone confirm whether or not the problem occurs when using Dependency-Track Jenkins plugin v5.0.0?

I am afraid that this is something that I cannot currently test.. not until I get my Jenkins servers migrated away from CentOS in a few weeks' time and can then upgrade Jenkins and upgrade the DT Jenkins plugin from v4.3.1

[mtsfoni]

I have 4.3.1 installed.
I could update to 5.0.0 after hours.
Do you suspect it might work in 5.0.0?

[colinfyfe]

@msymons, @mtsfoni - I've just pushed a test build through our Jenkins server with DT plugin 5.0.0 and it imported without issue. Doesn't mean it's definitely not happening for others using V5 of course but it's solid for us:

14:53:31 [INFO] --- cyclonedx-maven-plugin:2.7.11:makeAggregateBom (cyclonedx-aggregate) @ demo ---
14:53:32 [INFO] CycloneDX: Resolving Dependencies
14:53:38 [INFO] CycloneDX: Creating BOM version 1.4 with 167 component(s)
14:53:38 [INFO] CycloneDX: Writing and validating BOM (XML): /data/jenkins/jobs/demo-continuous-build/workspace/target/bom.xml
14:53:39 [INFO]            attaching as demo-0.0.14-SNAPSHOT-cyclonedx.xml
14:53:39 [INFO] CycloneDX: Writing and validating BOM (JSON): /data/jenkins/jobs/demo-continuous-build/workspace/target/bom.json
14:53:39 [INFO]            attaching as demo-0.0.14-SNAPSHOT-cyclonedx.json
...
14:56:35 [DependencyTrack] Publishing artifact to Dependency-Track - https://dt.demo.com:8443
14:56:35 [DependencyTrack] The artifact was successfully published. You may now navigate to https://dt.demo.com/projects/ to view the results.
14:56:35 [DependencyTrack] Polling Dependency-Track for BOM processing status
14:56:45 [DependencyTrack] Looking up id of newly created project with name "Demo" and version "0.0.14-SNAPSHOT"
14:56:45 [DependencyTrack] Processing findings

Full list of versions for this build

• Jenkins 2.462.1 (LTS)
  • OWASP Dependency-Track 5.0.0
  • Java 17.0.12+7 (Temurin)
  • CycloneDX Maven Plugin 2.7.11
• BOM version 1.4
• Dep Track v4.11.5

[msymons]

@colinfyfe. I should have been more precise... requesting that an XML BOM generated by CycloneDX module for .NET be tested with Dependency-Track Jenkins plugin v5.0.0. The initial report for this issue was for a problem uploading a BOM generated by the cyclonedx-gradle-plugin but all reports since the end of 2022 have related to the .NET module.

@mtsfoni, the DT v5.0.0 plugin release notes do not seem to contain anything relevant to our problem, but I have definitely performed software upgrades in the past that delivered fixes/functionality that were unexpected.

The main thing is that plugin update is an easy thing to do (providing all requirements are met) and if it does not help then that is something that can be reported in an issue logged in dependency-track-plugin. It makes life far easier when it is clear up-front that a defect impacts the most recent version of software.

[colinfyfe]

Sorry @msymons, when I see Jenkins, I go into automatic Java mode :) I've just thrown a very simple pipeline build together for one of our .NET projects. Same Jenkins server so still 2.462.1 with Dependency Track plugin 5.0.0. Abridged extract below:

[Pipeline] (Generate SBOM)
22:54:35  + dotnet CycloneDX Demo.sln
..
22:54:36  Creating CycloneDX BOM
22:54:36  Writing to: /redacted/jenkins/jobs/demo/workspace/bom.xml
[Pipeline] (Upload SBOM to OWASP Dependency-Track)
[Pipeline] withCredentials
[Pipeline] dependencyTrackPublisher
22:54:37  [DependencyTrack] Publishing artifact to Dependency-Track - https://dt.demo.com:8443/
22:54:37  [DependencyTrack] The artifact was successfully published. You may now navigate to https://dt.demo.com/projects/ to view the results.
22:54:37  [DependencyTrack] Polling Dependency-Track for BOM processing status
22:54:47  [DependencyTrack] Looking up id of newly created project with name "Jenkins Demo" and version "1.0"
22:54:47  [DependencyTrack] Processing findings

Compiled with .NET 8.0.7 and dotnet-CycloneDX 3.0.8. Hopefully that's more useful?

EDIT: Removed disable-package-restore and deprecated disable-gitlab-licenses options to run "default" options

[colinfyfe]

Some potentially good or bad news depending on your POV - I rolled our Jenkins plugin back to 4.3.1 and it still uploaded a dotnet CycloneDX BOM without issue meaning I can't reproduce the failure unfortunately - my test didn't prove that upgrading from 4.3.1 to 5.0.0 would resolve the issue.

[msymons]

@colinfyfe, yeah... we need to find an instance where a fail using plugin v4.3.1 either still fails with v5.0.0 or stops failing.