Automatically audit packages using both osv and pypi, or document that they may report different results

[juspence]

Is your feature request related to a problem? Please describe.

I'm always frustrated when I think there are no security vulnerabilities in my project, but there actually are. My team runs pip-audit as a daily CI job to warn us about CVEs in our Python dependencies.

By default pip-audit uses PyPI, and currently reports no issues for any of our dependencies. But while investigating a dependabot warning today, we reran pip-audit using "--vulnerability-service osv", and saw that there was a django-celery-results GHSA reported in OSV.

Describe the solution you'd like

Obviously pip-audit doesn't control the contents of either the OSV or PyPI databases, and I'm about to go request that those projects synchronize their data. But it would also be nice if pip-audit automatically checked both OSV and PyPI by default:

-s SERVICE, --vulnerability-service SERVICE
                        the vulnerability service to audit dependencies
                        against (choices: osv, pypi) (default: osv, pypi)

Describe alternatives you've considered

Add some documentation stating that OSV and PyPI vulnerabilities may differ, and that auditing against both services is recommended. This could be a small snippet in "pip-audit --help", like below:

-s SERVICE, --vulnerability-service SERVICE
                        the vulnerability service to audit dependencies
                        against. Each service may report different
                        results, so checking both is recommended.
                        (choices: osv, pypi) (default: pypi)

[di]

The two should be in sync. If you can provide the specific example, we can look into it.

[woodruffw]

Whoops, @di beat me to it. Yeah, I believe the only times there should be a discrepancy are if the scheduled task is still pending or has failed for some reason.

[juspence]

@di @woodruffw Please see pypa/advisory-database#65

In this case it was django-celery-results==2.2.0 / GHSA-fvx8-v524-8579, but seems like it could happen for anything and the two aren't always guaranteed to be in sync. Or maybe I'm reading the linked issue incorrectly.

[juspence]

After looking at the linked GHSA, this could be a bug with the OSV data, or (maybe) a bug with how pip-audit compares package names and versions.

We definitely are using django-celery-results==2.2.0, and the GHSA is for versions less than 1.2.1. Running below succeeds:

(venv) $ pip-audit -r requirements/base.txt --strict --vulnerability-service pypi
No known vulnerabilities found

But running below fails:

(venv) $ pip-audit -r requirements/base.txt --strict --vulnerability-service osv
Found 1 known vulnerability in 1 package
Name                  Version ID                  Fix Versions
--------------------- ------- ------------------- ------------
django-celery-results 2.2.0   GHSA-fvx8-v524-8579

With a requirements/base.txt entry like:

django-celery-results==2.2.0 \
    --hash=sha256:cc0285090a306f97f1d4b7929ed98af0475bf6db2568976b3387de4fbe812edc \
    --hash=sha256:d5f83fad9091e52cd6dbb3ca80632153ad14b6cdac4d73258e040f92717237cb
    # via -r requirements/base.in

[di]

Looks like PyPI doesn't know about any vulnerabilities for this project:

$ curl -s https://pypi.org/pypi/django-celery-results/json | jq ".vulnerabilities"
[]

We did have some hiccups with the ingestion in the last few days, it's possible we might need to do some manual intervention to sync here (cc @oliverchang)

[oliverchang]

We weren't properly pushing the GHSA entries to warehouse. I have google/osv.dev#429 out to fix this.

[oliverchang]

warehouse now knows about GHSA-fvx8-v524-8579:

> curl -s https://pypi.org/pypi/django-celery-results/json | jq ".vulnerabilities"
[
  {
    "aliases": [
      "CVE-2020-17495"
    ],
    "details": "django-celery-results through 1.2.1 stores task results in the database. Among the data it stores are the variables passed into the tasks. The variables may contain sensitive cleartext information that does not belong unencrypted in the database.",
    "fixed_in": [],
    "id": "GHSA-fvx8-v524-8579",
    "link": "https://osv.dev/vulnerability/GHSA-fvx8-v524-8579",
    "source": "osv"
  }
]

[woodruffw]

Thanks a ton @oliverchang!

@juspence: Does re-running the audit now give you the results you expect?

[juspence]

Yes, the GHSAs reported by pip-audit --vulnerability-service osv are now reported by pip-audit --vulnerability-service pypi as well! Thanks for fixing it!

[woodruffw]

Glad to hear it! Thanks again for reporting.