Inconsistent GHSA reports a fixed-in version in OSV but not in PyPI

[juspence]

Similar to pypa/pip-audit#274 but seems to be an issue with the advisory-database, not the pip-audit code.

There is a GHSA for django-celery-results that was (finally) fixed in version 2.4.0:
GHSA-fvx8-v524-8579

I see that the OSV vulnerability reporting service properly reports the "fixed-in" version:

(venv) $ pip-audit --strict -s osv
Found 1 known vulnerability in 1 package
Name                  Version ID                  Fix Versions
--------------------- ------- ------------------- ------------------
django-celery-results 2.3.1   GHSA-fvx8-v524-8579 2.4.0

But the PyPI vulnerability reporting service does not report any fixed-in version:

(venv) $ pip-audit --strict
Found 1 known vulnerability in 1 package
Name                  Version ID                  Fix Versions
--------------------- ------- ------------------- ------------------
django-celery-results 2.3.1   GHSA-fvx8-v524-8579

# Forcing the use of pypi doesn't change the result
(venv) $ pip-audit --strict -s pypi
Found 1 known vulnerability in 1 package
Name                  Version ID                  Fix Versions
--------------------- ------- ------------------- ------------------
django-celery-results 2.3.1   GHSA-fvx8-v524-8579

I see no advisories at all in PyPI directly:

(venv) $  curl -s https://pypi.org/pypi/django-celery-results/json | jq ".vulnerabilities"
[]

Is there some stale / cached data in the advisory-database that hasn't been removed? Or is this a bug in PyPI's API?

[di]

The unversioned URL https://pypi.org/pypi/django-celery-results/json provides information on the latest release, which is currently https://pypi.org/project/django-celery-results/2.4.0/ and is not included in the advisory for this vulnerability:

advisory-database/vulns/django-celery-results/PYSEC-2020-38.yaml

Lines 16 to 26 in ca8c884

	versions:
	- 1.0.0
	- 1.0.1
	- 1.0.2
	- 1.0.3
	- 1.0.4
	- 1.1.0
	- 1.1.1
	- 1.1.2
	- 1.2.0
	- 1.2.1

If you look at the JSON API for a specific version that is listed as affected, you see the advisories:

$ curl -s https://pypi.org/pypi/django-celery-results/1.0.0/json | jq ".vulnerabilities"
[
  {
    "aliases": [
      "CVE-2020-17495"
    ],
    "details": "django-celery-results through 1.2.1 stores task results in the database. Among the data it stores are the variables passed into the tasks. The variables may contain sensitive cleartext information that does not belong unencrypted in the database.",
    "fixed_in": [
      "2.0.0"
    ],
    "id": "PYSEC-2020-38",
    "link": "https://osv.dev/vulnerability/PYSEC-2020-38",
    "source": "osv",
    "summary": null,
    "withdrawn": null
  },
  {
    "aliases": [
      "CVE-2020-17495"
    ],
    "details": "django-celery-results through 1.2.1 stores task results in the database. Among the data it stores are the variables passed into the tasks. The variables may contain sensitive cleartext information that does not belong unencrypted in the database.",
    "fixed_in": [],
    "id": "GHSA-fvx8-v524-8579",
    "link": "https://osv.dev/vulnerability/GHSA-fvx8-v524-8579",
    "source": "osv",
    "summary": null,
    "withdrawn": null
  }
]

$ curl -s https://pypi.org/pypi/django-celery-results/1.2.1/json | jq ".vulnerabilities"
[
  {
    "aliases": [
      "CVE-2020-17495"
    ],
    "details": "django-celery-results through 1.2.1 stores task results in the database. Among the data it stores are the variables passed into the tasks. The variables may contain sensitive cleartext information that does not belong unencrypted in the database.",
    "fixed_in": [
      "2.0.0"
    ],
    "id": "PYSEC-2020-38",
    "link": "https://osv.dev/vulnerability/PYSEC-2020-38",
    "source": "osv",
    "summary": null,
    "withdrawn": null
  },
  {
    "aliases": [
      "CVE-2020-17495"
    ],
    "details": "django-celery-results through 1.2.1 stores task results in the database. Among the data it stores are the variables passed into the tasks. The variables may contain sensitive cleartext information that does not belong unencrypted in the database.",
    "fixed_in": [],
    "id": "GHSA-fvx8-v524-8579",
    "link": "https://osv.dev/vulnerability/GHSA-fvx8-v524-8579",
    "source": "osv",
    "summary": null,
    "withdrawn": null
  }
]

It seems like the actual issue is that the advisory in this repository should now include additional versions, and maybe a different fixed version? But hasn't been updated so PyPI is unaware.

[juspence]

Yep, I think that the "fixed_in" version in the PyPI data should be 2.4.0 to match the GHSA. That was the first django-celery-results version that had the result_extended set to off by default (which resolves the CVE).

[di]

My guess is that something is broken in the pipeline that updates these if the fixed_in changes.

[oliverchang]

The latest push for GHSA-fvx8-v524-8579 looks like:

Posting GHSA-fvx8-v524-8579 to PyPI: [{"id": "GHSA-fvx8-v524-8579", "project": "django-celery-results", "versions": ["1.0.0", "1.0.1", "1.0.2", "1.0.3", "1.0.4", "1.1.0", "1.1.1", "1.1.2", "1.2.0", "1.2.1", "2.0.0", "2.0.1", "2.1.0", "2.2.0", "2.3.0", "2.3.1"], "link": "https://osv.dev/vulnerability/GHSA-fvx8-v524-8579", "aliases": ["CVE-2020-17495"], "details": "django-celery-results prior to 2.4.0 stores task results in the database. Among the data it stores are the variables passed into the tasks. The variables may contain sensitive cleartext information that does not belong unencrypted in the database.\n\nIn version 2.4.0 this is no longer the default behaviour but can be re-enabled with the `result_extended` flag in which case care should be taken to ensure any sensitive variables are scrubbed - see [here](https://github.com/celery/django-celery-results/issues/154#issuecomment-734706270) for an example.", "events": [{"introduced": "0"}, {"fixed": "2.4.0"}]}]

This includes the fix version, and we got no visible errors from OSV's perspective. @di can you please check logs on PyPI's side?

[di]

Shouldn't those be updated in https://github.com/pypa/advisory-database/blob/1cb9a0e3d3bfef3c750abfadaf3ff56df19c409d/vulns/django-celery-results/PYSEC-2020-38.yaml prior to this getting pushed to PyPI?

[oliverchang]

The one I posted is a GHSA one, which doesn't live in the pypa repo.

[di]

But PYSEC-2020-38 has GHSA-fvx8-v524-8579 as an alias, shouldn't they be the same?

[juspence]

I think originally the GHSA did match the PYSEC advisory. The GHSA was (if I'm remembering right) edited to have a new fixed version (2.4.0) when that was released. I think (again, from memory) previously the GHSA matched the PYSEC advisory and reported 2.0.0 as the fixed version.

My understanding is that version partially fixed the issue, but the full fix didn't come along until 2.4.0, after which the GHSA got updated.

[westonsteimel]

Unless something has changed with the import automation since last I worked on it, it never updates anything in the advisory except for the versions array after initial creation of the advisory. That was an enhancement that no one had ever quite gotten around to

[westonsteimel]

It also wasn't creating advisories based off of any information parsed from GHSA info but only from the NVD cve entry

[di]

Looks like the NVD entry also hasn't been updated: https://nvd.nist.gov/vuln/detail/CVE-2020-17495#range-5840140

I guess this is google/osv.dev#254? Should we close?

[oliverchang]

But PYSEC-2020-38 has GHSA-fvx8-v524-8579 as an alias, shouldn't they be the same?

They're different instances. I think there's still a possible bug somewhere in the ingestion pipeline that caused PyPI to not pick up the update to GHSA-fvx8-v524-8579 ?

Referring to :

  {
    "aliases": [
      "CVE-2020-17495"
    ],
    "details": "django-celery-results through 1.2.1 stores task results in the database. Among the data it stores are the variables passed into the tasks. The variables may contain sensitive cleartext information that does not belong unencrypted in the database.",
    "fixed_in": [],
    "id": "GHSA-fvx8-v524-8579",
    "link": "https://osv.dev/vulnerability/GHSA-fvx8-v524-8579",
    "source": "osv",
    "summary": null,
    "withdrawn": null
  }

We did send a fix event for "GHSA-fvx8-v524-8579", but it's not reflected.

[di]

Got it. I thought there was more deduplication happening between aliases, but I see now that we get separate reports for each.

I fixed the root cause in the initial implementation of this here: pypi/warehouse#12767

However this will not change advisories that should have been updated prior to this fix. @oliverchang, would it be possible to do a one-off re-report to PyPI for everything that has received updates?

[oliverchang]

Kicked one off!

[di]

@oliverchang Did it complete? I saw a brief spike, but there were only a few hundred before it dropped off, and the advisory in question doesn't seem to have gotten an update:

[di]

Aha, the cache purging was not triggered, after purging it does seem like this has been resolved:

$ http https://pypi.org/pypi/django-celery-results/2.3.1/json | jq ".vulnerabilities"
[
  {
    "aliases": [
      "CVE-2020-17495"
    ],
    "details": "django-celery-results prior to 2.4.0 stores task results in the database. Among the data it stores are the variables passed into the tasks. The variables may contain sensitive cleartext information that does not belong unencrypted in the database.\n\nIn version 2.4.0 this is no longer the default behaviour but can be re-enabled with the `result_extended` flag in which case care should be taken to ensure any sensitive variables are scrubbed - see [here](https://github.com/celery/django-celery-results/issues/154#issuecomment-734706270) for an example.",
    "fixed_in": [
      "2.4.0"
    ],
    "id": "GHSA-fvx8-v524-8579",
    "link": "https://osv.dev/vulnerability/GHSA-fvx8-v524-8579",
    "source": "osv",
    "summary": null,
    "withdrawn": null
  }
]

Closing!