Managing false positives and scanning workflow implementation in general

[zifot]

Hi,

I've some observations and questions in the context of #36 being closed recently without adding support for managing false positives.

What is the workflow I try to achieve with guarddog

What I want to get is to:

1. fetch Python source packages of my dependencies,
2. scan them with guarddog (guarddog pypi scan),
3. manage false positives (ignore specific errors on specific lines on specific versions of source packages, identified by hash of the content, not semantic version declared in the package metadata),
4. build from scanned sources,
5. transfer build artifacts to the target image build process and install them there.

Where I'm now

I got (1) and (2) from above working. With some adventures due to pypa/pip#1884, but that's another story.

I'm stuck at (3). I hoped for #36 to deliver at least some minimal functionality. As it got closed, and the proposed solution does not work for me (I don't use github actions), I thought I could roll some custom approach on my own.

First thought was to use SARIF output as a starting point, but guarddog pypi scan, which I must use instead of verify to scan local sources, does not support it.

Doubts / questions

1. Could pypi scan output its results in SARIF format? Is the reason it currently does not do it a matter of priorities and/or lack of demand or is there some deeper issue here?
2. As you can see from what I try to build, I want to be sure that I install exactly what was scanned by guarddog. I went with this exact approach, because I don't see how using guarddog pypi scan could achieve same result - it seems to me that you could end up with scanning one thing and installing a different one, because scan is a separate process from the one that ultimately takes artifacts and install them - so your build step could use different artifacts, builds from different source etc. But maybe I'm missing something here?

Any feedback very much appreciated! :)

[christophetd]

Hello!

Apologies for the delayed response, this is our first GitHub discussion and I didn't have notifications set up.

Could pypi scan output its results in SARIF format?

Now tracked in #197. The challenging part is that SARIF is supposed to be for SAST output (i.e. reference a file and line), while GuardDog also acts on package metadata

As you can see from what I try to build, I want to be sure that I install exactly what was scanned by guarddog. I went with this exact approach, because I don't see how using guarddog pypi scan could achieve same result - it seems to me that you could end up with scanning one thing and installing a different one, because scan is a separate process from the one that ultimately takes artifacts and install them - so your build step could use different artifacts, builds from different source etc. But maybe I'm missing something here?

Your reasoning makes sense - the simplest thing we could do is implement a --no-delete-package flag that keeps the package on the local filesystem, so you could install it after scanning. Would that be helpful?

[zifot]

Hey @christophetd, thanks for the feedback!

Now tracked in #197. The challenging part is that SARIF is supposed to be for SAST output (i.e. reference a file and line), while GuardDog also acts on package metadata

Or maybe user could also deliver metadata from pypi (you have this metadata in mind?), next to the source itself, so that scan could still consume it?

Your reasoning makes sense - the simplest thing we could do is implement a --no-delete-package flag that keeps the package on the local filesystem, so you could install it after scanning. Would that be helpful?

Yes, I believe --no-delete-package would be super useful in general, for supporting build-what-you-have-just-scanned workflows when using pypi verify.

On top of that, in my particular context I have now, I could switch from scan to verify and then:

• I would automatically get SARIF output to implement custom silencing/ignoring logic, even without Support SARIF output in "scan" commands #197 (but happy to see that issue got created - I can imagine people needing to fetch sources out of band - i.e. without using pypi verify).
• I would not have to main custom fetching logic from pypi, at least outside of boostrapping guarddog environment (not have to circumvent Avoid generating metadata in pip download --no-deps ... pypa/pip#1884 - tldr: at the moment pip download executes code from the downloaded packages (!) which is a no go for me [1]),

[1] Here I'm assuming guarddog is not doing something similar and/or using pip download under the hood for its scan operation - is this a proper assumption? Don't know, didn't verify it yet.