Add catalogers configuration by wagoodman · Pull Request #1038 · anchore/syft

wagoodman · 2022-06-09T14:47:41Z

This pulls in @ramanan-ravi s work from #843 with a few adjustments (see #843 (comment) as for why the separate PR).

Closes #840
Closes #465

Signed-off-by: ramanan-ravi <ramanan@deepfence.io> Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

github-actions · 2022-06-09T14:54:06Z

Benchmark Test Results

Benchmark results from the latest changes vs base branch

name                                                       old time/op    new time/op    delta
ImagePackageCatalogers/alpmdb-cataloger-2                    14.9ms ± 2%    11.2ms ± 1%  -24.56%  (p=0.008 n=5+5)
ImagePackageCatalogers/ruby-gemspec-cataloger-2              1.65ms ± 4%    1.21ms ± 1%  -26.76%  (p=0.008 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2            4.11ms ± 2%    3.04ms ± 0%  -26.13%  (p=0.008 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2    1.37ms ± 5%    0.96ms ± 0%  -29.66%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         934µs ± 1%     654µs ± 0%  -30.00%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                    1.15ms ± 3%    0.77ms ± 0%  -32.80%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                     1.07ms ± 3%    0.74ms ± 1%  -31.28%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      18.0ms ± 1%    13.5ms ± 1%  -25.03%  (p=0.008 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.59ms ± 3%    1.11ms ± 0%  -30.52%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2          3.02µs ± 4%    2.10µs ± 1%  -30.53%  (p=0.008 n=5+5)
ImagePackageCatalogers/dotnet-deps-cataloger-2               1.75ms ± 1%    1.24ms ± 1%  -29.00%  (p=0.008 n=5+5)

name                                                       old alloc/op   new alloc/op   delta
ImagePackageCatalogers/alpmdb-cataloger-2                    5.25MB ± 0%    5.26MB ± 0%   +0.13%  (p=0.008 n=5+5)
ImagePackageCatalogers/ruby-gemspec-cataloger-2               193kB ± 0%     193kB ± 0%     ~     (p=0.841 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2             917kB ± 0%     917kB ± 0%     ~     (p=0.548 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     204kB ± 0%     205kB ± 0%     ~     (p=0.841 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         149kB ± 0%     149kB ± 0%     ~     (p=0.095 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                     194kB ± 0%     194kB ± 0%     ~     (p=0.841 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                      224kB ± 0%     224kB ± 0%     ~     (p=0.056 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      3.35MB ± 0%    3.35MB ± 0%     ~     (p=0.222 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.24MB ± 0%    1.24MB ± 0%   -0.06%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2            672B ± 0%      672B ± 0%     ~     (all equal)
ImagePackageCatalogers/dotnet-deps-cataloger-2                360kB ± 0%     360kB ± 0%     ~     (p=0.841 n=5+5)

name                                                       old allocs/op  new allocs/op  delta
ImagePackageCatalogers/alpmdb-cataloger-2                     85.4k ± 0%     85.4k ± 0%   +0.00%  (p=0.029 n=4+4)
ImagePackageCatalogers/ruby-gemspec-cataloger-2               3.96k ± 0%     3.96k ± 0%     ~     (all equal)
ImagePackageCatalogers/python-package-cataloger-2             15.7k ± 0%     15.7k ± 0%     ~     (p=1.000 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     5.24k ± 0%     5.24k ± 0%     ~     (p=0.587 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         3.02k ± 0%     3.02k ± 0%     ~     (p=0.556 n=4+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                     4.30k ± 0%     4.30k ± 0%     ~     (all equal)
ImagePackageCatalogers/rpmdb-cataloger-2                      5.53k ± 0%     5.53k ± 0%     ~     (all equal)
ImagePackageCatalogers/java-cataloger-2                       54.9k ± 0%     54.9k ± 0%     ~     (p=0.738 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                      5.14k ± 0%     5.14k ± 0%     ~     (p=1.000 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2            15.0 ± 0%      15.0 ± 0%     ~     (all equal)
ImagePackageCatalogers/dotnet-deps-cataloger-2                6.98k ± 0%     6.98k ± 0%     ~     (all equal)

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

Signed-off-by: Ramanan Ravikumar <ramanan@deepfence.io> Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

spiffcs · 2022-06-14T17:15:42Z

@wagoodman I think there are some conflicts on this one. After those are resolved I can give it a review

spiffcs · 2022-06-16T14:47:01Z

+# set the list of package catalogers to use when generating the SBOM
+# default = empty (cataloger set determined automatically by the source type [image or file/directory])
+# catalogers:
+#   - "ruby-gemfile"


nit: do these have to be in quotes or can we just remove that part?

I don't think so, I'll update 👍

spiffcs

Tentative approval from over here! If we fix the conflicts and I can give it a run on my local I think 👍

…uration Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* main: (70 commits) fix: add php catalogers to all catalogers (anchore#1065) feat: add use-all-catalogers flag (anchore#1050) Updates parsing of `yarn.lock` to use `resolved` URLs that are pulled from yarn and npm registries (anchore#926) remove OSS Meetup message (anchore#1057) add pom.xml cataloger (anchore#1055) Add support for CBL-Mariner distroless images (anchore#1045) Add catalogers configuration (anchore#1038) add template output (anchore#1051) update stereoscope to latest version (anchore#1052) update zip_read_closer to incorporate zip64 support (anchore#1041) Add pacman (alpm) parser support (anchore#943) Update of README.md (anchore#1027) bump cosign to v1.9.0 to resolve reporting of GHSA-66x3-6cw3-v5gj (anchore#1025) add workflows to test new project automation (anchore#1023) improve LanguageByName and add unit tests (anchore#1034) Read Description from dpkg status files (anchore#996) Add announcement for Anchore OSS Virtual Meetup (anchore#1033) add main module field to go bin metadata (anchore#1026) Add filters to package cataloger (anchore#1021) change draft to false for release process (anchore#1016) ... Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

* Option to enable specific language or ecosystem cataloger Signed-off-by: ramanan-ravi <ramanan@deepfence.io> Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * Disable dotnet cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * Option to enable specific language or ecosystem cataloger Signed-off-by: Ramanan Ravikumar <ramanan@deepfence.io> Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * rename "enable-cataloger" option to "catalogers" Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add cli test for --catalogers option Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * update readme with latest cataloger names Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * enable dotnet cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * fix linting Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * fix cataloger imports Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * update readme with alpmdb cataloger config example Signed-off-by: Alex Goodman <alex.goodman@anchore.com> Co-authored-by: ramanan-ravi <ramanan@deepfence.io>

wagoodman force-pushed the add-catalogers-configuration branch from 8a06195 to 07f0561 Compare June 9, 2022 14:50

Option to enable specific language or ecosystem cataloger

901314c

Signed-off-by: ramanan-ravi <ramanan@deepfence.io> Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

ramanan-ravi and others added 4 commits June 9, 2022 10:54

Disable dotnet cataloger

3c3f2ec

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

Option to enable specific language or ecosystem cataloger

76db62b

Signed-off-by: Ramanan Ravikumar <ramanan@deepfence.io> Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

rename "enable-cataloger" option to "catalogers"

5b13366

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

add cli test for --catalogers option

c57f40d

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

wagoodman force-pushed the add-catalogers-configuration branch from 2d8c4e1 to c57f40d Compare June 9, 2022 14:54

wagoodman mentioned this pull request Jun 9, 2022

Option to enable specific language or ecosystem cataloger #843

Merged

wagoodman added 3 commits June 9, 2022 11:02

update readme with latest cataloger names

9bc9160

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

enable dotnet cataloger

8b3ca7a

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

fix linting

20d1412

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

wagoodman marked this pull request as ready for review June 9, 2022 18:34

wagoodman enabled auto-merge (squash) June 9, 2022 18:34

wagoodman mentioned this pull request Jun 9, 2022

Enable/disable SBOM generation for specific language types #840

Closed

fix cataloger imports

c4f6460

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

wagoodman mentioned this pull request Jun 9, 2022

Improve cataloger selection capabilities #1039

Closed

4 tasks

wagoodman requested a review from a team June 9, 2022 19:01

wagoodman self-assigned this Jun 9, 2022

spiffcs mentioned this pull request Jun 14, 2022

Continue - Option to enable specific language or ecosystem cataloger #888

Closed

cpendery mentioned this pull request Jun 15, 2022

feat: add use-all-catalogers flag #1050

Merged

spiffcs reviewed Jun 16, 2022

View reviewed changes

Comment thread cmd/syft/cli/options/packages.go

spiffcs approved these changes Jun 16, 2022

View reviewed changes

wagoodman added 2 commits June 21, 2022 08:43

Merge remote-tracking branch 'origin/main' into add-catalogers-config…

b1aacc0

…uration Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

update readme with alpmdb cataloger config example

546eaec

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

wagoodman disabled auto-merge June 21, 2022 12:46

wagoodman enabled auto-merge (squash) June 21, 2022 12:47

wagoodman merged commit ea611da into main Jun 21, 2022

wagoodman deleted the add-catalogers-configuration branch June 21, 2022 13:06

spiffcs mentioned this pull request Jun 21, 2022

fix: add missing javascript-lock-cataloger in docker image catalogers #1022

Closed

tofay mentioned this pull request Jun 23, 2022

SBOM cataloger #1029

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add catalogers configuration#1038

Add catalogers configuration#1038
wagoodman merged 11 commits intomainfrom
add-catalogers-configuration

wagoodman commented Jun 9, 2022 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 9, 2022 •

edited

Loading

Uh oh!

spiffcs commented Jun 14, 2022

Uh oh!

spiffcs Jun 16, 2022

Uh oh!

wagoodman Jun 21, 2022

Uh oh!

Uh oh!

spiffcs left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

wagoodman commented Jun 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark Test Results

Uh oh!

spiffcs commented Jun 14, 2022

Uh oh!

spiffcs Jun 16, 2022

Choose a reason for hiding this comment

Uh oh!

wagoodman Jun 21, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

spiffcs left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wagoodman commented Jun 9, 2022 •

edited

Loading

github-actions Bot commented Jun 9, 2022 •

edited

Loading