Skip to content

fix: apk product/vendor generation for old metadata#1635

Merged
westonsteimel merged 1 commit into
mainfrom
fix-apk-ruby-packages
Mar 1, 2023
Merged

fix: apk product/vendor generation for old metadata#1635
westonsteimel merged 1 commit into
mainfrom
fix-apk-ruby-packages

Conversation

@westonsteimel
Copy link
Copy Markdown
Contributor

This fixes some instances where the improved APK CPE generation logic caused regressions for older alpine package APK metadata. It now generates multiple "upstream" candidates with both name and package type which reduces the amount of duplicated code in the apk cpe gen logic. This also improves the handling of stream version packages, so now we can correctly identify packages such as ruby3.2-rexml as the rexml ruby gem.

This fixes some instances where the improved APK CPE generation
logic caused regressions for older alpine package APK metadata.
It now generates multiple "upstream" candidates with both name
and package type which reduces the amount of duplicated code in
the apk cpe gen logic.  This also improves the handling of stream
version packages, so now we can correctly identify packages such
as ruby3.2-rexml as the rexml ruby gem.

Signed-off-by: Weston Steimel <weston.steimel@anchore.com>
@westonsteimel westonsteimel requested a review from a team March 1, 2023 11:45
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 1, 2023

Benchmark Test Results

Benchmark results from the latest changes vs base branch
goos: linux
goarch: amd64
pkg: github.com/anchore/syft/test/integration
cpu: Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
                                                          │ ./.tmp/benchmark-dad3dfd.txt │
                                                          │            sec/op            │
ImagePackageCatalogers/alpmdb-cataloger-2                                   11.72m ±  6%
ImagePackageCatalogers/ruby-gemspec-cataloger-2                             845.8µ ± 12%
ImagePackageCatalogers/python-package-cataloger-2                           3.006m ±  1%
ImagePackageCatalogers/php-composer-installed-cataloger-2                   647.2µ ±  1%
ImagePackageCatalogers/javascript-package-cataloger-2                       349.3µ ±  1%
ImagePackageCatalogers/dpkgdb-cataloger-2                                   465.9µ ±  2%
ImagePackageCatalogers/rpm-db-cataloger-2                                   436.7µ ±  1%
ImagePackageCatalogers/java-cataloger-2                                     10.78m ±  7%
ImagePackageCatalogers/graalvm-native-image-cataloger-2                     7.976µ ±  2%
ImagePackageCatalogers/apkdb-cataloger-2                                    785.1µ ±  1%
ImagePackageCatalogers/go-module-binary-cataloger-2                         17.90µ ±  1%
ImagePackageCatalogers/dotnet-deps-cataloger-2                              951.1µ ±  2%
ImagePackageCatalogers/portage-cataloger-2                                  282.1µ ±  1%
ImagePackageCatalogers/sbom-cataloger-2                                     103.7µ ±  1%
ImagePackageCatalogers/binary-cataloger-2                                   144.4µ ±  0%
geomean                                                                     451.7µ

                                                          │ ./.tmp/benchmark-dad3dfd.txt │
                                                          │             B/op             │
ImagePackageCatalogers/alpmdb-cataloger-2                                   5.060Mi ± 0%
ImagePackageCatalogers/ruby-gemspec-cataloger-2                             141.9Ki ± 0%
ImagePackageCatalogers/python-package-cataloger-2                           946.7Ki ± 0%
ImagePackageCatalogers/php-composer-installed-cataloger-2                   155.9Ki ± 0%
ImagePackageCatalogers/javascript-package-cataloger-2                       95.96Ki ± 0%
ImagePackageCatalogers/dpkgdb-cataloger-2                                   144.7Ki ± 0%
ImagePackageCatalogers/rpm-db-cataloger-2                                   170.7Ki ± 0%
ImagePackageCatalogers/java-cataloger-2                                     2.722Mi ± 0%
ImagePackageCatalogers/graalvm-native-image-cataloger-2                     1.523Ki ± 0%
ImagePackageCatalogers/apkdb-cataloger-2                                    206.1Ki ± 0%
ImagePackageCatalogers/go-module-binary-cataloger-2                         3.102Ki ± 0%
ImagePackageCatalogers/dotnet-deps-cataloger-2                              314.0Ki ± 0%
ImagePackageCatalogers/portage-cataloger-2                                  75.52Ki ± 0%
ImagePackageCatalogers/sbom-cataloger-2                                     13.06Ki ± 0%
ImagePackageCatalogers/binary-cataloger-2                                   21.19Ki ± 0%
geomean                                                                     110.7Ki

                                                          │ ./.tmp/benchmark-dad3dfd.txt │
                                                          │          allocs/op           │
ImagePackageCatalogers/alpmdb-cataloger-2                                    86.71k ± 0%
ImagePackageCatalogers/ruby-gemspec-cataloger-2                              2.159k ± 0%
ImagePackageCatalogers/python-package-cataloger-2                            15.49k ± 0%
ImagePackageCatalogers/php-composer-installed-cataloger-2                    3.458k ± 0%
ImagePackageCatalogers/javascript-package-cataloger-2                        1.253k ± 0%
ImagePackageCatalogers/dpkgdb-cataloger-2                                    2.646k ± 0%
ImagePackageCatalogers/rpm-db-cataloger-2                                    3.759k ± 0%
ImagePackageCatalogers/java-cataloger-2                                      38.27k ± 0%
ImagePackageCatalogers/graalvm-native-image-cataloger-2                       40.00 ± 0%
ImagePackageCatalogers/apkdb-cataloger-2                                     4.988k ± 0%
ImagePackageCatalogers/go-module-binary-cataloger-2                           101.0 ± 0%
ImagePackageCatalogers/dotnet-deps-cataloger-2                               5.010k ± 0%
ImagePackageCatalogers/portage-cataloger-2                                   1.487k ± 0%
ImagePackageCatalogers/sbom-cataloger-2                                       392.0 ± 0%
ImagePackageCatalogers/binary-cataloger-2                                     649.0 ± 0%
geomean                                                                      2.243k

Comment thread syft/pkg/apk_metadata.go
Type Type
}

func (m ApkMetadata) UpstreamCandidates() (candidates []UpstreamCandidate) {
Copy link
Copy Markdown
Contributor

@wagoodman wagoodman Mar 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is more of a preference than anything else -- since UpstreamCandidates (and the former Upstream) is only used for the benefit of CPE generation the more ideal spot is where CPE generation is being done, in syft/pkg/cataloger/common/cpe. This also has the benefit of being unexported so it can't be used in the API.

What are your thoughts about this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I would have liked that; however, it's also needed for the purl.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I guess I could move it under syft/pkg/cataloger/common/cpe and then import it in apkdb?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would still have to be exported though, right?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed the usage in purl processing. I take back my comment then. I think in the future this will be in a better position to refactor: later we'll be moving the CPE generation logic back to where packages are created (near the catalogers) which means that we could remove this from the metadata object at that time.

@westonsteimel westonsteimel merged commit 8e1205f into main Mar 1, 2023
@westonsteimel westonsteimel deleted the fix-apk-ruby-packages branch March 1, 2023 14:58
@kzantow kzantow added the bug Something isn't working label Mar 2, 2023
GijsCalis pushed a commit to GijsCalis/syft that referenced this pull request Feb 19, 2024
This fixes some instances where the improved APK CPE generation
logic caused regressions for older alpine package APK metadata.
It now generates multiple "upstream" candidates with both name
and package type which reduces the amount of duplicated code in
the apk cpe gen logic.  This also improves the handling of stream
version packages, so now we can correctly identify packages such
as ruby3.2-rexml as the rexml ruby gem.

Signed-off-by: Weston Steimel <weston.steimel@anchore.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants