Skip to content

Improve support for cataloging nix package relationships #3837

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wagoodman merged 12 commits into from
May 5, 2025
Merged

Improve support for cataloging nix package relationships #3837

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
b76c853
add nix DB cataloger
wagoodman Apr 23, 2025
e8b3887
add derivation path to nix store pkg metadata
wagoodman Apr 28, 2025
6a942d3
go mod tidy
wagoodman Apr 28, 2025
fef39eb
allow for derivation path to be optional
wagoodman Apr 28, 2025
648332c
repin build image and disable syscall filtering
wagoodman Apr 28, 2025
c3aa6d4
bump storage capacity
wagoodman Apr 28, 2025
49bb0cd
track nix derivation details on packages
wagoodman Apr 28, 2025
5b691b1
image fixture should have derivation examples
wagoodman Apr 29, 2025
e11224b
Merge remote-tracking branch 'origin/main' into improve-nix
wagoodman Apr 30, 2025
681a41a
address comments
wagoodman May 2, 2025
944906c
Merge remote-tracking branch 'origin/main' into improve-nix
wagoodman May 2, 2025
6ebaa69
Merge remote-tracking branch 'origin/main' into improve-nix
wagoodman May 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/validations.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,8 @@ jobs:
Unit-Test:
# Note: changing this job name requires making the same update in the .github/workflows/release.yaml pipeline
name: "Unit tests"
runs-on: ubuntu-24.04
# we need more storage than what's on the default runner
runs-on: ubuntu-22.04-4core-16gb
Copy link
Copy Markdown
Contributor

@kzantow kzantow Apr 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this concerning? Is there something else we should do to limit/reduce the overall test resources used?

Copy link
Copy Markdown
Contributor Author

@wagoodman wagoodman Apr 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I tried to limit these test fixtures, but it seems like when you're working with nix and building anything you'll need at least nixos/nix (or install with https://nixos.org/nix/install) which seems to be ~650MB right out of the door with no alterations. I'm adding jq and sqlite on top of it which seems to make the image swell to ~850MB... which is why I make these a multistage image so it's reduced to ~40MB.

But yeah, I'm a little concerned, we are getting up there in terms of disk usage for these fixtures (not including whats pulled down to build them):

$find . | grep stereoscope-fixture | xargs du -ch | tail -1

2.4G    total

I don't think I have an answer within the scope of this PR though.

Copy link
Copy Markdown
Contributor

@kzantow kzantow May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like tests didn't kick off the last commit on this PR, I hope this isn't due to some resource constraints on the larger runners.

steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 #v4.2.2

Expand Down
5 changes: 5 additions & 0 deletions cmd/syft/internal/options/catalog.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ import (
"github.com/anchore/syft/syft/pkg/cataloger/java"
"github.com/anchore/syft/syft/pkg/cataloger/javascript"
"github.com/anchore/syft/syft/pkg/cataloger/kernel"
"github.com/anchore/syft/syft/pkg/cataloger/nix"
"github.com/anchore/syft/syft/pkg/cataloger/python"
"github.com/anchore/syft/syft/source"
)
Expand All @@ -48,6 +49,7 @@ type Catalog struct {
Java javaConfig `yaml:"java" json:"java" mapstructure:"java"`
JavaScript javaScriptConfig `yaml:"javascript" json:"javascript" mapstructure:"javascript"`
LinuxKernel linuxKernelConfig `yaml:"linux-kernel" json:"linux-kernel" mapstructure:"linux-kernel"`
Nix nixConfig `yaml:"nix" json:"nix" mapstructure:"nix"`
Python pythonConfig `yaml:"python" json:"python" mapstructure:"python"`

// configuration for the source (the subject being analyzed)
Expand Down Expand Up @@ -75,6 +77,7 @@ func DefaultCatalog() Catalog {
Package: defaultPackageConfig(),
License: defaultLicenseConfig(),
LinuxKernel: defaultLinuxKernelConfig(),
Nix: defaultNixConfig(),
Dotnet: defaultDotnetConfig(),
Golang: defaultGolangConfig(),
Java: defaultJavaConfig(),
Expand Down Expand Up @@ -193,6 +196,8 @@ func (cfg Catalog) ToPackagesConfig() pkgcataloging.Config {
LinuxKernel: kernel.LinuxKernelCatalogerConfig{
CatalogModules: cfg.LinuxKernel.CatalogModules,
},
Nix: nix.DefaultConfig().
WithCaptureOwnedFiles(cfg.Nix.CaptureOwnedFiles),
Python: python.CatalogerConfig{
GuessUnpinnedRequirements: cfg.Python.GuessUnpinnedRequirements,
},
Expand Down
25 changes: 25 additions & 0 deletions cmd/syft/internal/options/nix.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
package options

import (
"github.com/anchore/clio"
"github.com/anchore/syft/syft/pkg/cataloger/nix"
)

type nixConfig struct {
CaptureOwnedFiles bool `json:"capture-owned-files" yaml:"capture-owned-files" mapstructure:"capture-owned-files"`
}

func defaultNixConfig() nixConfig {
def := nix.DefaultConfig()
return nixConfig{
def.CaptureOwnedFiles,
}
}

var _ interface {
clio.FieldDescriber
} = (*nixConfig)(nil)

func (o *nixConfig) DescribeFields(descriptions clio.FieldDescriptionSet) {
descriptions.Add(&o.CaptureOwnedFiles, `enumerate all files owned by packages found within Nix store paths`)
}
1 change: 1 addition & 0 deletions cmd/syft/internal/test/integration/package_catalogers_represented_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ func TestAllPackageCatalogersReachableInTasks(t *testing.T) {
// not reachable since they are deprecated
"dotnet-portable-executable-cataloger",
"dotnet-deps-cataloger",
"nix-store-cataloger",
"php-pecl-serialized-cataloger",
// not reachable by design
"sbom-cataloger",
Expand Down
7 changes: 3 additions & 4 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ require (
github.com/anchore/go-homedir v0.0.0-20250319154043-c29668562e4d
github.com/anchore/go-logger v0.0.0-20250318195838-07ae343dd722
github.com/anchore/go-macholibre v0.0.0-20220308212642-53e6d0aaf6fb
github.com/anchore/go-sync v0.0.0-20250326131806-4eda43a485b6
github.com/anchore/go-testutils v0.0.0-20200925183923-d5f45b0d3c04
github.com/anchore/go-version v1.2.2-0.20200701162849-18adb9c92b9b
github.com/anchore/packageurl-go v0.1.1-0.20250220190351-d62adb6e1115
Expand Down Expand Up @@ -60,6 +61,7 @@ require (
github.com/magiconair/properties v1.8.10
github.com/mholt/archives v0.1.1
github.com/moby/sys/mountinfo v0.7.2
github.com/nix-community/go-nix v0.0.0-20250101154619-4bdde671e0a1
github.com/olekukonko/tablewriter v0.0.5
github.com/opencontainers/go-digest v1.0.0
github.com/pelletier/go-toml v1.9.5
Expand Down Expand Up @@ -175,6 +177,7 @@ require (
github.com/mattn/go-localereader v0.0.2-0.20220822084749-2491eb6c1c75 // indirect
github.com/mattn/go-runewidth v0.0.16 // indirect
github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d // indirect
github.com/minio/minlz v1.0.0 // indirect
github.com/mitchellh/copystructure v1.2.0 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/mitchellh/go-wordwrap v1.0.1 // indirect
Expand Down Expand Up @@ -256,10 +259,6 @@ require (
modernc.org/memory v1.9.1 // indirect
)

require github.com/anchore/go-sync v0.0.0-20250326131806-4eda43a485b6

require github.com/minio/minlz v1.0.0 // indirect

retract (
v0.53.2
v0.53.1 // Published accidentally with incorrect license in depdencies
Expand Down
4 changes: 4 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -654,6 +654,10 @@ github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/ncruces/go-strftime v0.1.9 h1:bY0MQC28UADQmHmaF5dgpLmImcShSi2kHU9XLdhx/f4=
github.com/ncruces/go-strftime v0.1.9/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
github.com/nix-community/go-nix v0.0.0-20250101154619-4bdde671e0a1 h1:kpt9ZfKcm+EDG4s40hMwE//d5SBgDjUOrITReV2u4aA=
github.com/nix-community/go-nix v0.0.0-20250101154619-4bdde671e0a1/go.mod h1:qgCw4bBKZX8qMgGeEZzGFVT3notl42dBjNqO2jut0M0=
github.com/nsf/jsondiff v0.0.0-20210926074059-1e845ec5d249 h1:NHrXEjTNQY7P0Zfx1aMrNhpgxHmow66XQtm0aQLY0AE=
github.com/nsf/jsondiff v0.0.0-20210926074059-1e845ec5d249/go.mod h1:mpRZBD8SJ55OIICQ3iWH0Yz3cjzA61JdqMLoWXeB2+8=
github.com/nwaples/rardecode v1.1.3 h1:cWCaZwfM5H7nAD6PyEdcVnczzV8i/JtotnyW/dD9lEc=
github.com/nwaples/rardecode v1.1.3/go.mod h1:5DzqNKiOdpKKBH87u8VlvAnPZMXcGRhxWkRpHbbfGS0=
github.com/nwaples/rardecode/v2 v2.1.0 h1:JQl9ZoBPDy+nIZGb1mx8+anfHp/LV3NE2MjMiv0ct/U=
Expand Down
2 changes: 1 addition & 1 deletion internal/constants.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@ package internal
const (
// JSONSchemaVersion is the current schema version output by the JSON encoder
// This is roughly following the "SchemaVer" guidelines for versioning the JSON schema. Please see schema/json/README.md for details on how to increment.
JSONSchemaVersion = "16.0.27"
JSONSchemaVersion = "16.0.28"
)
9 changes: 8 additions & 1 deletion internal/task/package_tasks.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,12 @@ func DefaultPackageTaskFactories() Factories {
pkgcataloging.DirectoryTag, pkgcataloging.InstalledTag, pkgcataloging.ImageTag, pkgcataloging.LanguageTag, Java, Maven,
),
newSimplePackageTaskFactory(java.NewNativeImageCataloger, pkgcataloging.DirectoryTag, pkgcataloging.InstalledTag, pkgcataloging.ImageTag, pkgcataloging.LanguageTag, Java),
newSimplePackageTaskFactory(nix.NewStoreCataloger, pkgcataloging.DirectoryTag, pkgcataloging.InstalledTag, pkgcataloging.ImageTag, pkgcataloging.LanguageTag, "nix"),
newPackageTaskFactory(
func(cfg CatalogingFactoryConfig) pkg.Cataloger {
return nix.NewCataloger(cfg.PackagesConfig.Nix)
},
pkgcataloging.DirectoryTag, pkgcataloging.InstalledTag, pkgcataloging.ImageTag, pkgcataloging.LanguageTag, "nix",
),
newSimplePackageTaskFactory(lua.NewPackageCataloger, pkgcataloging.DirectoryTag, pkgcataloging.InstalledTag, pkgcataloging.ImageTag, pkgcataloging.LanguageTag, "lua"),

// other package catalogers ///////////////////////////////////////////////////////////////////////////
Expand Down Expand Up @@ -167,5 +172,7 @@ func DefaultPackageTaskFactories() Factories {
newSimplePackageTaskFactory(dotnet.NewDotnetDepsCataloger, pkgcataloging.DeprecatedTag), // TODO: remove in syft v2.0
newSimplePackageTaskFactory(dotnet.NewDotnetPortableExecutableCataloger, pkgcataloging.DeprecatedTag), // TODO: remove in syft v2.0
newSimplePackageTaskFactory(php.NewPeclCataloger, pkgcataloging.DeprecatedTag), // TODO: remove in syft v2.0
newSimplePackageTaskFactory(nix.NewStoreCataloger, pkgcataloging.DeprecatedTag), // TODO: remove in syft v2.0

}
}
Loading
Loading