More memory efficient mimetype detection #267

wagoodman · 2024-07-10T17:22:13Z

This PR incorporates gabriel-vasile/mimetype#553 via the temporary fork. Before this improvement this was the relative performance for a sample image via syft:

Specifically the mimetype detection was allocating 1.1GB of memory:

With the upstream change incorporated we're using ~800 less MB of memory (reduced 1.1GB to ~340MB) to do the same work:

Note the figures are for total allocated space and not inuse memory.

Partially addresses #233

Signed-off-by: Alex Goodman <[email protected]>

github-actions · 2024-07-10T17:25:30Z

Benchmark Test Results

Benchmark results from the latest changes vs base branch

make .tool/task
make[1]: Entering directory '/home/runner/work/stereoscope/stereoscope'
make[1]: Leaving directory '/home/runner/work/stereoscope/stereoscope'
.tool/task show-benchstat
?   	github.com/anchore/stereoscope	[no test files]
?   	github.com/anchore/stereoscope/examples	[no test files]
PASS
ok  	github.com/anchore/stereoscope/internal	0.003s
?   	github.com/anchore/stereoscope/internal/bus	[no test files]
PASS
ok  	github.com/anchore/stereoscope/internal/containerd	0.008s
PASS
ok  	github.com/anchore/stereoscope/internal/docker	0.005s
?   	github.com/anchore/stereoscope/internal/log	[no test files]
PASS
ok  	github.com/anchore/stereoscope/internal/podman	0.005s
?   	github.com/anchore/stereoscope/pkg/event	[no test files]
?   	github.com/anchore/stereoscope/pkg/event/parsers	[no test files]
goos: linux
goarch: amd64
pkg: github.com/anchore/stereoscope/pkg/file
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkTarIndex-4   	   33375	     35777 ns/op	    5699 B/op	      93 allocs/op
BenchmarkTarIndex-4   	   33187	     35743 ns/op	    5699 B/op	      93 allocs/op
BenchmarkTarIndex-4   	   33516	     35828 ns/op	    5703 B/op	      93 allocs/op
BenchmarkTarIndex-4   	   33344	     35889 ns/op	    5699 B/op	      93 allocs/op
BenchmarkTarIndex-4   	   33582	     35737 ns/op	    5698 B/op	      93 allocs/op
BenchmarkTarIndex-4   	   33516	     35827 ns/op	    5698 B/op	      93 allocs/op
BenchmarkTarIndex-4   	   33442	     36007 ns/op	    5700 B/op	      93 allocs/op
PASS
ok  	github.com/anchore/stereoscope/pkg/file	10.942s
PASS
ok  	github.com/anchore/stereoscope/pkg/filetree	0.005s
?   	github.com/anchore/stereoscope/pkg/filetree/filenode	[no test files]
PASS
ok  	github.com/anchore/stereoscope/pkg/image	0.006s
PASS
ok  	github.com/anchore/stereoscope/pkg/image/containerd	0.009s
PASS
ok  	github.com/anchore/stereoscope/pkg/image/docker	0.006s
PASS
ok  	github.com/anchore/stereoscope/pkg/image/oci	0.006s
PASS
ok  	github.com/anchore/stereoscope/pkg/image/oci/credhelpers	0.006s
?   	github.com/anchore/stereoscope/pkg/image/podman	[no test files]
PASS
ok  	github.com/anchore/stereoscope/pkg/image/sif	0.005s
?   	github.com/anchore/stereoscope/pkg/imagetest	[no test files]
PASS
ok  	github.com/anchore/stereoscope/pkg/tree	0.003s
PASS
ok  	github.com/anchore/stereoscope/pkg/tree/node	0.004s
goos: linux
goarch: amd64
pkg: github.com/anchore/stereoscope/test/integration
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkSimpleImage_GetImage/docker-archive-4 	     987	   1208382 ns/op	  290047 B/op	    2472 allocs/op
BenchmarkSimpleImage_GetImage/docker-archive-4 	     990	   1215180 ns/op	  289720 B/op	    2472 allocs/op
BenchmarkSimpleImage_GetImage/docker-archive-4 	     987	   1246097 ns/op	  289678 B/op	    2471 allocs/op
BenchmarkSimpleImage_GetImage/docker-archive-4 	     991	   1228771 ns/op	  289523 B/op	    2471 allocs/op
BenchmarkSimpleImage_GetImage/docker-archive-4 	     988	   1195063 ns/op	  289329 B/op	    2471 allocs/op
BenchmarkSimpleImage_GetImage/docker-archive-4 	     994	   1270525 ns/op	  289369 B/op	    2471 allocs/op
BenchmarkSimpleImage_GetImage/docker-archive-4 	     985	   1279316 ns/op	  289476 B/op	    2471 allocs/op
BenchmarkSimpleImage_GetImage/podman-4         	      63	  18281529 ns/op	  410106 B/op	    2676 allocs/op
BenchmarkSimpleImage_GetImage/podman-4         	      58	  19324548 ns/op	  409704 B/op	    2676 allocs/op
BenchmarkSimpleImage_GetImage/podman-4         	      63	  19146959 ns/op	  410186 B/op	    2676 allocs/op
BenchmarkSimpleImage_GetImage/podman-4         	      61	  19139812 ns/op	  409459 B/op	    2674 allocs/op
BenchmarkSimpleImage_GetImage/podman-4         	      62	  18680640 ns/op	  409708 B/op	    2675 allocs/op
BenchmarkSimpleImage_GetImage/podman-4         	      64	  18809938 ns/op	  409431 B/op	    2675 allocs/op
BenchmarkSimpleImage_GetImage/podman-4         	      64	  18849104 ns/op	  409408 B/op	    2676 allocs/op
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 345B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load build context
#3 transferring context: 209B done
#3 DONE 0.0s

#4 [1/3] ADD file-1.txt /somefile-1.txt
#4 CACHED

#5 [2/3] ADD file-2.txt /somefile-2.txt
#5 CACHED

#6 [3/3] ADD target /
#6 CACHED

#7 exporting to image
#7 exporting layers done
#7 writing image sha256:fd49a7e75819d889d09a10c48dfa50867cba5b0e5620be56c53a7b708ed631ce done
#7 naming to docker.io/library/stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 done
#7 naming to docker.io/library/stereoscope-fixture-image-simple:latest done
#7 DONE 0.0s
ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied"
--- FAIL: BenchmarkSimpleImage_GetImage
    image_fixtures.go:193: using existing image tar: 'test-fixtures/cache/stereoscope-fixture-image-simple-04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7.tar' (size: 22528, modified: 2024-07-10 17:23:50.048392572 +0000 UTC, mode: -rw-r--r--)
    image_fixtures.go:241: Build docker image: name="stereoscope-fixture-image-simple" tag="04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7"
    image_fixtures.go:291: saveImage running: docker image save stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7
    image_fixtures.go:286: 
        	Error Trace:	/home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:286
        	            				/home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:162
        	            				/home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:152
        	            				/home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:33
        	            				/home/runner/work/stereoscope/stereoscope/test/integration/fixture_image_simple_test.go:163
        	Error:      	Received unexpected error:
        	            	exit status 1
        	Test:       	BenchmarkSimpleImage_GetImage
        	Messages:   	could not import docker image to containerd (shell out)
BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4         	   53836	     22263 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4         	   53839	     22129 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4         	   53907	     22145 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4         	   54104	     22112 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4         	   54279	     22106 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4         	   54148	     22055 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/docker-archive-4         	   53942	     22068 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/podman-4                 	   54199	     22043 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/podman-4                 	   54394	     22055 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/podman-4                 	   54304	     22115 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/podman-4                 	   53862	     22155 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/podman-4                 	   54034	     22006 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/podman-4                 	   54447	     22046 ns/op	    2712 B/op	      21 allocs/op
BenchmarkSimpleImage_FetchSquashedContents/podman-4                 	   54042	     21974 ns/op	    2712 B/op	      21 allocs/op
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 345B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load build context
#3 transferring context: 209B done
#3 DONE 0.0s

#4 [1/3] ADD file-1.txt /somefile-1.txt
#4 CACHED

#5 [2/3] ADD file-2.txt /somefile-2.txt
#5 CACHED

#6 [3/3] ADD target /
#6 CACHED

#7 exporting to image
#7 exporting layers done
#7 writing image sha256:fd49a7e75819d889d09a10c48dfa50867cba5b0e5620be56c53a7b708ed631ce done
#7 naming to docker.io/library/stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7 done
#7 naming to docker.io/library/stereoscope-fixture-image-simple:latest done
#7 DONE 0.0s
ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied"
--- FAIL: BenchmarkSimpleImage_FetchSquashedContents
    image_fixtures.go:193: using existing image tar: 'test-fixtures/cache/stereoscope-fixture-image-simple-04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7.tar' (size: 22528, modified: 2024-07-10 17:23:50.048392572 +0000 UTC, mode: -rw-r--r--)
    image_fixtures.go:241: Build docker image: name="stereoscope-fixture-image-simple" tag="04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7"
    image_fixtures.go:291: saveImage running: docker image save stereoscope-fixture-image-simple:04e16e44161c8888a1a963720fd0443cbf7eef8101434c431de8725cd98cc9f7
    image_fixtures.go:286: 
        	Error Trace:	/home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:286
        	            				/home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:162
        	            				/home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:152
        	            				/home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:33
        	            				/home/runner/work/stereoscope/stereoscope/pkg/imagetest/image_fixtures.go:64
        	            				/home/runner/work/stereoscope/stereoscope/test/integration/fixture_image_simple_test.go:189
        	Error:      	Received unexpected error:
        	            	exit status 1
        	Test:       	BenchmarkSimpleImage_FetchSquashedContents
        	Messages:   	could not import docker image to containerd (shell out)
FAIL
exit status 1
FAIL	github.com/anchore/stereoscope/test/integration	40.181s
?   	github.com/anchore/stereoscope/test/integration/test-fixtures/registry	[no test files]
FAIL
goos: linux
goarch: amd64
pkg: github.com/anchore/stereoscope/pkg/file
cpu: AMD EPYC 7763 64-Core Processor                
ctr: 
           │ .tmp/benchmark-fb32042.txt │
           │           sec/op           │
TarIndex-4                  35.83µ ± 1%

           │ .tmp/benchmark-fb32042.txt │
           │            B/op            │
TarIndex-4                 5.565Ki ± 0%

           │ .tmp/benchmark-fb32042.txt │
           │         allocs/op          │
TarIndex-4                   93.00 ± 0%

pkg: github.com/anchore/stereoscope/test/integration
                                      │ .tmp/benchmark-fb32042.txt │
                                      │           sec/op           │
SimpleImage_GetImage/docker-archive-4                  1.229m ± 4%
SimpleImage_GetImage/podman-4                          18.85m ± 3%
geomean                                                4.813m

                                      │ .tmp/benchmark-fb32042.txt │
                                      │            B/op            │
SimpleImage_GetImage/docker-archive-4                 282.7Ki ± 0%
SimpleImage_GetImage/podman-4                         400.1Ki ± 0%
geomean                                               336.3Ki

                                      │ .tmp/benchmark-fb32042.txt │
                                      │         allocs/op          │
SimpleImage_GetImage/docker-archive-4                  2.471k ± 0%
SimpleImage_GetImage/podman-4                          2.676k ± 0%
geomean                                                2.571k

ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied"
                                                   │ .tmp/benchmark-fb32042.txt │
                                                   │           sec/op           │
SimpleImage_FetchSquashedContents/docker-archive-4                  22.11µ ± 1%
SimpleImage_FetchSquashedContents/podman-4                          22.05µ ± 0%
geomean                                                             22.08µ

                                                   │ .tmp/benchmark-fb32042.txt │
                                                   │            B/op            │
SimpleImage_FetchSquashedContents/docker-archive-4                 2.648Ki ± 0%
SimpleImage_FetchSquashedContents/podman-4                         2.648Ki ± 0%
geomean                                                            2.648Ki

                                                   │ .tmp/benchmark-fb32042.txt │
                                                   │         allocs/op          │
SimpleImage_FetchSquashedContents/docker-archive-4                   21.00 ± 0%
SimpleImage_FetchSquashedContents/podman-4                           21.00 ± 0%
geomean                                                              21.00
goos: linux
goarch: amd64
pkg: github.com/anchore/stereoscope/pkg/file
cpu: AMD EPYC 7763 64-Core Processor                
ctr: 
           │ .tmp/benchmark-fb32042.txt │
           │           sec/op           │
TarIndex-4                  35.83µ ± 1%

           │ .tmp/benchmark-fb32042.txt │
           │            B/op            │
TarIndex-4                 5.565Ki ± 0%

           │ .tmp/benchmark-fb32042.txt │
           │         allocs/op          │
TarIndex-4                   93.00 ± 0%

pkg: github.com/anchore/stereoscope/test/integration
                                      │ .tmp/benchmark-fb32042.txt │
                                      │           sec/op           │
SimpleImage_GetImage/docker-archive-4                  1.229m ± 4%
SimpleImage_GetImage/podman-4                          18.85m ± 3%
geomean                                                4.813m

                                      │ .tmp/benchmark-fb32042.txt │
                                      │            B/op            │
SimpleImage_GetImage/docker-archive-4                 282.7Ki ± 0%
SimpleImage_GetImage/podman-4                         400.1Ki ± 0%
geomean                                               336.3Ki

                                      │ .tmp/benchmark-fb32042.txt │
                                      │         allocs/op          │
SimpleImage_GetImage/docker-archive-4                  2.471k ± 0%
SimpleImage_GetImage/podman-4                          2.676k ± 0%
geomean                                                2.571k

ctr: failed to dial "/run/containerd/containerd.sock": connection error: desc = "transport: error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied"
                                                   │ .tmp/benchmark-fb32042.txt │
                                                   │           sec/op           │
SimpleImage_FetchSquashedContents/docker-archive-4                  22.11µ ± 1%
SimpleImage_FetchSquashedContents/podman-4                          22.05µ ± 0%
geomean                                                             22.08µ

                                                   │ .tmp/benchmark-fb32042.txt │
                                                   │            B/op            │
SimpleImage_FetchSquashedContents/docker-archive-4                 2.648Ki ± 0%
SimpleImage_FetchSquashedContents/podman-4                         2.648Ki ± 0%
geomean                                                            2.648Ki

                                                   │ .tmp/benchmark-fb32042.txt │
                                                   │         allocs/op          │
SimpleImage_FetchSquashedContents/docker-archive-4                   21.00 ± 0%
SimpleImage_FetchSquashedContents/podman-4                           21.00 ± 0%
geomean                                                              21.00

more efficient mimetype detection

cd486cc

Signed-off-by: Alex Goodman <[email protected]>

wagoodman added the performance label Jul 10, 2024

wagoodman requested a review from a team July 10, 2024 17:22

kzantow approved these changes Jul 10, 2024

View reviewed changes

wagoodman merged commit 27b66b7 into main Jul 10, 2024
7 checks passed

wagoodman deleted the improve-mimetype-detect-mem-use branch July 10, 2024 17:51

wagoodman mentioned this pull request Jul 10, 2024

Forks in use #266

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More memory efficient mimetype detection #267

More memory efficient mimetype detection #267

wagoodman commented Jul 10, 2024 •

edited

Loading

github-actions bot commented Jul 10, 2024

More memory efficient mimetype detection #267

More memory efficient mimetype detection #267

Conversation

wagoodman commented Jul 10, 2024 • edited Loading

github-actions bot commented Jul 10, 2024

Benchmark Test Results

wagoodman commented Jul 10, 2024 •

edited

Loading