Add append-file(s) command to crane

[bouk]

It would be useful if we could have a command that could do something like:

$ crane append-file -t gcr.io/... -b busybox --file=/usr/sbin/script.sh - << EOF
#!/usr/bin/env sh
echo "Hi there!"
EOF

Or imitate ko by doing:

$ go build -o app
$ crane append-file -t gcr.io/... -b gcr.io/distroless/static:nonroot --directory=/usr/bin app

And end up with a container image that contains the app binary at /usr/bin/app.

Basically roll some basic tarring functions into crane itself

[imjasonh]

I also recently found myself wanting something like this 😄

As a workaround, you can pipe tar's output to crane's stdin:

tar -zcf - some_file_or_directory | crane append -t gcr.io/... -f /dev/fd/0                                                                                                                                                  

The /dev/fd/0 thing is a bit baroque, most things use -f - to mean "read from stdin" (e.g., kubectl apply -f -). We could support this in crane append if it would help.

@jonjohnsonjr wdyt?

[jonjohnsonjr]

I believe you want something like this:

$ crane append -f <(tar -c /usr/bin -f -) -b busybox -t $IMAGE

Tar has flags for Modifying File and Member Names that can help reposition things within the archive, if it needs to go in a particular place, see --transform.

If that doesn't hit your use-case, let me know. I'm sure we can figure something out.

Basically roll some basic tarring functions into crane itself

I am wary of doing this because we're going to end up inventing a non-standard syntax for something that tar already does. We could implement a very small subset of tar's functionality (if you just care about the filename), but if we do this, people will ask about how to change permissions and stuff, and I'd rather not bake that into a command that also appends things.

I think we are missing some functionality in crane, which is why I feel bad about saying "no" to this. I'll file a separate issue for that.

The /dev/fd/0 thing is a bit baroque, most things use -f - to mean "read from stdin" (e.g., kubectl apply -f -). We could support this in crane append if it would help.

Process substitution is a nice alternative to that.

[jonjohnsonjr]

I'll file a separate issue for that.

See #934

[bouk]

Yeah I thought about using tar, but the problem is that GNU tar and whatever you get on macOS are quite different, so it would be nice if I didn't have to rely on specific OS tooling.

[jonjohnsonjr]

the problem is that GNU tar and whatever you get on macOS are quite different

Ah, sorry, I am unfortunately ignorant here as I only have linux devices currently. What kinds of differences are there, and can we paper over them without the abstraction being too leaky? Does windows also suffer from this?

One thing that has me worried is docker's package for this: https://github.com/moby/moby/tree/d5209b29b9777e0b9713d87847a5dc8ce9d93da6/pkg/archive

There seems to be a lot of platform-specific stuff, and I'd like to avoid re-implementing all of that or even just importing it because the dependency graph in docker/docker can be pretty scary.

One thing that feels kind of interesting to me would be exposing something like crane tar to help you construct a tarball in a platform-agnostic way (though I'm not sure if that is even possible) by allowing you to set anything in https://pkg.go.dev/archive/tar#Header, reading the file content from stdin, and writing the file content to stdout as a tar archive. A single-file layer is a bit weird, but I think that might satisfy some of your needs?

$ crane tar --name "/usr/sbin/script.sh" --mode 644 << EOF
#!/usr/bin/env sh
echo "Hi there!"
EOF > something.tar

Maybe we could compose this with #934

If we wanted multi-file archives maybe we could do something very silly like:

$ crane tar --name "/usr/bin/foo" < foo.sh | crane tar --append --name "/usr/bin/bar" < bar.sh | crane upload -z $REPO

... which would upload a gzipped tarball to $REPO that contains foo.sh and bar.sh under /usr/bin.

Tarring up a whole directory seems like a whole different beast, and I'm not sure if that's going to be any easier than just working around the differences in tar manually?

crane tar would also be a reasonable way to have sugar for creating whiteout files? Though it's not too hard to just:

$ crane tar --name "/usr/bin/.wh.foo" < /dev/null

🤷

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.