invalid tar header: unknown

[marshallford]

GKE cluster (1.12.7-gke.10) using the node image cos_containerd is failing to run containers built with buildah and pushed to GCR.

Failed to pull image "gcr.io/gke-clusters/testing:latest": rpc error: code = Unknown desc = failed to pull and unpack image "gcr.io/gke-clusters/testing:latest": failed to unpack image on snapshotter overlayfs: failed to extract layer sha256:f1b5933fe4b5f49bbe8258745cf396afe07e625bdab3168e364daf7c956b6b81: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount799987480: archive/tar: invalid tar header: unknown

➜ cat Dockerfile 
FROM alpine
RUN date

Projects/cluster1/test on 🐳 v18.09.5 
➜ buildah version
Version:         1.8.2
Go Version:      go1.12.4
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  v0.7.0-rc2
Git Commit:      e23314b1
Built:           Fri May 10 09:23:56 2019
OS/Arch:         linux/amd64

Projects/cluster1/test on 🐳 v18.09.5 
➜ cat test.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: test
  namespace: default
spec:
  containers:
  - name: test
    image: gcr.io/gke-clusters/testing:latest

Steps to reproduce the issue:

1. Build an image using the Dockerfile above with buildah
2. Push to GCR (or maybe any other container registry?)
3. kubectl apply -f test.yaml on k8s cluster
4. Use kubectl describe to view warning event

Describe the results you received:

Pod fails to spin up.

Describe the results you expected:

I'd expect the pod to pull and run the container.

Output of rpm -q buildah or apt list buildah:

➜ yay -s buildah
2 aur/buildah-git r1330.391a5bea-1 (+1 0.00%) 
    A tool which facilitates building OCI images
1 community/buildah 1.8.2-1 (5.4 MiB 23.6 MiB) (Installed)
    A tool which facilitates building OCI images
==> Packages to install (eg: 1 2 3, 1-3 or ^4)
==> ^C

Output of buildah version:

Version:         1.8.2
Go Version:      go1.12.4
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  v0.7.0-rc2
Git Commit:      e23314b1
Built:           Fri May 10 09:23:56 2019
OS/Arch:         linux/amd64

Output of cat /etc/*release:

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="0;36"
HOME_URL="https://www.archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"

Output of uname -a:

Linux dell 5.0.9-arch1-1-ARCH #1 SMP PREEMPT Sat Apr 20 15:00:46 UTC 2019 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# This file is is the configuration file for all tools
# that use the containers/storage library.
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers.
size = ""

# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
#mount_program = "/usr/bin/fuse-overlayfs"

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev"

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to UIDs/GIDs as they should appear outside of the container, and
# the length of the range of UIDs/GIDs.  Additional mapped sets can be listed
# and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and the a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base.
# device.
# mkfsarg = ""

# use_deferred_removal marks devicemapper block device for deferred removal.
# If the thinpool is in use when the driver attempts to remove it, the driver 
# tells the kernel to remove it as soon as possible. Note this does not free
# up the disk space, use deferred deletion to fully remove the thinpool.
# use_deferred_removal = "True"

# use_deferred_deletion marks thinpool device for deferred deletion.
# If the device is busy when the driver attempts to delete it, the driver
# will attempt to delete device every 30 seconds until successful.
# If the program using the driver exits, the driver will continue attempting
# to cleanup the next time the driver is used. Deferred deletion permanently
# deletes the device and all data stored in device will be lost.
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

# If specified, use OSTree to deduplicate files with the overlay backend
ostree_repo = ""

# Set to skip a PRIVATE bind mount on the storage home directory.  Only supported by
# certain container storage drivers
skip_mount_home = "false"

[marshallford]

Still facing this issue. I tried with podman 1.3.1 which uses vendored buildah 1.8.2 as well.

In the meantime I have also upgraded to 1.13.5-gke.10 but my guess is that this issue is related to the runtime not the k8s version.

Edit: runtime info

$ containerd --version
containerd github.com/containerd/containerd v1.2.6 894b81a4b802e4eb2a91d1ce216b8817763c29fb

[vrothberg]

Hi @marshallford,

thanks a lot for opening the issue.

I can reproduce the error locally using the aforementioned Dockerfile:

FROM alpine
RUN date

... pushing it to a local registry, e.g., localhost:5000/foo:bar

... and then pulling it with containerd via crictl -r /var/run/containerd/containerd.sock pull localhost:5000/foo:bar

I was using containerd v1.2.4 from Fedora 30. However, I also tried with containerd v1.1.7 where I cannot reproduce error. I also cannot reproduce the error when pulling with Docker 18.06.03 or with Podman, Buildah, CRI-O, Skopeo.

I think that there's a regression in containerd v1.2.x but oddly enough, the error does not occur when building the upper Dockerfile with Docker 18.06.03. @estesp, I was browsing containerd issues but did not find a similar one. Do you know of any changes to containerd v1.2.x that could contribute to this issue?

[TomSweeneyRedHat]

@QiWang19 can you take a look at this please?

[marshallford]

@vrothberg Thanks for the reply. Good thinking checking an older containerd version! For reference I also cannot reproduce on Docker 18.09.6.

[marshallford]

Should I cross post this issue in containerd/containerd?

[estesp]

Any chance you could export this image to a tar or push it to a public registry? I don't currently have a system with buildah but would be happy to dig into the invalid tar header error on import/pull.

[TomSweeneyRedHat]

Would a container image work @estesp? https://quay.io/user/buildah

[marshallford]

@estesp https://hub.docker.com/r/marshallford/buildah-1.8.2-containerd-test

Exact commands used to get sample to dockerhub:

sudo buildah bud --storage-driver vfs -t marshallford/buildah-1.8.2-containerd-test:latest .
sudo buildah --storage-driver vfs push marshallford/buildah-1.8.2-containerd-test:latest

@TomSweeneyRedHat, I believe he is referring to a sample image built with buildah to test against containerd.

[estesp]

Strange--I installed 1.2.6 exactly after not being able to reproduce on master and still getting no error on a direct ctr images pull of that image:

$ sudo ctr images pull docker.io/marshallford/buildah-1.8.2-containerd-test:latest
docker.io/marshallford/buildah-1.8.2-containerd-test:latest:                      resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:65ba127390b67e42f54e2d2abfa2505bf86b6c02f7366ba91b7d081ed43bea08: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:1ae208f3bf9c66ed4d14b12aab6652749395a8196c4c80232188501416e02296:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:ae4e710bd73b45c07b66bb6f85ecf12fbe3eadb4049903e5b613f85b30ad70fc:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e7c96db7181be991f19a9fb6975cdbbd73c65f4a2681348e63a141a2192a5f10:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.3 s                                                                    total:  2.6 Mi (2.0 MiB/s)
unpacking linux/amd64 sha256:65ba127390b67e42f54e2d2abfa2505bf86b6c02f7366ba91b7d081ed43bea08...
done

[marshallford]

Perhaps the Dockerhub container registry is modifying the image in a non standard way?

@vrothberg ideas?

[vrothberg]

Docker Hub is transforming the OCI image into a Docker v2 one, while the default registry:2 preserves the OCI image. But even when building a Docker v2 image (i.e., buildah bud --format=docker) and pushing it to a local registry, containerd still doesn't like it:

ctr: failed to extract layer sha256:a91695c2d5c558e126464a6e8229d3062ad78db76ec5ab685f1150d0f5929ca0: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount564865765: unexpected EOF: unknown

I pushed the same image to docker.io/valentinrothberg/image:docker. Containerd fails with the same error trying to pull that image from Docker Hub, so the issue does not seem to be related to OCI images or some other side-effects from Docker Hub.

@nalind, @rhatdan, do you have any suspicion of what could possible go south here? It would be nice to have containerd >= 1.2 and Buildah working together.

[marshallford]

This is still an issue with podman v1.4.0 (vendored Buildah v1.8.3).

[rhatdan]

podman v1.4.2 has Buildah v1.9.0 vendored into it. Please try this out.

[marshallford]

No dice. The testing container image you see referenced was built and pushed with Buildah v1.9.0 (not podman, just normal Buildah). I'm also using imagePullPolicy: Always in the pod yaml to ensure that I am pulling the new image.

➜ k get pods
NAME   READY   STATUS         RESTARTS   AGE
test   0/1     ErrImagePull   0          5s

➜ kubectl describe pod test
Name:               test
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               gke-cluster-1-pool-1-8f173a97-vg0k/10.128.0.20
Start Time:         Wed, 19 Jun 2019 20:28:19 -0500
Labels:             <none>
Annotations:        kubectl.kubernetes.io/last-applied-configuration:
                      {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"test","namespace":"default"},"spec":{"containers":[{"image":"gcr.io/g...
                    kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container test
Status:             Pending
IP:                 10.8.20.4
Containers:
  test:
    Container ID:   
    Image:          gcr.io/gke-clusters/testing:latest
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ErrImagePull
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        100m
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-6psq2 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-6psq2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-6psq2
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age   From                                         Message
  ----     ------     ----  ----                                         -------
  Normal   Scheduled  9s    default-scheduler                            Successfully assigned default/test to gke-cluster-1-pool-1-8f173a97-vg0k
  Normal   Pulling    8s    kubelet, gke-cluster-1-pool-1-8f173a97-vg0k  pulling image "gcr.io/gke-clusters/testing:latest"
  Warning  Failed     8s    kubelet, gke-cluster-1-pool-1-8f173a97-vg0k  Failed to pull image "gcr.io/gke-clusters/testing:latest": rpc error: code = Unknown desc = failed to pull and unpack image "gcr.io/gke-clusters/testing:latest": failed to unpack image on snapshotter overlayfs: failed to extract layer sha256:f1b5933fe4b5f49bbe8258745cf396afe07e625bdab3168e364daf7c956b6b81: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount749608218: archive/tar: invalid tar header: unknown
  Warning  Failed     8s    kubelet, gke-cluster-1-pool-1-8f173a97-vg0k  Error: ErrImagePull
  Normal   BackOff    7s    kubelet, gke-cluster-1-pool-1-8f173a97-vg0k  Back-off pulling image "gcr.io/gke-clusters/testing:latest"
  Warning  Failed     7s    kubelet, gke-cluster-1-pool-1-8f173a97-vg0k  Error: ImagePullBackOff

➜ buildah version
Version:         1.9.0
Go Version:      go1.12.5
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  v0.7.0-rc2
Git Commit:      00eb895d
Built:           Sat Jun 15 14:05:53 2019
OS/Arch:         linux/amd64

[rhatdan]

@vrothberg @mtrmac PTAL

[vrothberg]

I still think it's a regression in containerd. As mentioned above, containerd v1.1.7 has no issue but v1.2.x fails.

[marshallford]

@estesp was pulled into this thread via a cross-post on containerd/containerd. Perhaps the containerd conversation should be moved to that issue?

[mtrmac]

ctr: failed to extract layer sha256:a91695c2d5c558e126464a6e8229d3062ad78db76ec5ab685f1150d0f5929ca0: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount564865765: unexpected EOF: unknown

I pushed the same image to docker.io/valentinrothberg/image:docker.

I can’t see anything wrong with that layer as such (the above is the DiffID, the compressed layer is 6fcbf60dfdaa421d2b6e3d120320a86d8f839b47342f36b60972ed80224f3ec1).

OTOH, it is pretty strange that a RUN date command creates a layer with an empty /etc/resolv.conf file. Why is buildah doing that?!

[rhatdan]

When you run a container, it creates inodes for all of the internal mount points including /etc/resolv.conf.

[estesp]

I can now reproduce with @vrothberg's DockerHub image; let me see if I can understand what the applier method is choking on with that layer. I did look through commit history and nothing stands out as changing behavior here since 1.1.7 and today; but hopefully some debug will help figure it out.

[estesp]

Not anything clear yet; I can capture the actual tar.gz of the layer blob and parse it with a simple Go program as well as tar ztvf from the command line. When that layer is extracted inside containerd, the very first tr.Next() (the tar header reader) fails with EOF and "unknown". Almost like it can't figure out the tar format, but again, outside of containerd using the same Golang version (1.12.6) and a simple program based on archive/tar has no issue with the same content.

[vrothberg]

Thanks a lot for the update, @estesp! @giuseppe, do you have a suspicion?

[giuseppe]

from a quick look, it seems we are generating the wrong manifest.

We should generate application/vnd.oci.image.layer.v1.tar+gzip for the compressed layers, not application/vnd.oci.image.layer.v1.tar

[estesp]

Wow; you're absolutely right:

From delve output of the start of what it thinks is the tar file:

blk: archive/tar.block [31,139,8,0,0,9..

From hexdump {tar.gz from registry}:

0000000 8b1f 0008 0900

Same content; containerd believed the manifest media type and skipped uncompressing the content assuming it really was tar, and not tar.gz. Confirming the file is tar.gz:

root@ubu1804vm:/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256# file 6fcbf60dfdaa421d2b6e3d120320a86d8f839b47342f36b60972ed80224f3ec1
6fcbf60dfdaa421d2b6e3d120320a86d8f839b47342f36b60972ed80224f3ec1: gzip compressed data

[giuseppe]

@mtrmac @vrothberg looks like we need to update the manifest as well to reflect the compression we do in https://github.com/containers/image/blob/master/copy/copy.go#L808.

Maybe this a good chance to discuss containers/image#639. @estesp is that something that should be discussed through OCI? I've seen huge improvements using zstd instead of gzip for compressing the layers.

[mtrmac]

@mtrmac @vrothberg looks like we need to update the manifest as well to reflect the compression we do in https://github.com/containers/image/blob/master/copy/copy.go#L808.

Yes; earlier discussion is in containers/podman#2013 .

Maybe this a good chance to discuss containers/image#639.

I don’t see how that is strictly related, except that the code to have algorithm identifiers could help this (because we need to pass the manifest-schema-specific implementation some information about the compression/decompression that is happening, or maybe even let that implementation decide what to use).

[vrothberg]

Hm ... that's actually not the issue in this particular case. Docker Hub accepts the OCI image but one layer claims to be unencrypted. I'll investigate further. Apologies for the inconvenience.

[vrothberg]

I tracked it down. The manifest types are not correctly updated when a given blob is already present on the registry. For instance, the alpine:latest base layer will incorrectly be claimed to be unencrypted. I'll prepare a fix.

[TomSweeneyRedHat]

Given @vrothberg's last few comments, I'm going to reopen this one to make tracking easier.

[marshallford]

@vrothberg Have you had a chance to look at this? Perhaps a PR you can link to this issue? As always, thanks for all your work.

[vrothberg]

@marshallford, I had time to look into it and to think about how we can address this specific issue but I did not have the time to implement it. I opened containers/image#733 to track it. I can't give an ETA for a fix since there are many things on my table at the moment. Maybe others will pick up containers/image#733. I will close the issue here to move the discussion over to containers/image.

[akospinter]

Hi,

I am facing something similar issue with buildah and k3s cluster.

Steps to reproduce

• Building a simple app with buildah from mcr.microsoft.com/dotnet/core/aspnet:3.1 image.
• Pushing it to Amazon ECR
• Trying to use it in an K3S cluster pulling it from Amazon ECR

Symptoms

• The kubectl get pods command shows 'ErrImagePull' status for the given pod
• kubectl describe command shows the following output (I used "x_image" instead of the real image name)

Failed to pull image "x_image": rpc error: code = Unknown desc = failed to pull and unpack image "x_image": unpack: failed to extract layer sha256:c2adabaecedbda0af72b153c6499a0555f3a769d52370469d8f6bd6328af9b13: mount callback failed on /var/lib/rancher/k3s/agent/containerd/tmpmounts/containerd-mount084900266: archive/tar: invalid tar header: unknown

Build machine info

• Ubuntu 19.10
• buildah version 1.14.8 (image-spec 1.0.1-dev, runtime-spec 1.0.1-dev)

Host machine info

• Ubuntu Server 18.04.4 LTS
• k3s version v1.17.4+k3s1 (3eee8ac3)

Did something go wrong with buildah?

Thanks,
Ákos

[vrothberg]

Hi @akospinter, thanks for reaching out. The issue is being tracked in containers/image#733 which is not yet fixed.

[sdeoras]

I faced the same issue today with podman 2.1.1/fedora-32 and gcr.io on GKE/cos/containerd. Exporting to tarball and pushing via docker worked.

[AntonOfTheWoods]

@vrothberg , this issue has been around for over 1.5 years now, with apparently no end in sight. What do you think about putting it prominently in the documentation, so that people have appropriate expectations about setting up workflows using buildah?

I will submit a documentation patch if you are comfortable having something included.

[vrothberg]

Thanks for reaching out, @AntonOfTheWoods. I am positive that things will improve very soon. @nalind is currently working on a fix in containers/image#1089.

[qhaas]

Seeing this issue when trying to build singularity 3.7 images from docker images from a GitLab Enterprise Edition 13.4 docker registry that were built/pushed using podman 2.0.5 / buildah 1.15.1 / skopeo 1.1.1.

UPDATE: Think mine is caused by the issue linked by @vrothberg , but here is a simple example that replicates what I'm seeing:

$ podman --version
podman version 2.0.5
$ buildah --version
buildah version 1.15.1 (image-spec 1.0.1-dev, runtime-spec 1.0.2-dev)
$ singularity --version
singularity version 3.7.0-1.el8
$ cat test.sh
#!/bin/bash
podman run --name reg -d --rm -p 127.0.0.1:5000:5000 registry:2
podman build -t 127.0.0.1:5000/test:default -f Dockerfile.1 .
BUILDAH_FORMAT=docker podman build -t 127.0.0.1:5000/test:docker1 -f Dockerfile.1 .
podman push --tls-verify=false 127.0.0.1:5000/test:docker1
podman push --tls-verify=false 127.0.0.1:5000/test:default
$ cat Dockerfile.1
FROM centos:7

RUN dd if=/dev/zero of=/opt/foo.bin bs=16M count=1
$ ./test.sh
...
$ podman images
REPOSITORY                  TAG      IMAGE ID      CREATED         SIZE
127.0.0.1:5000/test         docker1  bdd038f6be04  51 seconds ago  228 MB
127.0.0.1:5000/test         default  bdd038f6be04  51 seconds ago  228 MB
docker.io/library/registry  2        678dfa38fcfa  5 days ago      26.8 MB
docker.io/library/centos    7        8652b9f0cb4c  5 weeks ago     212 MB

$ singularity pull --nohttps test-default.sif docker://127.0.0.1:5000/test:default
INFO:    Converting OCI blobs to SIF format
...
Storing signatures
2020/12/22 09:18:13  info unpack layer: sha256:1fe1f6442209bc48384abd225dc53a25d5f987d1ea2059ea18ec9b8e34d9c1de
FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: read next entry: archive/tar: invalid tar header

[Frederik-Baetens]

I see that containers/image#733 is fixed now, has this fix been vendored into buildah yet?

[vrothberg]

Yes, it has been vendored into Buildah v1.9.x.

[slartibart70]

Hi all,
i was trying to use podman-2.2.1 and buildah-1.19.4 created containerimages with containerd-1.3.3 in a completely fresh registry (docker-registry v2).
There we still have this nasty problem, containerd complaining:
archive/tar: invalid tar header: unknown
when it tries to extract a multi-layer image from the registry. All layers being pulled individually work fine, but pulling any image (top level or in-between) of the layered image always gives this complaint for the lowest layer (while trying to extract)

One workaround is to use buildah instead of podman when pushing images to the registry, because we can (as suggested in earlier comments) use the switch --disable-compression when pushing. Something that is not possible with podman push, though

I would say, please take a look again into this problem, maybe?

[rhatdan]

Does podman push --compress=false ... Work? Seems like this needs a three way algorithm. If user sets it, we should pass it through, if not then stick with default.

[vrothberg]

I think the fix predates Podman v2.2.1. Podman 3.0+ definitely ships with it.

[slartibart70]

Does podman push --compress=false ... Work? [...]

Yes indeed, this option does the trick as well

[slartibart70]

I think the fix predates Podman v2.2.1. Podman 3.0+ definitely ships with it.

One question: i was using buildah 1.19.4 to push the image(s). Not Podman.
And still the test with multi-layer images and containerd failed.
How does a newer version of podman address this problem?

[vrothberg]

Buildah v1.19.4 was using github.com/containers/image/v5 v5.10.1 which did not include a follow-up fix (i.e., containers/image@f990c32dd462). In fact, containers/image must be updated for Buildah's 1.19 series.

Thanks for pointing that out. I'll open a PR for Buildah.

[slartibart70]

Hi all,
in the meantime i verified the problem with the following apps:

• containerd-1.4.4-1.fc33.x86_64
• buildah-1.19.6-2.fc33.x86_64
• podman-3.0.1-1.fc33.x86_64

Pushing newly created images with and without compression into a docker registry:v2 and pulling multi-layered images with containerd works as expected.
nice!