Podman inspect fail with Error: readObjectStart

[ipanova]

Issue Description

If a registry serves unsigned v1 schema manifest, podman inspect fails with Error: readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

Steps to reproduce the issue

Steps to reproduce the issue

1. Setup a registry from which a docker v1 schema unsigned manifest will be served
2. perform podman pull
3. perfoirm podman inspect
4. podman inspect fails

Describe the results you received

$ podman inspect puffy.example.com/quay-busybox --log-level=debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Called inspect.PersistentPreRunE(podman inspect puffy.example.com/quay-busybox --log-level=debug)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/vagrant/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] systemd-logind: Unknown object '/'.
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/vagrant/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000/containers
DEBU[0000] Using static dir /home/vagrant/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/vagrant/.local/share/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 13
DEBU[0000] Looking up image "puffy.example.com/quay-busybox" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Trying "puffy.example.com/quay-busybox:latest" ...
DEBU[0000] parsed reference into "[overlay@/home/vagrant/.local/share/containers/storage+/run/user/1000/containers]@e3121c769e3948dd4a7e1764f4841d044efcfda47804a5384597b7b117054c4c"
DEBU[0000] Found image "puffy.example.com/quay-busybox" as "puffy.example.com/quay-busybox:latest" in local containers storage
DEBU[0000] Found image "puffy.example.com/quay-busybox" as "puffy.example.com/quay-busybox:latest" in local containers storage ([overlay@/home/vagrant/.local/share/containers/storage+/run/user/1000/containers]@e3121c769e3948dd4a7e1764f4841d044efcfda47804a5384597b7b117054c4c)
DEBU[0000] Inspecting image e3121c769e3948dd4a7e1764f4841d044efcfda47804a5384597b7b117054c4c
Error: readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
DEBU[0000] Shutting down engines

Describe the results you expected

podman inspect should succeed

podman info output

$ podman version
Client:       Podman Engine
Version:      4.4.1
API Version:  4.4.1
Go Version:   go1.18.10
Built:        Fri Feb 17 10:31:22 2023
OS/Arch:      linux/amd64

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

No

Additional environment details

No response

Additional information

I have setup registry that serves 2 repos. 1 repo serves unsigned v1 schema manifest. 2nd repo serves signed v2 schema manifest.
Podman pull works on both, podman inspect fails on unsigned v1 schema manifest.

$ curl -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws' 'https://puffy.example.com/v2/busybox/manifests/latest'  -X GET -L 
{
   "schemaVersion": 1,
   "name": "library/busybox",
   "tag": "latest",
   "architecture": "amd64",
   "fsLayers": [
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:3f4d90098f5b5a6f6a76e9d217da85aa39b2081e30fa1f7d287138d6e7bf0ad7"
      }
   ],
   "history": [
      {
         "v1Compatibility": "{\"architecture\":\"amd64\",\"config\":{\"Hostname\":\"\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"sh\"],\"Image\":\"sha256:919de79e94ce464098cec8a4796f27aa2d0c93985e27f6a157c8bbb557568013\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":null},\"container\":\"adfa050237a5fcf9f1e5ad6351b3665e4b99e0c12f33bdd3535b8fb001f9dd2e\",\"container_config\":{\"Hostname\":\"adfa050237a5\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) \",\"CMD [\\\"sh\\\"]\"],\"Image\":\"sha256:919de79e94ce464098cec8a4796f27aa2d0c93985e27f6a157c8bbb557568013\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":{}},\"created\":\"2023-07-18T23:19:33.655005962Z\",\"docker_version\":\"20.10.23\",\"id\":\"5e0da803ecd276c43424f0a0636b515ee875bb2c4f9c0a08c9d0248e26e1d8d0\",\"os\":\"linux\",\"parent\":\"9bafc02a2a5cf79ee87025155541453614fdfe65ea660f42ace95ff9a4451782\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"9bafc02a2a5cf79ee87025155541453614fdfe65ea660f42ace95ff9a4451782\",\"created\":\"2023-07-18T23:19:33.538571854Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ADD file:7e9002edaafd4e4579b65c8f0aaabde1aeb7fd3f8d95579f7fd3443cef785fd1 in / \"]}}"
      }
   ],
   "signatures": [
      {
         "header": {
            "jwk": {
               "crv": "P-256",
               "kid": "BY72:IORR:SGVI:BNPT:GHOW:TEXZ:JI5I:KEIQ:OQY7:V2E6:QMJR:D4SO",
               "kty": "EC",
               "x": "anLrzIzbOr9F89fId2uyoSN0RkaKgq76slu6YErW4qE",
               "y": "o67sigOk8c06qF-sPE7ysk9Z5GGwJqOAaZUZpYBCko8"
            },
            "alg": "ES256"
         },
         "signature": "2_jmJQermEyoCoIEJESs0AYBCySEcSHtnbTUY-4vJTY4HCLgke5_pCZ_VcQElUwe0Rn8LTh0p5VmPLnBVwQfZg",
         "protected": "eyJmb3JtYXRMZW5ndGgiOjIwODgsImZvcm1hdFRhaWwiOiJDbjAiLCJ0aW1lIjoiMjAyMy0wOS0yNlQxNDoyNzozNFoifQ"
      }
   ]

 podman inspect puffy.example.com/busybox | jq .[].ManifestType
"application/vnd.docker.distribution.manifest.v1+prettyjws"

VS:

$ curl -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws' 'https://puffy.example.com/v2/quay-busybox/manifests/latest'  -X GET -L 
{
   "tag": "latest",
   "name": "quay/busybox",
   "architecture": "amd64",
   "schemaVersion": 1,
   "history": [
      {
         "v1Compatibility": "{\"architecture\": \"amd64\", \"config\": {\"Hostname\": \"\", \"Domainname\": \"\", \"User\": \"\", \"AttachStdin\": false, \"AttachStdout\": false, \"AttachStderr\": false, \"Tty\": false, \"OpenStdin\": false, \"StdinOnce\": false, \"Env\": [\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\", \"TERM=xterm\", \"container=podman\", \"HOSTNAME=\"], \"Cmd\": [\"/bin/sh\"], \"Image\": \"\", \"Volumes\": null, \"WorkingDir\": \"/\", \"Entrypoint\": null, \"OnBuild\": null, \"Labels\": null}, \"container_config\": {\"Hostname\": \"\", \"Domainname\": \"\", \"User\": \"\", \"AttachStdin\": false, \"AttachStdout\": false, \"AttachStderr\": false, \"Tty\": false, \"OpenStdin\": false, \"StdinOnce\": false, \"Env\": null, \"Cmd\": null, \"Image\": \"\", \"Volumes\": null, \"WorkingDir\": \"\", \"Entrypoint\": null, \"OnBuild\": null, \"Labels\": null}, \"created\": \"2020-09-04T16:08:39.002329276Z\", \"os\": \"linux\", \"id\": \"1adad9945f604bf46b195bc5bbbd3a2d2d564e884cb8fca971dcdfff9c41ce32\", \"parent\": \"86e1debbda555ef2ce35d724d2ad40e9c674a79f9ca2d264563949c17b09d4de\", \"Size\": 368}"
      },
      {
         "v1Compatibility": "{\"id\": \"86e1debbda555ef2ce35d724d2ad40e9c674a79f9ca2d264563949c17b09d4de\", \"parent\": \"a86a07cf996fd04a5043d699fcf169b931469d486416f76847e18e01dfe267e5\", \"created\": \"2020-09-04T15:06:15.059006882Z\", \"comment\": \"Updated at 2020-09-04 15:06:14 +0000\", \"throwaway\": true, \"container_config\": {\"Cmd\": [\"sh echo \\\"2020-09-04 15:06:14 +0000\\\" > foo\"]}, \"Size\": 32}"
      },
      {
         "v1Compatibility": "{\"id\": \"a86a07cf996fd04a5043d699fcf169b931469d486416f76847e18e01dfe267e5\", \"parent\": \"aa753ca5664ad132bf1c89b79324f4c90db69757e97d24becfbba591ef074c31\", \"created\": \"2020-09-01T00:36:18.335938403Z\", \"throwaway\": true, \"container_config\": {\"Cmd\": [\"/bin/sh -c #(nop)  CMD [\\\"sh\\\"]\"]}, \"Size\": 32}"
      },
      {
         "v1Compatibility": "{\"id\": \"aa753ca5664ad132bf1c89b79324f4c90db69757e97d24becfbba591ef074c31\", \"created\": \"2020-09-01T00:36:18.002487153Z\", \"container_config\": {\"Cmd\": [\"/bin/sh -c #(nop) ADD file:4e5169fa630e0afede3b7db9a6d0ca063df3fe69cc2873a6c50e9801d61f563f in / \"]}, \"Size\": 763774}"
      }
   ],
   "fsLayers": [
      {
         "blobSum": "sha256:ee780d08a5b4de5192a526d422987f451d9a065e6da42aefe8c3b20023a250c7"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:9c075fe2c773108d2fe2c18ea170548b0ee30ef4e5e072d746e3f934e788b734"
      }
   ]
}
$podman inspect puffy.example.com/quay-busybox | jq .[].ManifestType
Error: readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

Suspected offending line is containers/common@c53283f#diff-fc461513b322becd13becb226b44b4164a4913c732302e079863d85571686b1eL111

After talking to @mtrmac here is current thinking:

(Guess being that https://github.com/containers/common/blob/e18cda8d7750746031de3d3d9305df059ac5f0ae/libimage/inspect.go#L186 tries to unmarshal a config object which does not exist in schema1, i.e. an empty byte array. That would also be consistent with the "unexpected end of JSON input" text.)
It is two bugs:

• The line above is incorrectly looking for a config in a schema1 image
• The case statement is incorrectly only matching for the unsigned schema1 MIME type, not the signed MIME type.
  Now, one nit in the above (or, alternatively, a damning note), is that the code has been in there for 2 years. So it’s both that we didn’t notice for 2 years, and that this is manifesting on an upgrade from a pretty old Podman.

Yes, If downgrading to podman 3, the inspect command works.
Here is the BZ that impacts customers https://bugzilla.redhat.com/show_bug.cgi?id=2240252 Current workaround is to edit locally stored manifest.json and add to it signatures: []

[ipanova]

As an additional note, skopeo inspect seems to cope with both signed and unsigned v1 schema manifest.

$ skopeo  inspect docker://puffy.example.com/quay-busybox | jq .Tag
"latest"
$ skopeo  inspect docker://puffy.example.com/busybox | jq .Tag
"latest"

[rhatdan]

Interested in opening a PR?

[vincentywdeng]

@ipanova
I'm very new to both podman and golang, if this is not urgent, I can work on it.
Some questions

• How to configure registry v1 ? The docker hub currenlty only have registry:2
• It seems that the bug is for common project, given that, should the PR created against that project?

[mtrmac]

@vincentywdeng

• “schema 1” is a data format, not directly related to the registry version. registry:2 should work for this.
• Yes, this should be fixed in containers/common .

[vincentywdeng]

Thanks @mtrmac , it seems that we need older docker/podman to have registry generating schema1 manifest, do you know which docker/podman version to choose?

[mtrmac]

I’d use skopeo copy --format … $source dir:$tmp to get a starting point, then manually edit the manifest in $tmp, and publish to a registry using skopeo copy --preserve-digests dir:$tmp docker://…. That gives you full and direct control over contents of the manifest.

(I also suspect the c/common code can be run and tested without a registry at all, maybe by writing that manifest to c/storage, or if the code is well-isolated, by providing a types.ImageReference that points to such a manually-crafted dir:. But if you want to use a registry to be fully realistic, that’s also valuable.)

[vincentywdeng]

I tried skopeo copy and unable to generate schema without signature

• skopeo copy --format … $source dir:$tmp , then edit the manifest.json to remove signature and then skopeo copy --preserve-digests dir:$tmp docker://…
• skopeo copy --remove-signatures --format v2s1 docker://quay.io/skopeo/stable:latest docker://localhost:5000/ywdeng/skopeo

When using curl to get manifest, both return v1 schema with signature. I'll check out c/common code then

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[mtrmac]

containers/common#1748 will fix this.