Pushing to an artifactory OCI registry results in 406 Not Acceptable

[lsoica]

Although Artifactory is OCI compliant, https://www.jfrog.com/confluence/display/JFROG/Docker+Registry, following the official docs for distributing policies (https://docs.kubewarden.io/distributing-policies.html) results in:

kwctl push annotated-policy.wasm registry://<registry fqdn>/palindromify-policy:v0.0.1
Error: could not push policy: could not push policy: An unexpected error occured: code=406 Not Acceptable, message='{
  "errors" : [ {
    "status" : 406,
    "message" : "Not Acceptable"
  } ]
}'

How can I troubleshoot further?

LE: Using the same registry with ORAS, works as expected https://github.com/oras-project/oras

oras push  <registry fqdn>/palindromify-policy:v0.0.1 --manifest-config "config.json:application/vnd.wasm.config.v1+json" ./annotated-policy.wasm:application/vnd.wasm.content.layer.v1+wasm
Uploading 1e0afe56fa4e annotated-policy.wasm
Pushed <registry fqdn>/palindromify-policy:v0.0.1
Digest: sha256:<sha>

Then, applying the policy works as expected.

[ereslibre]

Thank you @lsoica for the report. kwctl uses the oci-distribution crate from the krustlet project for pushing OCI artifacts to registries.

It's very possible that the problematic logic happens inside this crate when pushing, but I didn't confirm it yet.

[flavio]

@ereslibre I have an instance of artifactory running in their free tier, I can reproduce the issue.

wasm-to-oci works fine too. That's not a surprise though, since it is built on top of oras

[ereslibre]

@ereslibre I have an instance of artifactory running in their free tier, I can reproduce the issue.

Thanks for confirming, I will look into this.

[ereslibre]

It took a bit of speleology but the problem boils down to the Accept header not being sent when pushing the configuration and the layers. The request is being created on the oci-distribution crate.

Looking at the description on the distribution spec and I think JFrog is having some behavior that is not strictly defined in the spec: https://github.com/opencontainers/distribution-spec/blob/main/spec.md.

On the pulling manifest specification, it is described that the client should include an Accept header. However, on the pushing description, there is no such requirement.

I'm going to ping this issue with the path forward in order to fix the problem.

[ereslibre]

https://github.com/kubewarden/krustlet/compare/oci-distribution-fix-push-to-ghcr...kubewarden:oci-distribution-fix-push-to-jfrog passes through the 406 problem, but then the Location header the server should set is not properly detected. This is a different problem and will be tackled here as well.

[ereslibre]

Related:

• Proposal: add (optional) section on content-negotiation opencontainers/distribution-spec#212 (comment) (section "3.2. "Proactive Negotiation" rfc7231, section 3.4.1")
• https://www.jfrog.com/jira/browse/RTFACT-25069

[ereslibre]

I have opened krustlet/krustlet#670 fixing the relevant issues on the oci-distribution plate.

However, JFrog is not following the spec as expected as described in krustlet/krustlet#670 (comment). I'm reporting that to the JFrog team and reporting back here.

[ereslibre]

Despite I contacted JFrog one month ago regarding this issue I had no response from them. If krustlet/krustlet#670 gets merged, I would call this issue done from our side.

[ereslibre]

Closing. We are going to specify a list of OCI registries we have tried and succeeded.