'MANIFEST_INVALID' when uploading to Artifactory #534
Comments
|
Hey @Joeskyyy thanks for the report, we'll take a look and see what's wrong with our Artifactory interactions. |
|
I got the same thing, when I tried to push JIB pushed image from local registry:2 (https://docs.docker.com/registry/deploying/#run-a-local-registry) to Openshift (Red Hat Container Registry). When checking with API calls: Centos -image from Docker hub just pushed to local by Docker (renaming to keep thing clearer): |
|
Hmm, this might be that there is something in our manifest format that Openshift and Artifactory finds to be invalid. I wish those registries could give more detail as to why rather than just giving back "manifest invalid". We will look more into this and try to fix this as a high priority issue. @GoogleContainerTools/java-tools |
The manifest of the CentOS image, which works fine with OpenShift, seems to be scheme1 (although I could be totally out of place). The OpenShift doc says
Just a wild guess from the logs above and the following OpenShift doc: I guess we are generating schema2 manifests and the user's OpenShift repository cannot store it? Maybe the repository is trying to convert it to schema1 on the fly? |
|
/cc @mfojtik |
|
@Joeskyyy Can you try creating new local docker registry in Artifactory with V1 api support and try pushing image again. You will see flag as follow when you create new local docker registry in Artifactory.
|
|
OpenShift registry can store manifest v2 schema, even not configured by default can easily be enabled: https://docs.openshift.com/container-platform/3.9/install_config/registry/extended_registry_configuration.html#docker-registry-configuration-reference-middleware, and should accept them since 3.6. @Hi-Fi can you verify what version of OpenShift were you testing against? The reason is very well explained here: https://docs.openshift.com/container-platform/3.9/install_config/registry/extended_registry_configuration.html#middleware-repository-acceptschema2 and it's basically to support docker versions previous to Docker 1.10. I would think that nowadays there are very few (close to none) users with older versions of docker cli, but it would be good to support both schema1 and schema2 in the plugin, defaulting to schema2 but with the ability to override the value, and obviously a nicer error message explaining to the user what to do in case he get's a response from the registry that schema2 is not supported. Edited by coollog: @jorgemoralespou Thanks for the suggestion! I'll file an issue for this: #601 |
|
@jorgemoralespou: Openshift we have is: |
|
/cc @bparees Can you provide any more insight into what's going on here? |
|
@jorgemoralespou the registry logs might provide more insight. But the theory that a v2 schema is being pushed and the registry is rejecting it seems plausible (I think the normal docker client managed to recognize that situation and fallback to schema 1). |
|
Hi all, fyi I created a ticket for jfrog Artifactory trying to get more info as to what is causing this issue: |
|
For the Artifactory issue: from the Artifactory server log in the above Artifactory service ticket (https://www.jfrog.com/jira/browse/RTFACT-17134) created by @mzagar, I am suspecting it's the Artifactory JSON manifest parser that is breaking. Also from the log, Artifactory does seem like it's capable of handling manifest schema 2. |
|
Seems that files are uploaded OK to Artifactory, so only Manifest error is preventing containers to be usable from there:
|
|
With Openshift and Artifactory it seems that target/jib-cache/metadata-v2.json -file is almost empty content being: When doing successful against local registry:2 file has layer information (including files in each layer), but no e.g. schemaVersion, mediaType or config that are present in manifest that's downloaded from Artifactory item. |
|
Made some tcpdumping to see what's different. Results below (haven't yet figured out what is the thing causing the issue). Also not sure if (without clearing up those messages) the put should work twice or long after the other things are uploaded. But at least some differences can be seen from those. |
|
If only removing "+json" from MANIFEST_MEDIA_TYPE error message is: |
|
@Hi-Fi as I pointed out earlier, this looks like a bug in Artifactory : #534 (comment) |
|
@Hi-Fi but thanks for the tcpdump. It is indeed very interesting to compare what Jib and docker push. They don't seem much different, so I am intrigued. |
|
Looks like we are not sending |
|
That's OK, as it's chunked so it doesn't need to have that. |
|
Hmm, sometimes |
|
You mean the thing mentioned in 'config' element? If so, I actually compared the ones in Artifactory. The Docker version is about 4 times bigger. |
|
@Hi-Fi Yep - I believe it is about 4 times bigger because Docker adds the history field that contains a bunch of metadata about each layer. |
|
@Hi-Fi so one theory we've been speculating for long is that the Artifactory parser is not conformant to the standard and crashes if there is no history field. @coollog can we just put an empty history? The Artifactory ticket seems stalled, so it'd be interesting to see if it will make Artifactory happy in the next Jib release. |
|
@chanseokoh We can try the solution and test it first, but I'd be hesitant to have it be the default behavior, since it adds an unnecessary extra field and some registries may require the number of items in |
|
Seems that there's also some other differences than history -part. Files that end up to Artifactory are below (for Red Hat registry doesn't show at least as easily those uploads, so haven't check how that sees things. Probably same way). From JIB: From Docker client: Noticed also funny thing: From those examples I pasted, the docker client example was not working when I used it directly "as is". When I changed Content-Type to "application/vnd.docker.distribution.manifest.v2+json" (adding "+json") and using chunked transfer encoding, I was able to remake the PUT request. So the Content-Type is not causing the issue, it's just probably showing it some other way (or showing it correctly, as with Content-Length the error is just null). |
|
At least empty histrory didn't did the trick, same error about circuit breaker threshold. |
|
Thanks for getting this info! The Docker client is adding a bunch of extra fields that are Docker V2.2-specific and not required by OCI format. I'm wondering if Artifactory is expecting one of these fields to be present. |
|
Seems that Jfrog repos docker importer checks only a few things for manifest validity: https://github.com/jfrog/docker2artifactory/blob/c1639b1747a4dbb8f348b31ca72adbc9fecae1c4/tests/util/ArtifactoryUtil.py#L51. |
|
@Hi-Fi The log from RTFACT-17134 indicates that Artifactory is written in Java. That code seems to be from a test helper for a migration utility. |
|
@briandealwis: Yes, Artifactory is written in Java, but that docker2artifactory importer is util provided by jFrog, which (I think) should mean that it should check validity correctly before trying to copy containers from some other registry to Artifactory. So point just was, that at least in that copy there's not many fields checked, which could indicate that Artifactory doesn't require anything else. I could think that from JFrog side it should be quite easy to tell what parsing is tried to be done when that error occurs, but sadly they seem to be quite unresponsive. |
|
I used a method decompiler and walked through Artifactory's Basically this code walks through the layers and creates blob infos for each layer. Although it attempts to support containers with no history (which is allowed by the spec, and which Jib does not produce), it does not increment private static ManifestMetadata applyAttributesFromContent(...) {
...
final JsonNode history = config.get("history");
final JsonNode layers = manifest.get("layers");
final boolean foreignHasHistory = layers.size() == historyCounter;
int historyIndex = 0;
int layersIndex = 0;
while (historyIndex < historySize || layersIndex < layers.size()) {
final JsonNode historyLayer = (history == null) ? null : history.get(historyIndex);
final JsonNode layer = layers.get(layersIndex);
long size = 0L;
String digest = null;
// This condition appears to be problematic:
// - Jib images have no history, thus `historyLayer` is always `null`,
// and so `notEmptyHistoryLayer(null)` is always `false`
// - None of our layers are foreign
// and so layersIndex is never incremented
if (notEmptyHistoryLayer(historyLayer) || (!foreignHasHistory && isForeignLayer(layer))) { // !!!
size = layer.get("size").asLong();
totalSize += size;
digest = layer.get("digest").asText();
++layersIndex; // XXX never incremented
}
// build a BlobInfo from digest, size, and created
// add the blobInfo to the manifest metadata
// circuit breaker checks
}
// populate remaining more manifest metadata
}
...
private static boolean notEmptyHistoryLayer(final JsonNode historyLayer) {
return historyLayer != null && historyLayer.get("empty_layer") == null;
}It seems to me that either the conditional marked with |
|
@briandealwis Let me check with Dev Team |
|
If history is the cause, #875 then hopefully solves also this one. |
|
Confirmed: Adding as many history elements as there's layers make container push to Artifactory. Red Hat Container Registry still throws error with this quick try, so it might still require something else. But Artifactory works when there's equal number of (non-empty-layer) history elements than layers. For test I only added static ones to |
|
Closing this since it should be fixed by #877 (the fix will be in 0.9.10, which should be coming later today or tomorrow). |
|
Tested with Artifactory 6.0.2, works well. Big Thanks! |
|
Version |
Description of the issue: When running jib as part of a maven goal, I get the following:
When looking at the artifactory logs, it seems like the manifest being uploaded is
null:Expected behavior: The image should be uploaded to artifactory :D
Steps to reproduce:
<to><image>blockmvnto publish imageEnvironment:
jib-maven-pluginConfiguration:Log output: See description above for relevant logs
Additional Information:
Tried running as well with the
-Xflag in maven with nothing else extreme helpful. I do actually see part of the image layers in an_uploadsfolder in artifactory, but nomanifest.jsonlike I'd expect from an image. Happy to provide additional information if needed!The text was updated successfully, but these errors were encountered: