reg rm deletes more than just the specified tag

[patricklucas]

I was surprised by the behavior of reg rm in that it removes other tags that point to the same image digest as well as the specified tag.

The real-world example where I noticed the issue (I'm running the open source registry) is that I was removing the latest tag from a repo, but the result was an additional tag being removed as well, but not all the tags pointing to the same digest.

I've annotated the unique true images with the letters A/B/C/D.

$ reg tags my-registry/flink
INFO[0006] domain: my-registry
INFO[0006] server address: https://my-registry
A: 1.10
A: 1.10.0
A: 1.10.0-stream1
B: 1.10.0-stream1-scala_2.11
A: 1.10.0-stream1-scala_2.12
C: 1.9
C: 1.9.2
C: 1.9.2-stream2
D: 1.9.2-stream2-scala_2.11
C: 1.9.2-stream2-scala_2.12
A: latest

When I ran $ reg rm my-registry/flink:latest, the two A tags latest and 1.10.0-stream1-scala_2.12 were removed, but the other A tags remained. I expected that only the latest tag should go away.

A more contrived example is this:

$ docker pull alpine
...
$ for tag in one two three four five; do docker tag alpine my-registry/deleteme:$tag; done
$ for tag in one two three four five; do docker push my-registry/deleteme:$tag; done
...
$ reg tags my-registry/deleteme
INFO[0004] domain: my-registry
INFO[0004] server address: https://my-registry
five
four
one
three
two
$ reg rm my-registry/deleteme:one
INFO[0004] domain: my-registry
INFO[0004] server address: https://my-registry
Deleted my-registry/deleteme:one@sha256:ddba4d27a7ffc3f86dd6c2f92041af252a1f23a8e742c90e6e1297bfa1bc0c45
$ reg tags my-registry/deleteme
INFO[0004] domain: my-registry
INFO[0004] server address: https://my-registry

I.e. all five tags are gone after the single delete.

It's possible this is a behavior of the registry itself, though I don't remember running into this when using some custom scripts with curl to delete tags manually.

I'm able to reproduce the "partial deletion" if I push the scala_2.12 tag again as latest, then again delete latest. Here is the output, along with verification that the A image digests are indeed the same, and using debugging logging on the call to reg rm:

$ reg tags my-registry/flink
INFO[0005] domain: my-registry
INFO[0005] server address: https://my-registry
1.10
1.10.0
1.10.0-stream1
1.10.0-stream1-scala_2.11
1.10.0-stream1-scala_2.12
1.9
1.9.2
1.9.2-stream2
1.9.2-stream2-scala_2.11
1.9.2-stream2-scala_2.12

$ for tag in 1.10 1.10.0 1.10.0-stream1 1.10.0-stream1-scala_2.11 1.10.0-stream1-scala_2.12 latest; do echo -n "${tag}: "; reg digest my-registry/flink:$tag 2> /dev/null; done
1.10: sha256:1f9919dad9105e04c8270a852e27e34a325ad6e26f6747ffb4b2b8d68fe9c678
1.10.0: sha256:1f9919dad9105e04c8270a852e27e34a325ad6e26f6747ffb4b2b8d68fe9c678
1.10.0-stream1: sha256:1f9919dad9105e04c8270a852e27e34a325ad6e26f6747ffb4b2b8d68fe9c678
1.10.0-stream1-scala_2.11: sha256:b21f6c8df850c63dbf46a7aeac99fa3b4b880a8bde5a780cf681a3cfcdf38008
1.10.0-stream1-scala_2.12: sha256:1f9919dad9105e04c8270a852e27e34a325ad6e26f6747ffb4b2b8d68fe9c678
latest: sha256:1f9919dad9105e04c8270a852e27e34a325ad6e26f6747ffb4b2b8d68fe9c678

$ docker tag my-registry/flink:1.10.0-stream1-scala_2.12 my-registry/flink:latest

$ docker push my-registry/flink:latest

$ reg tags my-registry/flink
INFO[0004] domain: my-registry
INFO[0004] server address: https://my-registry
1.10
1.10.0
1.10.0-stream1
1.10.0-stream1-scala_2.11
1.10.0-stream1-scala_2.12
1.9
1.9.2
1.9.2-stream2
1.9.2-stream2-scala_2.11
1.9.2-stream2-scala_2.12
latest

$ for tag in 1.10 1.10.0 1.10.0-stream1 1.10.0-stream1-scala_2.11 1.10.0-stream1-scala_2.12 latest; do echo -n "${tag}: "; reg digest my-registry/flink:$tag 2> /dev/null; done
1.10: sha256:1f9919dad9105e04c8270a852e27e34a325ad6e26f6747ffb4b2b8d68fe9c678
1.10.0: sha256:1f9919dad9105e04c8270a852e27e34a325ad6e26f6747ffb4b2b8d68fe9c678
1.10.0-stream1: sha256:1f9919dad9105e04c8270a852e27e34a325ad6e26f6747ffb4b2b8d68fe9c678
1.10.0-stream1-scala_2.11: sha256:b21f6c8df850c63dbf46a7aeac99fa3b4b880a8bde5a780cf681a3cfcdf38008
1.10.0-stream1-scala_2.12: sha256:1f9919dad9105e04c8270a852e27e34a325ad6e26f6747ffb4b2b8d68fe9c678
latest: sha256:1f9919dad9105e04c8270a852e27e34a325ad6e26f6747ffb4b2b8d68fe9c678

$ reg rm -d my-registry/flink:latest
INFO[0004] domain: my-registry
INFO[0004] server address: https://my-registry
2020/03/10 15:16:27 registry.ping url=https://my-registry/v2/
2020/03/10 15:16:28 registry.manifests.get url=https://my-registry/v2/flink/manifests/latest repository=flink ref=latest
2020/03/10 15:16:28 registry.manifests.delete url=https://my-registry/v2/flink/manifests/sha256:1f9919dad9105e04c8270a852e27e34a325ad6e26f6747ffb4b2b8d68fe9c678 repository=flink digest=sha256:1f9919dad9105e04c8270a852e27e34a325ad6e26f6747ffb4b2b8d68fe9c678
Deleted my-registry/flink:latest@sha256:1f9919dad9105e04c8270a852e27e34a325ad6e26f6747ffb4b2b8d68fe9c678

$ reg tags my-registry/flink
INFO[0004] domain: my-registry
INFO[0004] server address: https://my-registry
1.10
1.10.0
1.10.0-stream1
1.10.0-stream1-scala_2.11
1.9
1.9.2
1.9.2-stream2
1.9.2-stream2-scala_2.11
1.9.2-stream2-scala_2.12

(1.10.0-stream1-scala_2.12 is missing, along with latest)

What is the exact intended behavior for reg rm? Do you think this is a problem with the registry?

[rskuipers]

I ran into this as well and am curious what the expected behavior is and how I would go about deleting just the tag I supply.

Using v0.16.1.

[rskuipers]

@patricklucas Did you find any alternatives to remove a tag without it removing other tags?

[patricklucas]

@rskuipers unfortunately not, I just re-pushed the image to the erroneously-deleted tag.

[leomongeg]

Hi,

Today I started to use this tool and get the same issue. I use this to remove the oldest tags from my private container registry automatically. I have the following list of tags:

• dev0.0.3
• dev0.0.4
• dev0.0.5

the command executed is:

reg rm my-container-registry.com/my-project/test-app:dev0.0.3 -d -r ${CONTAINER_REGISTRY} -u ${CONTAINER_REGISTRY_USER} -p ${CONTAINER_REGISTRY_PASS}

And the result was that the tags dev0.0.3 and dev0.0.4 was deleted.

It would be great to know what is the behavior of the reg rm, seems like the tag to be removed, if this tag shares something with other tags, these other tags are be removed too.

Regards,
Leonardo Monge García.

[athos-ribeiro]

This is because the reg rm command calls [1]. Which deletes an image by its reference. This means that all tags pointing to that image will also be removed. This is related to [2].

Note that currently, there is no atomic operation to delete a single tag through the docker registry API.

It should be possible to push an empty manifest to a specific repository moving the to-be-removed tag to it and then removing the empty manifest though.

[1] https://docs.docker.com/registry/spec/api/#deleting-an-image
[2] #100