Fixed Registry synchronization with v2 manifests

In the new manifest Version 2, schema 2; some fields are missing. In this case,
the "tag" field is no longer available as of this version. We used that because
the notification from the registry does not provide such information for now
(hopefully there will be a solution for Distribution 2.4). Thus, we pulled the
manifest and extracted the tag from there.

The code now detects the version of the manifest schema. If it's the latest
one, then we will fetch the tags of the given repo and compare them with what
we've got in the database.

Note that this solution could also be applied to the version 2 schema 1, but
fetching the manifest is faster and less prone to possible sync errors (even
though hese errors are not really important if crono is in place).

Moreover, I've added a table on the README.md file describing which versions of
Portus implement what.

Fixes #718

Signed-off-by: Miquel Sabaté Solà <[email protected]>

README.md

@@ -54,6 +54,30 @@ Some highlights:
 Take a tour by our [documentation](http://port.us.org/features.html) site to
 read more about this.
 
+## Supported versions
+
+Docker technologies have a fast iteration pace. This is a good thing, but it
+comes with some challenges. As requested by some of our users, the following
+table shows which versions of Docker and Docker Distribution are supported by
+each Portus version:
+
+| Portus | Docker Engine | Docker Distribution |
+|:------:|:-------------:|:-------------------:|
+| master | 1.6+ | 2.0+ |
+| 2.0.0 & 2.0.1 | 1.6 to 1.9 | 2.0 to 2.2 |
+| 2.0.2 | 1.6 to 1.9 | 2.0+ |
+| 2.0.3 (soon to be released) | 1.6+ | 2.0+ |
+
+Let's detail some of the version being specified:
+
+- Docker Engine `1.6` is the first version supported by Docker Distribution 2.
+  Therefore, this requirement is also the same for Portus.
+- As of Docker `1.10`, the Manifest Version 2, Schema 2 is the one being used.
+  This is only supported by Portus in the `master` branch and in `2.0.3`.
+- Docker Distribution `2.3` supports both Manifest versions, but some changes
+  had to be made in order to offer backwards compatibility. This is not
+  supported neither for Portus `2.0.0` nor `2.0.1`.
+
 ## Overview
 
 In this video you can get an overview of some of the features and capabilities

app/models/registry.rb

@@ -62,7 +62,7 @@ def get_namespace_from_event(event)
       return
     end
 
-    tag_name = get_tag_from_manifest(event["target"])
+    tag_name = get_tag_from_target(event["target"])
     return if tag_name.nil?
 
     [namespace, repo, tag_name]
@@ -105,19 +105,50 @@ def reachable?
 
   protected
 
+  # Fetch the tag being pushed through the given target object.
+  def get_tag_from_target(target)
+    case target["mediaType"]
+    when "application/vnd.docker.distribution.manifest.v1+json",
+      "application/vnd.docker.distribution.manifest.v1+prettyjws"
+      get_tag_from_manifest(target)
+    when "application/vnd.docker.distribution.manifest.v2+json",
+      "application/vnd.docker.distribution.manifest.list.v2+json"
+      get_tag_from_list(target["repository"])
+    else
+      raise "unsupported media type \"#{target["mediaType"]}\""
+    end
+
+  rescue StandardError => e
+    logger.info("Could not fetch the tag for target #{target}")
+    logger.info("Reason: #{e.message}")
+    nil
+  end
+
+  # Fetch the tag by making the difference of what we've go on the DB, and
+  # what's available on the registry. Returns a string with the tag on success,
+  # otherwise it returns nil.
+  def get_tag_from_list(repository)
+    tags = client.tags(repository)
+    return if tags.nil?
+
+    available = Repository.find_by(name: repository).tags.pluck(:name)
+    resulting = tags - available
+
+    # Note that it might happen that there are multiple tags not yet in sync
+    # with Portus' DB. This means that the registry might have been
+    # unresponsive for a long time. In this case, it's not such a problem to
+    # pick up the first label, and wait for the CatalogJob to update the
+    # rest.
+    resulting.first
+  end
+
   # Fetch the tag of the image contained in the current event. The Manifest API
   # is used to fetch it, thus the repo name and the digest are needed (and
   # they are contained inside the event's target).
   #
   # Returns the name of the tag if found, nil otherwise.
   def get_tag_from_manifest(target)
-    man = client.manifest(target["repository"], target["digest"])
-    man["tag"]
-
-  rescue StandardError => e
-    logger.info("Could not fetch the tag for target #{target}")
-    logger.info("Reason: #{e.message}")
-    nil
+    client.manifest(target["repository"], target["digest"])["tag"]
   end
 
   # Create the global namespace for this registry and create the personal

bin/client.rb

@@ -20,8 +20,8 @@
 case ARGV.first
 when "catalog"
   catalog = registry.client.catalog
-  pp catalog
-  pp "Size: #{catalog.size}"
+  puts catalog.inspect
+  puts "Size: #{catalog.size}"
 when "delete"
   if ARGV.length == 2
     puts "You have to specify first the name, and then the digest"
@@ -39,7 +39,7 @@
   else
     name, tag = ARGV[1], "latest"
   end
-  pp registry.client.manifest(name, tag)
+  puts JSON.pretty_generate(registry.client.manifest(name, tag))
 when "ping"
   # No registry was found, trying to ping another one.
   if registry.nil?

lib/portus/registry_client.rb

@@ -51,19 +51,16 @@ def manifest(repository, tag = "latest")
     #   - name: a string containing the name of the repository.
     #   - tags: an array containing the available tags for the repository.
     def catalog
-      link = "_catalog?n=100"
-      res = []
-
-      # We fetch repositories in pages of 100 because of a bug in the registry.
-      # See: https://github.com/docker/distribution/issues/1190.
-      until link.empty?
-        cat, link = catalog_page(link)
-        res += cat["repositories"]
-      end
-
+      res = paged_response("_catalog", "repositories")
       add_tags(res)
     end
 
+    # Returns an array containing the list of tags. If something goes wrong,
+    # then it raises an exception.
+    def tags(repository)
+      paged_response("#{repository}/tags/list", "tags")
+    end
+
     # Deletes a layer of the specified image. The layer is pointed by the digest
     # as given by the manifest of the image itself. Returns true if the request
     # was successful, otherwise it raises an exception.
@@ -81,12 +78,26 @@ def delete(name, digest)
 
     protected
 
-    # Fetches the next page in the catalog from the provided link. On success,
-    # it will return an array of the items:
+    # Returns all the items that could be extracted from the given link that
+    # are indexed by the given field in a successful response. If anything goes
+    # wrong, it raises an exception.
+    def paged_response(link, field)
+      res = []
+      link += "?n=100"
+
+      until link.empty?
+        page, link = get_page(link)
+        res += page[field]
+      end
+      res
+    end
+
+    # Fetches the next page from the provided link. On success, it will return
+    # an array of the items:
     #   - The parsed response body.
     #   - The link to the next page.
     # On error it will raise the proper exception.
-    def catalog_page(link)
+    def get_page(link)
       res = perform_request(link)
       if res.code.to_i == 200
         [JSON.parse(res.body), fetch_link(res["link"])]
@@ -114,8 +125,12 @@ def add_tags(repositories)
 
       result = []
       repositories.each do |repo|
-        res = perform_request("#{repo}/tags/list")
-        result << JSON.parse(res.body) if res.code.to_i == 200
+        begin
+          ts = tags(repo)
+          result << { "name" => repo, "tags" => ts } unless ts.nil?
+        rescue StandardError => e
+          Rails.logger.debug "Could not get tags for repo: #{repo}: #{e.message}."
+        end
       end
       result
     end

lib/portus/registry_notification.rb

@@ -9,6 +9,7 @@ class RegistryNotification
     def self.process!(data, handler)
       data["events"].each do |event|
         next unless relevant?(event)
+        Rails.logger.info "Handling Push event:\n#{JSON.pretty_generate(event)}"
         handler.handle_push_event(event)
       end
     end

spec/lib/portus/registry_client_spec.rb

@@ -268,10 +268,10 @@ def fetch_link_test(header)
 
       VCR.use_cassette("registry/catalog_lots_of_repos", record: :none) do
         WebMock.disable_net_connect!
-        stub_request(:get, "http://#{registry_server}/v2/busybox/tags/list")
+        stub_request(:get, "http://#{registry_server}/v2/busybox/tags/list?n=100")
           .to_return(body: "{\"name\": \"busybox\", \"tags\":[\"latest\"]} ", status: 200)
         (1..101).each do |i|
-          stub_request(:get, "http://#{registry_server}/v2/busybox#{i}/tags/list")
+          stub_request(:get, "http://#{registry_server}/v2/busybox#{i}/tags/list?n=100")
             .to_return(body: "{\"name\": \"busybox#{i}\", \"tags\":[\"latest\"]} ", status: 200)
         end
 
@@ -348,6 +348,24 @@ def fetch_link_test(header)
     end
   end
 
+  context "fetching lists from the catalog" do
+    it "returns the available tags even if there are more than 100 of them" do
+      create(:registry)
+      create(:admin, username: "portus")
+
+      VCR.use_cassette("registry/catalog_lots_of_tags", record: :none) do
+        registry = Portus::RegistryClient.new(
+          registry_server,
+          false,
+          "portus",
+          Rails.application.secrets.portus_password)
+
+        tags = registry.tags("busybox")
+        (1..102).each_with_index { |v, idx| expect(tags[idx]).to eq v.to_s }
+      end
+    end
+  end
+
   context "deleting a blob from an image" do
     it "deleting a blob that does not exist" do
       VCR.use_cassette("registry/delete_missing_blob", record: :none) do

spec/models/registry_spec.rb

@@ -18,17 +18,25 @@ def client
       def o.manifest(*_)
         raise StandardError, "Some message"
       end
+
+      def o.tags(*_)
+        raise StandardError, "Some message"
+      end
     else
       def o.manifest(*_)
         { "tag" => "latest" }
       end
+
+      def o.tags(*_)
+        ["latest", "0.1"]
+      end
     end
     o
   end
 
-  def get_tag_from_manifest_test(repo, digest)
-    target = { repository: repo, digest: digest }
-    get_tag_from_manifest(target)
+  def get_tag_from_target_test(repo, mtype, digest)
+    target = { "mediaType" => mtype, "repository" => repo, "digest" => digest }
+    get_tag_from_target(target)
   end
 end
 
@@ -64,7 +72,7 @@ def client
   end
 end
 
-RSpec.describe Registry, type: :model do
+describe Registry, type: :model do
   it { should have_many(:namespaces) }
 
   describe "after_create" do
@@ -132,18 +140,57 @@ def client
     it "returns a tag on success" do
       mock = RegistryMock.new(false)
 
-      ret = mock.get_tag_from_manifest_test("busybox", "sha:1234")
+      ret = mock.get_tag_from_target_test("busybox",
+                                          "application/vnd.docker.distribution.manifest.v1+json",
+                                          "sha:1234")
       expect(ret).to eq "latest"
     end
 
+    it "returns a tag on v2 manifests" do
+      owner     = create(:user)
+      team      = create(:team, owners: [owner])
+      namespace = create(:namespace, team: team)
+      repo      = create(:repository, name: "busybox", namespace: namespace)
+      create(:tag, name: "latest", repository: repo)
+
+      mock = RegistryMock.new(false)
+      ret  = mock.get_tag_from_target_test("busybox",
+                                           "application/vnd.docker.distribution.manifest.v2+json",
+                                           "sha:1234")
+      expect(ret).to eq "0.1"
+    end
+
     it "handles errors properly" do
+      m = RegistryMock.new(true)
+
+      expect(Rails.logger).to receive(:info).with(/Could not fetch the tag/)
+      expect(Rails.logger).to receive(:info).with(/Reason: Some message/)
+
+      ret = m.get_tag_from_target_test("busybox",
+                                       "application/vnd.docker.distribution.manifest.v1+prettyjws",
+                                       "sha:1234")
+      expect(ret).to be_nil
+    end
+
+    it "handles errors on v2" do
       mock = RegistryMock.new(true)
 
       expect(Rails.logger).to receive(:info).with(/Could not fetch the tag/)
       expect(Rails.logger).to receive(:info).with(/Reason: Some message/)
 
-      ret = mock.get_tag_from_manifest_test("busybox", "sha:1234")
+      ret  = mock.get_tag_from_target_test("busybox",
+                                           "application/vnd.docker.distribution.manifest.v2+json",
+                                           "sha:1234")
       expect(ret).to be_nil
     end
+
+    it "raises an error when the mediaType is unknown" do
+      mock = RegistryMock.new(true)
+
+      expect(Rails.logger).to receive(:info).with(/Could not fetch the tag/)
+      expect(Rails.logger).to receive(:info).with(/Reason: unsupported media type "a"/)
+
+      mock.get_tag_from_target_test("busybox", "a", "sha:1234")
+    end
   end
 end

spec/models/repository_spec.rb

@@ -80,6 +80,7 @@ def get_url(repo, tag)
         @event = build(:raw_push_manifest_event).to_test_hash
         @event["target"]["repository"] = repository_name
         @event["target"]["url"] = get_url(repository_name, tag_name)
+        @event["target"]["mediaType"] = "application/vnd.docker.distribution.manifest.v1+json"
         @event["request"]["host"] = "unknown-registry.test.lan"
         @event["actor"]["name"] = user.username
       end
@@ -97,6 +98,7 @@ def get_url(repo, tag)
         @event = build(:raw_push_manifest_event).to_test_hash
         @event["target"]["repository"] = repository_name
         @event["target"]["url"] = get_url(repository_name, tag_name)
+        @event["target"]["mediaType"] = "application/vnd.docker.distribution.manifest.v1+json"
         @event["request"]["host"] = registry.hostname
         @event["actor"]["name"] = "a_ghost"
       end
@@ -114,6 +116,7 @@ def get_url(repo, tag)
         @event = build(:raw_push_manifest_event).to_test_hash
         @event["target"]["repository"] = repository_name
         @event["target"]["url"] = get_url(repository_name, "digest")
+        @event["target"]["mediaType"] = "application/vnd.docker.distribution.manifest.v1+json"
         @event["target"]["digest"] = "digest"
         @event["request"]["host"] = registry.hostname
         @event["actor"]["name"] = user.username
@@ -235,6 +238,7 @@ def get_url(repo, tag)
         @event = build(:raw_push_manifest_event).to_test_hash
         @event["target"]["repository"] = name
         @event["target"]["url"] = get_url(name, tag_name)
+        @event["target"]["mediaType"] = "application/vnd.docker.distribution.manifest.v1+json"
         @event["target"]["digest"] = digest
         @event["request"]["host"] = registry.hostname
         @event["actor"]["name"] = user.username