Replication Protocol

[calvinrp]

Use cases:

• Third-party monitors need to subscribe to a registry's updates and verify integrity;
• Registries importing / mirroring other registries' packages and subscribing to updates;

The protocol and reference implementation needs to be solidified. I'm investigating and working on a proposal, but I'd like to start the discussion here and welcome any thoughts.

[lann]

I would be very happy if this allowed for replication from static content blob store / CDN. Bonus points if we can get fine-grained delta updates via range query.

Rough design idea:

For replicating top-level registry state, represent log updates as a stream of protobuf messages, something like:

message RegistryRecord {
  uint64 index = 1;
  string log_id = 2;
  string record_id = 3;
}

Add a new API endpoint, e.g. GET /v1/replicate?since=<index> which returns content source(s) that cover (a superset of) the requested range:

{
  "content_sources": [
    {"type": "http", "url": "https://blobs.bigcorp.com/warg/records_1000_1999.bin"},
    {"type": "http", "url": "https://blobs.bigcorp.com/warg/records_2000_2999.bin"}
  ]
}

Replicators can fetch these sources with normal HTTP cache / range-request semantics for efficient sync as long as the URLs are stable and contents append-only, even if the server doesn't shard the record stream.

For replicating package logs I think the current fetch API can technically get the job done, but we could refine it or add another e.g. /v1/replicate/logs if we need a more efficient approach.

[calvinrp]

I'm definitely on board with leaning into blob store and caching mechanisms. That's where my head is at. Investigating implementation options and going write up a proposal. Please continue to share your ideas.

[esoterra]

I think two-phase systems like this are a good idea. The data is dynamic enough that we'll probably always need some logic for telling you what data you need to e.g. fetch everything you don't know about X packages. Doing that in a first pass and then retrieving the cache-friendly data itself is appealing.

This approach can probably be used both for replication/fetching of both hashes and the content they refer to depending on what the clients need.

So it may make sense to make this API quite general letting clients indicate

• interest in a specific subset of logs or all of them,
• interest in checkpoints, record hashes, the records themselves, and/or the content of releases,
• and what they already know (e.g. all records for "foo" up until sha256:deadbeef.

[calvinrp]

For the use case of Monitors, they can use the public REST API endpoint /v1/fetch/logs for retrieving operator records to keep track of valid keys for signing checkpoints and read a stream of protobuf CheckpointCommit messages. For the moment, let's focus on the protobuf data format first and then work out how it will be distributed.

message CheckpointCommit {
    uint32 index = 1;
    CheckpointEnvelope checkpoint_envelope = 2;
    repeated Leaf leafs = 3;
}

message CheckpointEnvelope {
    Checkpoint checkpoint = 1;
    string key_id = 2;
    string signature = 3;
}

message Checkpoint {
    uint32 log_length = 1;
    string log_root = 2;
    string map_root = 3;
}

message Leaf {
    string log_id = 1;
    string record_id = 2;
}

I have index and log_length as uint32 type at the moment, since log_length is currently uint32 elsewhere in the code. But we could up it to uint64.

[lann]

I don't know that it is valuable to distribute a stream of all checkpoints to monitors, at least until we are committing checkpoints to their own verifiable log.

I think the simplest addition to the current APIs would be for monitors to poll the existing /fetch/checkpoint and - if it differs from the last known checkpoint - fetch the new leaves.

Thinking about the format of the leaves, we could really make it very dense by just serializing fixed-size pairs of <log-id-bytes><record-id-bytes> with no delimeters, algo specifiers, or extra encoding (a "packed log entry format"); that would also make it inherently seekable; you can get the Nth entry in a stream by seeking to (N-1)*<hash-size>*2. Monitors would know how to interpret the hash data because they already read the operator log which specifies the algorithm.