Restoring a yanked gem

[sonalkr132]

Suppose world is coming to an end 🌋 and only way it can be saved is if we can restore yanked versions of the gem. As you may know, gem yank removes the .gem file from S3 and our CDN.

someone could maliciously or accidentally remove gems that others depended on. However, we’ve been using an Amazon S3 bucket to store the gems for years now with versioning on - so if someone does remove gems that are necessary, we can easily restore them. (Read more).

We can restore S3 bucket of yanked version using this restore function. Now we need to index its version, expire memcache and fastly accordingly and destroy the deletion record. Taking hint from this comment I came up with this. It wouldn't be enough for our versions endpoint. After yanking a version our versions endpoint would look something like this:

created_at: 2017-01-10T17:22:53-07:00
---
loremoooooooo 0.1.10 aa1f05eb62bf701bd42359b5f9b47059
loremoooooooo 0.1.11 157dcfb0bb0368c1532c20d9d1bcdfb5
loremoooooooo -0.1.10 0991bfe2958a19e18d6ccb9c89a8ef28

Since it is intended to be append only, after restoring 0.1.10 it should look like following:

created_at: 2017-01-10T17:22:53-07:00
---
loremoooooooo 0.1.10 aa1f05eb62bf701bd42359b5f9b47059
loremoooooooo 0.1.11 157dcfb0bb0368c1532c20d9d1bcdfb5
loremoooooooo -0.1.10 0991bfe2958a19e18d6ccb9c89a8ef28
loremoooooooo 0.1.10 aa1f05eb62bf701bd42359b5f9b47059

I am not exactly sure how are we going to make 0.1.10 appear a third time.
Alternatively, we can create a new versions.list file which would expire clients' cache. We won't want to do that either because it's expensive.

cc: @indirect, @segiddins

[segiddins]

I thought we didn't want to make yanks reversible?

[sonalkr132]

It is not to make yanking reversible. Someday, we might want to restore a gem which has been yanked in rage quit or by mistake. Hence the script, which is meant to be used only by admins.

[indirect]

since we’ll only use this script if the world is ending 😆 it seems fine to recreate the versions list

[segiddins]

@sonalkr132 in that case, you could just delete the yanked_at and yanked_info_checksum fields and save a fair bit of complexity

[sonalkr132]

it seems fine to recreate the versions list

I was hoping we could achieve this without having to recreate versions list so that we can entertain requests like one recently received on help site.

you could just delete the yanked_at and yanked_info_checksum fields

I hope you meant delete the yanked_at and yanked_info_checksum fields and mark version indexed. It would remove the yanked version line, leaving our versions response as following:

created_at: 2017-01-10T17:22:53-07:00
---
loremoooooooo 0.1.10 aa1f05eb62bf701bd42359b5f9b47059
loremoooooooo 0.1.11 157dcfb0bb0368c1532c20d9d1bcdfb5

We can't remove a line once it has been added to versions response because clients will be making partial requests on basis of what they have already cached. I bet you already know that. 😛
If this is not what you mean or if I am mistake, please elaborate.

[segiddins]

if the downloaded file's checksum doesn't match the etag, the client will re-request the entire file

[sonalkr132]

the client will re-request the entire file

That is same as regenerating the versions.list file. When I say expensive, I don't mean expensive to regenerate the file. I mean it would be expensive for all the clients to download the complete file. Given the stakes we won't be able to do it very often.