postinstall script easily runs afoul of github rate limits

[kdvolder]

The rate limit on github (unauthenticated) api requests is only 60 request per hour. The postinstall script seems to be making these kinds of requests. So it is really quite easy to exceed that limit, especially when builds are being run on a shared CI build infrastructure.

When the limits are exceeded builds that depend on vscode-ripgrep will start seeing these kind of errors causing them to fail (and I found the error somewhat cryptic and hard to diagnose):

error /theia-app/node_modules/vscode-ripgrep: Command failed.
Exit code: 1
Command: node ./lib/postinstall.js
Arguments: 
Directory: /theia-app/node_modules/vscode-ripgrep
Output:
Finding release for v11.0.1-2
GET https://api.github.com/repos/microsoft/ripgrep-prebuilt/releases/tags/v11.0.1-2
Deleting invalid download cache
Downloading ripgrep failed: Error: Request failed: 403
    at ClientRequest.https.get.response (/theia-app/node_modules/vscode-ripgrep/lib/download.js:106:24)
    at Object.onceWrapper (events.js:286:20)
    at ClientRequest.emit (events.js:198:13)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:556:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:109:17)
    at TLSSocket.socketOnData (_http_client.js:442:20)
    at TLSSocket.emit (events.js:198:13)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at TLSSocket.Readable.push (_stream_readable.js:224:10)

I was able to confirm the nature of the problem is github rate limit by adding an explicit curl request to my build:

curl -H https://api.github.com/repos/microsoft/ripgrep-prebuilt/releases/tags/v11.0.1-2
{
  "message": "API rate limit exceeded for XXX.XXX.XXX.XXX (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
  "documentation_url": "https://developer.github.com/v3/#rate-limiting"
}

This is quite annoying really, as it can be quite hard to know which builds on a shared build infrastructure are actually responsible for exceeding the rate limit for various addresses in build-infrastructure's ip address pool.

Is there any recommended/known way to work around this problem and avoid running up against this limit?

[yuezk]

Find a workaround from microsoft/vscode#28434

Recommended workaround: produce an API key, set the GITHUB_TOKEN environment var, and vscode-ripgrep will pick it up.

[roblourens]

I added that note to the README

[DeeDeeG]

FWIW, it's very common to skip the GitHub API for this type of use-case. Deduce the binary's URL by looking at the predictable URL structure of GitHub's Release pages, zero in on the binary you want, and get that it with any old http client.

e.g. taking a look at this binary's URL:

https://github.com/BurntSushi/ripgrep/releases/download/12.1.1/ripgrep-12.1.1-x86_64-pc-windows-msvc.zip

can be broken down as:

string: https://github.com/BurntSushi/ripgrep/releases/download/
var: ${release_version_number_or_tag}
string: /ripgrep-
var: ${release version/tag again, re-use the variable}
string: -
var: ${arch}
string -
var: ${platform}
var: ${extension}

As far as I know, downloads from the releases page are backed by a robust CDN, and not subject to rate-limiting. Not sure if this is against terms of service, but it is de-facto quite common, if not the prevailing norm for downloading release binaries off of GitHub. Even programmatically or as a bootstrapping step.

[DeeDeeG]

I just had a look at the API call this module makes.

I see it hitting this API URL: https://api.github.com/repos/microsoft/ripgrep-prebuilt/releases/tags/v11.0.1-2

The json file at this URL ultimately contains asset URLs that are one and the same as those of the Releases page, e.g.: https://github.com/microsoft/ripgrep-prebuilt/releases/download/v11.0.1-2/ripgrep-v11.0.1-2-x86_64-apple-darwin.tar.gz

EDIT: Okay, I found some API-specific URLs, such as: https://api.github.com/repos/microsoft/ripgrep-prebuilt/releases/assets/13800808, which is for ripgrep-v11.0.1-2-aarch64-unknown-linux-gnu.tar.gz. So I suppose there is an API-specific way to download assets... But it's very flaky in CI.

Edit 2: Upon reviewing the code, it does apparently download from the API-specific asset URLs. I am not a lawyer, but I have read the terms of service, and it's very lax. It doesn't say "you must use the API for everything". It more says "don't abuse the API" and "Don't abuse our services or break the trust GitHub tends to extend toward its users, by acting in bad faith" (paraphrasing). I do not personally think downloading directly from the Releases page URLs would be against the letter or the spirit of the Terms of Service/legalese for GitHub.

Consider forming this URL locally in JavaScript rather than hitting the GitHub API at all. I'll gladly do a PR if it's welcome.

P.S. This error is mysteriously happening only on macOS machines in my CI. It doesn't happen on Windows or Linux VMs. Who can say why? But avoiding this type of issue altogether would be great, IMO. Thanks for considering.

[roblourens]

I'm not worried about it being illegal, I'm more worried about that url format randomly changing because it is arbitrary and not API. But I wrote this originally like 4 years ago, I don't really think it has changed in that time, it is probably defacto API at this point. But what problem are you having with it? It is totally stable for us, except for random network issues in CI sometimes. So if you're having an issue, I'm not sure that the API is responsible.

[DeeDeeG]

We get the same Downloading ripgrep failed: Error: Request failed: 403 message as the original poster for this issue.

Failing CI run

---

It's as if I have run out of API usage, but only on macOS. A fork I work at (with a separate GitHub API token) also had failures only on macOS.

The unauthenticated API limit is 60 for some time period, I believe. Authenticated should be about 5000 per hour. https://docs.github.com/en/rest/overview/resources-in-the-rest-api#rate-limiting

My personal token is only used for CI, and that ran only 5 runs that day, * about 4-5 platforms per run. At most that's 25 runs, 50 if I read the code differently and it is making two API calls (It might be making one API call to fetch the release metadata, another API call to request the specific asset/tarball/zip bundle).

I'm not sure if I might have inadvertently used the API more times than 60, but certainly not over 5000 per hour.

[DeeDeeG]

It resolved itself for me and the fork I work with recently, not sure if it has to do with the calendar day rolling over and a limit being reset, or if it's just flaky.

More thoughts/speculation (click to expand)

This issue came up suddenly and is gone for us suddenly again. Maybe we'll never encounter it again.

The issue is that unauthenticated CI users will tend to see this error very quickly, and authenticating is an inconvenient hurdle to overcome. And it hit us as authenticated users this time for who knows what reason.

I noticed that suspicious or too-frequent use can cause the 403 error. https://docs.github.com/en/rest/guides/best-practices-for-integrators#dealing-with-abuse-rate-limits Maybe using this type of API call with multiple parallel runs of CI is just a bad idea? It says multiple requests in parallel for the same user can cause rate limiting. It's just odd for it to repeatedly happen only on the macOS containers.

[DeeDeeG]

An aside note: I see the first API call always seems to go through.

This API call returns (among other things) human-friendly URLs that needn't incur another API hit.

A quick fix would be to call the API the first time as is currently done, identify the assets.{some_integer}.browser_download_url (same URL as a human would download from the Releases page) and download from that, rather than the assets.{some_integer}.url (API-specific download URL).

Not sure why I thought that. It's the first API call that's not going through.

[roblourens]

Hm, I don't know why that would be when we run many many builds of vscode and have not had this issue.

[DeeDeeG]

🤷 Now only Windows 64-bit is failing on three separate instances of similar CI. [EDIT: to be clear: these new Windows failures are wholly unrelated to vscode-ripgrep. (I thought I had typed this but I apparently had not.)] I suppose I'll let this one go. If a clear and informative answer ever arises I will post here, but I guess it was just a fluke. Thank you anyhow for your responses.

Edit 2: My point being, CI fails for mysterious reasons that come and go. I have seen this to be true on any given CI platform.

[DeeDeeG]

Last update: It turned out we weren't using the GITHUB_TOKEN env var for downloading vscode-ripgrep. So we were at the whims of the API as unauthenticated API consumers. Sorry for the noise.