Apply default rate limits for hosts/registries with known rate limit needs

[rarkins]

Describe the proposed change(s).

For example, the npmjs registry or GOPROXY does not need rate limiting applied, while the following are known to need:

• Repology
• crates.io
• Rubygems API

Let's identify all such hosts in a single file/location, in a file like lib/data/host-limits.ts. I think we can reuse matchHost syntax, it can maybe even be hostRules format but is not necessary (the matchHost value can be a key for the Record).

Importantly, if a host has a default limit in this file then we should make sure that users cannot increase the rate limit using hostRules. Ideally we can catch and warn if such an attempt is made. This protects fragile registries from falling victim to flooding from Renovate users unless they really go out of their way to overcome this.

[HonkingGoose]

If the file is only about rate limits, we could name it: lib/data/host-rate-limits.ts. That way the name explains what the file does.

Or do you intend to cover other kinds of limits in the same file?

[HonkingGoose]

We may want to add these hosts/registries to the list:

• JFrog Artifactory, based on: Configuring Renovate to fail when it can not authenticate #30743
• Gradle Plugin Portal, based on: Heavy traffic from Renovate on the Gradle Plugin Portal #27018 (reply in thread)

[rarkins]

Gradle plugins especially

[zharinov]

With regard to JFrog, it's not clear which host to match, but my guess it's jfrog.io. But the problem is I don't know the exact limits to apply, and probably we have to separate path by the ecosystem.

[HonkingGoose]

With regard to JFrog, it's not clear which host to match, but my guess it's jfrog.io. But the problem is I don't know the exact limits to apply, and probably we have to separate path by the ecosystem.

The best thing is to find a official answer on the correct rate-limit. If we can't find that info, we could try this:

1. Start with a low limit.
2. No rate-limit problems reported? Increase limit.
3. Keep increasing the limit gradually.
4. Go back to previous limit when we get reports of rate-limits.

That gets us close to the correct limit, at the cost of some trial and error.