Make parallelism a provider-level configuration

[ror6ax]

Terraform Version

1.13

Use Cases

Many TF users are devloping pluings for APIs with various rate limiting and thoughput requirements.
https://developer.hashicorp.com/terraform/cli/commands/apply#parallelism-n is a good solution for these, but asking the end user to remember enabling this flag for this one provider is a bad UX.

Attempted Solutions

Just asking end users to set https://developer.hashicorp.com/terraform/cli/commands/apply#parallelism-n for our provider. When they forget to do so, rate limiting errors occur (core issue we're working around) and those can be quite cryptic. It's not clear for them that a flag for TF is required.

Proposal

I'd like to be able to to bake parallelism into provider level field so the user is not required to set it, but can do so if they want.

Additionally, it would be great if parallelism could be managed on per-resource level but I understand that this would require significant redesign of graph walking algorithm.

References

• https://discuss.hashicorp.com/t/specify-parallelism-for-null-resource/20884
• https://discuss.hashicorp.com/t/per-provider-parallelism-options-for-disabling/57479

[jbardin]

Hi @ror6ax,

The parallelism flag isn't really meant to be a solution for API rate limits, it is used for the graph walk, applying equally to every object Terraform evaluates. That isn't really something that we would give providers control of, because it affects a great number of things outside of that provider's control, including other providers.

Providers themselves are the only source of what they need for rate limiting, and already have the ability to implement that themselves (some already do). If the provider is setting their own rate limiting, and implementing that rate limiting, how does Terraform need to be involved in the process?

Thanks!

[ror6ax]

Do you have any suggestions of a providers that deal with rate limiting well from tf design perspective?

We're dealing with an api that simply caves in under the amound of simultaneous requests with default parallelism value. It's not parallelism multiplied by request rate per thread, it's just running 10 requests at the same time 🤕

[jbardin]

I don't have examples offhand, it's often done under the hood by providers when they know a particular API can't handle certain requests concurrently. Even within a single resource type, different calls may have different limitations. The provider can expose a configuration value for whatever rate limits it wants to make configurable, what sort of design suggestions are you looking for?

More often than not it's a problem with rate limiting rather than true concurrency, and in that case it requires a little more effort than a simple semaphore to setup a rate limiting algorithm, but generally not something the Terraform CLI is concerned with.

[apparentlymart]

I also don't have any concrete examples to hand sadly, but just wanted to add that for some of the providers I saw doing this in the past they relied on mechanisms from the vendor-provided SDK moreso than inline logic in the provider. For APIs that are designed with explicit rate-limits there is often a documented protocol for negotiating with them, such as:

• Returning the "429 Too Many Requests" status code when a rate limit is hit, and including a Retry-After header in the response to indicate when the cooldown period will be over. In this case the SDK might sleep until the cooldown period is over and then retry automatically.
• Including rate-limit accounting information in the responses from even successful requests so that the client can "plan ahead" by noticing that it's likely to exceed a rate limit soon and proactively throttling itself to try to avoid that.

One example of a remote API with these capabilities is the Microsoft Azure Resource Manager API, whose implementations of the above ideas are documented in Error Code and Remaining Requests. (I don't know whether and how the hashicorp/azurerm provider uses these mechanisms, but the Azure SDK it relies on contains some code for retrying while respecting the Retry-After values that might be a useful reference.)

If the API your provider is interacting with does not expose information like this then the work for the provider developer is harder because they must typically rely on heuristics for self-rate-limiting and for how long to wait before automatically retrying certain operations.

---

One relatively straightforward way to implement a concurrency limit in Go is to use a buffered channel whose buffer capacity represents the concurrency limit. For example:

// This channel has a capacity of 10 elements, so sending to
// this channel will block if there are already 10 items in it.
var limit = make(chan struct{}, 10)

func Do(f func ()) {
    // try to send to the channel. this will block if all of the
    // capacity is already used by other concurrent calls.
    limit <- struct{}{}

    // now call the provided function and wait for it to finish
    // while the item we just wrote is still occupying a slot
    // in the channel's buffer.
    f()

    // now "receive" from the channel, freeing up one of
    // its buffer slots so that another send can be unblocked.
    <- limit
}

Many different functions in the provider can then call this Do function with an arbitrary callback and this will effectively ensure that at most 10 of them can be running concurrently at a time.

Last time I saw the Terraform codebase this was essentially how Terraform implemented its own concurrency limit. I don't know if that remains true, but nonetheless you can adopt a similar strategy in your provider regardless of what Terraform itself is doing.

As noted in the previous comment, this is not generally sufficient for avoiding a rate limit: it indirectly decreases throughput by forcing certain requests to wait for others to complete, but it doesn't directly impose an upper limit on how many requests can occur within a particular time interval. Whether this would be sufficient therefore depends on how your target system's rate limiting model is designed and what sequence of requests the provider ends up making based on your specific Terraform configuration.