Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions website/content/api-docs/agent/service.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -693,10 +693,10 @@ For the `Connect` field, the parameters are:
If this is true, then service mesh proxies, DNS queries, etc. will be able to
service discover this service.
- `Proxy` `(Proxy: nil)` -
[**Deprecated**](/consul/docs/connect/proxies/managed-deprecated) Specifies that
a managed service mesh proxy should be started for this service instance, and
optionally provides configuration for the proxy. The format is as documented
in [Managed Proxy Deprecation](/consul/docs/connect/proxies/managed-deprecated).
**Deprecated** Specifies that a managed service mesh proxy should be started
for this service instance, and optionally provides configuration for the proxy.
Managed proxies (which have been deprecated since Consul v1.3.0) have been
[removed](/consul/docs/connect/proxies) since v1.6.0.
- `SidecarService` `(ServiceDefinition: nil)` - Specifies an optional nested
service definition to register. For more information see
[Sidecar Service Registration](/consul/docs/connect/registration/sidecar-service).
Expand Down
2 changes: 1 addition & 1 deletion website/content/commands/exec.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ execute this command.
| `key:write` | `"_rexec"` prefix |
| `event:write` | `"_rexec"` prefix |

In addition to the above, the policy associated with the [agent token](/consul/docs/security/acl/acl-tokens#acl-agent-token) should have `write` on `"_rexec"` key prefix. This is for the agents to read the `exec` command and write its output back to the KV store.
In addition to the above, the policy associated with the [agent token](/consul/docs/security/acl/tokens#acl-agent-token) should have `write` on `"_rexec"` key prefix. This policy permits agents to read the `exec` command and write its output back to the KV store.

## Usage

Expand Down
2 changes: 1 addition & 1 deletion website/content/docs/agent/config/config-files.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -905,7 +905,7 @@ Refer to the [formatting specification](https://golang.org/pkg/time/#ParseDurati

- `default` ((#acl_tokens_default)) - When provided, this agent will
use this token by default when making requests to the Consul servers
instead of the [anonymous token](/consul/docs/security/acl/acl-tokens#anonymous-token).
instead of the [anonymous token](/consul/docs/security/acl/tokens#anonymous-token).
Consul HTTP API requests can provide an alternate token in their authorization header
to override the `default` or anonymous token on a per-request basis,
as described in [HTTP API Authentication](/consul/api-docs/api-structure#authentication).
Expand Down
31 changes: 31 additions & 0 deletions website/content/docs/agent/limits/usage/init-rate-limits.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
layout: docs
page_title: Initialize rate limit settings
description: Learn how to determine regular and peak loads in your network so that you can set the initial global rate limit configurations.
---

# Initialize rate limit settings

Because each network has different needs and application, you need to find out what the regular and peak loads in your network are before you set traffic limits. We recommend completing the following steps to benchmark request rates in your environment so that you can implement limits appropriate for your applications.

1. In the agent configuration file, specify a global rate limit with arbitrary values based on the following conditions:

- Environment where Consul servers are running
- Number of servers and the projected load
- Existing metrics expressing requests per second

1. Set the [`limits.request_limits.mode`](/consul/docs/agent/config/config-files#mode-1) parameter in the agent configuration to `permissive`. In the following example, the configuration allows up to 1000 reads and 500 writes per second for each Consul agent:

```hcl
request_limits {
mode = "permissive"
read_rate = 1000.0
write_rate = 500.0
}
```
1. Observe the logs and metrics for your application's typical cycle, such as a 24 hour period. Refer to [Monitor traffic rate limit data](/consul/docs/agent/limits/usage/monitor-rate-limits) for additional information. Call the [`/agent/metrics`](/consul/api-docs/agent#view-metrics) HTTP API endpoint and check the data for the following metrics:

- `rpc.rate_limit.exceeded` with value `global/read` for label `limit_type`
- `rpc.rate_limit.exceeded` with value `global/write` for label `limit_type`

1. If the limits are not reached, set the `mode` configuration to `enforcing`. Otherwise, continue to adjust and iterate until you find your network's unique limits.
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
---
layout: docs
page_title: Limit traffic rates for a source IP address
description: Learn how to set read and request rate limits on RPC and gRPC traffic from all source IP addresses to a Consul resource.
---

# Limit traffic rates from source IP addresses

This topic describes how to configure RPC and gRPC traffic rate limits for source IP addresses. This enables you to specify a budget for read and write requests to prevent any single source IP from overwhelming the Consul server and negatively affecting the network. For information about setting global traffic rate limits, refer to [Set a global limit on traffic rates](/consul/docs/agent/limits/usage/set-global-traffic-rate-limits). For an overview of Consul's server rate limiting capabilities, refer to [Limit traffic rates overview](/consul/docs/agent/limits).

<EnterpriseAlert>

This feature requires Consul Enterprise. Refer to the [feature compatibility matrix](/consul/docs/enterprise#consul-enterprise-feature-availability) for additional information.

</EnterpriseAlert>

## Overview

You can set limits on the rate of read and write requests from source IP addresses to specific resources, which mitigates the risks to Consul servers when consul clients send excessive requests to a specific resource type. Before configuring traffic rate limits, you should complete the initialization process to understand normal traffic loads in your network. Refer to [Initialize rate limit settings](/consul/docs/agent/limits/init-rate-limits) for additional information.

Complete the following steps to configure traffic rate limits from a source IP address:

1. Define rate limits in a control plan request limit configuration entry. You can set limits for different types of resources calls.

1. Apply the configuration entry to enact the limits.

You should also monitor read and write rate activity and make any necessary adjustments. Refer to [Monitor rate limit data](/consul/docs/agent/limits/usage/monitor-rate-limits) for additional information.

## Define rate limits

Create a control plane request limit configuration entry in the `default` partition. The configuration entry applies to all client requests targeting any partition. Refer to the [control plane request limit configuration entry](/consul/docs/connect/config-entries/control-plane-request-limit) reference documentation for details about the available configuration parameters.

Specify the following parameters:

- `kind`: This must be set to `control-plane-request-limit`.
- `name`: Specify the name of the service that you want to limit read and write operations to.
- `read_rate`: Specify overall number of read operations per second allowed from the service.
- `write_rate`: Specify overall number of write operations per second allowed from the service.

You can also configure limits on calls to the key-value store, ACL system, and Consul catalog.

## Apply the configuration entry

If your network is deployed to virtual machines, use the `consul config write` command and specify the control plane request limit configuration entry to apply the configuration. For Kubernetes-orchestrated networks, use the `kubectl apply` command.

<Tabs>
<Tab heading="HCL" group="hcl">

```shell-session
$ consul config write control-plane-request-limit.hcl
```

</Tab>
<Tab heading="JSON" group="json">

```shell-session
$ consul config write control-plane-request-limit.json
```

</Tab>
<Tab heading="Kubernetes" group="kubernetes">

```shell-session
$ kubectl apply control-plane-request-limit.yaml
```

</Tab>
</Tabs>

## Disable request rate limits

Set the [limits.request_limits.mode](/consul/docs/agent/config/config-files#mode-1) in the agent configuration to `disabled` to allow services to exceed the specified read and write requests limits. The `disabled` mode applies to all request rate limits, even limits specifed in the [control plane request limits configuration entry](/consul/docs/connect/config-entries/control-plane-request-limit). Note that any other mode specified in the agent configuration only applies to global traffic rate limits.
77 changes: 77 additions & 0 deletions website/content/docs/agent/limits/usage/monitor-rate-limits.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
---
layout: docs
page_title: Monitor traffic rate limit data
description: Learn about the metrics and logs you can use to monitor server rate limiting activity, include rate limits for read operations and writer operations
---

# Monitor traffic rate limit data

This topic describes Consul functionality that enables you to monitor read and write request operations taking place in your network. Use the functionality to help you understand normal workloads and set safe limits on the number of requests Consul client agents and services can make to Consul servers.

## Access rate limit logs

Consul prints a log line for each rate limit request. The log provides the necessary information for identifying the source of the request and the configured limit. The log provides the information necessary for identifying the source of the request and the configured limit. Consul prints the log `DEBUG` log level and can drop the log to avoid affecting the server health. Dropping a log line increments the `rpc.rate_limit.log_dropped` metric.

The following example log shows that RPC request from `127.0.0.1:53562` to `KVS.Apply` exceeded the limit:

```text
2023-02-17T10:01:15.565-0500 [DEBUG] agent.server.rpc-rate-limit: RPC
exceeded allowed rate limit: rpc=KVS.Apply source_addr=127.0.0.1:53562
limit_type=global/write limit_enforced=false
```

Refer to [`log_file`](/consul/docs/agent/config/config-files#log_file) for information about where to retrieve log files.

## Review rate limit metrics

Consul captures the following metrics associated with rate limits:

- Type of limit
- Operation
- Rate limit mode

Call the `/agent/metrics` API endpoint to view the metrics associated with rate limits. Refer to [View Metrics](/consul/api-docs/agent#view-metrics) for API usage information. In the following example, Consul dropped a call to the consul service because it exceeded the limit by one call:

```shell-session
$ curl http://127.0.0.1:8500/v1/agent/metrics
{
. . .
"Counters": [
{
"Name": "consul.rpc.rate_limit.exceeded",
"Count": 1,
"Sum": 1,
"Min": 1,
"Max": 1,
"Mean": 1,
"Stddev": 0,
"Labels": {
"service": "consul"
}
},
{
"Name": "consul.rpc.rate_limit.log_dropped",
"Count": 1,
"Sum": 1,
"Min": 1,
"Max": 1,
"Mean": 1,
"Stddev": 0,
"Labels": {}
}
],
. . .
}
```

Refer to [Telemetry](/consul/docs/agent/telemetry) for additional information.

## Request denials

When an HTTP request is denied for rate limiting reason, Consul returns one of the following errors:

- **429 Resource Exhausted**: Indicates that a server is not able to perform the request but that another server could potentially fulfill it. This error is most common on stale reads because any server may fulfill stale read requests. To resolve this type of error, we recommend immediately retrying the request to another server. If the request came from a Consul client agent, the agent automatically retries the request up to the limit set in the [`rpc_hold_timeout`](/consul/docs/agent/config/config-files#rpc_hold_timeout) configuration .

- **503 Service Unavailable**: Indicates that server is unable to perform the request and that no other server can fulfill the request, either. This usually occurs on consistent reads or for writes. In this case we recommend retrying according to an exponential backoff schedule. If the request came from a Consul client agent, the agent automatically retries the request according to the [`rpc_hold_timeout`](/consul/docs/agent/config/config-files#rpc_hold_timeout) configuration.

Refer to [Rate limit reached on the server](/consul/docs/troubleshoot/common-errors#rate-limit-reached-on-the-server) for additional information.
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
---
layout: docs
page_title: Set a global limit on traffic rates
description: Use global rate limits to prevent excessive rates of requests to Consul servers.
---

# Set a global limit on traffic rates

This topic describes how to configure rate limits for RPC and gRPC traffic to the Consul server.

## Introduction

Rate limits apply to each Consul server separately and limit the number of read requests or write requests to the server on the RPC and internal gRPC endpoints.

Because all requests coming to a Consul server eventually perform an RPC or an internal gRPC request, global rate limits apply to Consul's user interfaces, such as the HTTP API interface, the CLI, and the external gRPC endpoint for services in the service mesh.

Refer to [Initialize Rate Limit Settings](/consul/docs/agent/limits/init-rate-limits) for additional information about right-sizing your gRPC request configurations.

## Set a global rate limit for a Consul server

Configure the following settings in your Consul server configuration to limit the RPC and gRPC traffic rates.

- Set the rate limiter [`mode`](/consul/docs/agent/config/config-files#mode-1)
- Set the [`read_rate`](/consul/docs/agent/config/config-files#read_rate)
- Set the [`write_rate`](/consul/docs/agent/config/config-files#write_rate)

In the following example, the Consul server is configured to prevent more than `500` read and `200` write RPC calls:

<CodeTabs heading="Consul server agent">

```hcl
limits = {
rate_limit = {
mode = "enforcing"
read_rate = 500
write_rate = 200
}
}
```

```json
{
"limits" : {
"rate_limit" : {
"mode" : "enforcing",
"read_rate" : 500,
"write_rate" : 200
}
}
}

```

</CodeTabs>

## Monitor request rate traffic

You should continue to mmonitor request traffic to ensure that request rates remain within the threshold you defined. Refer to [Monitor traffic rate limit data](/consul/docs/agent/limits/usage/monitor-rate-limits) for instructions about checking metrics and log entries, as well as troubleshooting informaiton.

## Disable request rate limits

Set the [`limits.request_limits.mode`](/consul/docs/agent/config/config-files#mode-1) to `disabled` to allow services to exceed the specified read and write requests limits, even limits specified in the [control plane request limits configuration entry](/consul/docs/connect/config-entries/control-plane-request-limit). Note that any other mode specified in the agent configuration only applies to global traffic rate limits.
Loading