Skip to content

Commit

Permalink
Increase HTTP request queue capacity (#2449)
Browse files Browse the repository at this point in the history
In 2.13, the default inbound and outbound HTTP request queue capacity
decreased from 10,000 requests to 100 requests (in PR #2078). This
change results in proxies shedding load much more aggressively while
under high load to a single destination service, resulting in increased
error rates in comparison to 2.12 (see linkerd/linkerd2#11055 for
details).

This commit changes the default HTTP request queue capacities for the
inbound and outbound proxies back to 10,000 requests, the way they were
in 2.12 and earlier. In manual load testing I've verified that
increasing the queue capacity results in a substantial decrease in 503
Service Unavailable errors emitted by the proxy: with a queue capacity
of 100 requests, the load test described [here] observed a failure rate
of 51.51% of requests, while with a queue capacity of 10,000 requests,
the same load test observes no failures.

Note that I did not modify the TCP connection queue capacities, or the
control plane request queue capacity. These were previously configured
by the same variable before #2078, but were split out into separate vars
in that change. I don't think the queue capacity limits for TCP
connection establishment or for control plane requests are currently
resulting in instability the way the decreased request queue capacity
is, so I decided to make a more focused change to just the HTTP request
queues for the proxies.

[here]: linkerd/linkerd2#11055 (comment)
  • Loading branch information
hawkw authored Aug 3, 2023
1 parent 3224560 commit 9fa90df
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions linkerd/app/src/env.rs
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,7 @@ pub const DEFAULT_CONTROL_LISTEN_ADDR: &str = "0.0.0.0:4190";
const DEFAULT_ADMIN_LISTEN_ADDR: &str = "127.0.0.1:4191";
const DEFAULT_METRICS_RETAIN_IDLE: Duration = Duration::from_secs(10 * 60);

const DEFAULT_INBOUND_HTTP_QUEUE_CAPACITY: usize = 100;
const DEFAULT_INBOUND_HTTP_QUEUE_CAPACITY: usize = 10_000;
const DEFAULT_INBOUND_HTTP_FAILFAST_TIMEOUT: Duration = Duration::from_secs(1);
const DEFAULT_INBOUND_DETECT_TIMEOUT: Duration = Duration::from_secs(10);
const DEFAULT_INBOUND_CONNECT_TIMEOUT: Duration = Duration::from_millis(300);
Expand All @@ -246,7 +246,7 @@ const DEFAULT_INBOUND_CONNECT_BACKOFF: ExponentialBackoff =

const DEFAULT_OUTBOUND_TCP_QUEUE_CAPACITY: usize = 10;
const DEFAULT_OUTBOUND_TCP_FAILFAST_TIMEOUT: Duration = Duration::from_secs(3);
const DEFAULT_OUTBOUND_HTTP_QUEUE_CAPACITY: usize = 100;
const DEFAULT_OUTBOUND_HTTP_QUEUE_CAPACITY: usize = 10_000;
const DEFAULT_OUTBOUND_HTTP_FAILFAST_TIMEOUT: Duration = Duration::from_secs(3);
const DEFAULT_OUTBOUND_DETECT_TIMEOUT: Duration = Duration::from_secs(10);
const DEFAULT_OUTBOUND_CONNECT_TIMEOUT: Duration = Duration::from_secs(1);
Expand Down

0 comments on commit 9fa90df

Please sign in to comment.