RoundRobinLoadBalancer: better identify excess connections
#2173
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Motivation: Cold start or traffic spikes can result in opening excess connections to handle unexpected load. Current selection algorithm in `RoundRobinLoadBalancer` attempts to pick a random connection. This approach does not let to decrease number of open connections when traffic stabilizes. Modifications: Introduce `linearSearchSpace` configuration option for `RoundRobinLoadBalancerFactory` to start a linear search before falling back to a random selection. Result: Allows configuring more predictable selection of the connection. When configured with `IDLE_TIMEOUT` socket option, helps to reduce number of connections that can serve multiplexed or pipelined requests. Default value of 16 has no performance impact on non-pipelined HTTP/1.1 connections, but significantly reduces latencies for HTTP/2 requests and number of open HTTP/2 connections.
|
Great improvement, congratulations! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation:
Cold start or traffic spikes can result in opening excess connections to
handle unexpected load. Current selection algorithm in
RoundRobinLoadBalancerattempts to pick a random connection. Thisapproach does not let to decrease number of open connections when
traffic stabilizes.
Modifications:
Introduce
linearSearchSpaceconfiguration option forRoundRobinLoadBalancerFactoryto start a linear search before fallingback to a random selection.
Result:
Allows configuring more predictable selection of the connection. When
configured with
IDLE_TIMEOUTsocket option, helps to reduce the numberof connections that can serve multiplexed or pipelined requests.
Default value of 16 has no performance impact on non-pipelined HTTP/1.1
connections, but significantly reduces latencies for HTTP/2 requests and
number of open HTTP/2 connections. For 1024 concurrent HTTP/2 requests
p99 latency drops down 2.5x times without offloading and 4x with offloading,
number of open connections goes down from ~256 to <10.