Increase the max number of http connections in Azure native FS#22561
Increase the max number of http connections in Azure native FS#22561electrum merged 1 commit intotrinodb:masterfrom
Conversation
By default, OkHttp only allows 5 concurrent HTTP connections per host, and 64 total concurrent connections. Increase these values to avoid a performance degradation compared to the legacy FS, when executing queries with a large number of splits, on nodes with large number of split runner threads.
| int maximumConnectionPoolSize = (poolSize != null && poolSize > 0) ? poolSize : 5; | ||
| Dispatcher dispatcher = new Dispatcher(); | ||
| dispatcher.setMaxRequests(Runtime.getRuntime().availableProcessors() * 4); | ||
| dispatcher.setMaxRequestsPerHost(Runtime.getRuntime().availableProcessors() * 2); |
There was a problem hiding this comment.
This is the same as the default for the number of worker threads: https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/execution/TaskManagerConfig.java#L62
I didn't want to get it from that config to avoid adding a dependency on trino-main. I set the max requests as twice as much, but I don't have any specific reason for it. I'm open for suggestions how to choose a better value.
There was a problem hiding this comment.
is this worth exposing it as a configuration property?
Also nit: extract Runtime.getRuntime().availableProcessors() to a variable, and also inline dispatcher below
There was a problem hiding this comment.
I can't inline it, as the setters don't return anything.
If we'll make it configurable, it should only be a fallback in case we get the default wrong. Maybe we should ask differently - in which cases the number of worker threads should be customized?
|
I tested this manually on a 16 node cluster executing q09 from TPCDS. The wall time went down from 60s to 9s, matching legacy FS. |
@nineinchnick does this finding apply for AWS S3 or GCS clients? |
I don't think so. In both AWS and GCP, we're not injecting a custom HTTP client, and relying more on the default SDK configuration. |
4 participants
Description
By default, OkHttp only allows 5 concurrent HTTP connections per host, and 64 total concurrent connections. Increase these values to avoid a performance degradation compared to the legacy FS, when executing queries with a large number of splits, on nodes with large number of split runner threads.
Fixes #22548
Additional context and related issues
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text: