Make max requests configurable in Azure native FS#22932
Make max requests configurable in Azure native FS#22932wendigo merged 1 commit intotrinodb:masterfrom
Conversation
|
@raunaqmorarka or @ebyhr, I ran tests locally, but please run tests with secrets. |
|
/test-with-secrets sha=ad3f2192ea7a58046584a2157acbe4ad3001ce32 |
|
The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/10246479126 |
lib/trino-filesystem-azure/src/main/java/io/trino/filesystem/azure/AzureFileSystemConfig.java
Outdated
Show resolved
Hide resolved
nineinchnick
force-pushed
the
azure-max-connections-config
branch
2 times, most recently
from
1b6d712 to
cce712d
Compare
There was a problem hiding this comment.
you previously had per-host set to a lower value. does this mean we now need two config options?
There was a problem hiding this comment.
I didn't have any specific reason for using different values, so I'd rather have one config option.
|
This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua |
Max requests where increased from 5 to 2 * number of CPUs in bd289dc. With a very large number of nodes in a cluster, this can cause throttling and result in failed queries. Make it configurable allows to tune the value, if the default one is too big. This change also sets the max requests and max requests per host to the same value, to limit the number of new configuration options.
nineinchnick
force-pushed
the
azure-max-connections-config
branch
from
cce712d to
8b31b10
Compare
|
@raunaqmorarka @wendigo PTAL and maybe merge this, I forgot about it for a while. |
|
Let's merge it! |
|
I'm running this with cloud tests |
|
@nineinchnick can you send a docs PR? |
6 participants
Description
Max requests were increased from 5 to 2 * number of CPUs in
bd289dc (#22561). With a very large number of
nodes in a cluster, this can cause throttling and result in failed
queries. Make it configurable allows to tune the value, if the default
one is too big.
This change also sets the max requests and max requests per host to the
same value, to limit the number of new configuration options.
Fixes #22915
Additional context and related issues
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text: