Clients cache open sockets for up to 20 seconds after last request #632
Comments
stevepryde
added
bug
|
The SDK is using the hyper defaults for connection pooling right now, and it looks like by default, hyper doesn't have an upper bound on idle connections. I'm not sure what the correct fix is, but its definitely seems like something the SDK needs solve. It is possible to work around this by replacing the default connector so that you can change hyper's defaults, but this is definitely not easy, and we don't have a good example for it currently. |
|
Adding this to our Production Readiness tracking issue. |
|
It looks like the AWS SDK for Java V1 defaults to 50 idle connections with a max idle time of 60 seconds, and V2 is consistent. |
|
Go V2 appears to default to 100 idle connections and 90 seconds max idle time. |
|
I think you can workaround this issue with the following connector customization for now: use aws_sdk_dynamodb as dynamodb;
use aws_sdk_s3 as s3;
use aws_smithy_client::erase::DynConnector;
use aws_smithy_client::hyper_ext::Adapter as HyperAdapter;
fn create_smithy_conn() -> DynConnector {
// The `DynConnector` results in an allocation and dynamic dispatch for all client calls,
// so you may desire to type out the full type name for the return value instead.
DynConnector::new(
// The `HyperAdapter` converts a hyper connector into a Smithy connector that can be used with the SDK
HyperAdapter::builder()
.hyper_builder({
// Tell the `HyperAdapter` to set max idle connections on the underlying hyper client
let mut hyper_builder = hyper::Client::builder();
hyper_builder.pool_max_idle_per_host(50);
hyper_builder
})
// Use the default/built-in HTTPS connector
.build(aws_smithy_client::conns::https()),
)
}
#[tokio::main]
async fn main() {
let sdk_config = aws_config::load_from_env().await;
// Construct clients with the customized connectors
let s3_client = s3::Client::from_conf_conn((&sdk_config).into(), create_smithy_conn());
let dynamodb_client =
dynamodb::Client::from_conf_conn((&sdk_config).into(), create_smithy_conn());
println!("{:?}", s3_client.list_buckets().send().await);
println!("{:?}", dynamodb_client.list_tables().send().await);
}If you try that, let me know if it improves things. |
|
That works nicely. Thank you 😄 |
jdisanti
added
the
pending-release
|
A fix for this will go out in the next release. |
|
This was included in release-2022-12-14. |
jdisanti
removed
the
pending-release
|
Describe the bug
Each instance of
aws_sdk_*::Clientseems to cache potentially hundreds of open sockets that stay around for up to 20 seconds after use. This becomes a major problem on Lambda where your app has a file handle limit of 1024, especially when using a number of different AWS clients to do lots of parallel operations in the same app (each with their own cache).This can be mitigated slightly by dropping clients and recreating as needed but that's not always feasible and is probably also less efficient.
Expected Behavior
Either sockets should always be closed after every request, or the caller should be able to customize this behaviour.
Or perhaps a way to share the socket pool between multiple clients.
Current Behavior
After performing 50x S3 HeadObject requests concurrently, the S3
Clientstruct keeps all 50 file handles (sockets) open for up to 20 seconds, even though the requests complete almost instantly. I assume all otherClientstructs behave similarly.Reproduction Steps
I can reproduce this by spawning concurrent S3 HeadObject tasks and then polling the number of file handles used.
I assume any
Clientand any operation will do similar.Possible Solution
I haven't taken a deep enough dive into the client yet but something seems to be caching open sockets for up to 20 seconds. These do get reused if using the same client, but it would be good to have a way to avoid caching them at all.
Additional Information/Context
No response
Version
Environment details (OS name and version, etc.)
AWS Lambda (Amazon Linux 2 x86_64) and also Mac OS 12.6 M1/32G
Logs
No response
The text was updated successfully, but these errors were encountered: