-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error: IncompleteMessage: connection closed before message completed #2136
Comments
This is just due to the racy nature of networking. hyper has a connection pool of idle connections, and it selected one to send your request. Most of the time, hyper will receive the server's |
Hey, thank you for the swift response! I got it! So the connection is being reused, right? It is due the keep-alive option? Also, I could not reproduce the error when requesting a url over If that is the reason, I should experienced the issue when using |
I just found that aws s3 has a default max idle timeout of 20s, and hyper's default keep_alive_timeout is 90s. Settings the keep_alive_timeout to less than 20s on hyper client seems to have solved the problem! Thank you, your explanation really help me to understand why this was happening! |
I was looking at the java aws client, and I saw that they use the max-idle-timeout as 60s, but there is a second property called validate-after-inactivity (5s default) that allows the idle timeout be so high. It would be possible to implement the same behavior on hyper? Does it make sense? 😄 |
I believe the "revalidation" it does is to poll that it is readable. In hyper, we already register for when the OS discovers the connection has hung up. The race would still exist, if the "revalidation" happened at the same time the server was closing. |
``` Error: Request Error when talking to qbittorrent: error sending request for url (http://localhost:6006/api/v2/torrents/delete): connection closed before message completed Caused by: 0: error sending request for url (http://localhost:6006/api/v2/torrents/delete): connection closed before message completed 1: connection closed before message completed ``` Issue: hyperium/hyper#2136
Anyone getting this with
|
Well, I tried to use |
Doesn't I've faced this problem with ClickHouse HTTP crate, however, ClickHouse sends @seanmonstar, any ideas? |
Setting pool idle timeout to a value smaller than watchtower's poll interval can fix following error: [2022-08-25T04:03:22.811160892Z INFO solana_watchtower] Failure 1 of 3: solana-watchtower testnet: Error: rpc-error: error sending request for url (https://api.testnet.solana.com/): connection closed before message completed It looks like this happens because either RPC servers or ISPs drop HTTP connections without properly notifying the client in some cases. Similar issue: hyperium/hyper#2136.
Setting pool idle timeout to a value smaller than solana-watchtower's poll interval can fix following error: [2022-08-25T04:03:22.811160892Z INFO solana_watchtower] Failure 1 of 3: solana-watchtower testnet: Error: rpc-error: error sending request for url (https://api.testnet.solana.com/): connection closed before message completed It looks like this happens because either RPC servers or ISPs drop HTTP connections without properly notifying the client in some cases. Similar issue: hyperium/hyper#2136.
Setting pool idle timeout to a value smaller than solana-watchtower's poll interval can fix following error: [2022-08-25T04:03:22.811160892Z INFO solana_watchtower] Failure 1 of 3: solana-watchtower testnet: Error: rpc-error: error sending request for url (https://api.testnet.solana.com/): connection closed before message completed It looks like this happens because either RPC servers or ISPs drop HTTP connections without properly notifying the client in some cases. Similar issue: hyperium/hyper#2136.
Setting pool idle timeout to a value smaller than solana-watchtower's poll interval can fix following error: [2022-08-25T04:03:22.811160892Z INFO solana_watchtower] Failure 1 of 3: solana-watchtower testnet: Error: rpc-error: error sending request for url (https://api.testnet.solana.com/): connection closed before message completed It looks like this happens because either RPC servers or ISPs drop HTTP connections without properly notifying the client in some cases. Similar issue: hyperium/hyper#2136. (cherry picked from commit 798975f)
Setting pool idle timeout to a value smaller than solana-watchtower's poll interval can fix following error: [2022-08-25T04:03:22.811160892Z INFO solana_watchtower] Failure 1 of 3: solana-watchtower testnet: Error: rpc-error: error sending request for url (https://api.testnet.solana.com/): connection closed before message completed It looks like this happens because either RPC servers or ISPs drop HTTP connections without properly notifying the client in some cases. Similar issue: hyperium/hyper#2136. (cherry picked from commit 798975f)
Some optimizations in regards to downloading Favicon's. I also encounterd some issues with accessing some sites where the connection got dropped or closed early. This seems a reqwest/hyper thingy, hyperium/hyper#2136. This is now also fixed. General: - Decreased struct size - Decreased memory allocations - Optimized tokenizer a bit more to only emit tags when all attributes are there and are valid. reqwest/hyper connection issue: The following changes helped solve the connection issues to some sites. The endresult is that some icons are now able to be downloaded always instead of sometimes. - Enabled some extra reqwest features, `deflate` and `native-tls-alpn` (Which do not bring in any extra crates since other crates already enabled them, but they were not active for Vaultwarden it self) - Configured reqwest to have a max amount of idle pool connections per host - Configured reqwest to timeout the idle connections in 10 seconds
This fix is linked to this hyper issue hyperium/hyper#2136 It can't be reproduced locally and is a frequent occurrence in the CI. Note: It might be a better fix than this one.
I think it's unexpected for most people that this isn't automatically retried? If i ask the library to get a website for me i expect it to not fail because the keep alive timed out. If it should use keepalive by default it should also be able to handle it properly? Am I misunderstanding anything here? I think closing this as completed is misleading :-) |
It's just not possible conceptually, is it? See #2136 (comment) Rather the application developer has to decide if the request should get retried, e.g. if it's a well behaving GET request or if it's visible on the application level that the request did or did not have the desired effect yet. Side note: There seem to be some weird servers that silently time out connections, meaning that they do not close the connection when their timeout is reached but unconditionally close it as soon as they get reused later. While you can counteract that with a suitable |
### Description Applies the fix in #2384 everywhere an `HttpClient` is constructed via rusoto. It lowers the S3 timeout to 15s based on tips in [this thread](hyperium/hyper#2136 (comment)), to avoid `Error during dispatch: connection closed before message completed` errors. Note that we'll probably still run into these issues, but less frequently ([source](rusoto/rusoto#1766 (comment))). ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->
### Description Applies the fix in hyperlane-xyz#2384 everywhere an `HttpClient` is constructed via rusoto. It lowers the S3 timeout to 15s based on tips in [this thread](hyperium/hyper#2136 (comment)), to avoid `Error during dispatch: connection closed before message completed` errors. Note that we'll probably still run into these issues, but less frequently ([source](rusoto/rusoto#1766 (comment))). ### Drive-by changes <!-- Are there any minor or drive-by changes also included? --> ### Related issues <!-- - Fixes #[issue number here] --> ### Backward compatibility <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> ### Testing <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->
Applies the fix in #2384 everywhere an `HttpClient` is constructed via rusoto. It lowers the S3 timeout to 15s based on tips in [this thread](hyperium/hyper#2136 (comment)), to avoid `Error during dispatch: connection closed before message completed` errors. Note that we'll probably still run into these issues, but less frequently ([source](rusoto/rusoto#1766 (comment))). <!-- Are there any minor or drive-by changes also included? --> <!-- - Fixes #[issue number here] --> <!-- Are these changes backward compatible? Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling? Yes/No --> <!-- What kind of testing have these changes undergone? None/Manual/Unit Tests -->
Backport of #3283 Applies the fix in #2384 everywhere an `HttpClient` is constructed via rusoto. It lowers the S3 timeout to 15s based on tips in [this thread](hyperium/hyper#2136 (comment)), to avoid `Error during dispatch: connection closed before message completed` errors. Note that we'll probably still run into these issues, but less frequently ([source](rusoto/rusoto#1766 (comment))).
Hey, I'm experiencing a weird behavior with the hyper client when using https.
Sometimes my app in production fails to perform the request, but the same request works most of the time. I performed a load test locally to try to reproduce the problem, and I could reproduce: it is occurring ~0.02% of the times.
I guess that it could be something related to the hyper-tls, so I switched to hyper-rustls, but the same problem continue to occur.
So I tried to hit the url using http instead of https and the error went away!
The error I receive from hyper::Client::get is:
hyper::Error(IncompleteMessage): connection closed before message completed.
Follow a minimal working example to reproduce the error:
Cargo.toml:
src/main.rs:
PS: replace the
url
value with a validhttps
url. In my tests I used a small file on aws s3.I performed a local load test using hey:
Running the test for 2 minutes (
-z 120s
) was enough to see some errors appearing.Could anyone help me out? If I need to provide more information, or anything, just let me know.
Thank you!
The text was updated successfully, but these errors were encountered: