Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Creating connections can hang during server downtime #595

Closed
pvlugter opened this issue Jun 13, 2023 · 0 comments · Fixed by #596
Closed

Creating connections can hang during server downtime #595

pvlugter opened this issue Jun 13, 2023 · 0 comments · Fixed by #596
Labels
type: bug A general bug
Milestone

Comments

@pvlugter
Copy link
Contributor

During Postgres instance changes, maintenance or resizing or failover, on Google Cloud SQL in our case, we've been seeing that when applications are active during this downtime, the connection pool can reach its max size with allocated connections, while these connections are not actually usable (neither acquired nor idle). I see there have been some similar issues reported before, with the connection pool hanging after database restarts.

In what we're seeing, connections are closed when the server goes down, and these are invalidated and released to the pool as expected. During the downtime, new connections are accepted and then closed immediately on the Cloud SQL side. For the client, creating these connections never returns, because the SSL handshake future is never completed. Permits have been allocated from the pool, but with the connection creation Mono not returning, they also aren't invalidated.

We've tested changes that handle the early connection resets in the SSL adapter, and this looks good under our testing — I'll create a pull request. There could also be a timeout on the SSL handshake Mono, to safeguard against this never completing.

@pvlugter pvlugter added the status: waiting-for-triage An issue we've not yet triaged label Jun 13, 2023
@mp911de mp911de added type: bug A general bug and removed status: waiting-for-triage An issue we've not yet triaged labels Jun 15, 2023
@mp911de mp911de added this to the 1.0.2.RELEASE milestone Jun 15, 2023
@mp911de mp911de linked a pull request Jun 15, 2023 that will close this issue
mp911de pushed a commit that referenced this issue Jul 11, 2023
mp911de added a commit that referenced this issue Jul 11, 2023
[#595][#596]

Signed-off-by: Mark Paluch <[email protected]>
mp911de pushed a commit that referenced this issue Jul 11, 2023
mp911de added a commit that referenced this issue Jul 11, 2023
[#595][#596]

Signed-off-by: Mark Paluch <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug A general bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants