Backport of database: Avoid race condition in connection creation into release/1.16.x

[hc-github-team-secure-vault-core]

Backport

This PR is auto-generated from #26147 to be assessed for backporting due to the inclusion of the label backport/1.16.x.

The below text is copied from the body of the original PR.

---

When creating database connections, there is a race condition when multiple goroutines try to create the connection at the same time. This happens, for example, on leadership changes in a cluster.

Normally, the extra database connections are cleaned up when this is detected. However, some database
implementations, notably Postgres, do not seem to clean up in a timely manner, and can leak in these scenarios.

To fix this, we create a global lock when creating database connections to prevent multiple connections from being created at the same time.

We also clean up the logic at the end so that if (somehow) we ended up creating an additional connection, we use the existing one rather than the new one. This by itself would solve our problem long-term, however, would still involve many transient database connections being created and immediately killed on leadership changes.

It's not ideal to have a single global lock for database connection creation. Some potential alternatives:

• a map of locks from the connection name to the lock. The biggest downside is the we probably will want to garbage collect this map so that we don't have an unbounded number of locks.
• a small pool of locks, where we hash the connection names to pick the lock. Using such a pool generally is a good way to introduce deadlock, but since we will only use it in a specific case, and the purpose is to improve performance for concurrent connection creation, this is probably acceptable.

---

Overview of commits

• a65d913

[github-actions]

CI Results:
All Go tests succeeded! ✅

[github-actions]

Build Results:
All builds succeeded! ✅