Skip to content

[v18] Database authorization resource consumption enhancements#64311

Merged
wethreetrees merged 1 commit intobranch/v18from
wethreetrees/backport-63878-branch/v18
Mar 6, 2026
Merged

[v18] Database authorization resource consumption enhancements#64311
wethreetrees merged 1 commit intobranch/v18from
wethreetrees/backport-63878-branch/v18

Conversation

@wethreetrees
Copy link
Copy Markdown
Contributor

Backport #63878 to branch/v18

changelog: Improved performance and reduced resource usage of the database proxy for clusters with large numbers of registered databases.

Replace the call to CachingAccessPoint.GetDatabaseServers in
ProxyServer.Authorize with a DatabaseServerWatcher lookup using
CurrentResourcesWithFilter. Previously, every inbound database connection
allocated and iterated a full copy of all database servers in the cache,
then discarded all but the matching entries. Under high concurrency with
large numbers of registered databases this caused significant GC pressure
and OOM. The watcher lookup only allocates the servers matching the
requested database name.

The watcher is initialized once at startup in service.go and plumbed
through the Cluster interface.
@wethreetrees wethreetrees reopened this Mar 5, 2026
@wethreetrees wethreetrees changed the base branch from master to branch/v18 March 5, 2026 16:02
@wethreetrees
Copy link
Copy Markdown
Contributor Author

There were some merge conflicts, mostly in the reversetunnel, and a few adjacent changes snuck in through the cherry-pick (AppServerWatcher and a few others). While I am pretty sure I got everything cleaned up, additional checks would be greatly appreciated!

@wethreetrees wethreetrees marked this pull request as ready for review March 5, 2026 16:17
@github-actions github-actions bot added backport database-access Database access related issues and PRs size/md labels Mar 5, 2026
Copy link
Copy Markdown
Contributor

@espadolini espadolini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remember to run the same manual testing as the original PR

@wethreetrees
Copy link
Copy Markdown
Contributor Author

Remember to run the same manual testing as the original PR

Ran through the test plan and everything looks good 👍

Manual testing details

Test Environment

Cloud tenant created in staging

Name: tyler-dbwatcher-testing
Features:

  • RollingAuthUpdate
  • WildcardDNS
  • AccessMonitoring
  • BootstrapInit
  • InjectAWSCredSh
  • R53HealthCheck
  • TLSCertReload
  • KMS
  • EnablePyroscope
  • AutoUpdates
  • BootstrapCNC
  • SetEnvGOMEMLIMIT90
  • ACKDedicatedS3Buckets
  • EKSPodIdentity
  • UnmountServiceAccountToken
  • IAMJoinToken
  • DisableStaticJoinToken
  • DeferConfigChanges
  • QUICProxyPeering
  • DisableAWSCredentialsFile
  • VPADeployment
  • EnableDefaultBedrockSummarizer

Trusted cluster tests were run locally with two instances of teleport.

Test Cases

  • Register a new database
  • Connect to the newly registered database and confirm the database is found with tsh db connect (if you registered a fake database, you should get a “failed to connect” error)
  • Restart the proxy and confirm a database connection works immediately after startup, verifying the watcher initializes correctly on startup
  • Confirm that connecting to an unregistered database returns an appropriate error (e.g. database not found)
  • (Optional - can be skipped if a functional database was registered in the first test case) Register a functional database and confirm successful connection
  • Remove a database and confirm it is no longer connectable (e.g. database not found)
  • Register multiple database agents proxying the same database and confirm successful connection
  • (Optional - this is technically covered by the go test benchmark) Register >3000 databases and confirm connection to one database, verify that no significant memory increase occurs
  • (Trusted cluster - requires self-hosted setup, not testable on cloud) Connect to a database registered on a leaf cluster via the root proxy and confirm the database is found (if you registered a fake database, you should get a “failed to connect” error)

@wethreetrees wethreetrees reopened this Mar 5, 2026
Copy link
Copy Markdown
Contributor

@okraport okraport left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@greedy52
Copy link
Copy Markdown
Contributor

greedy52 commented Mar 6, 2026

Thanks Tyler! 🎉

Merged via the queue into branch/v18 with commit 05e16c0 Mar 6, 2026
46 checks passed
@wethreetrees wethreetrees deleted the wethreetrees/backport-63878-branch/v18 branch March 6, 2026 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport database-access Database access related issues and PRs size/md

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants