Skip to content

MySQL: avoid tiny writes to improve performance in read-heavy scenarios#31204

Merged
Tener merged 5 commits intomasterfrom
tener/mysql-io-copy
Aug 31, 2023
Merged

MySQL: avoid tiny writes to improve performance in read-heavy scenarios#31204
Tener merged 5 commits intomasterfrom
tener/mysql-io-copy

Conversation

@Tener
Copy link
Copy Markdown
Contributor

@Tener Tener commented Aug 30, 2023

This change is analogous to the Postgres change made in #29812.

Just like for Postgres, the MySQL engine is forwarding the server messages individually. This hurts performance because tiny writes will have that effect.

With this PR, we replace that mechanism with a io.Copy call, ensuring sufficiently large writes are made. To populate the relevant metrics, we analyze the copy of the message stream in a separate goroutine.

The performance impact was measured using custom scripts, with a particular focus on the read-heavy use-case of creating a backup.

I ran the tests on my local MacBook, with MySQL running in Docker.

version command runs average time slowdown
N/A (without Teleport) make create-backup 5 1.170s x1.00
master @ 27ab9514a6 make create-backup 5 9.470s x8.09
tener/mysql-io-copy @ 0cdf4a66cb make create-backup 5 1.526s x1.30

There is still a 30% performance penalty compared to the direct database connection. It is, however, much faster than the existing code by a vast margin and likely acceptable in practice.

I was hoping to add a synthetic benchmark similar to BenchmarkPostgresReadLargeTable, but it is challenging: the library we are using to mock the MySQL server in tests suffers from the same issue this PR is fixing, i.e., it is sending each result with an individual Write(), leading to poor performance.

Changelog: MySQL: improve performance in read-heavy scenarios.

Related: #26868.

This change is analogous to the Postgres change made in #29812.
@github-actions github-actions Bot added database-access Database access related issues and PRs size/sm labels Aug 30, 2023
@github-actions github-actions Bot requested review from lxea and rosstimothy August 30, 2023 11:45
Comment thread lib/srv/db/mysql/engine.go Outdated
@public-teleport-github-review-bot public-teleport-github-review-bot Bot removed the request for review from lxea August 31, 2023 14:09
@Tener Tener enabled auto-merge August 31, 2023 14:11
@Tener Tener added this pull request to the merge queue Aug 31, 2023
Merged via the queue into master with commit 9de4658 Aug 31, 2023
@Tener Tener deleted the tener/mysql-io-copy branch August 31, 2023 17:20
@public-teleport-github-review-bot
Copy link
Copy Markdown

@Tener See the table below for backport results.

Branch Result
branch/v12 Failed
branch/v13 Failed
branch/v14 Create PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

database-access Database access related issues and PRs size/sm

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants