Skip to content

try to fix setup mysql#19371

Merged
arthurschreiber merged 2 commits intovitessio:mainfrom
mhamza15:fix-setup-mysql
Feb 13, 2026
Merged

try to fix setup mysql#19371
arthurschreiber merged 2 commits intovitessio:mainfrom
mhamza15:fix-setup-mysql

Conversation

@mhamza15
Copy link
Collaborator

@mhamza15 mhamza15 commented Feb 12, 2026

Description

I've been seeing this consistently in CI:

--2026-02-12 15:54:41--  (try: 2)  http://mirrors.kernel.org/ubuntu/pool/main/liba/libaio/libaio1_0.3.112-13build1_amd64.deb
Connecting to mirrors.kernel.org (mirrors.kernel.org)|142.0.200.124|:80... failed: Connection timed out.
Retrying.

Which often leads to this when actually running the tests:

/tmp/mysql-8.0/bin/mysqld: exit status 127, output: /tmp/mysql-8.0/bin/mysqld: error while loading shared libraries: libaio.so.1: cannot open shared object file: No such file or directory

I'm really not sure that the proposed fix here is what is needed. Or why this started happening all of a sudden. But it seems to be working more consistently now. I guess that old URL is less reliable? 🤷‍♂️

Successful run(s): https://github.com/vitessio/vitess/actions/runs/21964925856?pr=19371

Related Issue(s)

Checklist

  • "Backport to:" labels have been added if this change should be back-ported to release branches
  • If this change is to be back-ported to previous releases, a justification is included in the PR description
  • Tests were added or are not required
  • Did the new or modified tests pass consistently locally and on CI?
  • Documentation was added or is not required

Deployment Notes

AI Disclosure

@github-actions github-actions bot added this to the v24.0.0 milestone Feb 12, 2026
@vitess-bot vitess-bot bot added NeedsWebsiteDocsUpdate What it says NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsIssue A linked issue is missing for this Pull Request NeedsBackportReason If backport labels have been applied to a PR, a justification is required labels Feb 12, 2026
@vitess-bot
Copy link
Contributor

vitess-bot bot commented Feb 12, 2026

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

  • Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

  • Apply the release notes (needs details) label if users need to know about this change.
  • New features should be documented.
  • There should be some code comments as to why things are implemented the way they are.
  • There should be a comment at the top of each new or modified test to explain what the test does.

New flags

  • Is this flag really necessary?
  • Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
  • vtctl command output order should be stable and awk-able.

Signed-off-by: Mohamed Hamza <mhamza@fastmail.com>
@mhamza15 mhamza15 added Type: CI/Build Backport to: release-22.0 Needs to be backport to release-22.0 Backport to: release-23.0 Needs to be backport to release-23.0 and removed NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsWebsiteDocsUpdate What it says NeedsIssue A linked issue is missing for this Pull Request NeedsBackportReason If backport labels have been applied to a PR, a justification is required labels Feb 12, 2026
Signed-off-by: Mohamed Hamza <mhamza@fastmail.com>
@mhamza15 mhamza15 marked this pull request as ready for review February 12, 2026 21:41
@mhamza15 mhamza15 requested review from arthurschreiber, nickvanw and timvaillancourt and removed request for frouioui and rohit-nayak-ps February 12, 2026 21:41
@arthurschreiber arthurschreiber merged commit 614a4ce into vitessio:main Feb 13, 2026
109 checks passed
vitess-bot bot added a commit that referenced this pull request Feb 13, 2026
Signed-off-by: Mohamed Hamza <mhamza@fastmail.com>
vitess-bot bot added a commit that referenced this pull request Feb 13, 2026
Signed-off-by: Mohamed Hamza <mhamza@fastmail.com>
tanjinx pushed a commit to slackhq/vitess that referenced this pull request Feb 13, 2026
Signed-off-by: Mohamed Hamza <mhamza@fastmail.com>
mhamza15 added a commit to mhamza15/vitess that referenced this pull request Feb 15, 2026
Signed-off-by: Mohamed Hamza <mhamza@fastmail.com>
tanjinx added a commit to slackhq/vitess that referenced this pull request Feb 16, 2026
… ERS (#677) (#798)

* `slack-19.0`: support a minority of lagging tablets in ERS (#677)

* `EmergencyReparentShard`: wait only for majority of most advanced relay logs

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* fix bad cherry-pick

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* fix header, rename var

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* cleanup

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* cleanup

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* `EmergencyReparentShard`: include SQL thread position in most-advanced candidate selection

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* additional tests

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* fix test

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* fix tests

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* add source uuid

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* test cleanup

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* fix MySQL56GTIDSet sort

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* lint

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* lint again

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* support sort optimization in both paths

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* fix comment

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* fix cond

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* revert to simpler sorter

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* fix subtest name

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* log skipped candidates

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* fix test

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* update .AtLeast(), move to map[string]*RelayLogPositions

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* fix bad conflict fix

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* check for empty pointer in gtid logic

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* fix .IsZero()

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* remove conditional on status.RelayLogPosition

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

---------

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Co-authored-by: Tanjin Xu <109303790+tanjinx@users.noreply.github.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>

* Fix libdbd-mysql-perl dependency issues on Ubuntu 24.04

Ubuntu 24.04 rebuilt libdbd-mysql-perl to depend on libperconaserverclient22
(from MySQL/Percona 8.4), which is not available when using MySQL/Percona 8.0.
This causes installation failures in CI workflows.

Since libdbd-mysql-perl and percona-toolkit are not needed for our tests,
this commit uses APT preferences pinning to block their installation:
- Pin-Priority: -1 prevents these packages from being installed
- Added --no-install-recommends flag to percona-xtrabackup-80 installation

Changes:
- Updated workflow template: test/templates/cluster_endtoend_test.tpl
- Regenerated 3 cluster workflows that use xtrabackup
- Manually updated 6 upgrade/downgrade workflows that use xtrabackup

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>

* Remove libdbd-mysql-perl from Docker install_dependencies.sh

The previous commit fixed CI workflows but missed the Docker build script.
install_dependencies.sh was still trying to install libdbd-mysql-perl and
percona-toolkit as BASE_PACKAGES, causing Docker builds to fail on Ubuntu 24.04.

These packages are not needed for Vitess functionality, so removing them
from the BASE_PACKAGES array.

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>

* Remove libdbd-mysql-perl from bootstrap Dockerfiles

The Dockerfiles in docker/bootstrap/ were still explicitly installing
libdbd-mysql-perl, which causes build failures on Ubuntu 24.04.

Removed libdbd-mysql-perl from:
- Dockerfile.mysql80
- Dockerfile.mysql84
- Dockerfile.percona80

This package is not needed for Vitess functionality.

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>

* Add --no-install-recommends to bootstrap Dockerfiles

The bootstrap Dockerfiles for mysql80 and mysql84 were installing packages
without --no-install-recommends flag. This caused percona-xtrabackup-80/84
to pull in libdbd-mysql-perl as a recommended dependency, which fails on
Ubuntu 24.04 due to libperconaserverclient22 dependency.

Added --no-install-recommends to prevent recommended packages from being
installed automatically.

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>

* Install libdbd-mysql-perl from Ubuntu repos before adding Percona repos

The key issue was that Ubuntu 24.04's libdbd-mysql-perl works fine with
system libraries, but the Percona repository has a version that depends
on libperconaserverclient22 (Percona 8.4), which conflicts with our
MySQL/Percona 8.0 setup.

Solution: Install libdbd-mysql-perl from Ubuntu repos BEFORE adding
Percona repositories. This way we get the Ubuntu version, and when we
later add Percona repos and install percona-xtrabackup, apt won't try
to upgrade libdbd-mysql-perl to the incompatible Percona version.

Changes:
- Updated workflow template to install libdbd-mysql-perl early
- Regenerated 3 cluster workflows
- Fixed 6 upgrade/downgrade workflows manually
- Updated all 3 bootstrap Dockerfiles
- Restored libdbd-mysql-perl to docker/utils/install_dependencies.sh

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>

* Add percona-toolkit back to install_dependencies.sh

Now that libdbd-mysql-perl is installed from Ubuntu repos before adding
Percona repositories, percona-toolkit can also be installed safely from
Ubuntu repos. Both packages will be the Ubuntu versions and won't have
the libperconaserverclient22 dependency issue.

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>

* Hold libaio1 package and add --no-install-recommends to prevent upgrades

After setup-mysql manually installs an old version of libaio1 (to work around
Ubuntu 24.04 issues), subsequent apt-get install commands were upgrading it,
causing MySQL 5.7 binaries to fail with 'cannot open shared object file'.

Changes:
1. Hold libaio1 package after manual installation (apt-mark hold)
2. Add --no-install-recommends to all apt-get install commands in workflows

This prevents apt from upgrading the manually installed libaio1 package.

Fixes: cluster_endtoend_vreplication_across_db_versions test failures

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>

* Remove --no-install-recommends flags from workflows

The --no-install-recommends flags were preventing necessary dependencies
from being installed, which may be causing the libaio.so.1 issue.

Reverting to allow recommended packages to be installed normally.

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>

* Revert libaio1 hold in setup-mysql action

Remove apt-mark hold libaio1 from setup-mysql action as it doesn't help
with the libaio.so.1 issue.

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>

* Remove --no-install-recommends from bootstrap Dockerfiles

Remove --no-install-recommends flags from DEBIAN_FRONTEND apt-get install
commands to allow recommended packages to be installed.

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>

* Add back --no-install-recommends in Dockerfile.common

Keep --no-install-recommends for the common base image to minimize
image size, but allow recommended packages in mysql80 and mysql84
Dockerfiles where they may be needed.

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>

* Revert all changes to docker/utils/install_dependencies.sh

Restore docker/utils/install_dependencies.sh to its original state from
before the libdbd-mysql-perl fixes. The file will have both libdbd-mysql-perl
and percona-toolkit in their original positions.

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>

* try to fix setup mysql (vitessio#19371)

Signed-off-by: Mohamed Hamza <mhamza@fastmail.com>

* Skip TestDownPrimaryPromotionRuleWithLagCrossCenter

This test validates PreventCrossCellFailover configuration which Slack
does not use in production.

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>

* Skip TestDownPrimaryPromotionRuleWithLag

PR #677 changed ERS to prioritize data safety by only considering the
majority of most-advanced replicas for promotion. This excludes lagging
replicas even if they have Prefer promotion rules.

This test expects a lagging replica with a Prefer rule to catch up and
then be promoted, but the new behavior removes it from consideration
before the catch-up phase. Slack prioritizes data safety over promotion
preferences in failover scenarios.

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>

---------

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>
Signed-off-by: Mohamed Hamza <mhamza@fastmail.com>
Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com>
Co-authored-by: Claude <svc-devxp-claude@slack-corp.com>
Co-authored-by: Mohamed Hamza <mhamza@fastmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Backport to: release-22.0 Needs to be backport to release-22.0 Backport to: release-23.0 Needs to be backport to release-23.0 Component: Build/CI Type: CI/Build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants