CI: ubuntu-latest now has MySQL 8.0.26, let us override it with latest 8.0.x by deepthi · Pull Request #9368 · vitessio/vitess

deepthi · 2021-12-13T21:57:31Z

Description

Actions workflows that run on ubuntu-latest have been failing for ~12 hours now.
This is because of changes made to the images
actions/runner-images#4708
actions/runner-images#4674

~~Since the image already includes a mysql install, perhaps we can stop trying to install it ourselves.~~
Let's try to install the latest 8.0.x version ourselves.

Related Issue(s)

Checklist

Should this PR be backported?
Tests were added or are not required
Documentation was added or is not required

Deployment Notes

Signed-off-by: deepthi <deepthi@planetscale.com>

mattlord · 2021-12-13T22:05:51Z

I'm OK with this as a temporary workaround but it breaks a reasonable assumption here, that is: you install the latest official MySQL version with apt-get install. That's what we normally want to do as we try and regularly upgrade to stay up to date. We do also in some cases alter behavior based on the patch version.

And even if it caused no immediate issues, not being able to rely on apt being able to manage the dependencies is likely to crop up in unexpected ways moving forward.

This is one potential short-term option though, for sure. Others may be:

Specifying a specific image that is fine, e.g. 18.04 or the specific version latest pointed to before this change
Explicitly adding the upstream MySQL APT repository (potential long-term solution if we need it)
Pinning and managing usage of a specific MySQL version on our side and then installing via dpkg directly (potential long-term solution if we need it)
...

deepthi · 2021-12-13T22:47:34Z

This is one potential short-term option though, for sure. Others may be:

Specifying a specific image that is fine, e.g. 18.04 or the specific version latest pointed to before this change

Explicitly adding the upstream MySQL APT repository (potential long-term solution if we need it)

Pinning and managing usage of a specific MySQL version on our side and then installing via dpkg directly (potential long-term solution if we need it)

I'm reluctant to move everything back to 18.04, we had tried to actually move everything forward to 20.04 at some point but didn't complete that work, see #7850
The others are good options, this was just the fastest, smallest change to get us unblocked.
I can try the second option and see if I can do it correctly :)

Signed-off-by: deepthi <deepthi@planetscale.com>

…the latest Signed-off-by: deepthi <deepthi@planetscale.com>

mattlord · 2021-12-13T23:54:18Z

.github/workflows/cluster_endtoend_mysql80.yml

+        sudo DEBIAN_FRONTEND="noninteractive" apt-get update
+
+        # Uninstall any previously installed MySQL first
+        sudo systemctl stop apparmor
+        sudo DEBIAN_FRONTEND="noninteractive" apt-get remove -y --purge mysql-server mysql-client mysql-common
+        sudo apt-get -y autoremove
+        sudo apt-get -y autoclean
+        sudo deluser mysql
+        sudo rm -rf /var/lib/mysql
+        sudo rm -rf /etc/mysql
+


I don't any of this is necessary, is it? Any previous installation and its files don't matter as we use non-default values anyway and this will slow down the CI jobs further.

mattlord · 2021-12-13T23:54:56Z

.github/workflows/cluster_endtoend_mysql80.yml

+        sudo apt-get install -y make unzip g++ etcd curl git wget eatmydata
        sudo service mysql stop
        sudo service etcd stop
+        sudo bash -c "echo '/usr/sbin/mysqld { }' > /etc/apparmor.d/usr.sbin.mysqld" # https://bugs.launchpad.net/ubuntu/+source/mariadb-10.1/+bug/1806263


I would remove this. This was about running MariaDB after MySQL was previously installed -- which is not relevant for us here.

mattlord · 2021-12-13T23:55:35Z

.github/workflows/cluster_endtoend_mysql80.yml

+        sudo DEBIAN_FRONTEND="noninteractive" dpkg -i mysql-apt-config*
        sudo apt-get update
-        sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata
+        sudo DEBIAN_FRONTEND="noninteractive" apt-get install -y mysql-server mysql-client


I would remove this and just keep the original apt-get install for all of our packages.

mattlord · 2021-12-13T23:56:42Z

.github/workflows/cluster_endtoend_mysql80.yml

+        sudo bash -c "echo '/usr/sbin/mysqld { }' > /etc/apparmor.d/usr.sbin.mysqld" # https://bugs.launchpad.net/ubuntu/+source/mariadb-10.1/+bug/1806263
        sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
-        sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
+        sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld || echo "could not remove mysqld profile"


If this step failing is not fatal, then why do it at all?

mattlord

I had a few comments/nits, but want to unblock you and thus everyone else. :-)

I can make another commit, or another PR, if you want.

deepthi · 2021-12-14T00:02:59Z

I'm going to merge this as-is to unblock everyone, and @mattlord will do a follow up PR cleaning up how we are installing mysql in some of the tests.

GuptaManan100 · 2021-12-14T05:08:30Z

.github/workflows/cluster_endtoend_mysql80.yml

+        sudo DEBIAN_FRONTEND="noninteractive" apt-get update
+
+        # Uninstall any previously installed MySQL first
+        sudo systemctl stop apparmor
+        sudo DEBIAN_FRONTEND="noninteractive" apt-get remove -y --purge mysql-server mysql-client mysql-common
+        sudo apt-get -y autoremove
+        sudo apt-get -y autoclean
+        sudo deluser mysql
+        sudo rm -rf /var/lib/mysql
+        sudo rm -rf /etc/mysql


This is a generated file, so we should be changing the code generation and not the file directly. The file that generates this is ci_workflow_gen.go

doeg · 2021-12-14T15:40:42Z

Thank you both for looking into this! ❤️

CI: ubuntu-latest already has MySQL 8.0, don't try to install it

0e17e71

Signed-off-by: deepthi <deepthi@planetscale.com>

CI: Install latest MySQL 8.0

3cdaca5

Signed-off-by: deepthi <deepthi@planetscale.com>

deepthi added Backport me! Component: Build/CI Type: CI/Build labels Dec 13, 2021

deepthi changed the title ~~CI: ubuntu-latest already has MySQL 8.0, don't try to install it~~ CI: ubuntu-latest now has MySQL 8.0.26, let us override it with latest 8.0.x Dec 13, 2021

CI: some tests don't need MySQL installed, for others, let's install …

a4981ff

…the latest Signed-off-by: deepthi <deepthi@planetscale.com>

mattlord reviewed Dec 13, 2021

View reviewed changes

mattlord approved these changes Dec 13, 2021

View reviewed changes

deepthi merged commit 4dd92c3 into vitessio:main Dec 14, 2021

deepthi deleted the ds-fix-ci branch December 14, 2021 00:28

mattlord mentioned this pull request Dec 14, 2021

CI: Simplify Changes to Explicitly Install and Configure MySQL8 #9370

Merged

3 tasks

GuptaManan100 reviewed Dec 14, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: ubuntu-latest now has MySQL 8.0.26, let us override it with latest 8.0.x#9368

CI: ubuntu-latest now has MySQL 8.0.26, let us override it with latest 8.0.x#9368
deepthi merged 3 commits intovitessio:mainfrom
planetscale:ds-fix-ci

deepthi commented Dec 13, 2021 •

edited

Loading

Uh oh!

mattlord commented Dec 13, 2021 •

edited

Loading

Uh oh!

deepthi commented Dec 13, 2021

Uh oh!

mattlord Dec 13, 2021

Uh oh!

mattlord Dec 13, 2021

Uh oh!

mattlord Dec 13, 2021

Uh oh!

mattlord Dec 13, 2021

Uh oh!

mattlord left a comment

Uh oh!

deepthi commented Dec 14, 2021

Uh oh!

GuptaManan100 Dec 14, 2021

Uh oh!

doeg commented Dec 14, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

deepthi commented Dec 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Checklist

Deployment Notes

Uh oh!

mattlord commented Dec 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deepthi commented Dec 13, 2021

Uh oh!

mattlord Dec 13, 2021

Choose a reason for hiding this comment

Uh oh!

mattlord Dec 13, 2021

Choose a reason for hiding this comment

Uh oh!

mattlord Dec 13, 2021

Choose a reason for hiding this comment

Uh oh!

mattlord Dec 13, 2021

Choose a reason for hiding this comment

Uh oh!

mattlord left a comment

Choose a reason for hiding this comment

Uh oh!

deepthi commented Dec 14, 2021

Uh oh!

GuptaManan100 Dec 14, 2021

Choose a reason for hiding this comment

Uh oh!

doeg commented Dec 14, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

deepthi commented Dec 13, 2021 •

edited

Loading

mattlord commented Dec 13, 2021 •

edited

Loading