CI: Simplify Changes to Explicitly Install and Configure MySQL8 #9370
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -32,7 +32,9 @@ const ( | |
| unitTestTemplate = "templates/unit_test.tpl" | ||
| unitTestDatabases = "mysql57, mariadb103, mysql80, mariadb102" | ||
|
|
||
| // An empty string will cause the default non platform specific template | ||
| // to be used. | ||
| clusterTestTemplateFormatStr = "templates/cluster_endtoend_test%s.tpl" | ||
|
|
||
| unitTestSelfHostedTemplate = "templates/unit_test_self_hosted.tpl" | ||
| unitTestSelfHostedDatabases = "" | ||
|
|
@@ -125,7 +127,7 @@ type unitTest struct { | |
| } | ||
|
|
||
| type clusterTest struct { | ||
| Name, Shard, Platform string | ||
|
Collaborator
There was a problem hiding this comment. This is elegant.
Member
Author
There was a problem hiding this comment. I had similar thoughts, but:
I'm fine changing it to whatever though.
Collaborator
There was a problem hiding this comment. alright, you've convinced me :) |
||
| MakeTools, InstallXtraBackup bool | ||
| Ubuntu20 bool | ||
| } | ||
|
|
@@ -255,6 +257,7 @@ func generateSelfHostedClusterWorkflows() error { | |
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| filePath := fmt.Sprintf("%s/cluster_endtoend_%s.yml", workflowConfigDir, cluster) | ||
| err = writeFileFromTemplate(clusterTestSelfHostedTemplate, filePath, test) | ||
| if err != nil { | ||
|
|
@@ -289,12 +292,18 @@ func generateClusterWorkflows() { | |
| for _, ubuntu20Cluster := range ubuntu20Clusters { | ||
| if ubuntu20Cluster == cluster { | ||
| test.Ubuntu20 = true | ||
| test.Platform = "mysql80" | ||
| break | ||
| } | ||
| } | ||
|
|
||
| path := fmt.Sprintf("%s/cluster_endtoend_%s.yml", workflowConfigDir, cluster) | ||
| var tplPlatform string | ||
| if test.Platform != "" { | ||
| tplPlatform = "_" + test.Platform | ||
| } | ||
| template := fmt.Sprintf(clusterTestTemplateFormatStr, tplPlatform) | ||
| err := writeFileFromTemplate(template, path, test) | ||
| if err != nil { | ||
| log.Print(err) | ||
| } | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| name: {{.Name}} | ||
| on: [push, pull_request] | ||
| concurrency: | ||
| group: format('{0}-{1}', ${{"{{"}} github.ref {{"}}"}}, '{{.Name}}') | ||
| cancel-in-progress: true | ||
|
|
||
| env: | ||
| LAUNCHABLE_ORGANIZATION: "vitess" | ||
| LAUNCHABLE_WORKSPACE: "vitess-app" | ||
| GITHUB_PR_HEAD_SHA: "${{`{{ github.event.pull_request.head.sha }}`}}" | ||
|
|
||
| jobs: | ||
| build: | ||
| name: Run endtoend tests on {{.Name}} | ||
| {{if .Ubuntu20}}runs-on: ubuntu-20.04{{else}}runs-on: ubuntu-18.04{{end}} | ||
|
|
||
| steps: | ||
| - name: Set up Go | ||
| uses: actions/setup-go@v2 | ||
| with: | ||
| go-version: 1.17 | ||
|
|
||
| - name: Set up python | ||
| uses: actions/setup-python@v2 | ||
|
|
||
| - name: Tune the OS | ||
| run: | | ||
| echo '1024 65535' | sudo tee -a /proc/sys/net/ipv4/ip_local_port_range | ||
|
|
||
| - name: Check out code | ||
| uses: actions/checkout@v2 | ||
|
|
||
| - name: Get dependencies | ||
| run: | | ||
| # Setup MySQL 8.0 | ||
| wget -c https://dev.mysql.com/get/mysql-apt-config_0.8.20-1_all.deb | ||
| echo mysql-apt-config mysql-apt-config/select-server select mysql-8.0 | sudo debconf-set-selections | ||
| sudo DEBIAN_FRONTEND="noninteractive" dpkg -i mysql-apt-config* | ||
| sudo apt-get update | ||
|
|
||
| # Install everything else we need, and configure | ||
| sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata | ||
| sudo service mysql stop | ||
| sudo service etcd stop | ||
| sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/ | ||
| sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld | ||
| go mod download | ||
|
|
||
| # install JUnit report formatter | ||
| go get -u github.com/vitessio/go-junit-report@HEAD | ||
|
|
||
| {{if .InstallXtraBackup}} | ||
|
|
||
| wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb | ||
| sudo apt-get install -y gnupg2 | ||
| sudo dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb | ||
| sudo apt-get update | ||
| sudo apt-get install percona-xtrabackup-24 | ||
|
|
||
| {{end}} | ||
|
|
||
| {{if .MakeTools}} | ||
|
|
||
| - name: Installing zookeeper and consul | ||
| run: | | ||
| make tools | ||
|
|
||
| {{end}} | ||
|
|
||
| - name: Setup launchable dependencies | ||
| run: | | ||
| # Get Launchable CLI installed. If you can, make it a part of the builder image to speed things up | ||
| pip3 install --user launchable~=1.0 > /dev/null | ||
|
|
||
| # verify that launchable setup is all correct. | ||
| launchable verify || true | ||
|
|
||
| # Tell Launchable about the build you are producing and testing | ||
| launchable record build --name "$GITHUB_RUN_ID" --source . | ||
|
|
||
| - name: Run cluster endtoend test | ||
| timeout-minutes: 30 | ||
| run: | | ||
| source build.env | ||
|
|
||
| set -x | ||
|
|
||
| # run the tests however you normally do, then produce a JUnit XML file | ||
| eatmydata -- go run test.go -docker=false -follow -shard {{.Shard}} | tee -a output.txt | go-junit-report -set-exit-code > report.xml | ||
|
|
||
| - name: Print test output and Record test result in launchable | ||
| run: | | ||
| # send recorded tests to launchable | ||
| launchable record tests --build "$GITHUB_RUN_ID" go-test . || true | ||
|
|
||
| # print test output | ||
| cat output.txt | ||
| if: always() | ||
|
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@GuptaManan100 pointed out that this file is actually generated (see Line 1) using https://github.com/vitessio/vitess/blob/main/test/ci_workflow_gen.go.
I'm not sure what happens to other tests if we change the template that is being used by the
maketarget to generate yaml files.I guess you will need to try and move these changes into the template and if it breaks other tests, then maybe create a new template for mysql80?
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, will need to do something there. Running
make generate_ci_workflowputscluster_endtoend_mysql80.ymlback into its original state where it breaks after the ubuntu-latest image change.Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added support here: 9f6bd06
So now
make generage_ci_workflowproduces what I had previously done in the generated file.