-
Notifications
You must be signed in to change notification settings - Fork 241
Update Gitlab Version
Updating gitlab for ucsc-ci.com:
-
EC2 Instance is "allspark"
-
AWS account: platform-dev
-
Region: us-west-2
-
SSH Key is an AWS secret (TODO: which?)
To connect (note that the server IP has probably changes; check the AWS EC2 console!):
ssh -i "allspark.pem" [email protected]
Then run:
sudo apt-get update
If the GPG Key is out of date, you may have to refresh it before running sudo apt-get update:
curl -s https://packages.gitlab.com/gpg.key | sudo apt-key add -
Don't run sudo apt-get upgrade to update gitlab.
Gitlab may require multiple installs to upgrade.
Follow the upgrade paths here:
https://docs.gitlab.com/ee/update/index.html#upgrade-paths
To see gitlab-ee versions:
sudo apt-cache policy gitlab-ee
Install each version in the order described by the upgrade path and then update the server. Double-check the versions. Do not try to downgrade Gitlab. For example:
sudo apt-get install gitlab-ee=13.0.14-ee.0
sudo gitlab-ctl reconfigure && sudo gitlab-ctl restart
sudo apt-get install gitlab-ee=13.1.11-ee.0
sudo gitlab-ctl reconfigure && sudo gitlab-ctl restart
sudo apt-get install gitlab-ee=13.8.4-ee.0
sudo gitlab-ctl reconfigure && sudo gitlab-ctl restart
If you cannot find the shared SSH private key, and your own SSH public key is not authorized on the server, you can use "EC2 Instance Connect" in the AWS console to get a shell on the machine to recover from.
If you manage to type the wrong version of Gitlab to install, you might see this:
The following packages will be DOWNGRADED:
gitlab-ee
You should stop and install the right version instead.
If you abort Gitlab deb package installation in the middle, you may need to sudo dpkg --configure -a
to get dpkg back into a good state.
After that, you can install Gitlab versions as normal, but you might see things like:
dpkg: warning: unable to delete old directory '/opt/gitlab/embedded/service/gitlab-rails/public/assets/sql.js/gh-pages/documentation/stylesheets': Directory not empty
Having the wrong stuff in /opt/gitlab/embedded
can break the Gitlab version migration scripts, which do things like install gems in there.
You can replace the files in there with:
sudo mv /opt/gitlab/embedded /opt/gitlab/embedded.bak
sudo apt install --reinstall gitlab-ee
You might need to disable the Gitlab automatic backup and version check logic if the existing Gitlab installation is sufficiently broken, in order to let the package install. This is very dangerous if you have not already made a backup!. You can do this by creating files /etc/gitlab/skip-auto-backup
and /etc/gitlab/skip-unmigrated-data-check
. Remember to remove them when you are done.
Do not stop the gitlab-runsvdir
systemctl service. The service has to be running for the upgrade process to work. This is because it is responsible for running the database server, and the upgrade scripts really want to talk to the database.
If you get a failure like:
* Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: runit_service[gitlab-kas] (gitlab-kas::enable line 122) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas ----
STDOUT: fail: /opt/gitlab/service/gitlab-kas: runsv not running
STDERR:
---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas ----
Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-kas returned 1
(Note the runsv not running
)
Then you need to start the service:
sudo systemctl start gitlab-runsvdir
Then make sure it is actually running in sudo systemctl status
.
The Gitlab package installation or reconfiguration scripts might fail like this:
* Mixlib::ShellOut::ShellCommandFailed occurred in Chef Infra Client run: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
It's not clear how to make the logs be revealed to you, but you can run the Rails migrations separately and see what they say:
sudo gitlab-rake db:migrate
If the result is complaints about NoMethodError
s or other things that should never happen, in Ruby code in /opt/gitlab/embedded
, then you might need to fix a partial or corrupted Gitlab installation.
If you want to watch the server logs as they happen, you can run:
sudo gitlab-ctl tail