Disable runner auto update

[portellaa]

Describe the bug
We are running the runner in docker containers and it is forcing us to update the runner and shuts it down.

Is there any way we can disable this?

https://github.com/ydataai/docker-github-runner -> project with the Dockerfile to build runner in containers

To Reproduce

1. Go to the folder where you have github-runner
2. execute ./run.sh
3. Run a job

Expected behavior
I want to run the runner without it forcing me to update and shutting down my docker

Runner Version and Platform

Version of your runner? v2.169.1

OS of the machine running the runner? Linux

Job Log Output

github@acace520a67a:~$ tail -f runner.log

√ Connected to GitHub

2020-05-18 16:21:01Z: Listening for Jobs
Runner update in progress, do not shutdown runner.
Downloading 2.262.1 runner
Waiting for current job finish running.
Generate and execute update script.
Runner will exit shortly for update, should back online within 10 seconds.

Runner and Worker's Diagnostic Logs

Doesn't seem relevant.

[TingluoHuang]

@portellaa please check #484 (comment)

[portellaa]

i read that before and it seems that basically the runner is auto updating itself in service mode, but i don't want that.

As far as i read, i'm not the only one requesting for this, the runner should have the ability to turn off automatic update, or don't have it enabled at all.

[TingluoHuang]

You can run runsvc.sh interactive and it will still run your runner in interactive mode.

[bryanmacfarlane]

This issue covers why the runner updates itself: #246

The service is moving forward. And some of those new features need runner changes. So it's not about disabling and compat, it's about new features and new code needed in the runner.

This is a dupe of 246

[portellaa]

It's not about being interactive or not, it's about user features, if for some reason i want to be stuck on that runner version, no matter the reason, i think that's up to me, right?

I'm not saying that it should be as this, but it's an odd behaviour, if i have that image tagged with the version of the runner, it will loose sense then.
What if the update breaks, that runner will be down and that's not cool. And this internal self update is something that shouldn't happen in a container, that's why it was nice to have a flag to disable it.

And svc.sh it attached to systemd cli and we would prefer to not have it in the machine.

Thanks for the info, feel free to close the issue.

[derek-burdick]

This is also a concern for the security and compliance minded folks. Partially mitigated if you can also ship signed binaries and DLLs.
Ideally if auto update is OFF, when looking at runners via UI you get visual signal they are outdated.
If the runner RunAsync loop just ignores the message, what happens? Will it just perpetually get a job to update?

[provgregoryabdo]

Every time an update is released, my kube runners go into a crash loop backoff. :(

[onedr0p]

@provgregoryabdo you can create your own action-runner image, I have created one and it gets updated on a schedule with Github actions e.g. https://github.com/onedr0p/actions-runner

[alessiogambi]

I am also facing the same problem. Adding the "--restart always" on docker does not help. Manually restarting the docker image also does not help, as the update-exit cycle repeats.

[Temtaime]

I am forced to build my own runner with my patches, even if i silently ignore refresh message, the runner doesn't receive new jobs.
Seems that you're stopping sending job messages to the runner if its version is not up to date.
That's ugly

[alessiogambi]

I managed to solve my issue by following @TingluoHuang's suggestion. Basically, I replaced run.sh with svcrun.sh as my entrypoint in the Dockerfile. However, svcrun.sh does not come with GitHub actions, you have to download it from somewhere else. See this comment.

[yaochaoutokyo]

I managed to bypass the issue by creating a docker image that always downloads the latest GitHub runner. I share my code here. Run the container with the option --restart=always, if the auto-update of the runner causes the container to shut down, the container will restart with the latest runner installed and won't enter into the loop of "start - update - shutdown"

[pchmelar]

We are running multiple runners in the background on our CI and every time a new update comes out it kills all the runners, so we have to manually restart them. This is a really frustrating issue, please add some option to disable auto update.

[viceice]

Maybe my script helps. see https://github.com/visualon/docker-images/blob/master/docker/github-runner/src/bin/start-runner.sh

it's working fine and sucessfully auto updates

[woter1832]

This issue has been open for some time with little response from Github. Whatever your reasoning, this is causing a lot of people unnecessary work and it's time you addressed the issue properly.

It's great that people have got their containers to auto-update, but isn't one of the foundations of containerisation, immutability, whereby they don't change during their life span?

If you are trying out containers for the first time, don't treat them as traditional servers. For example, you might be tempted to update your application inside of a running container or to patch a running container when vulnerabilities arise.

Containers are fundamentally not designed to work this way. They are designed to be stateless and immutable.

Source: https://cloud.google.com/solutions/best-practices-for-operating-containers#ensure_that_your_containers_are_stateless_and_immutable

Being forced to update a container every time you do a new release is a pain. As @portellaa says, "It's up to me". I appreciate there are ways to automate the build and deployment of a new container which means it is still immutable, however, for this to work well, we need some sort of public hook into your release. (Does this now exist? It didn't the last time I tried) For some, this level of automation is an added overhead. Your runner is the only one I have come across that breaks people's builds when you do even a patch update.

There are a plethora of CI/CD tools out there and this issue alone is preventing me from taking Github actions seriously.

If you want people to adopt actions and workflows, allow self-host containers and you can't / won't stop breaking builds every time you release a new version, then please consider:

1. Providing a mechanism that alerts people's pipelines to new releases.
2. Change the webpage "Starting your workflow run..." to "Your runner is out of date."
   

Thanks

[davidkarlsen]

Amen!
All in all I find GHs responsiveness around actions/CI not comforting (actions, infra, you name it, issues and support-cases sit to rot like they do not care...)
The take at containerization (or rather lack of it) is also not very convincing. Something needs to happen here or projects will be gone soon.

[timharris777]

I agree there should be a --no-updates or --disable-updates option for run.sh

[bryanmacfarlane]

heads up @hross as he's been the engineering lead around the runner for awhile. want to make sure this gets attentions since I'm in other areas. I wrote my thoughts in #246 which appears to be a dupe of this issue.

Thanks so much to @timharris777 and others for feedback.

[kub0x]

k8s pods enter CrashLoopBackOff state whenever there's an update. They update themselves, restart and start the cycle again. The only solution is to build the image with the latest runner version. This how I solved it.

What I did is to move the action-runner downloading from my Dockerfile to my Entrypoint's script start.sh.

There I just check the last version of the action runner, download and extract it.

Print latest tag's download URL : curl https://api.github.com/repos/actions/runner/releases/latest | grep browser_download_url | grep -i linux-x64 | cut -d ":" -f2,3 | tr -d '"'

Notice that the latter is for linux just change it. Passing '-u' option in the curl with your user name and token gives you more attemps for the rate limit, just in case you hit it.

Now whenever I get an update, runner gets updated which ends up on pod restarting and fetching the latest version. I know is not the best solution but it works.

P.D: I agree with the concept of images being inmutable, they should run as it where built. Any modification (update-patch) that implies a restart has an impact, as in the restart you may get back to the old version again.

[erikkn]

k8s pods enter CrashLoopBackOff state whenever there's an update. They update themselves, restart and start the cycle again. The only solution is to build the image with the latest runner version. This how I solved it.

What I did is to move the action-runner downloading from my Dockerfile to my Entrypoint's script start.sh.

There I just check the last version of the action runner, download and extract it.

Print latest tag's download URL : curl https://api.github.com/repos/actions/runner/releases/latest | grep browser_download_url | grep -i linux-x64 | cut -d ":" -f2,3 | tr -d '"'

Notice that the latter is for linux just change it. Passing '-u' option in the curl with your user name and token gives you more attemps for the rate limit, just in case you hit it.

Now whenever I get an update, runner gets updated which ends up on pod restarting and fetching the latest version. I know is not the best solution but it works.

P.D: I agree with the concept of images being inmutable, they should run as it where built. Any modification (update-patch) that implies a restart has an impact, as in the restart you may get back to the old version again.

@kub0x , seems like you make an unauthenticated call, how do you get around the rate-limiting :)?

[edudip-thomasp]

As many people already mentioned, and we have the same long discussion going on at issue #313, this feature is really needed. We also use the runner in Kubernetes and the self-update functionality is causing constant problems.

Why are you guys ignoring several issues where people ask you to provide a way to disable the self-update functionality?

[xucian]

please add this

[zhangyile]

please add this

[ghost]

please add this

[asad-awadia]

still nothing? My runners break every time they try to update

[ghost]

Same here. They simply don't seem to give a damn about us.

[ethomson]

We're working on adding this support now. I outlined our plans in this issue: #1396 (comment)

[thboop]

🎉 🎉 We've shipped this feature, please see the changelog for more information on how to use it.

[asad-awadia]

For compatibility with the GitHub Actions service, you will need to manually update your runner within 30 days of a new runner version being available.

[xucian]

For compatibility with the GitHub Actions service, you will need to manually update your runner within 30 days of a new runner version being available.

this. we should be forced to update only if there are breaking changes, not on an arbitrary time limit, like 30 days