CI flakes with "no such service: web"

[aminvakil]

It's been a couple of days I'm seeing this regression on docker-compose 1.28.0 and 1.29.2 on my fork:

https://github.com/aminvakil/self-hosted/actions/runs/2068250465
https://github.com/aminvakil/self-hosted/actions/runs/2074457629
https://github.com/aminvakil/self-hosted/actions/runs/2080468698
https://github.com/aminvakil/self-hosted/actions/runs/2083753352

This is a clean installation of course :)

Here is the error:

Bootstrapping and migrating Snuba ...
  Container self-hosted-0-clickhouse-1  Creating
  Container self-hosted-0-redis-1  Creating
  Container self-hosted-0-zookeeper-1  Creating
  Container self-hosted-0-clickhouse-1  Created
  Container self-hosted-0-zookeeper-1  Created
  Container self-hosted-0-kafka-1  Creating
  Container self-hosted-0-redis-1  Created
  Container self-hosted-0-kafka-1  Created
  Container self-hosted-0-redis-1  Starting
  Container self-hosted-0-zookeeper-1  Starting
  Container self-hosted-0-clickhouse-1  Starting
  Container self-hosted-0-redis-1  Started
  Container self-hosted-0-zookeeper-1  Started
  Container self-hosted-0-zookeeper-1  Waiting
  Container self-hosted-0-clickhouse-1  Started
  Container self-hosted-0-zookeeper-1  Healthy
  Container self-hosted-0-kafka-1  Starting
  Container self-hosted-0-kafka-1  Started
  Container self-hosted-0-kafka-1  Waiting
  Container self-hosted-0-redis-1  Waiting
  no such service: web
  An error occurred, caught SIGERR on line 3
  *NOT* cleaning up, to clean your environment run "docker compose stop".
  Error: Process completed with exit code 1.

This started to happen since last nightly run on https://github.com/getsentry/self-hosted CI too:

https://github.com/getsentry/self-hosted/runs/5843062307?check_suite_focus=true
https://github.com/getsentry/self-hosted/runs/5843062379?check_suite_focus=true

[chadwhitacre]

This is plaguing the 22.5.0 release:

getsentry/publish#1013

[chadwhitacre]

Okay, went down the 🐰 🕳️ on #1423 #1424 ... afaict this is due to no change we introduced either in self-hosted or upstream (sentry, etc.). Therefore my current hypothesis is that this is unique to GitHub Actions and due to some change introduced in that environment.

@aminvakil (or anyone else) Can you repro this locally? I tried but I'm on a new laptop and am running afoul of #914. 😩

[aminvakil]

@chadwhitacre Sorry for late reply, I will test this on a Debian 10 Buster in the next hours and report back.

[aminvakil]

On Debian 10 Buster:

$ docker version
Client: Docker Engine - Community
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:48:13 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:46:05 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Using docker-compose 1.29.2, I couldn't reproduce it on 22.1.0, 22.2.0, 22.3.0, 22.4.0 .

I ran docker-compose down -v --remove-orphans && docker volume prune -f && git clean -fdx after every run.

And tested it using ./install.sh && ./_integration-test/run.sh.

[chadwhitacre]

Dang, the plot thickens! Something really specific to GitHub Actions environment? 🤔

[aminvakil]

And why it started happening to my fork a couple days before it started to happen on main repo?

[boukeversteegh]

This issue just felt oddly familiar datastack-net/dockerized#39 .

The way to reproduce it locally might be to run the script without a TTY (e.g. true | ./install.sh | tee log.txt). Around the end of march, a lot of official docker images were updated (with the same tags), and updated the underlying OS and their shells.

I really don't know if this is related, but thought to let you know just in case.

[chadwhitacre]

Thanks for sharing, @boukeversteegh!

[github-actions]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[aminvakil]

We have removed docker-compose 1.x versions and I have not seen this on docker-compose 2.x versions ever, seems like it has been fixed by removing the problem :)

As no one has reported this outside of CI, I will close this.

[chadwhitacre]

We stopped testing docker-compose v1 in CI precisely because of this problem. Are we really ready to drop docker-compose v1 support overall?

[chadwhitacre]

self-hosted/.github/workflows/test.yml

Lines 35 to 39 in 518a9ec

	# Disabled due to https://github.com/getsentry/self-hosted/issues/1415
	# - compose_version: "1.28.0"
	# compose_path: "/usr/local/bin"
	# - compose_version: "1.29.2"
	# compose_path: "/usr/local/bin"

[aminvakil]

We stopped testing docker-compose v1 in CI precisely because of this problem. Are we really ready to drop docker-compose v1 support overall?

I don't think we can do so, for one ansible docker-compose plugin only supports < 2 , and many other users are using v1.x .

But about this issue, I would say opening it when we are testing this inside CI or users experience it, AFAICT this only happens in GHA and there is no bandwidth to fix this, is there?

[chadwhitacre]

@aminvakil So I guess we should bring this back in CI and find a workaround for GHA?

[aminvakil]

We will not drop support for docker-compose v1, but it's eol in a couple of months anyway https://www.docker.com/blog/announcing-compose-v2-general-availability/
and as far as users are not experiencing the issue happens to CI on GHA, we can keep this closed.

But about this issue, I would say opening it when we are testing this inside CI or users experience it, AFAICT this only happens in GHA and there is no bandwidth to fix this, is there?

Now that I read my sentence again, I get that it can be interpreted a different way.

I meant let's just open this issue if and only if users report that they are experiencing it inside their environment outside of GHA.

And we should not use docker-compose v1 anymore anyway and encourage (not mandatory though) users to use docker-compose v2.

We can make it mandatory on October 2022 when there will be no security fixes for docker-compose v1 then.

[hubertdeng123]

Docker compose v1 support is dropped completely now, seems safe to close this