Debug CI

[chadwhitacre]

CI is failing on master. Why?

#1272

[chadwhitacre]

curl: (22) The requested URL returned error: 502 Bad Gateway

https://github.com/getsentry/self-hosted/runs/4929127714?check_suite_focus=true#step:4:1402

[chadwhitacre]

Restarting debug ...

[chadwhitacre]

Actually, does it fail locally? 🤔

[chadwhitacre]

Passes locally.

[chadwhitacre]

curl: (22) The requested URL returned error: 502 Bad Gateway

https://github.com/getsentry/self-hosted/runs/5023770344?check_suite_focus=true#step:4:1378

That sounds like nginx is up but the app server is down. Why does it fail so late though? That's not the first time we hit the server, is it?

[BYK]

@chadwhitacre here is your error: https://github.com/getsentry/self-hosted/runs/5023770344?check_suite_focus=true#step:5:81

[aminvakil]

@chadwhitacre here is your error: https://github.com/getsentry/self-hosted/runs/5023770344?check_suite_focus=true#step:5:81

So maybe giving #1267 a little love helps here too?
And why it has started to fail now? What has been changed?

[chadwhitacre]

Hm, maybe something changed in Relay? 🧐

[chadwhitacre]

Trying to repro locally ...

[chadwhitacre]

Based on #1251 (comment) I made an attempt to get our relay invocation to not spit out credentials.json properly, but it did. 🤔

$ curl -L https://github.com/docker/compose/releases/download/1.28.0/run.sh -o docker-compose-1.28.0
$
$
$
$ ./test-relay 
docker-compose version 1.28.0, build d02a7b1
docker-py version: 4.4.1
CPython version: 3.9.0
OpenSSL version: OpenSSL 1.1.1i  8 Dec 2020
Creating sentry-self-hosted_relay_run ... done
{"secret_key":"foo","public_key":"bar","id":"baz"}
$
$
$
$ cat test-relay

#!/usr/bin/env zsh

dc="./docker-compose-1.28.0"
$dc version
$dc run \
    --no-deps \
    --volume "$(pwd)/relay/config.yml:/tmp/config.yml" \
    relay --config /tmp credentials generate --stdout

[BYK]

@chadwhitacre let's add a cat credentials.json and an ls before that in inspect failure step to see what we generate on CI?

[chadwhitacre]

Ooooh, GitHub sez I won. 😜

[chadwhitacre]

error: could not parse json config file (file /work/.relay/credentials.json)

My hunch is that the file is malformed rather than missing. We shall see ...

[chadwhitacre]

Hmm ... not seeing the cat? Guess we need that ls after all. :P

https://github.com/getsentry/self-hosted/runs/5024828441?check_suite_focus=true#step:4:520

[BYK]

You know what, I bet it's because we mount things to /tmp. What if it was something else, like /opt or something?

[BYK]

Maybe we should add that -T here right after $dcr? Since the file is there but it is empty, makes me think this is a stdout redirection related issue.

[chadwhitacre]

Is there a reason 07c025c won't work?

[BYK]

Because, read the giant comment I left there? 😁

[chadwhitacre]

I read it but I couldn't make sense of it.

relay generate credentials tries to read the config and the credentials even with the --stdout and --overwrite flags and then errors out when the credentials file exists but not valid JSON. We hit this case as we redirect output to the same config folder, creating an empty credentials file before relay runs.

So no shell redirect means no credentials file created before relay runs means no error parsing JSON?

[BYK]

Hard for me to recollect the whole memory but let me try:

1. We do have a config for Relay in place, either one the end-user created or the one we copied from the example file
2. When you run relay generate credentials even with --overwrite it tries to read both the config and the credentials (this happens only when you have the config in place)
3. Relay yells at us if it cannot read/parse credentials file and stops
4. Even when we say "ignore that, overwrite or write the output to stdout" Relay still acts like a fussy toddler

[chadwhitacre]

Hrmm ... the file exists but is seemingly empty. 🤔

https://github.com/getsentry/self-hosted/runs/5024926314?check_suite_focus=true#step:4:617

[BYK]

Okay so that only works with docker compose not docker-compose

[BYK]

Okay, I think the only reasonable fix here is to run docker compose pull upfront.

[chadwhitacre]

Okay that explains https://github.com/getsentry/self-hosted/runs/5029137872?check_suite_focus=true#step:4:163.

I can use the tail call in 5f67f7b.

[BYK]

Btw, relay generate credentials fails, right?

[chadwhitacre]

Does it? Link?

[BYK]

Does it? Link?

Dunno, thought you tried that too :D

Btw setting COMPOSE_INTERACTIVE_NO_CLI=1 may help with that output.

[chadwhitacre]

Dunno, thought you tried that too :D

Ah, you mean without --stdout? Yes, I tried that somewhere back there and yes it failed you were right. ;)

[chadwhitacre]

Oh! Locally I already have the image cached.

[chadwhitacre]

Boom! 💥

I removed the relay nightly image and I can repro locally:

relay/credentials.json ------------------
nightly: Pulling from getsentry/relay
Digest: sha256:c126666fd4f9a99388e276686eb5c81dd71a58c114c719b18a2f1d78653fd900
Status: Downloaded newer image for getsentry/relay:nightly
{"secret_key":"f66RDFJ-K-C6jHpsO7mGjuKXc_zXypdynQedNYzNmlU","public_key":"gyuqZ-p-TxVMj6xt0_aNDd8k8LBWsOmnR1RfISUH93I","id":"05775dda-6d9a-42ef-8ac8-c914ea9a3f33"}
=========================================
Creating sentry-self-hosted_relay_run ... done
error: could not parse json config file (file /work/.relay/credentials.json)
  caused by: expected ident at line 1 column 2
ERROR: 1
$

💃

[chadwhitacre]

yes it failed you were right

But! We can work around the "redirect creates the empty file which confuses relay" issue by redirecting to a tmp file and moving that into place. So we're good there. 👍

Last thing I want to check is Docker Compose 2 behavior ...

[chadwhitacre]

Alright I'm outta time. I think with some fresh eyes tomorrow it shouldn't take too long to get a PR together.

Thanks for your help @BYK! 😁 💃 🌮 👏

[chadwhitacre]

Oh wow do the different CI runs use the same docker cache? 🤔

[chadwhitacre]

Because I'm surprised that 1.28.0 is passing 🤔

[chadwhitacre]

Looks like we need the -T for 1 and 2.2.3+. Okay! Gonna scrap this for parts. 💃

[aminvakil]

Wow! Glad it was sorted out at last!