Guides: fix setup scripts to yield correct exit code

[snazy]

The statements in the shell scripts for the setup services are often concatenated using ;, which means that a previous' command exit code is not propagated and the service, although it failed, is determined to be successful.

This change updates those scripts to use && for the statement concatenation.

"Final" setup services (aka "polaris-setup") now have a final sleep 120. This is due to the behavior of docker compose up --detach --wait, which considers any service (without dependants) that exits with exit code 0 as a failure, leading to that docker-compose command yielding an error code. That would break the guides testing code (#3553). That sleep 120 in "polaris-setup" services does not cause a delay of the compose starting up - it is purely a "hack around" Docker Compose not having a notion of "setup services".

To avoid merge conflicts, this change also:

• updates affected curl invocations (as Guides: add mandatory curl --fail option #3610)
• removes superfluous restart: "no"

[MonkeyCanCode]

should we do implicitly --fail-with-body here too for curl?

[snazy]

My assumption is that the token would just be "invalid".
It's also that the pipe "eliminates" the exit code from curl, but jq would fail then.

[MonkeyCanCode]

That is fair enough. Yeah. Just thought to be consistent. But not a blocker.

[MonkeyCanCode]

This will return 0 as well no?

[snazy]

WDYM?

[MonkeyCanCode]

So we have at most 2 setup services for these docker compose and you are adding sleep 120 to pass the CI based on docker-compose (as exit of status code of zero will be treat as failed). Thus, this will also return status code of zero.

[snazy]

Ah, I think I know what you're referring to.
Those setup-bucket services behave in an "interesting" way, which is different than how the "final" polaris-setup services behave (in terms of exit behavior).
An "intermediate" setup-service like setup-bucket, on which for example polaris depends using the service_completed_successfully condition, is fine to exit and does not "break" a docker compose up --detach --wait.
A "final" setup-service like some polaris-setup, a service that no other service depends on, lets docker compose up --detach --wait fail when it exits.

The "crux" with those "final" setup-services is that the behavior isn't immediately obvious and depends on the timing of the relevant services and when exactly docker compose up --detach --wait "thinks" the whole compose thing is ready.

But this ^^ is all from experiments. It's tricky to find good documentation on this behavior. However, the behavior of depends_on (... a setup service) with service_completed_successfully condition makes sense.

[MonkeyCanCode]

So yeah, while I was doing a local prototyping, I ran into this issue as well where --exit-code-from is not happy with this. What I ended up doing is to use tail /dev/null to keep the service up then use health check on service ready instead of completed. Sleep 120 here will work but if for whatever reason these bash scripts took more than 120s (assuming they get very complex later on), this will bite us.

[snazy]

Man, docker-compose isn't a good friend at all :(

[snazy]

Neither tail /dev/null nor sleep 120 is particularly great.
I wanted to stay away from the added complexity of health-checks for the setup-scripts though.
Let me think a bit about this.

[snazy]

Probably best to let the polaris-setup services "tail forever" and have health-checks. Added that

[MonkeyCanCode]

Yes. That is what I ended up doing in my local prototyping with tail forever as status code of zero is not okay for compose service.

[snazy]

@MonkeyCanCode mind taking another look?