[25.0 backport] libcontainerd/supervisor: fix data race

[thaJeztah]

• backport libcontainerd/supervisor: fix data race #47300
• related to Stop subprocesses from getting unexpectedly killed #44215
• fixes data race during daemon startup (libcontainerd/supervisor.(*remote).monitorDaemon()) #47287

The monitorDaemon() goroutine calls startContainerd() then blocks on <-daemonWaitCh to wait for it to exit. The startContainerd() function would (re)initialize the daemonWaitCh so a restarted containerd could be waited on. This implementation was race-free because startContainerd() would synchronously initialize the daemonWaitCh before returning. When the call to start the managed containerd process was moved into the waiter goroutine, the code to initialize the daemonWaitCh struct field was also moved into the goroutine. This introduced a race condition.

Move the daemonWaitCh initialization to guarantee that it happens before the startContainerd() call returns.

(cherry picked from commit dd20bf4)

- What I did
I fixed a data race in libcontainerd/supervisor that I added in #44215.

- How I did it
The monitorDaemon() goroutine calls startContainerd() then blocks on <-daemonWaitCh to wait for it to exit. The startContainerd() function would (re)initialize the daemonWaitCh so a restarted containerd could be waited on. This implementation was race-free because startContainerd() would synchronously initialize the daemonWaitCh before returning. When the call to start the managed containerd process was moved into the waiter goroutine, the code to initialize the daemonWaitCh struct field was also moved into the goroutine. This introduced a race condition.

Move the daemonWaitCh initialization to guarantee that it happens before the startContainerd() call returns.

- How to verify it
Build a daemon with BUILDFLAGS=-race. Start it and check that no WARNING: DATA RACE messages are emitted.

Verify that the containerd supervisor is working correctly: it is launched when the daemon starts up, is restarted when killed, and is shut down when the daemon shuts down.

- Description for the changelog

• Fixed a potential race condition in the managed containerd supervisor

- A picture of a cute animal (not mandatory but encouraged)

[thaJeztah]

Hmm... CodeCov being flaky?

Run codecov/codecov-action@v4
  with:
    directory: ./bundles/test-integration
    env_vars: RUNNER_OS
    flags: integration,rootless
  env:
    GO_VERSION: 1.21.6
    GOTESTLIST_VERSION: v0.3.1
    TESTSTAT_VERSION: v0.1.3
    ITG_CLI_MATRIX_SIZE: 6
    DOCKER_EXPERIMENTAL: 1
    DOCKER_GRAPHDRIVER: overlay2
    TEST_INTEGRATION_USE_SNAPSHOTTER: 
    OTEL_EXPORTER_OTLP_ENDPOINT: http://172.17.0.1:4318/
    DOCKER_ROOTLESS: 1
    CACHE_DEV_SCOPE: dev
    TESTREPORTS_NAME: ubuntu-22.04-rootless
==> linux OS detected
https://cli.codecov.io/latest/linux/codecov.SHA256SUM
==> Running version latest
==> Running version v0.4.6
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-commit'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-commit -C c12bbf549b4d52a5f5b4e6425a882f770cb903c3
gpg: directory '/home/runner/.gnupg' created
gpg: keybox '/home/runner/.gnupg/pubring.kbx' created
gpg: /home/runner/.gnupg/trustdb.gpg: trustdb created
gpg: key 806BB28AED779869: public key "Codecov Uploader (Codecov Uploader Verification Key) <[email protected]>" imported
gpg: Total number processed: 1
gpg:               imported: 1

Error: write EPIPE
    at afterWriteDispatched (node:internal/stream_base_commons:160:15)
    at writeGeneric (node:internal/stream_base_commons:151:3)
    at Socket._writeGeneric (node:net:952:11)
    at Socket._write (node:net:964:8)
    at writeOrBuffer (node:internal/streams/writable:447:12)
    at _write (node:internal/streams/writable:389:10)
    at Socket.Writable.end (node:internal/streams/writable:665:17)
    at Socket.end (node:net:722:31)
    at module.exports (/home/runner/work/_actions/codecov/codecov-action/v4/node_modules/gpg/lib/spawnGPG.js:50:1)
    at Object.call (/home/runner/work/_actions/codecov/codecov-action/v4/node_modules/gpg/lib/gpg.js:28:1)

[thaJeztah]

Other failure is a flaky test that we should really fix (failing often)

=== FAIL: amd64.integration.daemon TestLiveRestore/volume_references/local_volume_with_mount_options (2.35s)
    daemon_test.go:581: assertion failed: error is not nil: Error response from daemon: remove test-live-restore-volume-references-local: volume has active mounts
        --- FAIL: TestLiveRestore/volume_references/local_volume_with_mount_options (2.35s)

=== FAIL: amd64.integration.daemon TestLiveRestore/volume_references (17.13s)
    --- FAIL: TestLiveRestore/volume_references (17.13s)

=== FAIL: amd64.integration.daemon TestLiveRestore (0.00s)

[thaJeztah]

Thx! I'll bring this one in; it was a clean cherry-pick, and reviewed in master.