ci: add host-level cleanup for ECS runners

.github/scripts/ecs-runner/README.md

@@ -0,0 +1,94 @@
+# ECS Runner host cleanup
+
+This directory contains host-level cleanup for dedicated Linux ECS GitHub
+Actions runner hosts. It complements per-job workflow cleanup; it does not
+replace it.
+
+## What it does
+
+- Runs `qwen-docker-cleanup` once a day at 02:30 UTC, with up to 30 minutes of
+  randomized delay.
+- Removes Qwen CI containers — sandbox-labelled or named `qwen-code-*` —
+  created more than 24 hours ago.
+- Prunes sandbox-labelled Qwen CI images and dangling images older than 24
+  hours.
+- Uses the existing Qwen sandbox daemon lock to avoid concurrent Docker
+  maintenance.
+- Changes the host `/tmp` retention policy from 30 days to 7 days through
+  `systemd-tmpfiles`.
+
+Use this only on dedicated CI hosts. A sandbox-labelled or `qwen-code-*`-named
+container older than 24 hours is considered leaked and may be removed even if
+it is still running.
+
+## Prerequisites
+
+- Linux with systemd
+- Docker available as `docker.service`
+- `flock` and `timeout`
+- Root access for installation
+- Runner workspaces under `/home/github-runner`
+
+## Install or update
+
+From a Qwen Code checkout, run:
+
+```bash
+sudo .github/scripts/ecs-runner/install-qwen-docker-cleanup.sh
+```
+
+The installer copies the cleanup command and systemd units into the host,
+installs the seven-day `/tmp` policy, reloads systemd, and enables the timer.
+Run the same command again after pulling a newer version to update the host.
+
+## Verify
+
+```bash
+systemctl is-enabled qwen-docker-cleanup.timer
+systemctl is-active qwen-docker-cleanup.timer
+systemctl list-timers qwen-docker-cleanup.timer
+systemd-tmpfiles --cat-config | grep '^D /tmp '
+```
+
+The expected `/tmp` policy is:
+
+```text
+D /tmp 1777 root root 7d
+```
+
+## Run manually
+
+Drain the host first if it may contain a legitimate job or container running
+for more than 24 hours. Then run:
+
+```bash
+sudo systemctl start qwen-docker-cleanup.service
+sudo journalctl -u qwen-docker-cleanup.service --since today
+```
+
+To apply the `/tmp` retention policy immediately:
+
+```bash
+sudo systemd-tmpfiles --clean --prefix=/tmp
+```
+
+## Disable or uninstall
+
+```bash
+sudo systemctl disable --now qwen-docker-cleanup.timer
+sudo systemctl clean --what=state qwen-docker-cleanup.timer
+sudo rm -f /usr/local/sbin/qwen-docker-cleanup
+sudo rm -f /etc/systemd/system/qwen-docker-cleanup.service
+sudo rm -f /etc/systemd/system/qwen-docker-cleanup.timer
+sudo rm -f /etc/tmpfiles.d/tmp.conf
+sudo systemctl daemon-reload
+```
+
+Removing `/etc/tmpfiles.d/tmp.conf` restores the operating system's packaged
+`/tmp` policy on the next `systemd-tmpfiles` invocation.
+
+## Scope
+
+This cleanup intentionally does not delete runner workspaces, package caches,
+containerd leases, or containerd snapshots directly. Those resources require
+separate disk-pressure monitoring and a host drain before manual cleanup.

.github/scripts/ecs-runner/install-qwen-docker-cleanup.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+if ((EUID != 0)); then
+  echo 'Run this installer as root.' >&2
+  exit 1
+fi
+
+source_dir=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
+install -Dm755 "$source_dir/qwen-docker-cleanup.sh" /usr/local/sbin/qwen-docker-cleanup
+install -Dm644 "$source_dir/qwen-docker-cleanup.service" /etc/systemd/system/qwen-docker-cleanup.service
+install -Dm644 "$source_dir/qwen-docker-cleanup.timer" /etc/systemd/system/qwen-docker-cleanup.timer
+install -Dm644 "$source_dir/qwen-ci-tmp.conf" /etc/tmpfiles.d/tmp.conf
+systemctl daemon-reload
+systemctl enable --now qwen-docker-cleanup.timer

.github/scripts/ecs-runner/qwen-ci-tmp.conf

@@ -0,0 +1,2 @@
+# Dedicated CI hosts generate large numbers of disposable test directories.
+D /tmp 1777 root root 7d

.github/scripts/ecs-runner/qwen-docker-cleanup.service

@@ -0,0 +1,14 @@
+[Unit]
+Description=Clean stale Qwen CI Docker resources
+Requires=docker.service
+After=docker.service
+
+[Service]
+Type=oneshot
+# Type=oneshot units do NOT inherit DefaultTimeoutStartSec (systemd disables
+# the start timeout by default for oneshot), so this is the unit's only bound.
+# Keep it: without one, a wedged docker call hangs the run forever holding the
+# /run mutex, and every later elapse then skips. 1h clears the script's worst
+# case instead of cutting the sweep off mid-prune.
+TimeoutStartSec=1h
+ExecStart=/usr/local/sbin/qwen-docker-cleanup

.github/scripts/ecs-runner/qwen-docker-cleanup.sh

@@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+set -uo pipefail
+
+# Host-level mutex: only one cleanup per host at a time. Lives under /run, so
+# it is root-owned by design and never collides with the shared sandbox daemon
+# lock in the runner's home below.
+exec 9>/run/qwen-docker-cleanup.lock || {
+  echo 'error: cannot open /run/qwen-docker-cleanup.lock (running as root?)' >&2
+  exit 1
+}
+flock --nonblock 9 || { echo 'skipped: another cleanup is already running' >&2; exit 0; }
+
+# The sandbox daemon lock is created runner-owned by CI jobs on first use
+# (${HOME}/.cache/qwen-code-ci/docker-sandbox-daemon.lock). Never create it
+# here: this service runs as root, and a root-owned directory or lock file in
+# the runner's home would make every later CI job's `exec 9>` fail with EACCES
+# under `set -e`.
+runner_home=$(getent passwd github-runner | cut -d: -f6 || true)
+daemon_lock="${runner_home:-/home/github-runner}/.cache/qwen-code-ci/docker-sandbox-daemon.lock"
+
+# Reap leaked containers older than 24h. `--all` includes stopped containers;
+# the age gate, not the daemon lock, is what protects still-running jobs, so
+# the lock is only taken around the labelled prune below.
+now=$(date +%s)
+reap_stale() {
+  local filter="$1" now="$2"
+  while read -r id; do
+    [[ -n "$id" ]] || continue
+    created=$(docker inspect --format '{{.Created}}' "$id" 2>/dev/null) || continue
+    created_at=$(date -d "$created" +%s 2>/dev/null) || continue
+    if ((now - created_at > 86400)); then
+      docker rm --force "$id" || echo "warning: failed to remove stale container $id" >&2
+    fi
+  done < <(docker ps --all --quiet --filter "$filter")
+}
+# e2e/release containers carry the sandbox label (set at image build time);
+# autofix/review containers come from the published image, which does not, and
+# are named qwen-code-*.
+reap_stale 'label=org.qwen-code.ci.sandbox=true' "$now"
+reap_stale 'name=qwen-code-' "$now"
+
+# Dangling images are untagged and unreferenced, so this prune needs no daemon
+# lock and runs even on hosts that never create it.
+docker image prune --force --filter 'until=24h' || echo 'warning: dangling image prune failed' >&2
+
+# Take the shared daemon lock exclusively, non-blocking, only around the
+# labelled prune — the same shape as the job's own prune step. Holding it
+# across the reap loop above would starve CI jobs waiting on `flock --shared
+# --wait 1800` for the whole step. Append (not `>`) so opening the lock never
+# truncates an inode another process is holding a flock on. Skip the prune
+# when the lock has never been created: it exists only after a docker
+# e2e/release leg has run, and creating it here as root would break later CI.
+if [[ ! -e "$daemon_lock" ]]; then
+  echo "skipped: no sandbox daemon lock at $daemon_lock; labelled image prune skipped" >&2
+  exit 0
+fi
+exec 8>>"$daemon_lock" || {
+  echo "error: cannot open sandbox daemon lock at $daemon_lock" >&2
+  exit 1
+}
+flock --nonblock 8 || { echo 'skipped: sandbox daemon lock busy' >&2; exit 0; }
+
+timeout 20m docker image prune --all --force \
+  --filter 'label=org.qwen-code.ci.sandbox=true' \
+  --filter 'until=24h' || echo 'warning: Qwen CI image prune failed' >&2

.github/scripts/ecs-runner/qwen-docker-cleanup.timer

@@ -0,0 +1,10 @@
+[Unit]
+Description=Run Qwen CI Docker cleanup daily
+
+[Timer]
+OnCalendar=*-*-* 02:30:00 UTC
+RandomizedDelaySec=30m
+Persistent=true
+
+[Install]
+WantedBy=timers.target