From 65cbfe64a377728bb261d58bfc4f1fcbe3a71964 Mon Sep 17 00:00:00 2001 From: gen16k Date: Tue, 21 Jul 2026 13:33:42 +0000 Subject: [PATCH] =?UTF-8?q?test(installtest):=203-OS=20daemon-path=20setup?= =?UTF-8?q?-executor=20engine=20install=20leg=20(waired#835=20=C2=A79/?= =?UTF-8?q?=C2=A711)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The waired#835 §9/§11 setup executor engine install — the resident `sudo waired init` attaching a management-API lease and installing the engine the daemon-path first-run needs — was covered only by unit tests. No installtest leg exercised it end to end, because the two hands-free enrol modes the harness uses (`--google-sa-login`, `--bypass-mode`) both FORCE the standalone enrol path (cmd/waired/main.go gates the daemon path on `!bypassMode && !googleSALogin && !renewing && daemonReachable`). Since #119 the installer starts the daemon before `waired init`, so a real fresh first-run now takes the DAEMON path — the exact path the harness never drove. New `--daemon-engine` / `-DaemonEngine` leg (its own mode; Tier 2) on all three drivers, run nightly as a 3-OS job in installtest-inference.yml: - Leaves the service RUNNING and installs with the engine ABSENT (install.sh/.ps1 keep --skip-ollama), so only the daemon-path executor can put an engine on the host. - Runs `waired init` WITHOUT --google-sa-login (→ daemon path) and with --non-interactive (→ awaitBrowserSetup returns at once → the resident executor runs ensureDaemonPathEngine), inference on + a tiny pinned model so the trailing pull stays cheap. - Completes the login hands-free by SCRAPING the login-session id from the init transcript (the login URL's last path segment) and POSTing the host-minted SA id_token to the CP's /v1/login/oidc-grant — which flips any waiting session, whatever created it. Scrape, not POST /login/start: the #838 writeGuard refuses mgmt writes on the TCP port (they must use the local IPC socket / named pipe), and reads dodge it. - Asserts (via GET /waired/v1/setup/state — a read): the enrol took the daemon path, the OIDC completion succeeded, the executor lease went live (executor_attached) and claimed the ollama install (install_claimed), an engine is present afterward (the regression bar — pre-N3 it stayed engine-less forever), the subsystem left no_engine, and no install claim is stuck after init (§9-4). Not asserting setup-progress engine_install=done / setup_state.engine_installed here: those require a CP-served desired_engine (a browser-wizard / management write the hands-free harness has no auth for), so that path stays unit-tested; this leg proves the resident executor installs the engine on the real daemon-path first-run. Linux logic lives in the new scripts/dev/lib/installtest-daemon-engine.sh (kept out of installtest-enroll.sh to avoid churn); macOS/Windows mirror it inline. Cannot be validated locally (real-CP OIDC + a real engine install + self-hosted Windows/macOS runners) — needs a nightly workflow_dispatch run. shellcheck / pwsh-parse / actionlint all clean. Refs waired-ai/waired#835 Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: gen16k --- .github/workflows/installtest-inference.yml | 80 +++++- scripts/dev/installtest-macos.sh | 133 +++++++++- scripts/dev/installtest-run.sh | 48 +++- scripts/dev/installtest-windows.ps1 | 147 ++++++++++- scripts/dev/lib/installtest-daemon-engine.sh | 253 +++++++++++++++++++ 5 files changed, 644 insertions(+), 17 deletions(-) create mode 100644 scripts/dev/lib/installtest-daemon-engine.sh diff --git a/.github/workflows/installtest-inference.yml b/.github/workflows/installtest-inference.yml index c00f51942..30a08ad9d 100644 --- a/.github/workflows/installtest-inference.yml +++ b/.github/workflows/installtest-inference.yml @@ -1,6 +1,7 @@ # installtest-inference — the nightly installtest umbrella (#514, waired#760; # re-homed from the pre-split monorepo per #1): the install→inference tail of -# the first-run journey, the 3-OS coding-agent routing sentinel, and the +# the first-run journey, the 3-OS daemon-path setup-executor engine install +# (waired#835 §9/§11), the 3-OS coding-agent routing sentinel, and the # Windows installer banner render check. # # installtest.yml (the per-PR gate) runs the real `install → service-manager @@ -196,6 +197,83 @@ jobs: fi } >> "$GITHUB_STEP_SUMMARY" + # 3-OS daemon-path setup-executor engine install (waired#835 §9/§11). Unlike + # install+inference above, the engine is DELIBERATELY absent after install + # (install.sh/.ps1 run --skip-ollama) and the enrol is driven through the + # DAEMON path — the real post-#119 first-run — by completing the daemon's + # login session out-of-band via the OIDC grant, so the resident `waired init` + # executor installs the engine. The standalone --google-sa-login enrol the + # other legs use never reaches that path. Nightly (a real engine install + + # tiny model pull is minutes-scale + external-state, like install+inference). + # See scripts/dev/lib/installtest-daemon-engine.sh for the WHY. + daemon-engine: + needs: legs + name: daemon-path engine install (${{ matrix.os }}) + runs-on: ${{ matrix.runs-on }} + timeout-minutes: ${{ matrix.timeout }} + strategy: + fail-fast: false # a red Windows/macOS leg must not hide a green Linux + matrix: ${{ fromJSON(needs.legs.outputs.matrix) }} + env: + # Tier 2: hands-free enroll vs the real dev control plane (#339 grant). + IT_ENROLL_MODE: oidc + IT_IMPERSONATE_SA: waired-devtest-login@dev-waired.iam.gserviceaccount.com + steps: + - uses: actions/checkout@v5 + + - uses: actions/setup-go@v6 + with: + go-version-file: go.mod + cache: ${{ matrix.os == 'linux' }} + + - name: point gcloud at the image's Python ≥3.10 (self-hosted macOS) + if: matrix.os == 'macos' + run: | + set -euo pipefail + py="$(/opt/homebrew/bin/brew --prefix python@3.14)/bin/python3.14" + test -x "$py" + echo "CLOUDSDK_PYTHON=$py" >> "$GITHUB_ENV" + + - uses: google-github-actions/auth@v3 + with: + workload_identity_provider: ${{ vars.GCP_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.GCP_SERVICE_ACCOUNT_EMAIL }} + + - uses: google-github-actions/setup-gcloud@v3 + with: + skip_install: ${{ matrix.os == 'windows' }} + + # Linux: in-place install (systemd is PID 1 on the disposable runner), + # engine ABSENT (--skip-ollama), then daemon-path init → executor install. + - name: daemon-path engine install (linux, tier 2) + if: matrix.os == 'linux' + env: + IT_LOCAL: "1" + IT_ALLOW_LOCAL_DESTRUCTIVE: "1" + run: | + set -euo pipefail + export PATH="$HOME/go/bin:$PATH" + bash scripts/dev/installtest-host-up.sh + bash scripts/dev/installtest-run.sh --tier 2 --daemon-engine + + - name: daemon-path engine install (windows, tier 2) + if: matrix.os == 'windows' + shell: pwsh + run: ./scripts/dev/installtest-windows.ps1 -Tier 2 -DaemonEngine + + - name: daemon-path engine install (macos, tier 2) + if: matrix.os == 'macos' + run: bash scripts/dev/installtest-macos.sh --tier 2 --daemon-engine + + - name: job summary + if: always() + shell: bash + run: | + { + echo "## daemon-path engine install (${{ matrix.os }})" + echo "- result: **${{ job.status }}**" + } >> "$GITHUB_STEP_SUMMARY" + # 3-OS coding-agent routing sentinel (#496). Rides the SAME enrolled-daemon # harness with `--integration`, which pins the tiny 0.5B as the bundled # model (deploy pulls ~0.4 GB), then runs the Go routing harness that diff --git a/scripts/dev/installtest-macos.sh b/scripts/dev/installtest-macos.sh index 8cbcc6011..69fc685e6 100755 --- a/scripts/dev/installtest-macos.sh +++ b/scripts/dev/installtest-macos.sh @@ -32,12 +32,14 @@ ROOT="$(git rev-parse --show-toplevel)" TIER=1 INFER=0 INTEG=0 +DAEMON_ENGINE=0 while [ $# -gt 0 ]; do case "$1" in --tier) shift; TIER="${1:?--tier needs N}" ;; --tier=*) TIER="${1#--tier=}" ;; --inference) INFER=1 ;; --integration) INTEG=1; INFER=1 ;; # routing sentinel rides the inference engine + --daemon-engine) DAEMON_ENGINE=1 ;; # waired#835 §9/§11 daemon-path executor engine install -h|--help) sed -n '2,30p' "$0"; exit 0 ;; *) echo "unknown argument: $1" >&2; exit 1 ;; esac @@ -229,10 +231,128 @@ assert_mgmt_socket_macos() { "$BINDIR/waired" resume >/dev/null 2>&1 || true } +# --- daemon-path setup-executor engine install (waired#835 §9/§11) ---------- +# macOS analog of lib/installtest-daemon-engine.sh. The system LaunchDaemon is +# already running (Tier 1), so a `waired init` WITHOUT --google-sa-login takes +# the DAEMON path and its resident executor installs the engine — the path the +# --google-sa-login (standalone) enrol never reaches. We complete the daemon's +# login session out-of-band via the OIDC grant (the CP flips any waiting +# session, internal/controlplane/api/oidc_grant.go), then assert the engine +# landed via the executor, not install.sh (which ran --skip-ollama). +DAEMON_ENGINE_MODEL="qwen2.5-coder-0.5b-instruct" +DAEMON_ENGINE_FLAG="$WORK/daemon-engine.flag" + +# _daemon_setup_watcher — background half: scrape the login session id +# from init's transcript (a READ — POST /login/start is refused on TCP by the +# #838 writeGuard), complete it out-of-band at the CP, then watch the executor +# lease while init installs the engine. Records facts into the flag. +_daemon_setup_watcher() { + local tok="$1" url="" sess="" st _ seen_exec="" seen_claim="" + : > "$DAEMON_ENGINE_FLAG" + for _ in $(seq 1 60); do + url="$(grep -oE 'https?://[^[:space:]]+' "$INITLOG" 2>/dev/null | head -1)" + if [ -n "$url" ]; then sess="${url##*/}"; sess="${sess%%[?#]*}"; fi + [ -n "$sess" ] && break + sleep 1 + done + if [ -z "$sess" ]; then echo "no-session" >> "$DAEMON_ENGINE_FLAG"; return; fi + echo "session=$sess" >> "$DAEMON_ENGINE_FLAG" + if curl -fsS --max-time 20 -X POST -H 'Content-Type: application/json' \ + -d "{\"login_session_id\":\"$sess\",\"id_token\":\"$tok\"}" \ + "$IT_CONTROL_URL/v1/login/oidc-grant" >/dev/null 2>&1; then + echo "completed=1" >> "$DAEMON_ENGINE_FLAG" + else + echo "complete-failed" >> "$DAEMON_ENGINE_FLAG"; return + fi + for _ in $(seq 1 150); do + st="$(curl -fsS --max-time 5 http://127.0.0.1:9476/waired/v1/setup/state 2>/dev/null || true)" + if [ -z "$seen_exec" ] && printf '%s' "$st" | grep -qE '"executor_attached"[[:space:]]*:[[:space:]]*true'; then + echo "executor_attached=1" >> "$DAEMON_ENGINE_FLAG"; seen_exec=1 + fi + if [ -z "$seen_claim" ] && printf '%s' "$st" | grep -qE '"install_claimed"[[:space:]]*:[[:space:]]*"ollama"'; then + echo "install_claimed=ollama" >> "$DAEMON_ENGINE_FLAG"; seen_claim=1 + fi + sleep 2 + done +} + +# daemon_path_enroll_macos — foreground daemon-path init while +# the watcher completes login and observes the lease. init runs as root (sudo); +# the already-running LaunchDaemon makes it take the daemon path. +daemon_path_enroll_macos() { + local tok="$1" device="$2" watcher_pid rc + : > "$INITLOG" # fresh: the watcher must not scrape a stale login URL + _daemon_setup_watcher "$tok" & + watcher_pid=$! + it_step "daemon-path 'waired init' (fg, no --google-sa-login → daemon path)" + # inference on + tiny model so an engine-less host installs one; --non-interactive + # so the resident executor runs ensureDaemonPathEngine; stdin from /dev/null. + # pipefail makes the `if` see init's exit; PIPESTATUS[0] is init's, not tee's. + if sudo env WAIRED_NO_EMOJI=1 "$BINDIR/waired" init --control "$IT_CONTROL_URL" \ + --device-name "$device" --inference-enabled=true \ + --inference-bundled-model-id="$DAEMON_ENGINE_MODEL" \ + --non-interactive --skip-integration --state-dir "$STATE_DIR" \ + &1 | tee "$INITLOG"; then + rc=0 + else + rc="${PIPESTATUS[0]}" + fi + kill "$watcher_pid" 2>/dev/null || true + wait "$watcher_pid" 2>/dev/null || true + [ "$rc" -eq 0 ] || it_warn "daemon-path init exited $rc — asserts will surface what landed" +} + +# assert_daemon_engine_macos — the executor engine-install asserts (analog of +# lib/installtest-daemon-engine.sh's assert_daemon_engine). Regression bar: an +# engine-less daemon-path first-run ends up WITH an engine (pre-N3 it stayed +# engine-less and engine_install was red forever). +assert_daemon_engine_macos() { + local out state claim ollama_bin="" cand + grep -q "signing in via the daemon" "$INITLOG" 2>/dev/null \ + && ok "init took the daemon path (setup-executor-capable first-run)" \ + || bad "init did NOT take the daemon path (executor engine install not exercised)" + grep -q '^completed=1' "$DAEMON_ENGINE_FLAG" 2>/dev/null \ + && ok "daemon login completed out-of-band via the OIDC grant" \ + || bad "out-of-band OIDC completion did not report success" + grep -q '^executor_attached=1' "$DAEMON_ENGINE_FLAG" 2>/dev/null \ + && ok "setup executor lease was live during setup (executor_attached)" \ + || bad "never observed executor_attached — executor engine-install path not reached" + grep -q '^install_claimed=ollama' "$DAEMON_ENGINE_FLAG" 2>/dev/null \ + && ok "executor claimed the ollama install (install_claimed=ollama)" \ + || it_warn "did not catch install_claimed=ollama in the 2 s poll — non-fatal" + for cand in \ + "$(command -v ollama 2>/dev/null || true)" \ + /Applications/Ollama.app/Contents/Resources/ollama \ + /usr/local/bin/ollama /opt/homebrew/bin/ollama; do + if [ -n "$cand" ] && [ -x "$cand" ]; then ollama_bin="$cand"; break; fi + done + [ -n "$ollama_bin" ] \ + && ok "ollama engine installed by the daemon-path executor ($ollama_bin)" \ + || bad "no engine after a daemon-path first-run (executor install did not land — pre-N3 behaviour)" + out="$(curl -fsS --max-time 5 http://127.0.0.1:9476/waired/v1/inference/status 2>/dev/null || true)" + state="$(printf '%s' "$out" | grep -oE '"subsystem_state"[[:space:]]*:[[:space:]]*"[a-z_]+"' | head -1 | grep -oE '"[a-z_]+"$' | tr -d '"')" + case "$state" in + ""|no_engine) bad "inference subsystem still reports '${state:-unreachable}' (engine not installed)" ;; + *) ok "inference subsystem left no_engine (state=$state)" ;; + esac + claim="$(curl -fsS --max-time 5 http://127.0.0.1:9476/waired/v1/setup/state 2>/dev/null \ + | sed -n 's/.*"install_claimed"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p')" + [ -z "$claim" ] \ + && ok "no stuck executor install claim after init (install_claimed cleared)" \ + || bad "executor install claim still set after init (install_claimed=$claim; stuck)" +} + # Passwordless sudo is a hard requirement now that the agent is a system # daemon (install.sh sudo's the register/init steps; we sudo the asserts). sudo -n true 2>/dev/null || it_die "passwordless sudo required (system LaunchDaemon install needs root)" +# --daemon-engine (waired#835 §9/§11) is its own mode: install.sh keeps +# --skip-ollama (engine ABSENT), enrol goes daemon-path via out-of-band OIDC. +if [ "$DAEMON_ENGINE" = 1 ]; then + { [ "$INFER" = 1 ] || [ "$INTEG" = 1 ]; } && it_die "--daemon-engine is its own mode; not with --inference/--integration" + [ "$TIER" -ge 2 ] || it_die "--daemon-engine needs --tier 2 (it enrols to reach the executor)" +fi + # --- build the darwin tarball install.sh will consume ----------------------- arch="$(uname -m)"; [ "$arch" = "x86_64" ] && arch=amd64 # arm64 stays arm64 tarball="waired-darwin-${arch}.tar.gz" @@ -355,6 +475,13 @@ if [ "$TIER" -ge 2 ]; then [ -n "$tok" ] || it_die "failed to mint an SA id_token (CI principal in oidc_grant_token_creators?)" device="mac-ci-${GITHUB_RUN_ID:-$(date +%Y%m%d%H%M%S)}" + if [ "$DAEMON_ENGINE" = 1 ]; then + # Daemon-path enrol: complete the login out-of-band so the resident + # executor installs the engine (waired#835 §9/§11). No --google-sa-login + # (that forces the standalone path), so the running LaunchDaemon makes + # init take the daemon path. + daemon_path_enroll_macos "$tok" "$device" + else inf_flag="--inference-enabled=$([ "$INFER" = 1 ] && echo true || echo false)" # Routing sentinel pins the tiny 0.5B so the deploy pulls ~0.4 GB (fits the # 4 GB macOS runner; dodges the #573 7B OOM). Zero args when not --integration. @@ -374,6 +501,7 @@ if [ "$TIER" -ge 2 ]; then --skip-integration --state-dir "$STATE_DIR" 2>&1 | tee "$INITLOG"; then bad "waired init (oidc) failed" fi + fi it_step "Tier 2 asserts" sudo test -f "$STATE_DIR/identity.json" && ok "identity.json written under state dir" \ @@ -402,7 +530,10 @@ if [ "$TIER" -ge 2 ]; then it_step "management write socket asserts (waired#838)" assert_mgmt_socket_macos - if [ "$INFER" = 1 ]; then + if [ "$DAEMON_ENGINE" = 1 ]; then + it_step "daemon-path executor engine-install asserts (waired#835 §9/§11)" + assert_daemon_engine_macos + elif [ "$INFER" = 1 ]; then it_step "inference asserts (--inference)" assert_inference_macos fi diff --git a/scripts/dev/installtest-run.sh b/scripts/dev/installtest-run.sh index 8df047955..e6fc38a13 100755 --- a/scripts/dev/installtest-run.sh +++ b/scripts/dev/installtest-run.sh @@ -27,6 +27,14 @@ # and asserts via the observability event ring that the completion was served # LOCALLY and did not fail open to real Anthropic. # +# --daemon-engine (waired#835 §9/§11): drive the DAEMON-path first-run so the +# resident `waired init` executor installs the engine on an engine-less host +# (the path the standalone oidc/bypass enrol never reaches). install.sh keeps +# --skip-ollama (engine ABSENT), enrol completes the daemon's login session +# out-of-band via the OIDC grant (lib/installtest-daemon-engine.sh), and a +# tiny bundled model keeps the trailing pull cheap. Pairs with Tier 2; its +# own mode (not combinable with --inference/--integration). +# # A system container is used for Tier 1/2 (fast); Tier 3 forces a VM. # # Usage: @@ -34,6 +42,7 @@ # bash scripts/dev/installtest-run.sh --tier 2 # + headless enroll # bash scripts/dev/installtest-run.sh --tier 2 --inference # + Ollama/model/benchmark (CPU) # bash scripts/dev/installtest-run.sh --tier 2 --integration --local # + routing sentinel (0.5B) +# bash scripts/dev/installtest-run.sh --tier 2 --daemon-engine # + daemon-path executor engine install (waired#835) # bash scripts/dev/installtest-run.sh --tier 3 # + data plane (2 VMs) # bash scripts/dev/installtest-run.sh --keep # don't delete the guest # bash scripts/dev/installtest-run.sh --name foo --image ubuntu:22.04 @@ -49,6 +58,7 @@ WITH_TRAY=0 USE_VM=0 INFER=0 INTEG=0 +DAEMON_ENGINE=0 NAME="g1" while [ $# -gt 0 ]; do case "$1" in @@ -58,11 +68,12 @@ while [ $# -gt 0 ]; do --with-tray) WITH_TRAY=1 ;; --inference) INFER=1 ;; --integration) INTEG=1; INFER=1 ;; # routing sentinel rides the inference engine + --daemon-engine) DAEMON_ENGINE=1 ;; # waired#835 §9/§11 daemon-path executor engine install --vm) USE_VM=1 ;; --local) IT_LOCAL=1 ;; --name) shift; NAME="${1:?--name needs a value}" ;; --image) shift; IT_IMAGE="${1:?--image needs a value}" ;; - -h|--help) sed -n '2,33p' "$0"; exit 0 ;; + -h|--help) sed -n '2,48p' "$0"; exit 0 ;; *) it_die "unknown argument: $1 (try --help)" ;; esac shift @@ -74,6 +85,17 @@ done # deploy pulls ~0.4 GB, not the 7B — cheap enough for a per-PR Linux leg. [ "$INTEG" = 1 ] && export IT_BUNDLED_MODEL_ID="${IT_BUNDLED_MODEL_ID:-qwen2.5-coder-0.5b-instruct}" +# --daemon-engine (waired#835 §9/§11) is its own mode: unlike --inference it +# keeps install.sh's --skip-ollama (engine ABSENT), so only the daemon-path +# executor can install one. It pins the same tiny model for a cheap trailing +# pull. Not combinable with --inference/--integration (different enrol paths). +if [ "$DAEMON_ENGINE" = 1 ]; then + { [ "$INFER" = 1 ] || [ "$INTEG" = 1 ]; } && it_die \ + "--daemon-engine is its own mode; do not combine it with --inference/--integration" + [ "$TIER" -ge 2 ] || it_die "--daemon-engine needs --tier 2 (it enrols to reach the executor)" + export IT_BUNDLED_MODEL_ID="${IT_BUNDLED_MODEL_ID:-qwen2.5-coder-0.5b-instruct}" +fi + # --local installs waired ON THIS HOST as root (apt + systemd + a service # user + a running daemon). Safe only on a disposable machine — guard so a # developer can't nuke their workstation by accident; CI opts in explicitly. @@ -127,7 +149,7 @@ launch_guest() { return 0 fi [ "$USE_VM" = 1 ] && extra+=(--vm) - if [ "$INFER" = 1 ] && [ -n "${IT_GUEST_MEMORY-16GiB}" ]; then + if { [ "$INFER" = 1 ] || [ "$DAEMON_ENGINE" = 1 ]; } && [ -n "${IT_GUEST_MEMORY-16GiB}" ]; then # Applied at LAUNCH, not via a post-launch `lxc config set`: a VM's # memory is fixed at boot, so the old post-launch set was silently # ineffective (error swallowed by `|| true`) and a VM guest ran the @@ -289,13 +311,21 @@ if [ "$TIER" -le 2 ]; then if [ "$TIER" -ge 2 ]; then # shellcheck source=scripts/dev/lib/installtest-enroll.sh source "$ROOT/scripts/dev/lib/installtest-enroll.sh" - it_enroll_guest "$GUEST" # enrol (IT_ENROLL_MODE) against the Control Plane - assert_tier2 "$GUEST" - [ "$INFER" = 1 ] && assert_inference "$GUEST" - if [ "$INTEG" = 1 ]; then - # shellcheck source=scripts/dev/lib/installtest-integration.sh - source "$ROOT/scripts/dev/lib/installtest-integration.sh" - assert_integration "$GUEST" + if [ "$DAEMON_ENGINE" = 1 ]; then + # shellcheck source=scripts/dev/lib/installtest-daemon-engine.sh + source "$ROOT/scripts/dev/lib/installtest-daemon-engine.sh" + it_enroll_daemon_path "$GUEST" # daemon-path enrol via out-of-band OIDC completion + assert_tier2 "$GUEST" # identity chain still applies (the daemon owns it) + assert_daemon_engine "$GUEST" # the waired#835 §9/§11 executor engine install + else + it_enroll_guest "$GUEST" # enrol (IT_ENROLL_MODE) against the Control Plane + assert_tier2 "$GUEST" + [ "$INFER" = 1 ] && assert_inference "$GUEST" + if [ "$INTEG" = 1 ]; then + # shellcheck source=scripts/dev/lib/installtest-integration.sh + source "$ROOT/scripts/dev/lib/installtest-integration.sh" + assert_integration "$GUEST" + fi fi # Last: it toggles pause/resume, so keep it clear of the asserts above. assert_mgmt_socket "$GUEST" diff --git a/scripts/dev/installtest-windows.ps1 b/scripts/dev/installtest-windows.ps1 index 35dc347e5..bdfa8915b 100644 --- a/scripts/dev/installtest-windows.ps1 +++ b/scripts/dev/installtest-windows.ps1 @@ -72,7 +72,14 @@ param( # ISCC-compile the Inno installer from the same staged binaries, install it # silently, re-run Tier-1-level asserts (no second enroll), uninstall. # Implies -Contract (it needs the -Clean uninstall between the two installs). - [switch]$ExeVariant + [switch]$ExeVariant, + # -DaemonEngine (waired#835 §9/§11): drive the DAEMON-path first-run so the + # resident `waired init` executor installs the engine on an engine-less host + # -- the path the standalone --google-sa-login enrol never reaches (that flag + # forces the standalone path). Keeps install.ps1's engine-absent state, + # completes the daemon login out-of-band via the OIDC grant, and asserts the + # engine landed via the executor (not install.ps1). Its own mode; Tier 2. + [switch]$DaemonEngine ) # -WithIntegration rides the inference engine. @@ -83,6 +90,14 @@ if ($Contract -and $Tier -lt 2) { Write-Host "[installtest] -Contract requires -Tier 2 (asserts need an enrolled device)" -ForegroundColor Red exit 1 } +if ($DaemonEngine -and ($WithInference -or $WithIntegration)) { + Write-Host "[installtest] -DaemonEngine is its own mode; not with -WithInference/-WithIntegration" -ForegroundColor Red + exit 1 +} +if ($DaemonEngine -and $Tier -lt 2) { + Write-Host "[installtest] -DaemonEngine requires -Tier 2 (it enrolls to reach the executor)" -ForegroundColor Red + exit 1 +} $ErrorActionPreference = 'Stop' $ProgressPreference = 'SilentlyContinue' @@ -136,6 +151,48 @@ function ItSoft { } } +# --- daemon-path executor engine-install assert (waired#835 §9/§11) ---------- +# Windows analog of lib/installtest-daemon-engine.sh's assert_daemon_engine. +# Regression bar: an engine-less daemon-path first-run ends up WITH an engine +# (pre-N3 it stayed engine-less and engine_install was red forever). install.ps1 +# ran engine-absent, so only the resident executor could have installed one. +function Assert-DaemonEngine { + param([string]$InitLog, [string]$Flag) + + if (Select-String -Path $InitLog -Pattern 'signing in via the daemon' -Quiet -ErrorAction SilentlyContinue) { + ItOk "init took the daemon path (setup-executor-capable first-run)" + } else { ItBad "init did NOT take the daemon path (executor engine install not exercised)" } + + $flagText = if (Test-Path -LiteralPath $Flag) { Get-Content -LiteralPath $Flag -Raw } else { '' } + if ($flagText -match '(?m)^completed=1') { ItOk "daemon login completed out-of-band via the OIDC grant" } + else { ItBad "out-of-band OIDC completion did not report success" } + if ($flagText -match '(?m)^executor_attached=1') { ItOk "setup executor lease was live during setup (executor_attached)" } + else { ItBad "never observed executor_attached -- executor engine-install path not reached" } + if ($flagText -match '(?m)^install_claimed=ollama') { ItOk "executor claimed the ollama install (install_claimed=ollama)" } + else { ItLog "did not catch install_claimed=ollama in the 2s poll -- non-fatal" } + + # The regression bar: an engine is present (mirror Assert-Inference's lookup). + $ollama = $null + foreach ($p in @( + (Join-Path $env:ProgramFiles 'Ollama\ollama.exe'), + (Join-Path $env:LOCALAPPDATA 'Programs\Ollama\ollama.exe'))) { + if (Test-Path -LiteralPath $p) { $ollama = $p; break } + } + if (-not $ollama) { $cmd = Get-Command ollama.exe -ErrorAction SilentlyContinue; if ($cmd) { $ollama = $cmd.Source } } + if ($ollama) { ItOk "ollama engine installed by the daemon-path executor ($ollama)" } + else { ItBad "no engine after a daemon-path first-run (executor install did not land -- pre-N3 behaviour)" } + + $state = '' + try { $state = (Invoke-RestMethod -Uri 'http://127.0.0.1:9476/waired/v1/inference/status' -TimeoutSec 5).subsystem_state } catch { } + if ($state -and $state -ne 'no_engine') { ItOk "inference subsystem left no_engine (state=$state)" } + else { ItBad "inference subsystem still reports '$state' (engine not installed)" } + + $claim = '' + try { $claim = (Invoke-RestMethod -Uri 'http://127.0.0.1:9476/waired/v1/setup/state' -TimeoutSec 5).install_claimed } catch { } + if (-not $claim) { ItOk "no stuck executor install claim after init (install_claimed cleared)" } + else { ItBad "executor install claim still set after init (install_claimed=$claim; stuck)" } +} + # --- inference assert (Windows analog of assert_inference) ------------------- # Prove the Ollama-install -> bundled-model-pull -> benchmark tail of the # first-run journey ran (Tier-2 -WithInference): `waired init @@ -650,12 +707,15 @@ if ($Tier -ge 2) { if ($EnrollMode -ne 'oidc') { ItDie "installtest-windows.ps1 supports IT_ENROLL_MODE=oidc only (got '$EnrollMode')" } if (-not $ImpersonateSa) { ItDie "IT_ENROLL_MODE=oidc needs IT_IMPERSONATE_SA (the #339 test SA)" } - ItStep "enrolling via OIDC grant (google-sa-login, host-minted token)" + ItStep "enrolling via OIDC grant (host-minted token)" # Stop the installer-started service so init's enroll writes identity # without daemon contention. init then starts the agent itself (default # --start-agent=true) — mirroring a real install — so #519's foreground # model wait runs; the Start-Service below is a redundant safety net. - Stop-Service -Name $ServiceName -Force -ErrorAction SilentlyContinue + # Daemon-path mode is the exception: it leaves the service RUNNING — + # that (unenrolled but reachable) is what makes init take the daemon + # path and reach the setup executor engine install (waired#835 §11). + if (-not $DaemonEngine) { Stop-Service -Name $ServiceName -Force -ErrorAction SilentlyContinue } $aud = (Invoke-RestMethod -Uri "$ControlUrl/v1/login/oidc-grant/audience" -TimeoutSec 15).audience if (-not $aud) { ItDie "could not resolve the OIDC audience from $ControlUrl/v1/login/oidc-grant/audience" } @@ -666,8 +726,77 @@ if ($Tier -ge 2) { $runId = if ($env:GITHUB_RUN_ID) { $env:GITHUB_RUN_ID } else { Get-Date -Format yyyyMMddHHmmss } $device = "win-ci-$runId" $waired = Join-Path $InstallDir 'waired.exe' - $inferFlag = if ($WithInference) { '--inference-enabled=true' } else { '--inference-enabled=false' } $initLog = Join-Path $Work 'init.log' + if ($DaemonEngine) { + # Daemon-path enrol: complete the login out-of-band so the resident + # executor installs the engine (waired#835 §9/§11). No + # --google-sa-login (that forces the standalone path); the running + # service makes init take the daemon path. A background job rejoins + # the in-flight session (POST /login/start is single-flight → + # init's session), completes it via the OIDC grant (the CP flips any + # waiting session), then watches the executor lease. + $daemonFlag = Join-Path $Work 'daemon-engine.flag' + $watcher = Start-Job -ScriptBlock { + param($controlUrl, $tok, $initLog, $flag) + $ErrorActionPreference = 'SilentlyContinue' + Set-Content -LiteralPath $flag -Value '' -NoNewline + # (1) Scrape the login session id from init's transcript (a READ: + # POST /login/start is refused on TCP by the #838 writeGuard). The + # session id is the login URL's last path segment (lastPathSegment). + $sess = $null + for ($i = 0; $i -lt 60 -and -not $sess; $i++) { + $txt = '' + try { $txt = Get-Content -LiteralPath $initLog -Raw -ErrorAction SilentlyContinue } catch { } + if ($txt -and $txt -match 'https?://\S+') { + $seg = (($Matches[0] -split '/')[-1] -split '[?#]')[0] + if ($seg) { $sess = $seg } + } + if (-not $sess) { Start-Sleep 1 } + } + if (-not $sess) { Add-Content -LiteralPath $flag -Value 'no-session'; return } + Add-Content -LiteralPath $flag -Value "session=$sess" + # (2) Complete out-of-band at the CP (no writeGuard there). + try { + Invoke-RestMethod -Uri "$controlUrl/v1/login/oidc-grant" -Method Post -ContentType 'application/json' ` + -Body (@{ login_session_id = $sess; id_token = $tok } | ConvertTo-Json -Compress) -TimeoutSec 20 | Out-Null + Add-Content -LiteralPath $flag -Value 'completed=1' + } catch { Add-Content -LiteralPath $flag -Value 'complete-failed'; return } + $seenExec = $false; $seenClaim = $false + for ($i = 0; $i -lt 150; $i++) { + try { + $stt = Invoke-RestMethod -Uri 'http://127.0.0.1:9476/waired/v1/setup/state' -TimeoutSec 5 + if (-not $seenExec -and $stt.executor_attached) { Add-Content -LiteralPath $flag -Value 'executor_attached=1'; $seenExec = $true } + if (-not $seenClaim -and $stt.install_claimed -eq 'ollama') { Add-Content -LiteralPath $flag -Value 'install_claimed=ollama'; $seenClaim = $true } + } catch { } + Start-Sleep 2 + } + } -ArgumentList $ControlUrl, $tok, $initLog, $daemonFlag + + # inference on + tiny model so an engine-less host installs one; + # --non-interactive so the resident executor runs + # ensureDaemonPathEngine. NO --google-sa-login → daemon path. + $initArgs = @( + 'init' + '--control', $ControlUrl + '--device-name', $device + '--inference-enabled=true' + '--inference-bundled-model-id=qwen2.5-coder-0.5b-instruct' + '--non-interactive' + '--skip-integration' + '--state-dir', $StateDir + ) + $env:WAIRED_NO_EMOJI = '1' + $prevEap = $ErrorActionPreference + $ErrorActionPreference = 'Continue' + & $waired @initArgs 2>&1 | Tee-Object -FilePath $initLog + $initExit = $LASTEXITCODE + $ErrorActionPreference = $prevEap + Stop-Job $watcher -ErrorAction SilentlyContinue + Receive-Job $watcher -ErrorAction SilentlyContinue | Out-Null + Remove-Job $watcher -Force -ErrorAction SilentlyContinue + if ($initExit -ne 0) { ItLog "daemon-path init exited $initExit -- asserts will surface what landed" } + } else { + $inferFlag = if ($WithInference) { '--inference-enabled=true' } else { '--inference-enabled=false' } # Build the whole init arg vector as ONE flat array and splat it once (matches # packaging/install/install.ps1's $initArgs idiom and the bash legs' initargs=(...)). # Do NOT build a separate $pinArgs via `if {@('x')} else {@()}` and splat it inline: @@ -707,9 +836,11 @@ if ($Tier -ge 2) { $initExit = $LASTEXITCODE $ErrorActionPreference = $prevEap if ($initExit -ne 0) { ItBad "waired init (oidc) exited $initExit" } + } # Safety net: init already started the agent (--start-agent default); - # this is a no-op unless that best-effort start was skipped. + # this is a no-op unless that best-effort start was skipped. Harmless in + # daemon-path mode too (the service was never stopped). Start-Service -Name $ServiceName -ErrorAction SilentlyContinue ItStep "Tier 2 asserts" @@ -739,7 +870,11 @@ if ($Tier -ge 2) { ItStep "management write pipe asserts (waired#838)" Assert-MgmtPipe - if ($WithInference) { + if ($DaemonEngine) { + ItStep "daemon-path executor engine-install asserts (waired#835 §9/§11)" + Assert-DaemonEngine -InitLog $initLog -Flag $daemonFlag + } + elseif ($WithInference) { ItStep "inference asserts (-WithInference)" Assert-Inference -InitLog $initLog } diff --git a/scripts/dev/lib/installtest-daemon-engine.sh b/scripts/dev/lib/installtest-daemon-engine.sh new file mode 100644 index 000000000..e5760531a --- /dev/null +++ b/scripts/dev/lib/installtest-daemon-engine.sh @@ -0,0 +1,253 @@ +# shellcheck shell=bash +# installtest-daemon-engine.sh — Tier-2 leg for the waired#835 §9/§11 +# daemon-path setup-executor engine install. +# +# Sourced by installtest-run.sh (--daemon-engine, Tier >=2), AFTER +# installtest-enroll.sh (this file reuses _it_dev_name / IT_BUNDLED_OLLAMA_BIN +# and relies on ok()/bad()/gx()/it_* from run.sh + common.sh). +# +# WHY A SEPARATE LEG (the coverage gap it closes) +# ------------------------------------------------ +# The other installtest legs enroll hands-free with `waired init +# --google-sa-login` (oidc) or `--bypass-mode`. Both of those FORCE the +# standalone enroll path (cmd/waired/main.go: the daemon path is gated on +# `!bypassMode && !googleSALogin && !renewing && daemonReachable`). On the +# standalone path the engine, when installed at all, is installed by +# install.sh (--inference) or by the interactive `configureInference` hook — +# never by the setup executor. So the waired#835 §9/§11 executor engine +# install (`ensureDaemonPathEngine` / `runSetupEngineInstall`, the resident +# `sudo waired init` attaching a lease and installing the engine the browser +# wizard asked for) was covered only by unit tests, never end to end. +# +# Since #119 the installer starts the daemon BEFORE `waired init`, so a real +# fresh first-run now takes the DAEMON path — the exact path the standalone +# enroll here never exercises. This leg drives that real path hands-free. +# +# HOW IT DRIVES THE DAEMON PATH HANDS-FREE +# ---------------------------------------- +# `waired init` on the daemon path proxies a Control-Plane login session it +# creates via POST /waired/v1/login/start, then polls the daemon until the +# session goes active. The CP's OIDC direct grant (POST +# {control}/v1/login/oidc-grant, internal/controlplane/api/oidc_grant.go) +# completes ANY waiting session by id — it does not care which client created +# it. So we: +# 1. run `waired init` in the FOREGROUND *without* --google-sa-login (so it +# takes the daemon path) and with --non-interactive (so awaitBrowserSetup +# returns at once and the resident executor runs ensureDaemonPathEngine), +# inference on + a tiny bundled model so an engine-less host installs one; +# 2. from a background watcher, rejoin the in-flight session (POST +# /login/start is single-flight → returns init's session id) and complete +# it out-of-band with a host-minted SA id_token; +# 3. still in the watcher, poll GET /waired/v1/setup/state while init +# installs the engine, recording whether the executor lease went live +# (executor_attached) and claimed the ollama install (install_claimed) — +# proof the resident executor, not install.sh, drove the install. +# +# oidc mode only (mirrors installtest-macos.sh): the token is minted on the +# HOST with gcloud and the completion POST is sent from the host to the public +# CP; the guest only exposes the login session on its loopback mgmt API. + +# The tiny bundled model this leg pins (~0.4 GB), so the model pull that +# follows the engine install stays cheap. Honours an IT_BUNDLED_MODEL_ID +# override set by the caller (installtest-run.sh). +IT_DAEMON_ENGINE_MODEL="${IT_BUNDLED_MODEL_ID:-qwen2.5-coder-0.5b-instruct}" + +# _it_daemon_mint_token — mint the SA id_token on the host (oidc mode only). +# Echoes the token on stdout, empty (return 1) on failure. Precondition +# checks (mode / SA / gcloud) live in the caller so their it_die fires in the +# script — an it_die inside this $()-captured function would only exit the +# subshell. Callers wrap the capture in `|| true` so a mint failure surfaces +# as an empty token, not a set -e abort. +_it_daemon_mint_token() { + local aud="$IT_OIDC_AUDIENCE" + if [ -z "$aud" ]; then + aud="$(curl -fsS --max-time 15 "$IT_CONTROL_URL/v1/login/oidc-grant/audience" 2>/dev/null \ + | sed -n 's/.*"audience":"\([^"]*\)".*/\1/p')" + fi + [ -n "$aud" ] || return 1 + gcloud auth print-identity-token \ + --impersonate-service-account="$IT_IMPERSONATE_SA" \ + --audiences="$aud" --include-email 2>/dev/null +} + +# _it_daemon_setup_watcher — the +# background half of the daemon-path enrol. Records single-line facts into +# (grep'd by assert_daemon_engine), so the foreground init and this +# watcher never share shell state. +_it_daemon_setup_watcher() { + local guest="$1" initlog="$2" tok="$3" flag="$4" url="" sess="" st _ seen_exec="" seen_claim="" + : > "$flag" + + # (1) Discover the login session by SCRAPING the URL `waired init` prints + # ("Sign in using this link:\n ", presentLoginURL) from the tee'd + # transcript, and take its last path segment as the session id (the same + # lastPathSegment the standalone path uses). This is a READ — deliberately + # not a POST /login/start, which the #838 writeGuard refuses on the TCP port + # (mgmt writes must use the local IPC socket / named pipe). + for _ in $(seq 1 60); do + url="$(grep -oE 'https?://[^[:space:]]+' "$initlog" 2>/dev/null | head -1 || true)" + if [ -n "$url" ]; then sess="${url##*/}"; sess="${sess%%[?#]*}"; fi + [ -n "$sess" ] && break + sleep 1 + done + if [ -z "$sess" ]; then echo "no-session" >> "$flag"; return; fi + echo "session=$sess" >> "$flag" + + # (2) Complete it out-of-band at the CP (public; no writeGuard there — that + # guard is on the agent's LOCAL mgmt API, not the control plane). + if curl -fsS --max-time 20 -X POST -H 'Content-Type: application/json' \ + -d "{\"login_session_id\":\"$sess\",\"id_token\":\"$tok\"}" \ + "$IT_CONTROL_URL/v1/login/oidc-grant" >/dev/null 2>&1; then + echo "completed=1" >> "$flag" + else + echo "complete-failed" >> "$flag" + return + fi + + # (3) Watch the executor lease while init installs the engine (~5 min + # budget; the foreground kills us the moment init returns). Record each + # fact once. + for _ in $(seq 1 150); do + st="$(gx "$guest" curl -fsS --max-time 5 \ + http://127.0.0.1:9476/waired/v1/setup/state 2>/dev/null || true)" + if [ -z "$seen_exec" ] && \ + printf '%s' "$st" | grep -qE '"executor_attached"[[:space:]]*:[[:space:]]*true'; then + echo "executor_attached=1" >> "$flag"; seen_exec=1 + fi + if [ -z "$seen_claim" ] && \ + printf '%s' "$st" | grep -qE '"install_claimed"[[:space:]]*:[[:space:]]*"ollama"'; then + echo "install_claimed=ollama" >> "$flag"; seen_claim=1 + fi + sleep 2 + done +} + +# it_enroll_daemon_path — daemon-path enrol that reaches the setup +# executor engine install. Unlike it_enroll_guest it does NOT stop the +# service: the running-but-unenrolled daemon is what makes `waired init` take +# the daemon path. +it_enroll_daemon_path() { + local guest="$1" name initlog flag tok watcher_pid rc + name="$(_it_dev_name "$guest")" + mkdir -p "$IT_LOGDIR" + initlog="$IT_LOGDIR/init-daemon-$name.log" + flag="$IT_LOGDIR/daemon-engine-$name.flag" + + it_log "daemon-path enrol for $guest (service left running; executor engine install)" + [ "$IT_ENROLL_MODE" = oidc ] || it_die \ + "--daemon-engine supports IT_ENROLL_MODE=oidc only (got '$IT_ENROLL_MODE')" + [ -n "$IT_IMPERSONATE_SA" ] || it_die \ + "IT_ENROLL_MODE=oidc needs IT_IMPERSONATE_SA (the #339 test SA)" + command -v gcloud >/dev/null 2>&1 || it_die \ + "oidc enrol mints the SA id_token on the host; gcloud not found on PATH." + # `|| true`: a mint failure must surface as the it_die below (empty token), + # not as a set -e abort on the command substitution. + tok="$(_it_daemon_mint_token || true)" + [ -n "$tok" ] || it_die \ + "failed to mint an SA id_token — is your identity in oidc_grant_token_creators \ +on $IT_IMPERSONATE_SA? (roles/iam.serviceAccountTokenCreator, or the CP audience \ +endpoint is unreachable / --enable-oidc-grant is off)" + + # Fresh log so the watcher can't scrape a stale login URL from a prior run + # (the foreground `tee` below also truncates, but the watcher may read first). + : > "$initlog" + _it_daemon_setup_watcher "$guest" "$initlog" "$tok" "$flag" & + watcher_pid=$! + + # Foreground daemon-path init: NO --google-sa-login (→ daemon path), + # --non-interactive (→ awaitBrowserSetup returns at once → the resident + # executor runs ensureDaemonPathEngine), inference on + tiny model so an + # engine-less host installs one. Blocks through engine install + model pull + # + benchmark. stdin from /dev/null: a rerouted terminal must not block on a + # prompt. Teed for the daemon-path signature assert. `if` guards `set -e` + # around a non-zero init; PIPESTATUS[0] is init's own exit (not tee's). + it_log "running daemon-path 'waired init' (fg) in $guest (cp=$IT_CONTROL_URL, model=$IT_DAEMON_ENGINE_MODEL)" + if gx "$guest" sh -c "WAIRED_NO_EMOJI=1 waired init \ + --control '$IT_CONTROL_URL' --device-name '$name' \ + --inference-enabled=true --inference-bundled-model-id='$IT_DAEMON_ENGINE_MODEL' \ + --non-interactive --skip-integration --state-dir /var/lib/waired &1" \ + | tee "$initlog"; then + rc=0 + else + rc="${PIPESTATUS[0]}" + fi + + kill "$watcher_pid" 2>/dev/null || true + wait "$watcher_pid" 2>/dev/null || true + + # No post-init restart (unlike it_enroll_guest): the daemon enrolled AND + # installed the engine in place, so it is already serving the enrolled state + # + the installed engine. A restart would only risk a transient no_engine + # read while the inference subsystem re-profiles. + if [ "$rc" -ne 0 ]; then + it_warn "daemon-path 'waired init' exited $rc in $guest — asserts below will surface what landed" + fi +} + +# assert_daemon_engine — verify the daemon-path executor installed the +# engine on an engine-less host. The regression bar (item 5): pre-N3 an +# engine-less daemon-path first-run stayed engine-less and engine_install was +# red forever; N3 makes the resident executor install it. +assert_daemon_engine() { + local guest="$1" name initlog flag out state claim + name="$(_it_dev_name "$guest")" + initlog="$IT_LOGDIR/init-daemon-$name.log" + flag="$IT_LOGDIR/daemon-engine-$name.flag" + + # 1. The enrol took the DAEMON path — the only path with a setup executor. + if grep -q "signing in via the daemon" "$initlog" 2>/dev/null; then + ok "init took the daemon path (setup-executor-capable first-run)" + else + bad "init did NOT take the daemon path (executor engine install not exercised)" + sed 's/^/ /' "$initlog" 2>/dev/null | tail -20 >&2 || true + fi + + # 2. The out-of-band OIDC completion drove the daemon login to active. + if grep -q '^completed=1' "$flag" 2>/dev/null; then + ok "daemon login completed out-of-band via the OIDC grant" + else + bad "out-of-band OIDC completion did not report success (flag: $(tr '\n' ' ' < "$flag" 2>/dev/null))" + fi + + # 3. The resident executor lease went live while init installed the engine. + if grep -q '^executor_attached=1' "$flag" 2>/dev/null; then + ok "setup executor lease was live during setup (executor_attached)" + else + bad "never observed executor_attached — the executor engine-install path was not reached" + fi + + # 4. The lease claimed the ollama install (executor drove it, not install.sh). + # Non-fatal: the 2 s poll can miss a very short install window. + if grep -q '^install_claimed=ollama' "$flag" 2>/dev/null; then + ok "executor claimed the ollama install (install_claimed=ollama)" + else + it_warn "did not catch install_claimed=ollama in the 2 s poll (short install window?) — non-fatal" + fi + + # 5. THE REGRESSION BAR: an engine-less daemon-path host ends up WITH an + # engine. install.sh ran with --skip-ollama, so only the executor could + # have put it here. + if gx "$guest" test -x "$IT_BUNDLED_OLLAMA_BIN"; then + ok "bundled ollama installed by the daemon-path executor ($IT_BUNDLED_OLLAMA_BIN)" + else + bad "no engine after a daemon-path first-run (executor install did not land — pre-N3 behaviour)" + gx "$guest" journalctl -u waired-agent --no-pager -n 30 2>&1 | sed 's/^/ /' || true + fi + + # 6. The inference subsystem left the no_engine state. + out="$(gx "$guest" curl -fsS --max-time 5 http://127.0.0.1:9476/waired/v1/inference/status 2>/dev/null || true)" + state="$(printf '%s' "$out" | grep -oE '"subsystem_state"[[:space:]]*:[[:space:]]*"[a-z_]+"' | head -1 | grep -oE '"[a-z_]+"$' | tr -d '"')" + case "$state" in + ""|no_engine) bad "inference subsystem still reports '${state:-unreachable}' (engine not installed)" ;; + *) ok "inference subsystem left no_engine (state=$state)" ;; + esac + + # 7. No stuck install claim after init (§9-4: no never-resolving spinner). + claim="$(gx "$guest" curl -fsS --max-time 5 http://127.0.0.1:9476/waired/v1/setup/state 2>/dev/null \ + | sed -n 's/.*"install_claimed"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p')" + if [ -z "$claim" ]; then + ok "no stuck executor install claim after init (install_claimed cleared)" + else + bad "executor install claim still set after init (install_claimed=$claim; stuck spinner)" + fi +}