Skip to content

ci(installtest): move the Linux --daemon-engine guard to authkey - #389

Merged
gen16k merged 1 commit into
mainfrom
fix/388-daemon-engine-authkey
Aug 2, 2026
Merged

gen16k merged 1 commit into
mainfrom
fix/388-daemon-engine-authkey

Conversation

@gen16k

@gen16k gen16k commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

The Linux --daemon-engine installtest leg has not run a single assert since
2026-07-28. It dies 17 seconds in, at a mode guard that outlived the mode it
guards, and has done so on every scheduled nightly since:

[installtest] daemon-path enrol for wired-it-g1 (service left running; executor engine install)
[installtest] --daemon-engine supports IT_ENROLL_MODE=oidc only (got 'authkey')
##[error]Process completed with exit code 1.

Root cause

#290 replaced installtest's oidc enrol mode with authkey across the workflow
and all three OS drivers. Its own body says:

Two legs deliberately keep the old shape:

  • --daemon-engine (macOS/Windows) still completes an ordinary login
    out-of-band with the raw token.

Linux is missing from that sentence and from the change. #290's diff to
scripts/dev/lib/installtest-daemon-engine.sh touched three comments and left
the guard alone:

where mode it wants
.github/workflows/installtest-inference.yml:238 passes authkey
scripts/dev/installtest-macos.sh:651 authkey only — ported
scripts/dev/installtest-windows.ps1:1167 authkey only — ported
scripts/dev/lib/installtest-daemon-engine.sh:140 oidc only — not ported

Reverting the workflow is not an option: oidc mode no longer exists.
installtest-enroll.sh accepts authkey|interactive, and --google-sa-login
was deleted from the CLI in the same PR.

Why moving the guard is the whole fix

it_enroll_daemon_path never depended on the ambient enrol mode. It depends on
IT_IMPERSONATE_SA, gcloud on PATH, and the control plane's
/v1/login/oidc-grant — exactly the three things authkey mode already
requires (installtest-enroll.sh:67-70). The leg keeps the raw SA id_token
and completes the login out-of-band instead of spending the auth key, which is
the shape installtest-macos.sh (daemon_path_enroll_macos) and
installtest-windows.ps1 (-DaemonEngine) have used since #290.

Why it matters more now

After #138, no installer pre-installs the engine on any OS — waired init is
the only thing that puts one on a host. Its two engine-install paths split like
this:

path implementation leg
no wizard ensureDaemonPathEngine --inference
wizard-driven (executor lease) runSetupEngineInstall --daemon-engine

With this leg dead, the wizard-driven half had no live end-to-end coverage on
Linux at all.

Also in this PR

Comment drift left by #290, all the same class:

Verification

  • shellcheck — installtest-daemon-engine.sh clean; the SC2015/SC1091 info
    findings in installtest-enroll.sh / installtest-run.sh are pre-existing at
    untouched lines (same set on origin/main). Note these files are not in
    scripts/ci/install-script-lint.sh's target list, nor is
    installtest-windows.ps1 in ps-script-lint.ps1's.
  • bash -n on all three shell files; PowerShell parser check on the .ps1.
  • Not verifiable locally: the leg needs the dev control plane and an
    impersonated SA. installtest-inference.yml (os=linux) dispatched on this
    branch — the bar is that the leg reaches
    ==> daemon-path 'waired init' (fg ...) and prints a Tier-2 summary instead of
    exiting at the guard. Result recorded in a comment below.
  • Not testnet-relevant (scripts/dev/ is absent from
    testnet-relevant-paths.txt) and not a docs surface.

Fixes #388
Refs #300
Refs #138

@gen16k

gen16k commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

Verification: the leg is alive and green

installtest-inference.yml dispatched on this branch (os=linux, gpu=off) —
run 30735616525,
daemon-path engine install (linux): success, 30 passed, 0 failed, 0 skipped.

It clears the guard and runs the whole leg, including every assert this leg
exists for:

[installtest] daemon-path enrol for wired-it-g1 (service left running; executor engine install)
[installtest] running daemon-path 'waired init' (fg) in wired-it-g1 (cp=..., model=qwen2.5-coder-0.5b-instruct)
[installtest]  ok  init took the daemon path (setup-executor-capable first-run)
[installtest]  ok  daemon login completed out-of-band via the OIDC grant
[installtest]  ok  setup executor lease was live during setup (executor_attached)
[installtest]  ok  executor claimed the ollama install (install_claimed=ollama)
[installtest]  ok  bundled ollama installed by the daemon-path executor (/var/lib/waired/runtimes/ollama/bin/ollama)
[installtest]  ok  inference subsystem left no_engine (state=ready)
[installtest]  ok  no stuck executor install claim after init (install_claimed cleared)
[installtest] ==> Tier 2 summary: 30 passed, 0 failed, 0 skipped

Compare the same job on the previous dispatch, before this change
(run 30713258212):
it exited at the guard 17 seconds in, having run 8 Tier-1 asserts and none of
the above.

So runSetupEngineInstall — the executor installing the engine the browser
wizard asked for — is now proven end to end on Linux for the first time. Worth
noting install_claimed=ollama landed: that is the assert that distinguishes
"the executor drove the install" from "something else put a binary there", and
it is the one the macOS leg currently misses.

One non-fatal warn, pre-existing and not a #179 signal — the harness says so
itself: no desired_engine at the end of the leg, so engine_installed is false by definition. The control plane publishes no desired engine at that point, so
SetupState never computes engine_installed.

Other legs in the same dispatch

CI note

unit tests (windows) went red once on
TestReconciler_SafetyNetSilencedByRecentDirectEvidence — the known flake
already tracked in #357 / #384 (a 15 ms margin against Windows' ~15.6 ms timer
tick). This PR contains no Go. Locally the test is stable at -count=200 on
Linux, where the sleep slop is ~1 ms. Job rerun.

@gen16k

gen16k commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

Correction to the comment above: routing sentinel (linux) in that dispatch
failed, it did not pass. I read the wrong row — the green "routing sentinel
(linux, 0.5B)" is this PR's own check from routing-sentinel.yml, a different
workflow. The daemon-path engine install (linux) result and everything else
above stands.

Full dispatch result (run 30735616525):

leg result failing asserts
daemon-path engine install (linux) success 30 passed, 0 failed
install+inference (linux) failure 27 passed, 1 failed — no benchmark THROUGHPUT figure in init transcript
routing sentinel (linux) failure 28 passed, 2 failed — the same THROUGHPUT assert, plus TestIntegration/{claude, claude-anthropic-model-id, openclaw}

The THROUGHPUT assert is the known #300 baseline. The sentinel's TestIntegration
failures are not in that baseline — routing sentinel (linux) was green on
main's 2026-07-31 nightly and on the previous branch dispatch (30713258212).

This PR cannot be the cause: the sentinel leg enrolls through it_enroll_guest
and never sources installtest-daemon-engine.sh at all. The only files it does
touch that I changed are installtest-enroll.sh and installtest-run.sh, and
the diff there is comment-only:

$ git diff origin/main -- scripts/dev/lib/installtest-enroll.sh scripts/dev/installtest-run.sh \
    | grep -E '^[+-]' | grep -vE '^(\+\+\+|---)' | grep -vE '^[+-]\s*#'
(no output)

Rather than assert that from the diff alone, I dispatched the identical workflow
on main for an A/B:
run 30735808842.
Result to follow before this merges.

@gen16k

gen16k commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

Second dispatch settles the routing-sentinel question

run 30736163050,
same branch, same inputs:

leg 1st dispatch (30735616525) 2nd dispatch (30736163050) main baseline (30735808842)
daemon-path engine install (linux) success (30/0) success failure (dies at the guard)
routing sentinel (linux) failure success success

So the sentinel red in the first dispatch was not this branch. Its root cause is
visible in that run's engine log — the model runner died under the leg:

llama-server process has terminated: signal: segmentation fault (core dumped)
sched.go:651 msg="Load failed" ... error="llama-server process has terminated: signal: segmentation fault"
integration_test.go:102: drive gave up after 2 attempt(s) in 40s:
    the local model runner is dead (engine said "process has terminated")

which is the known #97 / #29 flake. It also explains the THROUGHPUT assert
failing in both legs of that run rather than just one: with :9475 refusing
connections, the boot benchmark logged
dial tcp 127.0.0.1:9475: connect: connection refused and never produced a
figure. One dead engine, three red asserts.

The daemon-path engine install (linux) result is the one that matters here and
it is now reproducible: green twice on this branch, red on main in between.
That is the A/B for this PR.

install+inference (linux) remains on the known #300 baseline (the THROUGHPUT
assert), identical to main's own run.

@gen16k
gen16k force-pushed the fix/388-daemon-engine-authkey branch from 6f2cae6 to e3a8def Compare August 2, 2026 07:28
The leg has not run an assert since 2026-07-28. It dies 17 seconds in:

    [installtest] --daemon-engine supports IT_ENROLL_MODE=oidc only (got 'authkey')

#290 replaced installtest's `oidc` enrol mode with `authkey` across the
workflow and all three OS drivers, and said so in its own body -- "Two legs
deliberately keep the old shape: --daemon-engine (macOS/Windows) still
completes an ordinary login out-of-band with the raw token". Linux is
missing from that sentence and from the change: the diff to this file
touched three comments and left the guard demanding a mode that no longer
exists (installtest-enroll.sh accepts authkey|interactive, and
--google-sa-login was deleted in the same PR).

Reverting the workflow is not an option, and nothing else has to move.
it_enroll_daemon_path never depended on the ambient enrol mode; it depends
on IT_IMPERSONATE_SA, gcloud, and the CP's /v1/login/oidc-grant, which is
exactly what authkey mode already requires. The leg keeps the RAW SA token
and completes the login out-of-band rather than spending the auth key --
the same shape installtest-macos.sh and installtest-windows.ps1 have used
since #290.

This matters more after #138: no installer pre-installs the engine on any
OS now, so `waired init` is the only thing that puts one on a host. Of its
two engine-install paths, --inference covers ensureDaemonPathEngine and
this leg is the only e2e that reaches runSetupEngineInstall -- the
wizard-driven half, which on Linux therefore had no live coverage at all.

Also refreshes the comments #290 left behind: the header still claimed the
engine is "installed by install.sh (--inference)" (false since #138) and
still explained --auth-key as a local-enroll selector, which #290 itself
made wrong. Plus two labels that drifted the same way -- run.sh's
"(oidc|bypass|interactive)" mode list and a Windows failure string reading
"waired init (oidc)" on the auth-key branch.

Verified: shellcheck (daemon-engine lib clean; the SC2015/SC1091 info
findings in enroll.sh/run.sh are pre-existing at untouched lines), bash -n,
pwsh parse. The leg itself needs the dev control plane -- dispatched
installtest-inference.yml (os=linux) on this branch.

Fixes #388

Signed-off-by: gen16k <gen16k@gmail.com>
@gen16k
gen16k force-pushed the fix/388-daemon-engine-authkey branch from e3a8def to 548c400 Compare August 2, 2026 07:36
@gen16k
gen16k merged commit 3de0a19 into main Aug 2, 2026
28 of 29 checks passed
@gen16k
gen16k deleted the fix/388-daemon-engine-authkey branch August 2, 2026 07:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci/installtest: Linux --daemon-engine leg dies at a stale IT_ENROLL_MODE=oidc guard (no e2e for the wizard-driven engine install)

1 participant