Repository navigation
test(e2e): add interactive onboard policy-preset step-ordering test (#6042) #8618
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
prekshivyas
merged 23 commits into
NVIDIA:main
from
wakqasahmed:test/onboard-policy-preset-sequencing-6042
Aug 13, 2026
Merged
Changes from 11 commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
49c18e4
test(e2e): add interactive onboard policy-preset step-ordering test (…
wakqasahmed 6b2fe72
fix(e2e): use the shared docker fixture instead of an inline Docker c…
wakqasahmed 9ce0eda
fix(e2e): redact the generated API key from onboard test assertion me…
wakqasahmed a79fdb7
Merge branch 'main' into test/onboard-policy-preset-sequencing-6042
cv 4b0ea3d
Merge branch 'main' into test/onboard-policy-preset-sequencing-6042
cv eb5cfb4
fix(e2e): address maintainer review on the onboard PTY regression
wakqasahmed abe9231
merge(main): refresh PR #8618
apurvvkumaria b237902
merge(main): refresh PR #8618
apurvvkumaria 6aa218c
merge(main): refresh PR #8618
apurvvkumaria 38b5d26
test(e2e): complete onboard PTY review coverage
apurvvkumaria 006518e
merge: refresh onboard policy sequencing base
apurvvkumaria 7868381
test(e2e): terminate the PTY child session
prekshivyas b19cbeb
merge(main): refresh PR #8618
apurvvkumaria a27049e
Merge branch 'main' into test/onboard-policy-preset-sequencing-6042
cv ffaecff
Merge branch 'main' into test/onboard-policy-preset-sequencing-6042
cv 20df452
Merge branch 'main' into test/onboard-policy-preset-sequencing-6042
cv 747579b
merge(e2e): update #8618 with main
apurvvkumaria df8299c
Merge branch 'main' into test/onboard-policy-preset-sequencing-6042
cv b8cface
merge(main): refresh PR #8618
cv c4b5e05
ci(e2e): incorporate current main checks
cv 9172a61
ci(e2e): incorporate current main checks
cv 9bf0d0b
merge(e2e): retain formatted current-main checks
cv 3895ca5
merge(main): resolve PR #8618 conflict
cv File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,199 @@ | ||
| // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| import type { ChildProcessProgress } from "../fixtures/observed-child-process.ts"; | ||
| import { spawnObservedChild } from "../fixtures/observed-child-process.ts"; | ||
|
|
||
| // Drives an interactive CLI through a real PTY, the same technique | ||
| // `test/helpers/installer-express-prompt-pty-harness.ts` uses for the | ||
| // installer's express prompt. The real onboard wizard behaves differently | ||
| // under a piped, non-TTY stdin than under a real terminal (raw-mode | ||
| // keypress selectors, `isTTY`-gated prompts), so a faithful regression test | ||
| // for interactive-only behavior must drive a real PTY rather than pipe | ||
| // stdin. | ||
| // | ||
| // Rules fire independently and out of order: a rule whose trigger never | ||
| // appears (for example, a first-run license notice already accepted on a | ||
| // prior run) must not block a later rule from firing when its own trigger | ||
| // appears. | ||
| // | ||
| // The child process itself is launched through the shared | ||
| // `spawnObservedChild` boundary (the suite's single audited asynchronous | ||
| // child-process call) so it still tracks a content-free progress activity | ||
| // and canonical lifecycle checkpoints; this module attaches its own | ||
| // listeners on top to capture output and match rule triggers. | ||
|
|
||
| export interface InteractiveCommandRule { | ||
| readonly trigger: string; | ||
| readonly response: string; | ||
| } | ||
|
|
||
| export interface InteractiveCommandResult { | ||
| readonly exitCode: number; | ||
| readonly output: string; | ||
| readonly firedTriggers: readonly string[]; | ||
| readonly timedOut: boolean; | ||
| } | ||
|
|
||
| export interface DriveInteractiveCommandOptions { | ||
| readonly activityLabel: string; | ||
| readonly cmd: readonly [string, ...string[]]; | ||
| readonly cwd?: string; | ||
| readonly env: NodeJS.ProcessEnv; | ||
| readonly progress: ChildProcessProgress; | ||
| readonly rules: readonly InteractiveCommandRule[]; | ||
| readonly timeoutMs: number; | ||
| } | ||
|
|
||
| // Runs inside a Python child so it can fork a real pseudo-terminal; | ||
| // Node has no built-in PTY primitive and this repo does not depend on | ||
| // node-pty. The child's own deadline is generous; the Node-side timer | ||
| // below is the enforced hard bound and SIGKILLs the whole process tree. | ||
| const PTY_DRIVER_SCRIPT = ` | ||
| import json, os, pty, select, signal, sys, time | ||
|
|
||
| # Read from stdin, not argv: the payload embeds every scripted response, | ||
| # including any credential a rule supplies (e.g. an onboard API key), and a | ||
| # process argument stays visible to anything that can list the command | ||
| # line for as long as the child runs. | ||
| payload = json.loads(sys.stdin.read()) | ||
| cmd = payload["cmd"] | ||
| rules = payload["rules"] | ||
| timeout_s = payload["timeoutSeconds"] | ||
|
|
||
| pid, fd = pty.fork() | ||
| if pid == 0: | ||
| os.execvp(cmd[0], cmd) | ||
|
|
||
| output = bytearray() | ||
| os.set_blocking(fd, False) | ||
| deadline = time.monotonic() + timeout_s | ||
| fired = [False] * len(rules) | ||
| exit_code = None | ||
| while time.monotonic() < deadline: | ||
| ready, _, _ = select.select([fd], [], [], 0.2) | ||
| if ready: | ||
| try: | ||
| chunk = os.read(fd, 65536) | ||
| except OSError: | ||
| chunk = b"" | ||
| if not chunk: | ||
| # PTY close (EOF, or EIO on Linux) after the child has already | ||
| # exited normally must not fall through to the timeout branch | ||
| # below: reap it here and record its real exit code so a | ||
| # successful run is never misreported as DRIVER_TIMEOUT. | ||
| _, status = os.waitpid(pid, 0) | ||
| exit_code = os.waitstatus_to_exitcode(status) | ||
| break | ||
| output.extend(chunk) | ||
| sys.stdout.buffer.write(chunk) | ||
| sys.stdout.flush() | ||
| text = output.decode("utf-8", errors="ignore") | ||
| for i, rule in enumerate(rules): | ||
| if fired[i]: | ||
| continue | ||
| if rule["trigger"] in text: | ||
| os.write(fd, rule["response"].encode()) | ||
| sys.stderr.write("FIRED\\t" + rule["trigger"] + "\\n") | ||
| fired[i] = True | ||
| waited = os.waitpid(pid, os.WNOHANG) | ||
| if waited[0] == pid: | ||
| exit_code = os.waitstatus_to_exitcode(waited[1]) | ||
| break | ||
| if exit_code is None: | ||
| try: | ||
| os.kill(pid, signal.SIGKILL) | ||
| except ProcessLookupError: | ||
| pass | ||
| os.waitpid(pid, 0) | ||
| sys.stderr.write("DRIVER_TIMEOUT\\n") | ||
| sys.exit(124) | ||
| sys.exit(exit_code) | ||
| `; | ||
|
|
||
| function resolvePython(): string { | ||
| return process.env.NEMOCLAW_E2E_PYTHON3_BIN || "python3"; | ||
| } | ||
|
|
||
| export function driveInteractiveCommand( | ||
| options: DriveInteractiveCommandOptions, | ||
| ): Promise<InteractiveCommandResult> { | ||
| const payload = JSON.stringify({ | ||
| cmd: options.cmd, | ||
| rules: options.rules.map((rule) => ({ trigger: rule.trigger, response: rule.response })), | ||
| // Comfortably longer than the Node-side hard timeout below so the | ||
| // driver's own bookkeeping never races the enforced bound. | ||
| timeoutSeconds: Math.ceil(options.timeoutMs / 1000) + 30, | ||
| }); | ||
| // Kept directly in this function's body, not inside the Promise executor | ||
| // below, so the sole audited async child-process boundary stays attached | ||
| // to a named, reviewed callsite. `detached: true` makes the Python driver | ||
| // the leader of its own process group/session, so its pty.fork()'d | ||
| // onboard child (which inherits that group) can be reaped as a unit on | ||
| // timeout instead of surviving as an orphan. | ||
| const child = spawnObservedChild(resolvePython(), ["-c", PTY_DRIVER_SCRIPT], { | ||
| activityLabel: options.activityLabel, | ||
| progress: options.progress, | ||
| spawn: { cwd: options.cwd, env: options.env, detached: true }, | ||
| }); | ||
| // Written to stdin rather than passed as a process argument: the payload | ||
| // carries every scripted response, including any credential a rule | ||
| // supplies (see PTY_DRIVER_SCRIPT's matching comment). | ||
| child.stdin?.end(payload); | ||
|
|
||
| return new Promise((resolve, reject) => { | ||
| let output = ""; | ||
| const firedTriggers: string[] = []; | ||
| let stderrRest = ""; | ||
| let timedOut = false; | ||
| let settled = false; | ||
|
|
||
| const timer = setTimeout(() => { | ||
| timedOut = true; | ||
| // Kill the whole process group (negative pid), not just the driver | ||
| // pid: the driver's forked onboard child shares that group and would | ||
| // otherwise keep running past this test. | ||
| try { | ||
| if (child.pid) process.kill(-child.pid, "SIGKILL"); | ||
| } catch { | ||
| // Group may already be gone (driver exited between the deadline | ||
| // check and this signal); fall back to the direct pid below. | ||
| } | ||
| child.kill("SIGKILL"); | ||
| }, options.timeoutMs); | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
|
|
||
| // Additional listeners alongside spawnObservedChild's own content-free | ||
| // observer; this module needs the real transcript to match rule | ||
| // triggers and to report ordered step evidence. | ||
| child.stdout?.on("data", (chunk: Buffer) => { | ||
| output += chunk.toString("utf-8"); | ||
| }); | ||
| child.stderr?.on("data", (chunk: Buffer) => { | ||
| const lines = (stderrRest + chunk.toString("utf-8")).split("\n"); | ||
| stderrRest = lines.pop() ?? ""; | ||
| for (const line of lines) { | ||
| const fired = line.match(/^FIRED\t(.*)$/); | ||
| if (fired) firedTriggers.push(fired[1]); | ||
| } | ||
| }); | ||
| child.once("error", (error) => { | ||
| if (settled) return; | ||
| settled = true; | ||
| clearTimeout(timer); | ||
| reject(error); | ||
| }); | ||
| child.once("close", (code) => { | ||
| if (settled) return; | ||
| settled = true; | ||
| clearTimeout(timer); | ||
| const fired = stderrRest.match(/^FIRED\t(.*)$/); | ||
| if (fired) firedTriggers.push(fired[1]); | ||
| resolve({ | ||
| exitCode: timedOut ? 124 : (code ?? 1), | ||
| output, | ||
| firedTriggers, | ||
| timedOut, | ||
| }); | ||
| }); | ||
| }); | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.