Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 100 additions & 0 deletions integration-tests/test-helper.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ function isProcessAlive(pid: number): boolean {
}
}

/** Emitted by the stand-in CLI below; nothing else in the run writes it. */
const FORWARD_CANARY = 'PTY_FORWARD_CANARY_11002';

describe('TestRig', () => {
const originalKeepOutput = process.env['KEEP_OUTPUT'];

Expand Down Expand Up @@ -89,6 +92,103 @@ describe('TestRig', () => {
.toBe(false);
});

// Skipped on Windows: node-pty's kill() terminates the child unconditionally
// there, so it cannot outlive cleanup() and the premise below does not hold.
it.skipIf(process.platform === 'win32')(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R4-2: Disclosure rather than a code defect: this file sits outside every npm workspace, so this review run executed zero tests against the diff — build-test ran zero suites and the efficacy probe ran 0 revert probes, 0 mutants and 0 hunk probes (harnessValidated: null). The only collector of this file is the CI Integration Tests (no-AK, No Sandbox) check (root script test:integration:no-ak:sandbox:none, ci.yml integration_no_ak job), which was still pending on this PR's head at review time. If the new test itself is defective — assertions pass for the wrong reason, or it hangs or flakes under CI's KEEP_OUTPUT environment — nothing in this review run would surface it; land the PR only after that check completes green on the current head, since it is the only witness the #11002 fix has.

Witness:

build-test report: ok:true, buildSet:[], test:[], testScope.workspaces: []
  caveat: "1 changed file(s) sit outside every workspace and are not inert
  (e.g. integration-tests/test-helper.test.ts)"
gh pr checks at review time: "Integration Tests (no-AK, No Sandbox)" pending

No code change required.

中文说明

这是披露而非代码缺陷:该文件位于所有 npm workspace 之外,因此本轮审查没有对该 diff 执行任何测试——build-test 运行了 0 个套件,有效性探针运行了 0 个还原探针、0 个变异体、0 个 hunk 探针(harnessValidated: null)。该文件唯一的收集者是 CI 的 Integration Tests (no-AK, No Sandbox) 检查项(根脚本 test:integration:no-ak:sandbox:noneci.ymlintegration_no_ak 作业),而在审查时它在本 PR 的 head 上仍处于 pending 状态。如果新测试本身有缺陷——断言因错误的原因通过,或在 CI 的 KEEP_OUTPUT 环境下挂起或抖动——本轮审查无法发现;请等该检查项在当前 head 上变绿后再合并,因为它是 #11002 修复的唯一见证。无需代码改动。

— qwen3.8-max via Qwen Code /review (v0.23.0)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disclosure acknowledged — it asks for no code change and none was made. Recording the evidence that closes its ask, and leaving the thread open because that ask is a merge-time condition no commit in this round can discharge.

The named collector did run green on the reviewed head. Integration Tests (no-AK, No Sandbox) (root script test:integration:no-ak:sandbox:none, the only collector of integration-tests/test-helper.test.ts) completed SUCCESS at 2026-09-04T16:16:59Z, having started at 15:46:29Z — after head d1d375bed4 was committed at 15:45:58Z. So the sole witness for the #11002 fix exists and passed on the head this review read.

Local witness for the same file, this round. npx vitest run --root ./integration-tests ./test-helper.test.ts (default mode, repo config including retry: 2) → 8 passed (8), 290ms. In installed-release mode after R4-1's fix → 7 passed | 1 skipped, 79ms.

The disclosed gap is by design, not missing coverage. Integration Tests (CLI, No Sandbox) is merge_group-only and shows SKIPPED on every pull request; ci.yml records that the separate no-AK check was given its own name precisely so a skipped CLI lane is not read as "the changed integration test never ran" (#9895 round 15).

Why this stays open. This round pushes a new head, so "green on the head that lands" has to be re-met by the no-AK check on that commit — a merge-time gate for the maintainer.

中文说明

已知悉该披露——它未要求任何代码改动,本轮也未做代码改动。此处记录可以满足其诉求的证据;该线程保持未解决,因为它的诉求是一个合并时点条件,本轮任何提交都无法代为完成。

它所点名的收集检查项已在被审查的 head 上跑绿。 Integration Tests (no-AK, No Sandbox)(根脚本 test:integration:no-ak:sandbox:none,是 integration-tests/test-helper.test.ts 的唯一收集者)于 2026-09-04T16:16:59Z 以 SUCCESS 完成,开始于 15:46:29Z——晚于 head d1d375bed4 的提交时间 15:45:58Z。因此 #11002 修复的唯一见证确实存在,并且在本次审查所读取的 head 上通过了。

本轮针对同一文件的本地见证。 npx vitest run --root ./integration-tests ./test-helper.test.ts(默认模式,沿用仓库配置,含 retry: 2)→ 8 passed (8),290ms。在按 R4-1 修复后的已安装发布模式下 → 7 passed | 1 skipped,79ms。

该披露的审查缺口是设计使然,并非覆盖缺失。 Integration Tests (CLI, No Sandbox) 仅在 merge_group 触发,在每个 pull request 上都显示 SKIPPEDci.yml 中记载,单独设立 no-AK 检查项并赋予其独立名称,正是为了避免把 CLI 检查项的 skip 读成"改动的集成测试从未运行"(#9895 第 15 轮)。

为何保持未解决。 本轮会推送新的 head,因此"在最终合入的 head 上跑绿"这一条件需要由 no-AK 检查项在该提交上重新满足——这是留给维护者的合并时点关卡。

"detaches a session's output forwarding during cleanup",
async () => {
// KEEP_OUTPUT is what the OpenTUI leg sets, and it is what makes the rig
// forward every PTY byte into this worker's stdout.
process.env['KEEP_OUTPUT'] = 'true';
const rig = new TestRig();
await rig.setup('cleanup detaches interactive output');
// Stands in for the CLI bundle, which traps the SIGHUP that node-pty's
// signal-less kill() sends and keeps rendering while its exit cleanup
// drains. The child therefore outlives cleanup() on purpose and keeps
// producing bytes: what is under test is whether the harness still
// forwards them into a stdout pipe vitest is about to destroy (#11002).
rig.bundlePath = rig.createFile(
'slow-exit-cli.js',
Comment on lines +110 to +111

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R4-1: Under the harness's supported INTEGRATION_TEST_USE_INSTALLED_GEMINI=true mode (the npm-bundle verification mode documented in _getCommandAndArgs's JSDoc), rig.bundlePath is silently dropped from argv — runInteractive spawns the installed qwen CLI instead of this stand-in script, which never emits the canary. A release-validation run of the suite in that mode then burns the full 20s poll (×3 with retry: 2) and fails with the stand-in CLI never produced output — a diagnostic that points at the stand-in file that was never spawned, sending whoever debugs it to a file that is not in the failure path. No lane in this repo sets the variable today, so nothing automated goes red; the cost lands on whoever runs the suite in installed-release mode.

Witness:

Probe (scratch tree at 8340b5ff0, real installed qwen):
DEFAULT arm  ✓ test-helper.test.ts (8 tests | 7 skipped) 158ms
BUG arm (INTEGRATION_TEST_USE_INSTALLED_GEMINI=true)
             × detaches a session's output forwarding during cleanup 20011ms
               AssertionError: the stand-in CLI never produced output:
               expected false to be true
FIX arm (skipIf extended, installed mode)  Tests 8 skipped (8)
FIX arm (skipIf extended, default mode)    1 passed | 7 skipped (8)

Extend the existing skip to cover that mode:

it.skipIf(
  process.platform === 'win32' ||
    process.env['INTEGRATION_TEST_USE_INSTALLED_GEMINI'] === 'true',
)(

The guard must key on the exact switch runInteractive honors: process.env.INTEGRATION_TEST_USE_INSTALLED_GEMINI === 'true' (integration-tests/test-helper.ts:943-944; same string at :279).

中文说明

在脚手架自身支持的 INTEGRATION_TEST_USE_INSTALLED_GEMINI=true 模式(_getCommandAndArgs 的 JSDoc 所载的 npm 包验证模式)下,rig.bundlePath 会被静默地从 argv 中丢弃——runInteractive 会改为派生已安装的 qwen CLI,而不是这个替身脚本,于是 canary 永远不会出现。在该模式下运行发布验证时,测试会耗满整个 20 秒轮询(retry: 2 下重复约 3 次),并以 the stand-in CLI never produced output 失败——这条诊断指向一个从未被派生的替身文件,会把排查者引向失败路径之外的文件。目前仓库里没有任何检查项设置该变量,因此自动化流程不会变红;代价落在以已安装发布模式运行套件的人身上。

修复:扩展现有的 skip 以覆盖该模式(见上方代码块)。守卫必须绑定 runInteractive 实际读取的开关:process.env.INTEGRATION_TEST_USE_INSTALLED_GEMINI === 'true'integration-tests/test-helper.ts:943-944:279 处为同一字符串)。

— qwen3.8-max via Qwen Code /review (v0.23.0)

[
"process.on('SIGHUP', () => {});",
`setInterval(() => process.stdout.write('${FORWARD_CANARY}\\n'), 20);`,
'setTimeout(() => process.exit(0), 30000);',
'',
].join('\n'),
);

const { ptyProcess } = rig.runInteractive();
// node-pty hands every onData registration its own disposable, so this
// witness keeps receiving after cleanup() detaches the harness's one.
const delivered: string[] = [];
const deliveryWitness = ptyProcess.onData((data) =>
delivered.push(String(data)),
);
try {
await expect
.poll(() => rig._interactiveOutput.includes(FORWARD_CANARY), {
message: 'the stand-in CLI never produced output',
timeout: 20_000,
})
.toBe(true);

const forwarded: string[] = [];
vi.spyOn(process.stdout, 'write').mockImplementation((chunk) => {
forwarded.push(String(chunk));
return true;
});

// Positive control, through the same KEEP_OUTPUT gate the harness uses:
// a forwarding path that never ran satisfies the empty assertion below
// exactly as well as one cleanup() has detached.
await expect
.poll(
() => forwarded.some((chunk) => chunk.includes(FORWARD_CANARY)),
{
message: 'stdout forwarding never went live before cleanup',
timeout: 20_000,
},
)
.toBe(true);

await rig.cleanup();
// Still alive: it swallowed SIGHUP. That is the window the real CLI's
// graceful shutdown opens between cleanup() and the child's own exit.
expect(isProcessAlive(ptyProcess.pid)).toBe(true);

forwarded.length = 0;
delivered.length = 0;
// Wait for a byte the surviving child provably delivered instead of
// sleeping a fixed window: a stalled worker runs the expired timer
// before the loop polls the pty fd, and an empty window then passes
// with the forwarding still attached.
await expect
.poll(
() => delivered.some((chunk) => chunk.includes(FORWARD_CANARY)),
{
message:
'the child that outlived cleanup() delivered no further bytes',
timeout: 10_000,
},
)
.toBe(true);

expect(
forwarded.filter((chunk) => chunk.includes(FORWARD_CANARY)),
'cleanup() left the session forwarding PTY bytes into stdout',
).toEqual([]);
} finally {
deliveryWitness.dispose();
vi.restoreAllMocks();
try {
process.kill(ptyProcess.pid, 'SIGKILL');
} catch {
// Already gone
}
}
},
);

it.each([
[
'telemetry events',
Expand Down
23 changes: 17 additions & 6 deletions integration-tests/test-helper.ts
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,12 @@ export class TestRig {
testName?: string;
_lastRunStdout?: string;
_interactiveOutput = '';
private readonly interactiveProcesses: pty.IPty[] = [];
// Each interactive PTY paired with the listener forwarding its bytes to
// this worker's stdout, so cleanup() can detach that forwarding.
private readonly interactiveSessions: Array<{
ptyProcess: pty.IPty;
forwardOutput: pty.IDisposable;
}> = [];

constructor() {
this.bundlePath = join(__dirname, '..', 'dist/cli.js');
Expand Down Expand Up @@ -497,9 +502,16 @@ export class TestRig {
// A session a test never closed keeps its CLI child forwarding PTY bytes
// into this worker's stdout; after vitest tears the worker down those
// writes EPIPE and fail an otherwise all-green run (#10969).
for (const ptyProcess of this.interactiveProcesses.splice(0)) {
//
// Killing is not enough: node-pty's signal-less kill() sends SIGHUP, which
// the CLI traps into an asynchronous graceful shutdown, so the child still
// renders after cleanup() returns and one more forwarded byte can reach a
// stdout pipe vitest has since destroyed. Detaching the listener closes
// that window however long the child takes to die (#11002).
for (const session of this.interactiveSessions.splice(0)) {
session.forwardOutput.dispose();
try {
ptyProcess.kill();
session.ptyProcess.kill();
} catch {
// Process may have already exited
}
Expand Down Expand Up @@ -944,14 +956,13 @@ export class TestRig {
...e2eRendererEnv(renderer),
} as { [key: string]: string },
});
this.interactiveProcesses.push(ptyProcess);

ptyProcess.onData((data) => {
const forwardOutput = ptyProcess.onData((data) => {
this._interactiveOutput += data;
if (env.KEEP_OUTPUT === 'true' || env.VERBOSE === 'true') {
process.stdout.write(data);
}
});
this.interactiveSessions.push({ ptyProcess, forwardOutput });

const promise = new Promise<{
exitCode: number;
Expand Down
Loading