Skip to content
Closed
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,12 @@ jobs:
# Kill-switch: MAINTAINER_ECS_RUNNER_DISABLED.
runs-on: '${{ (github.repository == ''QwenLM/qwen-code'' && vars.MAINTAINER_ECS_RUNNER_DISABLED != ''true'') && fromJSON(''["self-hosted", "linux", "x64", "ecs-qwen"]'') || fromJSON(''["ubuntu-latest"]'') }}'
# A wedged shard must free the pool runner well before GitHub's
# 360-minute default; 60 matches ci.yml's pool jobs.
timeout-minutes: 60
# 360-minute default. The ceiling is extended on the ECS pool only,
# mirroring ci.yml's pool lanes: run 33345905817 (issue #10591) expired
# at the old flat 60 on a contended shared host before its log reported
# any test result. Hosted fallbacks keep the pre-contention ceiling so
# a genuine hang there does not burn the extra 30 minutes.
timeout-minutes: "${{ fromJSON((github.repository == 'QwenLM/qwen-code' && vars.MAINTAINER_ECS_RUNNER_DISABLED != 'true') && '90' || '60') }}"

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified real at the current head: node --test --test-concurrency=1 .github/scripts/ci-runner-routing.test.mjs fails at line 552 (strictEqual expected 60, actual is the new routing-scoped expression string) — 23 tests, 22 pass, 1 fail. The full HELPER_TESTS set confirms this is the sole cause of the red required Test (ubuntu-latest, Node 22.x) check: 500 tests, exactly this one failure.

This round nevertheless does NOT implement the pin update: .github/scripts/ is outside this PR's footprint (the PR touches only .github/workflows/e2e.yml and scripts/tests/e2e-workflow.test.js), and the round-3 attempt at exactly this pin update was rejected by the autofix gate and reverted for that reason. Repeating the rejected change would fail the gate again, and there is no in-footprint change that can turn the check green while the pin still reads 60. This needs a maintainer decision on where the pin update lands:

  1. Allow this PR to expand into .github/scripts/, so a next round updates the pin here (substitute-then-evaluate: 90 on the pool routing, 60 on both hosted fallbacks, mirroring the runs-on condition; plus the stale "Pin the ci.yml pool precedent" comment); or
  2. A maintainer lands the pin update separately (direct commit, or a small standalone PR merged first), and this PR rebases onto it; or
  3. Revert this PR's timeout change entirely (not recommended — it abandons the Main CI failed: E2E Tests on cd5d5af2fbd7 #10591 fix).

Leaving this thread open until a maintainer chooses.

中文说明

已在当前 head 上确认为真实问题:node --test --test-concurrency=1 .github/scripts/ci-runner-routing.test.mjs 在第 552 行失败(strictEqual 期望 60,实际为新的路由限定表达式字符串)——23 个测试,22 通过,1 失败。完整 HELPER_TESTS 集合确认这是必需的 Test (ubuntu-latest, Node 22.x) 检查变红的唯一原因:共 500 个测试,仅此一处失败。

但本轮不实施该断言更新:.github/scripts/ 在本 PR 的 footprint 之外(本 PR 只改动 .github/workflows/e2e.yml 和 scripts/tests/e2e-workflow.test.js),且第 3 轮对同一断言的更新尝试已被 autofix gate 以该理由拒绝并回滚。重复被拒绝的改动只会再次被 gate 拒绝,而在断言仍钉死 60 的情况下,footprint 内没有任何改动能让该检查变绿。需要维护者决定断言更新落在哪里:

  1. 允许本 PR 扩展到 .github/scripts/,下一轮即可在此更新断言(先替换再求值:资源池路由 90、两种托管回退均为 60,与 runs-on 条件保持一致;同时修正过时的 "Pin the ci.yml pool precedent" 注释);或
  2. 由维护者单独落地断言更新(直接提交,或先合并一个小的独立 PR),之后本 PR 重新变基;或
  3. 整体回退本 PR 的 timeout 改动(不建议——等于放弃 Main CI failed: E2E Tests on cd5d5af2fbd7 #10591 的修复)。

在维护者作出选择之前,本线程保持打开。

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified real at the current head: node --test --test-concurrency=1 .github/scripts/ci-runner-routing.test.mjs fails at line 552 (strictEqual expected 60, actual is the new routing-scoped expression string) — 23 tests, 22 pass, 1 fail. The full HELPER_TESTS set confirms this is the sole cause of the red required Test (ubuntu-latest, Node 22.x) check: 500 tests, exactly this one failure.

This round nevertheless does NOT implement the pin update: .github/scripts/ is outside this PR's footprint (the PR touches only .github/workflows/e2e.yml and scripts/tests/e2e-workflow.test.js), and the round-3 attempt at exactly this pin update was rejected by the autofix gate and reverted for that reason. Repeating the rejected change would fail the gate again, and there is no in-footprint change that can turn the check green while the pin still reads 60. This needs a maintainer decision on where the pin update lands:

  1. Allow this PR to expand into .github/scripts/, so a next round updates the pin here (substitute-then-evaluate: 90 on the pool routing, 60 on both hosted fallbacks, mirroring the runs-on condition; plus the stale "Pin the ci.yml pool precedent" comment); or
  2. A maintainer lands the pin update separately (direct commit, or a small standalone PR merged first), and this PR rebases onto it; or
  3. Revert this PR's timeout change entirely (not recommended — it abandons the Main CI failed: E2E Tests on cd5d5af2fbd7 #10591 fix).

Leaving this thread open until a maintainer chooses.

中文说明

已在当前 head 上确认为真实问题:node --test --test-concurrency=1 .github/scripts/ci-runner-routing.test.mjs 在第 552 行失败(strictEqual 期望 60,实际为新的路由限定表达式字符串)——23 个测试,22 通过,1 失败。完整 HELPER_TESTS 集合确认这是必需的 Test (ubuntu-latest, Node 22.x) 检查变红的唯一原因:共 500 个测试,仅此一处失败。

但本轮不实施该断言更新:.github/scripts/ 在本 PR 的 footprint 之外(本 PR 只改动 .github/workflows/e2e.yml 和 scripts/tests/e2e-workflow.test.js),且第 3 轮对同一断言的更新尝试已被 autofix gate 以该理由拒绝并回滚。重复被拒绝的改动只会再次被 gate 拒绝,而在断言仍钉死 60 的情况下,footprint 内没有任何改动能让该检查变绿。需要维护者决定断言更新落在哪里:

  1. 允许本 PR 扩展到 .github/scripts/,下一轮即可在此更新断言(先替换再求值:资源池路由 90、两种托管回退均为 60,与 runs-on 条件保持一致;同时修正过时的 "Pin the ci.yml pool precedent" 注释);或
  2. 由维护者单独落地断言更新(直接提交,或先合并一个小的独立 PR),之后本 PR 重新变基;或
  3. 整体回退本 PR 的 timeout 改动(不建议——等于放弃 Main CI failed: E2E Tests on cd5d5af2fbd7 #10591 的修复)。

在维护者作出选择之前,本线程保持打开。

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified real at the current head: node --test --test-concurrency=1 .github/scripts/ci-runner-routing.test.mjs fails at line 552 (strictEqual expected 60, actual is the new routing-scoped expression string) — 23 tests, 22 pass, 1 fail. The full HELPER_TESTS set confirms this is the sole cause of the red required Test (ubuntu-latest, Node 22.x) check: 500 tests, exactly this one failure.

This round nevertheless does NOT implement the pin update: .github/scripts/ is outside this PR's footprint (the PR touches only .github/workflows/e2e.yml and scripts/tests/e2e-workflow.test.js), and the round-3 attempt at exactly this pin update was rejected by the autofix gate and reverted for that reason, as this comment itself notes. Repeating the rejected change would fail the gate again, and there is no in-footprint change that can turn the check green while the pin still reads 60. This needs a maintainer decision on where the pin update lands:

  1. Allow this PR to expand into .github/scripts/, so a next round updates the pin here (substitute-then-evaluate: 90 on the pool routing, 60 on both hosted fallbacks, mirroring the runs-on condition; plus the stale "Pin the ci.yml pool precedent" comment); or
  2. A maintainer lands the pin update separately (direct commit, or a small standalone PR merged first), and this PR rebases onto it; or
  3. Revert this PR's timeout change entirely (not recommended — it abandons the Main CI failed: E2E Tests on cd5d5af2fbd7 #10591 fix).

Leaving this thread open until a maintainer chooses.

中文说明

已在当前 head 上确认为真实问题:node --test --test-concurrency=1 .github/scripts/ci-runner-routing.test.mjs 在第 552 行失败(strictEqual 期望 60,实际为新的路由限定表达式字符串)——23 个测试,22 通过,1 失败。完整 HELPER_TESTS 集合确认这是必需的 Test (ubuntu-latest, Node 22.x) 检查变红的唯一原因:共 500 个测试,仅此一处失败。

但本轮不实施该断言更新:.github/scripts/ 在本 PR 的 footprint 之外(本 PR 只改动 .github/workflows/e2e.yml 和 scripts/tests/e2e-workflow.test.js),且正如本评论所述,第 3 轮对同一断言的更新尝试已被 autofix gate 以该理由拒绝并回滚。重复被拒绝的改动只会再次被 gate 拒绝,而在断言仍钉死 60 的情况下,footprint 内没有任何改动能让该检查变绿。需要维护者决定断言更新落在哪里:

  1. 允许本 PR 扩展到 .github/scripts/,下一轮即可在此更新断言(先替换再求值:资源池路由 90、两种托管回退均为 60,与 runs-on 条件保持一致;同时修正过时的 "Pin the ci.yml pool precedent" 注释);或
  2. 由维护者单独落地断言更新(直接提交,或先合并一个小的独立 PR),之后本 PR 重新变基;或
  3. 整体回退本 PR 的 timeout 改动(不建议——等于放弃 Main CI failed: E2E Tests on cd5d5af2fbd7 #10591 的修复)。

在维护者作出选择之前,本线程保持打开。

# Skip on fork PRs: forks have no access to repository secrets
# (OPENAI_*, DOCKERHUB_*), so the matrix would fail unconditionally
# and show misleading red status. Same-repo PRs run normally.
Expand Down
52 changes: 52 additions & 0 deletions scripts/tests/e2e-workflow.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -94,4 +94,56 @@ describe('e2e workflow', () => {
expect(runStep.run).toContain('mktemp -d /var/tmp/qwen-ci-XXXXXX');
expect(runStep.run).toContain('trap \'rm -rf "$TMPDIR"');
});

describe('job timeout lane contract', () => {
// Run 33345905817 (issue #10591) expired at the old flat 60-minute
// ceiling on a contended shared ECS host before its log reported any
// test result. The extended ceiling is
// scoped to the pool routing — the same condition `runs-on` uses:
// hosted fallbacks (forks and MAINTAINER_ECS_RUNNER_DISABLED=true)
// keep the pre-contention bound, so a genuine hang there does not
// burn the extra 30 minutes. Evaluate the real timeout expression for
// both routings instead of pinning a bare number — the same
// substitute-then-evaluate technique ci-platform-lanes.test.js uses —
// so a regression to an unconditional ceiling fails here instead of
// reading as a passing constant.
const timeoutMinutesOn = ({ repository, ecsDisabled }) => {
const expr = String(yml.jobs['e2e-test-linux']['timeout-minutes'])
.replace(/^\$\{\{\s*/, '')
.replace(/\s*\}\}$/, '')
.replace(/github\.repository/g, JSON.stringify(repository))
.replace(
/vars\.MAINTAINER_ECS_RUNNER_DISABLED/g,
JSON.stringify(ecsDisabled),
)
.replace(/fromJSON\(/g, '(');
if (/github\.|vars\.|needs\.|steps\.|fromJSON\(/.test(expr)) {
throw new Error(
`e2e-test-linux timeout carries a term this guard does not model: ${expr}`,
);
}
return Number(new Function(`return (${expr});`)());
};

it('keeps a contended ECS shard above the install/build/test budget', () => {
expect(
timeoutMinutesOn({ repository: 'QwenLM/qwen-code', ecsDisabled: '' }),
).toBe(90);
});

it('keeps the hosted fallback on the pre-contention ceiling', () => {
expect(
timeoutMinutesOn({
repository: 'some-fork/qwen-code',
ecsDisabled: '',
}),
).toBe(60);
expect(
timeoutMinutesOn({
repository: 'QwenLM/qwen-code',
ecsDisabled: 'true',
}),
).toBe(60);
});
});
});
Loading