test: stabilize nightly — skip engine-init failures, convert xfails to skips, fix http URL validation regression by dmitry-tokarev-nv · Pull Request #8443 · ai-dynamo/dynamo

dmitry-tokarev-nv · 2026-04-21T17:19:54Z

Summary

A bundle of nightly-stabilisation changes for the vLLM and SGLang pipelines. Five distinct concerns, each commit-scoped; none overlap.

1. Skip flaky/broken engine-init and cancellation tests

test_request_cancellation_vllm_prefill_cancel in tests/fault_tolerance/cancellation/test_vllm.py — commit 712761e1f4

Both [nats] and [tcp] parametrizations have failed in every nightly vllm-pipeline / Multi-gpu test cuda12.9 run on main for the last 8 nights (16/16 failures). The vLLM engine dies with EngineDeadError when cancelling mid-prefill, so the expected Aborted Prefill Request ID: log line is never emitted and the 500 ms poll budget expires. Same class of bug as the twin test_request_cancellation_vllm_decode_cancel already skipped under DYN-2606 in #8006. Tracked under OPS-4448 (@kthui).

Example failing nightlies: 24713014658, 24657547996, 24624817902, 24600680008, 24587232619, 24556217800, 24500822470, 24445034832.

test_serve_deployment[mm_{e_pd,epd}_qwen3-vl-2b] + [mm_epd_qwen3-vl-2b-video] in tests/serve/multimodal_profiles/vllm.py — commits 32ce9e998e + e4e89963ef

All three have failed in every nightly vllm-pipeline / Test cuda12.9 (amd64) run for the last 7+ nights. Error signature: RuntimeError: Engine core initialization failed (vllm/v1/engine/utils.py:1057 wait_for_engine_startup) — same family as the Qwen2-Audio case skipped in #8411. Only the e_pd/epd disagg topologies fail; the sibling agg and p_d topologies pass, so this is specific to disagg_multimodal_e_pd.sh / disagg_multimodal_epd.sh. Tracked under OPS-4445.

Example failing nightlies: 24713014658, 24657547996, 24624817902, 24600680008, 24587232619, 24556217800, 24500822470, 24445034832.

2. Convert vllm migration xfails → skips

In tests/fault_tolerance/migration/test_vllm.py — commit c1cc2fa45d

Three tests have function-level @pytest.mark.xfail(strict=False, ...) for known architectural limitations:

test_request_migration_vllm_prefill — prefill migration unsupported
test_request_migration_vllm_kv_transfer — request_id reuse + delay_free_blocks KV cache issue
test_request_migration_vllm_decode — same request_id / KV cache issue

Last 7 nightly vllm multi-gpu runs: 144 XFAIL, 0 XPASS, 216 SKIPPED. Every non-skipped parameterisation fails exactly as the xfail reasons predict. Running them anyway still incurs per-param GPU setup cost. Converting to @pytest.mark.skip with the same reason strings — no functional change (no XPASS means nothing silently-passing is being masked), just stops wasting GPU time and cleans up the report.

3. Fix http:// URL validation regression (NOT a skip — real fix)

In tests/serve/test_vllm.py — commit d634588d6d

Today's nightly flagged two new failures:

test_serve_deployment[multimodal_agg_llava]
test_serve_deployment[aggregated_toolcalling]

Both error with:

Exception: Failed to load image from http://...:
  http:// URLs are not allowed; set DYN_MM_ALLOW_INTERNAL=1 to enable

The strict URL validator landed in #8282 (2026-04-20) correctly plumbed DYN_MM_ALLOW_INTERNAL=1 through the factory for VLLM_MULTIMODAL_PROFILES-generated configs (tests/utils/multimodal.py::make_multimodal_configs), but missed these two hand-authored VLLMConfig entries that still use http:// URLs (cocodataset HTTP image for llava, localhost:8806 fixture server for toolcalling). Adds the same env var on both configs.

4. Extend sglang migration skip to TCP transport (OPS-4446)

In tests/fault_tolerance/migration/test_sglang.py — commit b3ac5ce25e

#8404 skipped only the [nats] variant of
test_request_migration_sglang_aggregated[migration_enabled-max_seq_len_disabled-worker_failure-chat-stream-*]
because the TCP sibling was stable at the time. From the very first post-merge run after #8404 landed, the [tcp] variant began failing with the identical symptom (AssertionError: Delay before response > 6 secs, got ~13 secs at utils.py:417) in every run: 24733651809, 24733908729, 24734216001, 24735128282, and onwards.

Cause is test-ordering: when NATS ran first it warmed the path; with NATS skipped, TCP now pays the cold-start cost itself. Dropping the request_plane == "nats" clause so both transports skip under the same OPS-4446 ticket. Also replaces the stale TODO(<LINEAR-ID>) comment left over from #8404 with the OPS-4446 reference that the skip reason string already uses.

5. Skip persistent TRTLLM nightly failures + convert cancellation xfails to skips

In tests/fault_tolerance/cancellation/test_trtllm.py, tests/gpu_memory_service/test_quiesce_resume.py, tests/gpu_memory_service/test_shadow_failover.py, and tests/serve/test_trtllm.py — commits 2dabd67f33 + 8084d1bf70

Convert xfails → skips (same pattern as section 2). Module-level @pytest.mark.xfail(strict=True, "Cancellation is temporarily disabled") and function-level @pytest.mark.xfail(strict=False, "Test fails only on CI") on test_request_cancellation_trtllm_kv_transfer_cancel produced 7/7 XFAIL and 0 XPASS across recent nightlies. Switched to @pytest.mark.skip with the same reason text.

Skip 4 persistent failures. All four have failed in the last 6+/7 nightly amd64 runs:

test_gms_basic_quiesce_resume_trtllm
test_gms_read_only_import_trtllm
test_gms_shadow_engine_failover_trtllm
test_deployment[disaggregated_same_gpu-2]

GMS tests fail with AssertionError: assert 'error' == 'ok' (quiesce/resume/shadow ops returning 'error'); the disagg_same_gpu serve test fails with RuntimeError: Main server process exited with code 1 while waiting for health check and RuntimeError: Executor worker returned error. Likely share a common trtllm-executor root cause. Tracked under OPS-4450.

Linear tickets

Skip	Ticket	Owner
vllm prefill_cancel	OPS-4448	@kthui
qwen3-vl-2b disagg (e_pd, epd image, epd video)	OPS-4445	@furionw
sglang migration (nats + tcp)	OPS-4446	—
vllm migration xfail→skip	n/a — skips preserve existing architectural-limitation reasons	—
TRTLLM nightly (GMS + disagg_same_gpu)	OPS-4450	—
http URL regression	n/a — real fix, not a skip	FYI @zhongdaor-nv (author of #8282)

Files

tests/fault_tolerance/cancellation/test_trtllm.py |  4 ++--
tests/fault_tolerance/cancellation/test_vllm.py   |  1 +
tests/fault_tolerance/migration/test_sglang.py    | 12 +++++-------
tests/fault_tolerance/migration/test_vllm.py      |  8 +++-----
tests/gpu_memory_service/test_quiesce_resume.py   |  2 ++
tests/gpu_memory_service/test_shadow_failover.py  |  1 +
tests/serve/multimodal_profiles/vllm.py           | 24 +++++++++++++++++++++---
tests/serve/test_trtllm.py                        |  1 +
tests/serve/test_vllm.py                          |  2 ++

Test plan

All <LINEAR-ID> placeholders replaced
Next nightly vllm-pipeline jobs (Multi-gpu + amd64) pass with the skips and env fixes applied
Next post-merge sglang-amd64 job no longer fails on the tcp variant of the migration test

🤖 Generated with Claude Code

test_request_cancellation_vllm_prefill_cancel[nats] and [tcp] have both failed in every one of the last 8 nightly vllm multi-gpu runs on main (16/16 consecutive failures). Failure signature is EngineDeadError: the vLLM engine crashes when the request is cancelled mid-prefill, so the expected "Aborted Prefill Request ID:" log line never appears and the 500ms poll times out. This is the same class of engine-level cancellation bug as the decode twin (test_request_cancellation_vllm_decode_cancel), which is already skipped under DYN-2606 (#8006). Mirror that skip here with a TODO placeholder until the Linear ticket ID is filled in. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

coderabbitai · 2026-04-21T17:22:50Z

Walkthrough

A @pytest.mark.skip decorator with a TODO reference was added to the test_request_cancellation_vllm_prefill_cancel test in the fault tolerance suite to skip execution due to a reported nightly CI failure.

Changes

Cohort / File(s)	Summary
Test Skip Decorator `tests/fault_tolerance/cancellation/test_vllm.py`	Added `@pytest.mark.skip` decorator to `test_request_cancellation_vllm_prefill_cancel` test to skip execution in marked runs; existing timeout, nightly, and GPU allocation markers remain unchanged.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	The PR title mentions 'stabilize nightly' and multiple changes (skip engine-init failures, convert xfails to skips, fix http URL validation regression), but the actual change only skips one vLLM prefill cancellation test. The title is overly broad and doesn't match the actual scope of changes in this PR.	Refocus the title to reflect the actual change: 'test: skip vllm prefill_cancel test pending engine fix' or similar, removing references to engine-init failures and HTTP URL validation regression that are not present in this changeset.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	PR description is comprehensive and follows template structure with Overview, Details, and Related Issues sections clearly documented.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/fault_tolerance/cancellation/test_vllm.py`:
- Line 399: The skip decorator on the test (pytest.mark.skip) currently uses the
placeholder "TODO(<LINEAR-ID>)"; replace that placeholder with the actual Linear
ticket ID referenced in the PR (or link to DYN-2606 if that is the correct
tracker) so the skip reason is actionable — update the skip reason string in
tests/fault_tolerance/cancellation/test_vllm.py for the relevant skipped test
decorator and ensure consistency with the skip message used for
test_request_cancellation_vllm_decode_cancel to mirror its format.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 4aee0eeb-f557-4c22-b692-fd80ff34b6ec

📥 Commits

Reviewing files that changed from the base of the PR and between e8ecf6f and 712761e.

📒 Files selected for processing (1)

tests/fault_tolerance/cancellation/test_vllm.py

Three multimodal serve deployment tests have failed in every one of the last 7 nightly vllm amd64 runs on main (7/7 consistent): - test_serve_deployment[mm_e_pd_qwen3-vl-2b] (e_pd topology) - test_serve_deployment[mm_epd_qwen3-vl-2b] (epd topology) - test_serve_deployment[mm_epd_qwen3-vl-2b-video] (epd topology) Failure is identical in all three: vLLM v1 engine core initialization crashes during startup of Qwen/Qwen3-VL-2B-Instruct under the disagg launch scripts (disagg_multimodal_e_pd.sh, disagg_multimodal_epd.sh), so the health-check URL never comes up and the test times out. RuntimeError: Engine core initialization failed. See root cause above. at vllm/v1/engine/utils.py:1057 wait_for_engine_startup The matching agg and p_d topology tests pass, so this is specific to the e_pd/epd scripts. Same class of vLLM engine-init crash as #8411 for mm_agg_qwen2-audio-7b — not flaky, needs real investigation. Ticket placeholder left on each skipped TopologyConfig. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

@xfail

Three vllm migration tests are marked @xfail(strict=False) for known architectural limitations: - test_request_migration_vllm_prefill (prefill migration unsupported) - test_request_migration_vllm_kv_transfer (request_id/KV cache reuse) - test_request_migration_vllm_decode (same request_id/KV cache reuse) The last seven nightly vllm multi-gpu runs logged 144 XFAIL results across these three tests with 0 XPASS — every non-skipped parameter combination fails exactly as predicted by the xfail reason. Running them anyway still incurs the per-parameterisation GPU setup cost and generates noise in the nightly report. Convert to @Skip with the same reason strings. No functional change expected: if any of these start passing, the skip reason is the signal to re-enable, same as xfail(strict=False) would have been. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

…lcalling Two legacy VLLMConfig entries in tests/serve/test_vllm.py started failing in today's nightly vllm amd64 job with: Exception: Failed to load image from http://...: http:// URLs are not allowed; set DYN_MM_ALLOW_INTERNAL=1 to enable The strict URL validator landed in #8282 and correctly plumbed the env var through the factory for VLLM_MULTIMODAL_PROFILES-generated configs in tests/utils/multimodal.py, but missed these two hand- authored configs that still use http:// URLs (cocodataset HTTP image and the local test fixture server at localhost:8806). Set DYN_MM_ALLOW_INTERNAL=1 on both configs so they can load http:// payloads like before, matching what the factory already does for the mm_* generated configs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

…(OPS-4446) PR #8404 skipped only the NATS variant of test_request_migration_sglang_aggregated[migration_enabled-max_seq_len_disabled-worker_failure-chat-stream-*] because the TCP variant was stable at the time. Starting from the first post-merge run after that skip landed, the TCP variant began failing with the same symptom (AssertionError: Delay before response > 6 secs, got ~13 secs at utils.py:417) in every run: 24733651809, 24733908729, 24734216001, 24735128282, and onwards. The cause is ordering: when NATS ran first, it warmed the path; with NATS skipped, TCP now pays the cold-start cost itself. Drop the request_plane == "nats" clause so both transports skip, and also replace the stale TODO(<LINEAR-ID>) comment with the OPS-4446 reference that the skip reason string already uses. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

…kips Two concerns in the trtllm nightly `Test cuda13.1 (amd64)` job: 1) Convert xfails → skips in tests/fault_tolerance/cancellation/test_trtllm.py The module-level xfail (pytestmark, strict=True, "Cancellation is temporarily disabled") and the function-level xfail on test_request_cancellation_trtllm_kv_transfer_cancel (strict=False, "Test fails only on CI") have both produced only XFAIL (0 XPASS) across the last 7 nightlies, matching the vllm migration pattern in commit c1cc2fa. Switch to @pytest.mark.skip with the same reason text to stop burning GPU on tests that are definitionally expected to fail. 2) Skip 4 persistent trtllm failures All four have failed in the last 6+ nightlies with the same family of trtllm engine-crash errors (`AssertionError: assert 'error' == 'ok'`, `RuntimeError: Executor worker returned error`, main server dying with exit 1 during health-check): - test_gms_basic_quiesce_resume_trtllm - test_gms_read_only_import_trtllm - test_gms_shadow_engine_failover_trtllm - test_deployment[disaggregated_same_gpu-2] Marked with TODO(<LINEAR-ID>) placeholder — to be swapped for the real Linear ticket(s) before merge. Likely share a single ticket given the identical engine-crash family. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

…gg tests The three tests were skipped in #8443 while OPS-4445 was investigated. The preceding commit fixes the encode worker's nested vllm.LLM() memory precheck; re-enable the tests under pre_merge so they gate regressions. Leaves the qwen2-audio-7b post_merge skip in place — that is a distinct failure on the agg topology (no encode worker, not addressed here). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

dmitry-tokarev-nv requested review from a team as code owners April 21, 2026 17:19

pull-request-size Bot added the size/XS label Apr 21, 2026

github-actions Bot added the test label Apr 21, 2026

coderabbitai Bot reviewed Apr 21, 2026

View reviewed changes

Comment thread tests/fault_tolerance/cancellation/test_vllm.py Outdated

pull-request-size Bot added size/S and removed size/XS labels Apr 21, 2026

copy-pr-bot Bot temporarily deployed to GITLAB April 21, 2026 17:28 Inactive

dmitry-tokarev-nv changed the title ~~test(fault_tolerance): skip vllm prefill_cancel test pending engine fix~~ test: skip persistent nightly vLLM failures (prefill_cancel + qwen3-vl-2b disagg) Apr 21, 2026

test(serve): link qwen3-vl-2b disagg skips to OPS-4445

e4e8996

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

copy-pr-bot Bot temporarily deployed to GITLAB April 21, 2026 17:32 Inactive

pull-request-size Bot added size/M and removed size/S labels Apr 21, 2026

copy-pr-bot Bot temporarily deployed to GITLAB April 21, 2026 17:41 Inactive

copy-pr-bot Bot temporarily deployed to GITLAB April 21, 2026 18:13 Inactive

dmitry-tokarev-nv changed the title ~~test: skip persistent nightly vLLM failures (prefill_cancel + qwen3-vl-2b disagg)~~ test(vllm): stabilize nightly — skip engine-init failures, convert xfails to skips, fix http URL validation regression Apr 21, 2026

dmitry-tokarev-nv commented Apr 21, 2026

View reviewed changes

Comment thread tests/fault_tolerance/cancellation/test_vllm.py Outdated

Apply suggestion from @dmitry-tokarev-nv

c145464

Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

copy-pr-bot Bot temporarily deployed to GITLAB April 21, 2026 18:25 Inactive

dmitry-tokarev-nv changed the title ~~test(vllm): stabilize nightly — skip engine-init failures, convert xfails to skips, fix http URL validation regression~~ test: stabilize nightly — skip engine-init failures, convert xfails to skips, fix http URL validation regression Apr 21, 2026

copy-pr-bot Bot temporarily deployed to GITLAB April 21, 2026 18:52 Inactive

copy-pr-bot Bot temporarily deployed to GITLAB April 21, 2026 19:20 Inactive

sara4dev reviewed Apr 21, 2026

View reviewed changes

Comment thread tests/fault_tolerance/cancellation/test_trtllm.py

sara4dev requested changes Apr 21, 2026

View reviewed changes

test(trtllm): link skipped nightly failures to OPS-4450

8084d1b

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

copy-pr-bot Bot temporarily deployed to GITLAB April 21, 2026 19:33 Inactive

formatting

02d348a

Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

copy-pr-bot Bot temporarily deployed to GITLAB April 21, 2026 20:07 Inactive

sara4dev approved these changes Apr 21, 2026

View reviewed changes

dmitry-tokarev-nv merged commit cc583b2 into main Apr 21, 2026
65 of 66 checks passed

dmitry-tokarev-nv deleted the dtokarev/skip-vllm-prefill-cancel-nightly branch April 21, 2026 20:10

furionw mentioned this pull request Apr 21, 2026

fix(vllm): cap gpu_memory_utilization for encoder-only vision model load #8466

Merged

copy-pr-bot Bot had a problem deploying to GITLAB April 21, 2026 21:11 Failure

Navigation Menu

Repository navigation

test: stabilize nightly — skip engine-init failures, convert xfails to skips, fix http URL validation regression - #8443

test: stabilize nightly — skip engine-init failures, convert xfails to skips, fix http URL validation regression#8443
dmitry-tokarev-nv merged 10 commits into
mainfrom
dtokarev/skip-vllm-prefill-cancel-nightly

dmitry-tokarev-nv commented Apr 21, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Apr 21, 2026 •

edited

Loading

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This branch had an error being deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

dmitry-tokarev-nv commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

1. Skip flaky/broken engine-init and cancellation tests

2. Convert vllm migration xfails → skips

3. Fix http:// URL validation regression (NOT a skip — real fix)

4. Extend sglang migration skip to TCP transport (OPS-4446)

5. Skip persistent TRTLLM nightly failures + convert cancellation xfails to skips

Linear tickets

Files

Test plan

Uh oh!

coderabbitai Bot commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This branch had an error being deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dmitry-tokarev-nv commented Apr 21, 2026 •

edited

Loading

coderabbitai Bot commented Apr 21, 2026 •

edited

Loading