Skip to content

test(replay): [cherry-pick 1.7.0] detect AISimulate under the aisim distribution name (#18286) - #18287

Open
BenHamm wants to merge 1 commit into
release/1.7.0from
bhamm/replay-tests-aisim-dist-1.7
Open

BenHamm wants to merge 1 commit into
release/1.7.0from
bhamm/replay-tests-aisim-dist-1.7

Conversation

@BenHamm

@BenHamm BenHamm commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Overview

Cherry-pick of #18286 to release/1.7.0. This branch pins aisim==0.14.0, so without this change the replay integration gate never finds the aisimulate distribution and silently skips test_replay_planner_scaling.py and test_replay_smoke.py::test_planner_replay_accepts_multi_shard_dynamo_trace. Unlike main, the coverage is lost on this branch today.

Key Changes

  • _require_aisimulate_distribution accepts either distribution name (aisim first, then aisimulate). Clean cherry-pick -x; the resulting file is identical to main.

Validation

  • Same checks as test(replay): detect AISimulate under the aisim distribution name #18286, run on this branch: gate unit check 8/8 pass (before: only-aisim cases skip); with aisim present the two files collect 18 tests (was 1, module skipped), and the smoke test gets past the gate instead of skipping.
  • pre-commit run --files lib/bindings/python/tests/replay/replay_utils.py: passed.

Issues and Ongoing Work

🤖 Generated with Claude Code

AISimulate 0.14.0 ships as the `aisim` distribution (the import name is
still `aisimulate`), and release/1.7.0 pins `aisim==0.14.0`. The replay
integration gate in tests/replay/replay_utils.py only looked up
`distribution("aisimulate")`, so with `aisim` installed it raised
PackageNotFoundError and skipped: test_replay_planner_scaling.py was
skipped at module level and
test_replay_smoke.py::test_planner_replay_accepts_multi_shard_dynamo_trace
skipped in its body, with no failure to flag the lost coverage.

Accept either distribution name, `aisim` first and then `aisimulate`, so
the gate works on main (still pinned to `aisimulate==0.13.0.dev...`) and
on branches that pin `aisim`. The tests still skip when neither is
installed, and the skip reason now names both distributions.

Internal tracking: Linear AISIM-2278.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ben Hamm <ben.hamm@gmail.com>
(cherry picked from commit 45de897)
@BenHamm
BenHamm requested a review from a team as a code owner November 6, 2026 02:58
@github-actions github-actions Bot added the test label Nov 6, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor

❌ Dynamo PR CI failed — run 42153664182 (attempt 1) on c9bf05799f

Gate checks: ❌ backend-status-check · ❌ deploy-status-check · ✅ dynamo-status-check

Framework Build
vLLM ❌ 1 ✅ 2
SGLang ❌ 1 ✅ 2
TRT-LLM ❌ 1 ✅ 2
Other Jobs
frontend ❌ 1 ✅ 2

Failure details

4 jobs failed, all in the Copy image to target registry step: every push of the built image to ACR is rejected with unknown: The operation is disallowed on this registry, repository or image on blob upload, across all retries. The image builds themselves passed.

❌ sglang-runtime / Copy to ACR cuda13.0: ACR blob upload disallowed

Job: sglang-runtime / Copy to ACR cuda13.0 · Failed step: Copy image to target registry · Logs: gh run view --job 127687105121 -R ai-dynamo/dynamo --log-failed

level=warning msg="Failed, retrying in 8s ... (4/4). Error: copying image 1/4 from manifest list: writing blob: initiating layer upload to /v2/ai-dynamo/dynamo/blobs/uploads/ in ***: unknown: The operation is disallowed on this registry, repository or image. View troubleshooting steps at https://aka.ms/acr/faq/#why-does-my-pull-or-push-request-fail-with-disallowed-operation-"
⚠️ Attempt 1 failed, retrying in 10s...
⚠️ Attempt 2 failed, retrying in 20s...
❌ All attempts failed

The copy of the built image to the Azure Container Registry is refused by the registry on layer upload (ACR "disallowed operation", typically a registry-side lock/write-disabled repository or permission policy), and all retries fail the same way. This is a registry/infrastructure-side rejection, not a build or test failure; once the registry accepts writes, a rerun of failed jobs is likely enough.

❌ vllm-runtime / Copy to ACR cuda13.0: ACR blob upload disallowed

Job: vllm-runtime / Copy to ACR cuda13.0 · Failed step: Copy image to target registry · Logs: gh run view --job 127688516110 -R ai-dynamo/dynamo --log-failed

level=warning msg="Failed, retrying in 8s ... (4/4). Error: copying image 1/4 from manifest list: writing blob: initiating layer upload to /v2/ai-dynamo/dynamo/blobs/uploads/ in ***: unknown: The operation is disallowed on this registry, repository or image. View troubleshooting steps at https://aka.ms/acr/faq/#why-does-my-pull-or-push-request-fail-with-disallowed-operation-"
⚠️ Attempt 1 failed, retrying in 10s...
⚠️ Attempt 2 failed, retrying in 20s...
❌ All attempts failed

The copy of the built image to the Azure Container Registry is refused by the registry on layer upload (ACR "disallowed operation", typically a registry-side lock/write-disabled repository or permission policy), and all retries fail the same way. This is a registry/infrastructure-side rejection, not a build or test failure; once the registry accepts writes, a rerun of failed jobs is likely enough.

❌ frontend / Copy to ACR cuda, amd64: ACR blob upload disallowed

Job: frontend / Copy to ACR cuda, amd64 · Failed step: Copy image to target registry · Logs: gh run view --job 127690636183 -R ai-dynamo/dynamo --log-failed

level=warning msg="Failed, retrying in 8s ... (4/4). Error: copying system image from manifest list: writing blob: initiating layer upload to /v2/ai-dynamo/dynamo/blobs/uploads/ in ***: unknown: The operation is disallowed on this registry, repository or image. View troubleshooting steps at https://aka.ms/acr/faq/#why-does-my-pull-or-push-request-fail-with-disallowed-operation-"
⚠️ Attempt 1 failed, retrying in 10s...
⚠️ Attempt 2 failed, retrying in 20s...
❌ All attempts failed

The copy of the built image to the Azure Container Registry is refused by the registry on layer upload (ACR "disallowed operation", typically a registry-side lock/write-disabled repository or permission policy), and all retries fail the same way. This is a registry/infrastructure-side rejection, not a build or test failure; once the registry accepts writes, a rerun of failed jobs is likely enough.

❌ trtllm-runtime / Copy to ACR cuda13.1: ACR blob upload disallowed

Job: trtllm-runtime / Copy to ACR cuda13.1 · Failed step: Copy image to target registry · Logs: gh run view --job 127691828684 -R ai-dynamo/dynamo --log-failed

level=warning msg="Failed, retrying in 8s ... (4/4). Error: copying image 1/4 from manifest list: writing blob: initiating layer upload to /v2/ai-dynamo/dynamo/blobs/uploads/ in ***: unknown: The operation is disallowed on this registry, repository or image. View troubleshooting steps at https://aka.ms/acr/faq/#why-does-my-pull-or-push-request-fail-with-disallowed-operation-"
⚠️ Attempt 1 failed, retrying in 10s...
⚠️ Attempt 2 failed, retrying in 20s...
❌ All attempts failed

The copy of the built image to the Azure Container Registry is refused by the registry on layer upload (ACR "disallowed operation", typically a registry-side lock/write-disabled repository or permission policy), and all retries fail the same way. This is a registry/infrastructure-side rejection, not a build or test failure; once the registry accepts writes, a rerun of failed jobs is likely enough.

For agents
{"pr": 18287, "run_id": 42153664182, "run_attempt": 1, "head_sha": "c9bf05799f69d96efd95ab44d82c3d0354b94e34", "failures": [{"job": "sglang-runtime / Copy to ACR cuda13.0", "job_id": 127687105121, "failed_step": "Copy image to target registry", "signature": "ACR push disallowed: \"The operation is disallowed on this registry, repository or image\"", "tests": [], "log_cmd": "gh run view --job 127687105121 -R ai-dynamo/dynamo --log-failed"}, {"job": "vllm-runtime / Copy to ACR cuda13.0", "job_id": 127688516110, "failed_step": "Copy image to target registry", "signature": "ACR push disallowed: \"The operation is disallowed on this registry, repository or image\"", "tests": [], "log_cmd": "gh run view --job 127688516110 -R ai-dynamo/dynamo --log-failed"}, {"job": "frontend / Copy to ACR cuda, amd64", "job_id": 127690636183, "failed_step": "Copy image to target registry", "signature": "ACR push disallowed: \"The operation is disallowed on this registry, repository or image\"", "tests": [], "log_cmd": "gh run view --job 127690636183 -R ai-dynamo/dynamo --log-failed"}, {"job": "trtllm-runtime / Copy to ACR cuda13.1", "job_id": 127691828684, "failed_step": "Copy image to target registry", "signature": "ACR push disallowed: \"The operation is disallowed on this registry, repository or image\"", "tests": [], "log_cmd": "gh run view --job 127691828684 -R ai-dynamo/dynamo --log-failed"}]}

Posted automatically by Devin for run 42153664182. Updated on every full-CI run of this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant