Repository navigation
[CI][ROCm] Export kernel source maps for CI selection - #59100
Merged
Merged
Conversation
Preserve HIPify and header provenance in native build maps, export the matching ROCm artifact, and forward single-node AMD recorder output. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
Export the ROCm kernel map separately after required build outputs succeed, and attempt a map-only export when the image already exists. Preserve strict image, wheel, smoke, and explicit map failures while treating optional map preparation, export, and upload failures as unavailable evidence. Reuse native content caches and add process regressions for fresh and cached builds, stale output, failure propagation, and disabled/print-only recording. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
Collect dependencies from each compiled GPU target and union them with host dependencies before provenance and freshness checks. Validate object targets, expand compiler response files, and preserve build outputs during dependency extraction. Forward vLLM worker method and shutdown timeout through the AMD runner. Add real HIP build regressions for device-only headers and compiler argument handling. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
Member
Author
|
Linked runs with the two PRs (this + vllm-project/ci-infra#685): |
AndreasKaratzas
marked this pull request as ready for review
October 1, 2026 00:38
AndreasKaratzas
requested review from
Harry-Chen,
LucasWilkinson,
dllehr-amd,
khluu,
tjtanaa and
tlrmchlsmth
as code owners
October 1, 2026 00:38
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
Member
Author
|
/ci rerun |
Collaborator
CI selector (shadow): no narrower answer, every step runs: 193 test steps (269 jobs) instead of 193 (269 jobs)Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works. Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.
Why: fail-open: CMakeLists.txt is a docker-image build input (docker/Dockerfile COPY); the CI image is rebuilt from it; running everything Today's rules run everything for this diff. Selector would run (193)
Would skip (today's rules run them) (0)none Would add (today's rules do not run them) (0)none AMD mirrors: would skip (0)none AMD mirrors: would add (0)none 14 changed files · base |
Member
Author
|
Another run complete: https://buildkite.com/vllm/amd-ci/builds/14676/list |
Member
Author
|
Ful CI run: |
When run-amd-test.sh finds no checkout to mount, the container still sourced ci_setup.sh and recorded into a directory that --rm discards. Pass KERNREC_DISABLE=1 so the job runs unrecorded and says why (ci-infra#685 honors it). Also trigger the hipify provenance test on changes to cmake/hipify.py. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Member
Author
|
Status of the AMD failures in https://buildkite.com/vllm/ci/builds/95743 (ROCm 7.2 image). The branch is based on
Recorder fixes from this investigation, pushed to ci-infra#685 (
|
main requires Python 3.11, so ruff (UP017) rewrites datetime.timezone.utc on the PR's merge ref. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com>
tahsintunan
approved these changes
Oct 21, 2026
Comment on lines
+131
to
+153
| configure_kernrec_docker_args() { | ||
| kernrec_docker_args=() | ||
| [[ "$1" == *recorders/kernrec/ci_setup.sh* ]] || return 0 | ||
| local checkout="${BUILDKITE_BUILD_CHECKOUT_PATH:-}" | ||
| if [[ -z "$checkout" || ! -d "$checkout" ]] || | ||
| ! mkdir -p "$checkout/.kernrec" || ! chmod 0777 "$checkout/.kernrec"; then | ||
| echo "kernrec: checkout unavailable; container recordings cannot be uploaded" >&2 | ||
| kernrec_docker_args=(-e KERNREC_DISABLE=1) | ||
| return 0 | ||
| fi | ||
| kernrec_docker_args=( | ||
| -v "$checkout/.kernrec:/tmp/kernrec-checkout/.kernrec" | ||
| -e "KERNREC_CHECKOUT_PATH=/tmp/kernrec-checkout" | ||
| -e BUILDKITE_JOB_ID | ||
| -e BUILDKITE_STEP_KEY | ||
| -e BUILDKITE_LABEL | ||
| -e BUILDKITE_BUILD_NUMBER | ||
| -e BUILDKITE_COMMIT | ||
| -e KERNREC_PYTHON | ||
| -e VLLM_WORKER_MULTIPROC_METHOD | ||
| -e VLLM_WORKER_SHUTDOWN_TIMEOUT_SECONDS | ||
| ) | ||
| } |
Contributor
There was a problem hiding this comment.
Question: This only runs on the dind path, and every AMD step is dind: false today. Is it still needed?
Member
Author
There was a problem hiding this comment.
You're right, it isn't needed. Every AMD step is dind: false, so the runner exits in the native branch before it reaches docker run. The only jobs that can still be dind (no_plugin and multi-node) are already excluded from kernrec in ci-infra's _kernrec_applies. I removed the helper, the sourcing guard and tests/tools/test_amd_runner.py, so run-amd-test.sh now matches main.
| else ([], []) | ||
| ), | ||
| ) | ||
| monkeypatch.setattr(producer, "device_dependencies", lambda *args: set()) |
Contributor
There was a problem hiding this comment.
nit: with this stubbed, device_dependencies never runs end to end in CI (the real-HIP test skips here). Could we mock run instead?
Member
Author
There was a problem hiding this comment.
It does run end to end in CI, just not on the CPU step. The :amd: (MI250) Docker Build Metadata mirror runs the same file on the ROCm image, and on this PR (vllm/ci 92017 and 95743) both real-HIP tests passed there, with build_map going through device_dependencies against real clang. Dropping the stub and relying on the run mock would mean faking clang's -### output and depfiles, which tests our imitation rather than clang. I'd rather keep this fixture focused on host and HIPify provenance.
Contributor
There was a problem hiding this comment.
Ah, missed that the MI250 mirror runs it. Makes sense, thanks.
Every AMD step runs natively (dind: false), so run-amd-test.sh exits before the docker run branch, and the jobs that can still be dind are not armed for kernrec. Remove the helper and its test; run-amd-test.sh now matches main. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com>
vllm/ci #96785, the AMD kernel-recording run of this branch: - AMD Docker Build Metadata failed collecting tests/tools/test_kernel_symbol_map.py: it loads /vllm-workspace/tools/ci/rocm_kernel_symbol_map.py, and native AMD pods unpack /vllm-workspace from export_vllm, which shipped only two files of tools/. Export tools/ci with the rest of the test workspace. - MiniMax M3 MonoKernel runs 19.5-20 min on main against a 20 min limit, so any slowdown times it out (it passed every test there and was cut at 20.5 min). Give it 40. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
Conflict: .buildkite/test_areas/models_basic.yaml. Main now gives the MiniMax M3 MonoKernel step 45 minutes; this branch had raised it from 20 to 40. Take main's 45, which reverts this branch's change to the file. docker/Dockerfile.rocm merged cleanly: main's UCX v1.23.0-rc1 bump (#62827) plus this branch's symbol-map stages and tools/ci export. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
AndreasKaratzas
added a commit
to AndreasKaratzas/vllm
that referenced
this pull request
Oct 24, 2026
…deployment vllm-project#59100 exports a ROCm kernel symbol map from the old build (ci-bake-rocm.sh, ci-rocm.hcl, kernel-symbol-map-rocm), which this branch replaces. Ported to the new build: - Dockerfile.rocm: CI csrc builds (ARCH set; release builds skip it) write the map from the build tree before removing it, with tools/ci/rocm_kernel_symbol_map.py, bounded and never failing the build. kernel-symbol-map stamps it with this build's commit, so a csrc layer reused from another commit still yields a map the collector accepts. The dev stage installs regex, which the map tool imports. - docker-bake-rocm.hcl: a kernel-symbol-map target (local output). - build.sh ci: with VLLM_KERNEL_SYMBOL_MAP=1 (set by ci-infra on image-build-amd), export and upload kernel_symbol_map.rocm.json.gz; a failure only warns, and a map from an earlier build is never uploaded. - Tests: the old orchestration tests are replaced by build.sh tests in test_rocm_build.py; test_kernel_symbol_map.py runs in the CPU step, its AMD mirror and the legacy AMD step. The stale build-config.sh dependency is dropped and the CPU step now depends on the ROCm build files. Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>
1 of 3 tasks
bantianyue
pushed a commit
to bantianyue/vllm
that referenced
this pull request
Oct 28, 2026
…9100) Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com> Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NVIDIA image builds already export kernel source maps for the CI selector. ROCm needs the same ownership information, but its native build transforms CUDA sources and headers into HIP inputs. The map uses the retained compilation and dependency records to recover those original paths and refuses incomplete or unsupported builds. This supplies the source half of the companion native SDK recording path, allowing CI to join observed kernel names to repository changes without guessing ownership.
Companion: ci-infra native recording and selection for both backends. Its consumers should be deployed before enabling the new map and recorder producers.
AI assistance was used to implement these changes and run the validation.