[AndreasKaratzas]

• Add a ROCProfiler SDK 1.3.2 recorder for dispatched AMD kernels, including graph replay.
• Require clean native traces with no drops, recorder errors, or unresolved names before a row can authorize skipping a test.
• Add matching error accounting to the CUPTI recorder and rebuild its shipped CUDA library.
• Set up recording for eligible AMD jobs with SDK build, version, and loader checks that preserve workload execution when collection is unavailable.
• Publish and fetch CUDA and ROCm tables and source maps independently, with separate latest pointers.
• Read both backends in selection, PR reporting, and validation, preserving observations and unusable matching rows when another pair suggests dropping a test.
• Arm AMD jobs through the pipeline generator, collect their artifacts, and request matching source maps from the image builds.
• Record AITER-only imports and editable AITER packages in the Python recorder, together with relevant ROCm execution settings.

The selector already uses CUPTI launch records and image-built source maps to determine which tests exercise a CUDA kernel change. AMD CI needs equivalent coverage for HIP, AITER, Triton, RCCL, and graph workloads. Finalized rocprofv3 JSON does not expose enough loss accounting to make missing kernel names safe evidence for skipping tests. The recorder therefore uses the pinned native SDK with lossless buffers and explicit completion checks, and the CUDA recorder gains matching error accounting. Records stay separate by backend while the selector combines their observations conservatively.

Companion: vLLM source maps and AMD Docker artifact forwarding. Deploy the consumers before enabling producers, then refresh the recorded coverage.

AI assistance was used to implement these changes and run the following checks.

[AndreasKaratzas]

Linked runs with the two PRs (this + vllm-project/vllm#59100):

• https://buildkite.com/vllm/amd-ci/builds/13842/list
• https://buildkite.com/vllm/ci/builds/92017/list

[AndreasKaratzas]

Another run complete: https://buildkite.com/vllm/amd-ci/builds/14676/list

[AndreasKaratzas]

Ful CI run:

• https://buildkite.com/vllm/ci/builds/95743/list

[tahsintunan]

Looks good overall. I made a few inline comments.

[tahsintunan]

VLLM_CI_KERNREC=1 now turns on ROCm recording too, so we can't run CUDA recording alone. It also makes the collector wait on AMD jobs and image-build-amd, so a slow or cancelled AMD build holds up the CUDA table. We could maybe give AMD its own switch.

[AndreasKaratzas]

I added VLLM_CI_KERNREC_AMD=1, which arms the ROCm recorder on AMD steps and the symbol map on image-build-amd. VLLM_CI_KERNREC=1 is CUDA-only again, as on main, and the collector only waits on and depends on what is actually armed. With just the CUDA switch on, it no longer touches image-build-amd or AMD jobs.

Slow AMD jobs weren't really holding it up. The wait stops at the 180-minute deadline, and in the dailies AMD finishes long before NVIDIA: at most 87 and 83 min on #96247 and #96139, against NVIDIA tails of 1445 and 269 min.

With both switches on, a cancelled image-build-amd can still fail the shared collector, the same way fnrec-collect can today.

[tahsintunan]

Question: if a write fails right after the header (e.g. disk full), could # gpu_active be lost too, making an active process look idle?

[AndreasKaratzas]

Yes, that could happen. After the first failed write the recorder closed the fd, so a file with just the header and the sdk_version line passed the idle check. The parser can't tell that apart from a real idle process, so I fixed it in the recorder: on a write failure, kernrec_rocm.cpp now truncates the recording to 0 bytes before closing it. An empty file is never clean or idle. I added a check to test_rocm_sdk.cpp that caps the file size with RLIMIT_FSIZE right after the header. It fails on the old code and passes now. CUDA has no idle rule and counts stdio errors in the footer, so it isn't affected.

[tahsintunan]

The footer rule makes sense in principle, but it's costly here: in 96247 (current CUDA table), 186 of 234 recorded steps have a job with a footerless process, so they'd become unusable. I compared step kernel sets across 96139 and 96247 and the footerless steps are as stable as the rest (29/30 identical on vLLM kernels), so they don't look lossy in practice. Could CUDA get a softer rule, or make worker exits write the footer?

[AndreasKaratzas]

Thanks for measuring this. I get the same cost: 4,400 of 18,118 recordings in 96247 have no footer, across 186 of 234 steps. I'd still keep the rule for CUDA, because the same two builds don't show footerless steps as lossless. On vLLM symbol-map kernels, skipping steps with a failed job, all-footer steps were 38/38 identical and footerless steps 156/172, and most of the differing vLLM names show up only in footerless processes.

The main cause is that EngineCore gets SIGKILLed right after SIGTERM: with shutdown_timeout=0 the process manager waits 0 s on CUDA, and the log shows "force killing remaining process EngineCore". No exit hook can run there, and an idle-style exception would only save 12 of those 186 steps. So I'd go with your second option, but in vLLM, as a follow-up: give EngineCore a shutdown grace period on recording runs, like the existing ROCm 15 s one, so these processes write real footers.

Also note 96247 used main's recorder with old-style footers, so it has no usable rows under this PR either way.

[AndreasKaratzas]

Follow-up is up: vllm-project/vllm#63260 lets an explicitly set VLLM_WORKER_SHUTDOWN_TIMEOUT_SECONDS give EngineCore a cleanup grace after the immediate abort, on every platform (defaults unchanged). This PR now exports =30 on CUDA recording runs too (f369133), as it already did on ROCm. On MI300 with CUDA's 0 s behaviour simulated, EngineCore footers go from 0/10 to 10/10. It covers the ~931 EngineCore/worker recordings killed on that path in 96247, not the other footerless causes.

[tahsintunan]

Thanks, nice catch. Fixing root cause makes more sense

[tahsintunan]

Minor: this can publish a table the selector can't use (e.g. all rows unusable after a setup failure, or one that fails the loader's checks) over the last good one. I think we can skip publishing in those cases.

[AndreasKaratzas]

Right. The gate only checked that the table had rows, so a build where every job failed would still publish and move latest.json. collect.sh now needs at least one row that passes kernel_table.usable, plus a map version the loader accepts. Otherwise that backend isn't published.

New test case unusable-rerun: a good pair is already published, then a rerun where every job failed. It fails on the old collect.sh and passes now. I didn't pull the full selector loader into collect.sh, because it imports the rules/pipeline_generator package. ci-fetch-kernel-record still runs those loaders before it replaces anything.

[tahsintunan]

nit: could we pin the ziglang and CUPTI versions in cross-build.sh so this .so is reproducible?

[AndreasKaratzas]

Done. cross-build.sh now pins ziglang 0.14.1, CUPTI 13.0.85, cuda-runtime 13.0.96 and cuda-crt 13.0.88-1, and each can be overridden from the environment. With the defaults it rebuilds the committed libkernrec.so byte for byte (sha256 82b3ca62...). Zig was the version that mattered: the .so was built with clang 19.1.7 from ziglang 0.14.1, and an unpinned run today pulls 0.17.0 and gives different bytes. The 13.0.x CUPTI/runtime/crt patch versions don't change the output.

[tahsintunan]

LGTM