Repository navigation
[ci-selector] Fail the collect steps when an armed recorder delivered nothing - #881
Merged
Merged
Conversation
… nothing The kernel collect step passed through six days of lost CUDA recordings (#685 to #858): with nothing uploaded it logged "nothing to fold; done" and exited 0, and in vllm/ci #97886 ROCm published while CUDA had zero rows, which also exited 0. - The generator passes the kernel collect step KERNREC_EXPECTED_JOBS_CUDA and _ROCM, counted per backend like FNREC_EXPECTED_JOBS. - kernrec/collect.sh judges each armed backend: fewer than KERNREC_MIN_DELIVERED_SHARE (0.8) of its jobs delivering, or no valid table/map pair, fails the (soft_fail) step and leaves that backend unpublished. A whole backend still publishes. Without the counts it behaves as before. - fnrec/collect.sh fails when the generator armed jobs and none delivered. - Both leave a *_collect_summary.json artifact and an annotation, for the dashboard to read. - The ci-selector workflow runs both collect tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: khluu <khluu000@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The kernel collect step stayed green through six days of lost NVIDIA recordings (#685 → #858):
0 recording files from 0 jobs; nothing to fold; doneand exited 0 (vllm/ci #97223 … #97701).steps: 0/cuda has no valid table/map pair, and the step still exited 0, because it only fails when no backend publishes.Nothing signalled that the CUDA kernel table had been stuck at build 96933.
What
_collect_step): the kernel collect step getsKERNREC_EXPECTED_JOBS_CUDAandKERNREC_EXPECTED_JOBS_ROCM, the armed jobs per backend (parallelism counted), from the same list it waits on. Same idea asFNREC_EXPECTED_JOBS.kernrec/collect.sh: for each armed backend, fewer thanKERNREC_MIN_DELIVERED_SHARE(default 0.8) of its jobs delivering a recording, or no valid table/map pair, fails the step and leaves that backend unpublished (its previous pair stays current). A whole backend still publishes. Without the counts (older generator, manual fold) it behaves exactly as before.fnrec/collect.sh: an armed build (FNREC_EXPECTED_JOBS > 0) that delivered nothing fails instead of "nothing to fold".kernrec_collect_summary.json/fnrec_collect_summary.json(expected, delivered, rows, verdict) as an artifact and a Buildkite annotation, on every exit path. The vllm-dashboard health check will read the summaries.soft_fail, so the build stays green; the collect job itself goes red.ci-selector-tests.ymlruns both collect tests, which nothing ran before.Tests
kernrec/test_collect.sh: the 13 existing cases unchanged, plus 6 with armed counts —armed-whole(exit 0, both publish),armed-cuda-lost(#97886: ROCm publishes, exit 1),armed-empty(#97701: exit 1),armed-short(1 of 5, exit 1, not published),armed-broken-cuda(exit 1, earlier CUDA pair untouched),unarmed-empty(exit 0, as before). Every case checks the summary artifact.fnrec/test_collect.sh:no-recordingsnow exit 1 when armed;no-rec-unarmedexit 0; all 9 pass.pytest tests/pipeline_generator: 429 passed, including a new per-backend count test.🤖 Generated with Claude Code