[khluu]

Problem: the fnrec and kernrec collect steps depended on every recording step, with allow_dependency_failure: true. A cancelled or timed-out dependency still ends the step waiting_failed, and a waiting_failed step can't be retried. The full runs 95480, 95553, 95634 and 95663 each had a timed-out shard (language-models-test-extended-pooling 3/4, hung in a ColBERT test; vllm#62279) and published nothing. The ci-selector has been deciding from build 95238's records since Oct 15.

Change:

• The collect steps now depend only on what the recording steps themselves wait on: the image build, which uploads the kernel symbol map.
• Their first command polls each recording step through buildkite-agent step get state every 5 minutes. It moves on once all are finished or ignored, or once COLLECT_AFTER_MINUTES (180) has passed since the pipeline was generated, whichever comes first. Then it folds what was uploaded.
• A job still running at the deadline is left out, and the table reads its shard as missing, as it already does for lost shards.
• An unreadable state (older agent, missing step) counts as unfinished, so the step waits out the deadline rather than collecting early.
• The step's timeout is now 180 + 90 minutes.

Cost: the two collect jobs sit on a postmerge CPU agent while waiting, up to 3 hours per full run.

Tests:

• Updated for the new wiring (CI_SELECTOR_COLLECT_WAIT_FOR replaces depends_on).
• New test_collect_waits_for_recording_steps_in_its_command_up_to_a_deadline, which also checks every runtime $ is escaped from upload-time interpolation.
• I ran the wait script by hand against a fake buildkite-agent, for both the "all finished" and the "stuck until deadline" paths.
• Generator suite: 318 passed.

🤖 Generated with Claude Code