fix: harden FPM collector completion and failure handling

[Arsene12358]

Why and what changed

Newer Dynamo self-benchmark writes a worker-probe sidecar alongside its rank timing files. AISimulate could treat that sidecar as a rank measurement. On full grids, it could also stop the engines after rank files appeared but before local merged results and the advertised probe finished writing. Separately, a later Slurm node could fail during startup while the collector waited on an earlier node’s unfinished future.

Classify probe sidecars separately and wait for correctly bound local merged/probe artifacts before teardown. A separate 120-second finalization window starts after local rank measurements complete, does not reset during polling, and remains capped by the collection deadline. Recorded partial or failed probe observations remain visible to downstream validation. Observe Slurm node futures in completion order so the first observed failure promptly triggers existing cancellation of owned child processes and preserves both nodes’ logs and the original failure.

Review map

• Risk level: medium.
• Start with collector/fpm_forward/runtime/fpm_exec.sh for lifecycle ordering and artifact binding, then collector/fpm_forward/native_artifact.py for rank-file discovery and collector/fpm_forward/slurm.py for failure propagation.
• Public or serialized contract changed: optional FPM_FINALIZATION_TIMEOUT_SECONDS override, default 120 and range 1–3600 seconds. Existing artifact schemas are unchanged.
• Compatibility or rollback concern: older producers whose merged artifacts do not advertise a probe remain supported. No stored timing data needs migration.

Evidence

• Tests: 394 collector tests passed on 8bc44d6d4993efc0ecf1260acb507aa57821ca0d, independently rerun by task and whole-branch reviewers. Coverage includes runtime wrapper, artifact parsing, runner, Slurm, and measurement evidence.
• Fast CI: passed on 8bc44d6d4993efc0ecf1260acb507aa57821ca0d.
• Full CI: not run locally.
• CodeRabbit reviewed commit: none; automatic review was skipped because this is a draft.
• Codex reviewed commit: 8bc44d6d4993efc0ecf1260acb507aa57821ca0d; independent task and fresh whole-branch reviews, no findings.
• Negative or boundary cases: delayed merged/probe output, both local DP completion orders, stale or mismatched bindings, invalid timeout overrides, separate finalization timeout, collection-deadline cap, legacy producers, partial probe observations, a failed second node while the first remains alive, retained stdout/stderr, and existing interruption/admission races. Ruff lint/format, shell syntax, and whitespace checks passed.
• Expected-value derivation: artifact fixtures follow the actual producer schema and bind to their known run, grid, mode, paths, and local source ranks. Pending/complete assertions follow those producer states.
• Before/after: preserved full-grid artifacts contain all eight ranks' completed measurements but lack advertised probes. The old checker returned complete; the new checker returns finalization-pending. Four complete prefill/decode smoke shards still pass. A fresh eight-GPU GB300 prefill/decode smoke with the patched collector completed successfully; the larger replication pilot was cancelled after an engine-startup port bind failure and produced no measurements. Full-grid replication with the patched collector remains unvalidated.

Reproduce the collector test subset from python/aisimulate (the timeout plugin is disabled for macOS):

PYTHONPATH=src:. PYTHONDONTWRITEBYTECODE=1 python -m pytest -p no:timeout -p no:cacheprovider -o addopts= -q tests/unit/collector/test_fpm_exec.py tests/unit/collector/test_fpm_runtime_wrapper.py tests/unit/collector/test_fpm_worker_probe_artifact.py tests/unit/collector/test_fpm_runner.py tests/unit/collector/test_fpm_slurm.py tests/unit/collector/test_fpm_measurement_evidence.py

Modeling or data provenance

This changes collector control flow and artifact classification. Validation uses synthetic boundary fixtures plus preserved GB300 DEP8 artifacts from pinned vLLM 0.27.0 and current Dynamo PR source revisions. Timing values, collection grids, aggregation, and quality thresholds are unchanged. Probe-file completion does not establish measurement quality or worker-restoration success. Slurm failure handling cancels owned child commands; it does not cancel the caller’s allocation.

Tracking

• Closes: N/A.
• Related PRs or issues: follow-up to feat: add FPM self-service onboarding and validation #248; validated with Dynamo #15110 and Dynamo #15257.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}