feat(profiling): nsight-slurm profiling type (Nsight Cloud for Slurm)

[nv-yna]

Summary

Adds profiling.type: nsight-slurm, which launches every worker srun step through the
Nsight Cloud for Slurm wrapper
(nsight-slurm srun ... -- <cmd>) instead of prefixing the worker command with nsys profile.
One coordinator per job collects time-aligned per-rank Nsight Systems reports into a shared report
workspace (<log_dir>/nsight-slurm-reports, with manifest and common collection ids).

Companion to #418 (playbook-flag nsys path). Both are independent of observability.enabled,
so the profiling overhead can be measured against the default-mode baseline (job 4 / PR #417).

Stacked on #417 (yna/observability-combined); the diff vs that branch is the last commit.

What changes

• schema (src/srtctl/core/schema.py): ProfilingType.NSIGHT_SLURM; fields
  nsight_slurm_home, nsight_slurm_tool_path (default /usr/local/bin/nsys),
  nsight_slurm_profiling_mode (at-launch | manual | cuda-api, default cuda-api),
  nsight_slurm_tool_options (default = the TRT-LLM profiling playbook set minus connector-owned
  flags), nvtx_injection_path. Worker env carries the TLLM_PROFILE_START_STOP window plus
  TLLM_PROFILE_LOG_RANKS=all, DYN_ENABLE_RUST_NVTX=1, NVTX_INJECTION64_PATH.
  Validation: absolute home, known mode, connector-owned flags (-o, --force-overwrite,
  -c/--capture-range*, --start-later) rejected, cuda-api requires backend.type: trtllm.
• start_srun_process (src/srtctl/core/slurm.py): optional srun_launcher /
  launcher_env. With a launcher, native srun options are forwarded unchanged and terminated by
  --; --export=... is omitted (the wrapper appends its own --export=ALL) and the export vars
  ride in the wrapper's process environment.
• NsightSlurmStageMixin (src/srtctl/cli/mixins/nsight_slurm_stage.py): before the first
  worker step, once: configure tool-path / tool-command profile / profiling-mode / tool-options / report-output, enable pyxis, coordinator start; coordinator stop --force in sweep cleanup.
  All wrapper calls logged to <log_dir>/nsight-slurm.out; SLURM_SUBMIT_DIR redirected to the
  log dir so the wrapper's job state lives under the run.
• worker_stage: both srun launch paths take the launcher kwargs.
• docs/profiling.md: nsight-slurm section.
• dry-run (src/srtctl/cli/submit.py): Profiling panel, incl. the nsight-slurm launcher rows.
• teardown (NsightSlurmStageMixin.flush_nsight_slurm, a ProcessRegistry pre-cleanup hook): end the nsys
  sessions gracefully and wait for the reports, since cuda-api mode only writes them when the session ends.
• mounts (start_srun_process(extra_container_mounts=...), ProfilingConfig.nsight_slurm_container_mounts):
  the connector's mounts ride in srtctl's single --container-mounts; disable pyxis on the wrapper.

Tests

• tests/test_profiling_nsight_slurm.py (13): config helpers, env, validation (incl. cuda-api
  backend gate), launcher argv/env through a mocked Popen, stage sequencing through a fake
  <home>/bin/nsight-slurm that records argv/env/cwd, idempotence, missing-install error.
• Full suite: 1801 passed, 2 skipped (19 nsight-slurm tests incl. the registry hook race and the growing-report wait, + 3 dry-run tests). ruff check / ruff format --check clean on src/srtctl.
  (pre-commit's pinned ruff v0.8.4 flags two pre-existing UP038 lines in do_sweep.py:239 /
  schema.py:2045 that predate this branch; left untouched.)

Status / open items

• End-to-end run on hecate. Tool installed for aarch64 on lustre (nsight-slurm 0.13.3, superproject
  d9a7d30, coordinator df7e496, nsight-cloud-storage 57a1570; job 571136). First profiling job 571147
  configured the wrapper, started the coordinator and ran the connector inside every worker container, but
  the 4 ranks on the coordinator's own node hit [Nsight ERROR] Timeout connecting to Coordinator while all
  28 remote ranks joined; the step exited 1 and srtctl aborted the sweep.
  Root cause (verified on a compute node with a pyzmq probe): the wrapper publishes
  tcp://<SLURMD_NODENAME>:<port>; broker and client set zmq.IPV6=1, so ZMQ resolves with
  AF_INET6|AI_V4MAPPED and connects to the first result only. On hecate a node's own name resolves to
  link-local fe80::… addresses (no v4-mapped entry at all), so tcp://<own name>:<port> times out
  while tcp://<ipv4>:<port> and remote names work. Fix 24aa66d: srtctl sets
  SLURMD_NODENAME=<head_node_ip> for the wrapper CLI calls so the coordinator address is an IPv4 literal.
  (Upstream note for nsight-cloud-slurm: publish an IP literal, or iterate over resolved addresses.)
  Rerun 571265: all 32 ranks joined the coordinator, but the workers never came up: the wrapper's
  enable pyxis mode appends a second --container-mounts, and pyxis applies only the last one
  (verified on a 1-node allocation), so /model, /logs and /configs were gone inside the containers and the
  dynamo install preamble failed. Fix 62df1ef: srtctl no longer uses enable pyxis; it mounts the install
  root (ro), the connector onto /usr/local/bin/nsight-slurm-connector, and the log dir at its host path
  inside its own single --container-mounts (new start_srun_process(extra_container_mounts=...)).
  Verified on 1 node: connector starts, /configs visible, app runs.
  Run 3 (571904): all 32 ranks joined, workers came up, both capture windows ran (connectors logged the
  nsys RangeCollection state), but (a) prefill worker 1 hit the same TRT-LLM hang as PR feat(profiling): adopt the Dynamo Benchmark Playbook nsys recipe for TRT-LLM workers #418's runs, at the
  same prefill iteration 2324 (deterministic in the warmup stream with nsys attached), and (b) 0 reports:
  in cuda-api mode the wrapper keeps the nsys session open after the range (--capture-range-end repeat), so
  reports only exist when the session ends, and srtctl's teardown SIGKILLed the steps. Fix 58477a8:
  flush_nsight_slurm() before registry.cleanup() (nsight-slurm stop --job, then SIGTERM to the wrapper
  processes of the worker steps, then wait for *.nsys-rep to appear and settle).
  Run 4 (572233): same hang (prefill iteration 2319), and the flush ran too late: the critical-process
  monitor thread calls registry.cleanup() itself. Also, the connector keeps nsys output in its runtime
  workspace (container-local /tmp) and only copies it out on a collection-stop it never sees for the
  nsys 2026.3 range states (RangeCollection/RangeGeneration are unknown to connector 1.5.0).
  Fix b009d49: ProcessRegistry.add_pre_cleanup_hook() (flush runs before any teardown path kills steps)
  and the runtime workspace bind-mounted from <log_dir>/nsight-slurm-runtime at /nsrt (AF_UNIX socket
  path limit) with rescue-copy of *.nsys-rep into <report_root>/rescued/.
  Run 5 (572590): decode hang at iteration 7624 (= PR feat(profiling): adopt the Dynamo Benchmark Playbook nsys recipe for TRT-LLM workers #418's run 1), and two more findings: (a) the finally
  block and the monitor thread raced into cleanup(), the second caller skipped the hook and killed the steps;
  (b) the lustre-backed runtime workspaces (32 rank dirs, ipc socket fine) stayed empty through
  RangeCollection/RangeGeneration: without -o, cuda-api nsys writes nothing until the session ends.
  Fix 5261265: hooks run under a dedicated lock (concurrent cleanup() waits), and the tool-options carry
  -o <report_root>/direct/%q{SLURM_JOB_ID}_%q{SLURMD_NODENAME}_rank%q{SLURM_PROCID} so every rank writes
  straight to the shared filesystem at session end / SIGTERM.
  Runs 6-9 (572866, 572948, 572964, 572999) never reached the workers: hecate jobs had lost their Kerberos
  credential on the compute nodes. After re-auth:
  • 575849: the wrapper refused a non-empty report workspace (our pre-created direct/) -> 576229e moves the
    direct output next to it.
  • 576022: flush now runs first, but SIGTERM to the wrapper process is relayed to srun, which SIGKILLs the
    step -> c1e3725 signals the connector tasks (nsight-slurm-connector steps) with --signal=TERM.
  • 576684: connector tasks got SIGTERM, workers exited, but nsys waited for re-parented children -> ad4d2a2
    adds --wait=primary.
  • 576975: nsys now writes reports into <log_dir>/nsight-slurm-direct/ (11 files appeared, progress bars in
    the logs), but the flush judged the file COUNT settled after 20 s and cleanup killed nsys mid-write (all
    0 bytes) -> 89c0353 settles on bytes, requires non-empty reports, settle 60 s / timeout 600 s.
    Validation run of 89c0353 pending (needs a fresh cluster login). Every run so far also hit the TRT-LLM hang
    from PR feat(profiling): adopt the Dynamo Benchmark Playbook nsys recipe for TRT-LLM workers #418 (prefill iteration ~2324 or decode 7624), which is what triggers the teardown flush.
• srtctl dry-run shows a Profiling panel (type, windows, launcher, tool-options, report-output) — dc42fa8.

🤖 Generated with Claude Code