fix(vllm): add env-gated CPython GC freeze policy for FPM self-benchmark

[liyuanzhe1991]

Overview:

Single-sample self-benchmark wall_time measurements are intermittently poisoned by
3–200x outliers. Root cause: CPython gen2 (full) collections walk every tracked
object, so their pauses grow with the heap — measured 0.4s → 9s over a 30k-point
sweep — and under TP/DP lockstep a single process's pause stalls the whole forward
pass. Triggers follow CPython's 25% long-lived-growth rule, producing outliers at
geometrically spaced, run-independent iteration indices (ratio ≈ 1.23, identical
across nodes/days/topologies) at a fleet-constant ~0.08%/sample rate.

This PR adds an opt-in, benchmark-scoped GC policy that keeps those pauses out
of measured windows. It is env-gated and off by default — zero behavior change
unless explicitly enabled, engine-core activation is benchmark-gated, and normal
GC is fully restored when the benchmark deactivates.

Details:

• New dynamo/vllm/gc_policy.py: on activation, disable automatic gen2
  (gc.set_threshold(t0, t1, 1<<30)), gc.freeze() the engine-init heap once,
  then a daemon thread runs a bare gc.freeze() per tick (O(1) generation-list
  splice: no traversal, no pause). Refcounting and gen0/gen1 stay fully active, so
  only cyclic garbage defers; young-gen scans stay bounded by allocation rate
  instead of total heap size.
• Lifecycle is symmetric: the engine-core process starts the policy from
  _bench_init only when a benchmark is configured, and _bench_deactivate calls
  stop_gc_policy() — stops the freeze loop, restores the saved gen2 threshold,
  and unfreezes + collects the deferred garbage. gc_maintain()
  (unfreeze → collect → freeze) is available for explicit reclamation from untimed
  windows in long runs.
• Worker processes are covered via vLLM's worker_extension_cls
  (FpmGcWorkerExtension), auto-injected in benchmark mode when
  DYN_FPM_GC_POLICY is set and the user configured no extension; a conflicting
  extension raises rather than silently leaving workers unpoliced. No vLLM changes.
• Two rejected variants are documented in the module docstring, both reproduce
  measurable stalls: periodic gc.collect() re-introduces growing full-heap walks
  (outlier rate 0.082% → 0.712%), and per-tick gc.get_freeze_count() walks the
  whole permanent generation (0.4–4s once millions of objects are frozen).

Validation (GLM-5.2-NVFP4, 4×B200 attention-DP=4, vLLM 0.25.1)

A/B on identical 30,480-measurement sweeps (same node, same GPUs, same point list;
only the GC policy toggled):

run	outliers >3× coordinate median	max pause
baseline	25 (0.082%) — geometric gen2 ladder	8,900 ms
GC policy on	2 (0.007%) — the eager first-touch pair, fixed by the warmup PR	1,131 ms

End-to-end: fresh single-sample collections with this policy reach median-of-5
database quality (LOO prefill 4.74%/max 125% vs reference 4.17%/129%; holdout
2.24%/max 12.2% vs 1.96%/13.5%) at ~1/3.5 the collection cost, with zero
curve-neighbor poisoned rows. No OOM across four full sweeps with gen2 disabled
(worker RSS ≈ +1–2 GB/h, decelerating); no change to the non-outlier latency
distribution.

Where should the reviewer start?

• components/src/dynamo/vllm/gc_policy.py — the policy, including design
  rationale and rejected variants in the module docstring
• components/src/dynamo/vllm/instrumented_scheduler.py — the two
  benchmark-gated hooks (_bench_init start, _bench_deactivate stop); 9 lines
  total
• components/src/dynamo/vllm/args.py — benchmark-mode worker_extension_cls
  injection and collision handling
• components/src/dynamo/vllm/tests/test_gc_policy.py — lifecycle, idempotency,
  threshold restore, and frozen-cycle reclamation tests

Related Issues

🚫 This PR is NOT linked to an issue:

• Confirmed — no related issue

---

Summary by CodeRabbit

• New Features

  • Added an optional garbage-collection policy for self-benchmarking workflows.
  • When enabled with DYN_FPM_GC_POLICY=freeze, memory cleanup is managed during benchmark runs to reduce garbage-collection pauses and improve measurement consistency.
  • Cleanup is automatically started and stopped as benchmarking begins and ends.
  • Added maintenance support for reclaiming cyclic garbage between timed measurement windows.

• Bug Fixes

  • Conflicting garbage-collection settings now produce a clear configuration error.

[devin-ai-integration]

Devin Review found 2 potential issues.

[coderabbitai]

Walkthrough

Adds an environment-controlled FPM garbage-collection freeze policy, worker extension controls, benchmark lifecycle hooks, engine configuration validation, and tests covering startup, maintenance, shutdown, restart, and interval parsing.

Changes

FPM GC policy

Layer / File(s)	Summary
GC policy and worker controls components/src/dynamo/vllm/gc_policy.py, components/src/dynamo/vllm/tests/test_gc_policy.py	Adds configurable periodic freezing, maintenance and shutdown behavior, worker extension methods, import-time startup, and tests for lifecycle, reclamation, and interval parsing.
Engine extension configuration components/src/dynamo/vllm/args.py	Injects the required worker extension when the GC policy is enabled and rejects conflicting extension configuration.
Benchmark lifecycle integration components/src/dynamo/vllm/instrumented_scheduler.py	Starts the GC policy during benchmark initialization and stops it during benchmark deactivation.

Estimated code review effort: 3 (Moderate) | ~25 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is specific, concise, and accurately summarizes the env-gated CPython GC freeze policy for vLLM FPM self-benchmark.
Description check	✅ Passed	The description includes Overview, Details, reviewer start, and Related Issues sections, and it clearly marks the PR as not linked to an issue.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/src/dynamo/vllm/args.py`:
- Around line 362-382: Update the DYN_FPM_GC_POLICY handling in the engine
configuration path to inject or validate FpmGcWorkerExtension only when the
environment value enables the policy’s freeze behavior. Treat values such as
“off” as disabled, preserving custom worker extensions and avoiding injection
when freeze is not enabled; keep the existing extension validation for enabled
freeze mode.

In `@components/src/dynamo/vllm/gc_policy.py`:
- Around line 85-95: The _freeze_loop function broadly catches and suppresses
gc.freeze() failures, allowing the process to continue with the policy disabled.
Catch only a specifically expected exception, or after logging an unexpected
failure re-raise it and transition the policy through its cleanup path so normal
GC thresholds are restored.
- Around line 61-68: Update _interval_seconds to reject non-finite parsed
values, including positive or negative infinity and NaN, alongside invalid
strings. Treat them like other invalid inputs by logging the existing warning
and returning the 60.0-second fallback; retain the existing minimum clamp for
finite values.
- Around line 171-177: Keep the import-time start_gc_policy() behavior in
components/src/dynamo/vllm/gc_policy.py:171-177 unchanged for worker-extension
loading. In InstrumentedScheduler._bench_init at
components/src/dynamo/vllm/instrumented_scheduler.py:1628-1630, start the
engine-core GC policy only after benchmark configuration validation and
successful initialization; ensure failures roll back any already-started policy.
Update the related cleanup or lifecycle logic at
components/src/dynamo/vllm/instrumented_scheduler.py:2834-2838 and adjust
expectations in components/src/dynamo/vllm/tests/test_gc_policy.py:18-25 to
verify startup does not persist after failed initialization.

In `@components/src/dynamo/vllm/tests/test_gc_policy.py`:
- Around line 46-57: Update test_stop_is_idempotent_and_allows_restart so the
complete GC policy start/stop and restart sequence is enclosed in a try/finally
block, with gc_policy.stop_gc_policy() in finally. Preserve the existing
assertions while ensuring cleanup runs after any assertion failure.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 658f3bb5-0fce-4ceb-9a79-34474affb880

📥 Commits

Reviewing files that changed from the base of the PR and between 769ee07 and b3db0aa.

📒 Files selected for processing (4)

• components/src/dynamo/vllm/args.py
• components/src/dynamo/vllm/gc_policy.py
• components/src/dynamo/vllm/instrumented_scheduler.py
• components/src/dynamo/vllm/tests/test_gc_policy.py

[tedzhouhk]

Blocking on post-benchmark GC equivalence for real serving. _bench_deactivate() restores only the engine-core process; workers auto-start the policy on extension import and no call site invokes fpm_gc_stop, so they continue serving with automatic gen2 collection disabled and the freeze daemon active. I reopened the existing worker-lifecycle thread with the required normal-completion and abort coverage.

[tianhaox]

Question on the completion path: if _wait_and_load_benchmark itself raises (e.g. artifact validation failure), _stop_worker_gc_policy never runs. If that path is guaranteed to terminate the process, this is fine as-is; if any caller can survive it and proceed to serving, the stop belongs in a finally. Worth a one-line clarification either way.

[liyuanzhe1991]

@tianhaox Good catch — that path could survive into serving on the prefill worker. The stop now sits on both exit paths (see _await_benchmark_then_restore_workers); the original error still propagates.

[tedzhouhk]

Re-reviewed at 73bae73. The benchmark lifecycle now restores the vLLM serving freeze, stops worker GC policy on both success and failure before model registration, preserves the primary error, and has process-level lifecycle coverage. Targeted GC-policy tests and current CI pass.