[tedzhouhk]

Note

This PR now carries the whole live retune stack. It started as the DGDR v1beta2 trace workload (the operator part below). The five PRs stacked on it were reviewed separately and merged into this branch on 2026-11-02:

PR	Layer
this PR	Operator: spec.workload.trace replays a recorded trace from a PVC in DGDR v1beta2 Runs
#17733	Replay: DGDR trace Runs replay with common random numbers
#17734	Planner: retune.executor: dgdr configuration and deployment premises; frontend parser builder
#17735	Planner: window cutter and reconciler on the trace claim
#17736	Planner: the DGDR sweep executor and Kubernetes client
#17562	Planner: supervisor, Lease state, router actuator, wiring, Helm RBAC and docs

Each layer's PR keeps its detailed description and its review follow-ups. This description summarizes the combined change, which lands on main as one squash commit.

Overview

Periodic retune (#17095, #17540) runs only inside offline planner replay today. This PR lets a live Planner on Kubernetes retune its KV router with retune.executor: dgdr:

1. Frontends record their request traces to a shared ReadWriteMany claim.
2. At the end of each window, the Planner cuts the window's segments into one trace file on that claim.
3. It replays the file through DGDR v1beta2 Requests for a grid of router parameters. Each Request uses the new trace workload, and every trace Run replays with common random numbers, so Candidates of different Runs are comparable. It scores the Candidates on retune.objective.
4. Unless it runs in advisory mode, it sets the winner on every frontend router with PATCH /router_policy.

# DGDR v1beta2 Request (the Planner creates these; users can too)
spec:
  workload:
    trace:
      format: Dynamo                    # optional; defaults to Dynamo, the only value
      source:
        persistentVolumeClaim:
          claimName: request-traces     # PVC in the Request namespace
          subPath: retune/w-1.jsonl.gz  # one file, relative to the claim root
      replay:
        arrivalSpeedupRatio: 2.0        # optional

Live retune is EXPERIMENTAL, for development clusters only. It searches the three router knobs a DGDR Request can express (overlap_score_credit, prefill_load_scale, temperature) with fixed replicas, so the Planner must run retune-only, with load and throughput scaling off. The default executor: local keeps today's behavior, and the Planner's Helm RBAC is gated by the new value dgdrV1Beta2.plannerLiveRetune (default false).

User guides: docs/fern/pages/kubernetes/auto-deployment/planner-live-retune.md (new) and the "Replay a recorded trace (v1beta2)" section of the DGDR reference.

Key Changes

99 files, +34.1k/−0.3k lines, including regenerated CRDs and API reference.

Operator: DGDR v1beta2 trace workload (deploy/operator)

• API. New spec.workload.trace{format, source.persistentVolumeClaim{claimName, subPath}, replay.arrivalSpeedupRatio}, exclusive with static, following the trace workload in DEP DEP: Evolve DGDR for replay-backed multi-candidate search #13092. format is optional, with the enum value and CRD default Dynamo; lowering translates it to AISimulate's native dynamo. CEL and the shared schema check that claimName is a DNS-1123 subdomain, that subPath is relative with no empty, . or .. segment, no NUL and at most 1024 characters, and that the speedup is greater than 0. The Run snapshot rule from feat(operator): integrate DGDR v1beta2 candidate flow #17308 now accepts static or trace.
• Markers and lint. Every serialized v1beta2 field is marked +optional or +required (no schema change for existing fields). kube-api-linter's optionalorrequired rule enforces it: make lint-api (part of make lint) and the operator Dockerfile's linter stage run it over api/v1beta2 with .golangci-kal.yml.
• Validation. Structural validators in shared_v1beta2.go mirror the CRD rules for a v1beta1 carrier stored without admission. inputOverrides stay authoritative: they are applied last as a JSON Merge Patch and may restate, replace or delete any traffic the trace lowers; the operator restores nothing. The typed claim alone decides the Job's mount and the claim the Run depends on. The Request controller no longer validates the trace; a Run create the CRD rejects as Invalid reports InvalidSpec instead of retrying.
• Lowering. traffic.source{type: trace, format: dynamo, paths: [<fixed mount path>]}, traffic.load.type: trace_timestamps, and traffic.load.speedup when set, as an underlay for the overrides.
• Run Job. Only the trace file is mounted, read-only through subPath at /var/run/aisimulate/trace/trace.jsonl.gz (or trace.jsonl), into the sweeper alone. No fsGroup; a trace Job's pod gets group 65532 as a supplemental group.
• Claim lifecycle. Before it creates a trace Job, the Run adds its finalizer nvidia.com/dgdr-run-<run-uid> to the claim with an optimistic-lock patch. Deleting the claim cancels the Run through the foreground stop path (stopAndCompleteRun); the Run fails with TraceSourceDeleted only once the Job is gone, then releases its finalizer. A missing claim keeps the Run Pending (TraceSourceMissing) until it appears; a block or lost claim fails it (TraceSourceUnavailable). A waiting sweeper or publisher container only sets the ContainerWaiting diagnostic. The new DynamoGraphDeploymentRunTraceClaimReconciler releases finalizers of deleted Runs once none of their Jobs is left.
• RBAC and watches. PVCs, Run Job pods, Run and Job deletions; persistentvolumeclaims get;list;watch;patch, pods get;list;watch, jobs delete.
• Golden. internal/aisimulate/testdata/trace_request_lowering.golden.json pins the Planner's shard Request and the Run input it lowers to. TestDGDRV1beta2TraceRequestLoweringGolden admits that Request through the webhooks and CRD schema and lowers its Run; a mismatch writes <golden>.new. The planner's executor tests read the same file.

Replay: common random numbers (components/src/dynamo/replay)

• DynamoReplayRunner, which AISimulate's Sweeper loads and every DGDR trace Run executes, passes _deterministic=True (ordinal request ids, the canonical router seed) to both trace replay calls. This applies to every DGDR trace Run, not only the Planner's: a Run's scores no longer depend on the other candidates or the Run's parallelism. Synthetic workloads are unchanged.

Planner configuration and premises (planner/config, planner/retune, frontend and common configuration)

• RetuneConfig.executor (local by default, or dgdr) and RetuneConfig.dgdr (DgdrExecutorConfig, DgdrRunConfig), required if and only if the executor is dgdr.
• Configuration premises for dgdr: P1 (Kubernetes, decision latency, explicit timeout_seconds ≤ interval_seconds − 600), P2 (retune-only), P3 (only the three DGDR knobs, prefill_load_model: none), P5 (an all-scalar engine with one parallelism preset and no replicas), P13 (the grid fits run.max_shards shards of at most 100 points within the timeout), and safe claim paths. P13 and the executor share one shard planner (balanced_shard_parts, largest_shard_points, dgdr_rounds, dgdr_need_seconds).
• retune/live/frontend_config.py parses the DGD's frontend command, args and env into the startup KvRouterConfig and evaluates the spec premises P4 and P6–P10 (evaluate_premises(dgd, *, planner_config)). retune.controller.startup_router_domain holds the router checks that depend on the frontend startup; the controller and P4 both call it.
• To parse another process's frontend command, the frontend and common configuration modules gain ArgEnv and frontend_args.build_frontend_parser(env); the router, KV router and frontend argument groups take an optional env. Existing callers pass none and are unchanged.
• Offline replay rejects executor: dgdr (replay/api.py); replay retune runs executor: local.

Window cut and reconciler (planner/retune/live)

• window_cut.py merges each frontend pod's request_end jsonl.gz segments into <windows_dir>/<dgd>/<window_end_ms>.jsonl.gz, with the same rows as retune.window.read_trace_window. It runs in a niced, budgeted child process, validates each gzip member (CRC32, ISIZE), excuses a truncated tail at the end of any segment, fails a mostly corrupt window (trace_corrupt), bounds memory by max_window_rows, and publishes atomically with mode 0644 (the Run's sweeper reads as uid 1000 with group 65532). The output's gzip header comment holds the cut's counts, so a later leader reuses a published window without reading segments again.
• reconciler.py garbage-collects finished Requests, window files no live pod reads, cut temp files and old segments, asking the caller before every delete or Kubernetes read so a round stops when leadership is lost. It is also the ref-counted SweepRegistry that protects in-flight windows.

DGDR sweep executor (planner/retune/live)

• dgdr.py (run_dgdr_sweep) splits the router grid into balanced shards and creates one Request per shard with the trace workload, the pinned retune.dgdr.engine, the DGD's replica counts and their GPU total, and generator: direct. An off-grid baseline gets its own Request, and a common-random-numbers probe re-scores one grid point (relative error ≤ 1e-9, else crn_mismatch).
• It adopts a previous leader's Requests, creates new ones with Strict field validation, checks each Run's status.aisimulate.input against the Request with a subset check (the operator golden is the oracle), waits on Runs, and scores Candidates with retune.objective. Open-Run reasons (ContainerWaiting, TraceSourceMissing, RunNameConflict, …) show up in the shard status and errors. It never raises and never deletes; on shutdown or lost leadership it returns at once and leaves the Requests to the next leader.
• kube.py (KubernetesRetuneKube) implements the Kubernetes calls over the official client with request timeouts and status-carrying errors; Leases round-trip without model deserialization.

Live retune on the Planner (planner, Helm, docs)

• supervisor.py (LiveRetuneSupervisor) runs each window: epoch-aligned windows that survive restarts, leadership, premise and time-budget checks, advisory and active modes, and a two-phase rollback to the startup parameters. Each decision is persisted before any router sees it, and each router's latched initial before the actuator moves that router. stop aborts the sweep and releases leadership before the engine shuts down.
• state.py: the leadership Lease <dgd>-planner-retune and the state Lease <dgd>-planner-retune-state, with writes fenced by leader term and deploy sequence, resourceVersion preconditions on decommission deletes, and an unknown format version refusing initialization.
• actuator.py reads and sets the eight /router_policy parameters on every frontend router with revision-checked PATCH and reset, reverts manual changes outside advisory mode, and treats a router as gone only once every pod instance it ran on is gone, finished or restarted. metrics.py and bootstrap.py add the Prometheus metrics and the wiring.
• retune/controller.py gains optional hooks (external sweep_fn, launch and harvest gates, generation, a shared single-thread executor); the replay path passes none and is unchanged. engine_adapter.py, core/base.py and __main__.py wire the supervisor; a live Planner now rejects only executor: local. A Kubernetes Planner without retune checks in the background whether a removed live retune left its Leases and cleans up. SIGTERM cancels the run so the Lease is released before exit.
• Helm planner.yaml: with dgdrV1Beta2.enabled and dgdrV1Beta2.plannerLiveRetune, the Planner Role and ClusterRole gain create/get/list/delete on DGDR Requests, get on Runs, list on Candidates and get/create/update/delete on Leases, with helm-unittest cases.
• Docs: the live retune guide, retune.executor and retune.dgdr in planner-configuration.mdx, and executor notes in the planner guide (en and zh-CN), planner-design.md and aisimulate-planner-replay.mdx.

Behavior notes

• Recipes whose workers advertise their own router configuration (--router-mode or DYN_ROUTER_MODE on a worker, such as minimax-m3 and solar-open2) fail P4 by design: the Run replays only the frontend's configuration.
• /router_policy is the frontend's unauthenticated admin API; the guide explains how to deny it at L7.
• The chart has one Planner role per install, so with plannerLiveRetune=true Planners without retune get the live retune rules too. The guide lists what that allows.

Validation

CI on this head (08c5eb9661) runs the whole stack together for the first time. Local checks from the last review follow-up:

Check	Head	Result
go build, go vet (no tags and clustertest)	03106cf4d3	Clean
go test ./api/... ./internal/aisimulate/... ./internal/webhook/... ./internal/controller/ (envtest, golden in compare mode)	03106cf4d3	Pass
make lint (golangci-lint and lint-api)	03106cf4d3	0 issues
Planner unit tests, integration/test_live_retune_e2e.py, frontend parser builder and common configuration tests	08c5eb9661	2490 passed, 4 skipped (engines not installed)
pre-commit over every layer's follow-ups	08c5eb9661	Clean
docs_lint.py	08c5eb9661	0 errors

The operator code did not change above 03106cf4d3. The Helm RBAC tests were checked against helm template output; helm-unittest was not run locally after the follow-ups. Each layer's PR has its per-layer runs.

Development cluster. Before the review follow-ups, the pre-split branch ran on an H200 development cluster (vcluster, operator with dgdrV1Beta2.enabled=true, Qwen/Qwen3-0.6B on vLLM, one KV-router frontend recording traces, interval_seconds: 1200, run.parallelism: 2, max_shards: 3):

Case	What happened
1 worker, grid overlap_score_credit: {choices: [0.5, 1.0]}, objective GoodputPerGPU	The cut wrote 3143 requests in 2.7 s; 3 Requests, all Runs succeeded, 2 of 2 candidates harvested. Decision kept (one worker makes the knobs irrelevant). A manual PATCH /router_policy was reverted within 10 s.
2 workers, frontend at DYN_ROUTER_TEMPERATURE=1.0, grid temperature: {choices: [0.0, 1.0]}, objective TTFT	The cut wrote 3101 requests in 3.1 s; 3 Runs succeeded, sweep 28.7 s. Decision applied: simulated mean TTFT 4.493 ms at 1.0 and 3.778 ms at 0.0 (−15.9 %). The frontend applied temperature 0.0 at revision 1 two seconds later.

That run used the earlier trace fields (source.pvc.path, format: dynamo) and a chart without plannerLiveRetune; it has not been repeated since.

Not verified:

• A cluster run of the stack after the review follow-ups.
• Online TTFT after an applied decision, the advisory → promote_last_advisory path, and several frontends.
• The default grid's run time (108 points, 3 shards of 36) against real replay time and memory.
• Real foreground garbage collection of a Job and its pods on claim deletion, release across a live operator restart, and RBAC in a non-system:masters install.

Issues and Ongoing Work

• Part of the retune work in feat(planner): engine-owned retune with replay router control #17095 and the runtime reconfiguration DEP DEP (light): Planner runtime reconfiguration with predictor warmup #16835; relates to DEP DEP: Evolve DGDR for replay-backed multi-candidate search #13092 (DGDR replay-backed search).
• Known limitations:
  • A Run whose container cannot start (missing trace file, symlink leaving the claim) has no timeout; it stays open with ContainerWaiting until the cause is fixed or the Run is cancelled. On a volume type whose node directory the kubelet can write, such as hostPath, a missing file becomes an empty directory and the Run fails in AISimulate.
  • Pods of finished Jobs keep a deleted claim Terminating through kubernetes.io/pvc-protection until their Runs are retired. If the operator is removed while trace Runs are in flight, their finalizers stay on the claim; the DGDR reference shows how to remove them.
  • Until AISimulate DGD generation rejects candidates without an explicit block size #17581 is fixed, a v1beta2 Request without an explicit kv_cache.block_size fails in the default AIC generator. The Planner's Requests use generator: direct.
• Follow-ups: a smoke measurement on a real trace to set the defaults; aligning the DEP text with this executor.

Pull Request Checklist

• I used Dynamo's /pr-checklist skill.

🤖 Generated with Claude Code

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a1a3ff-3509-7484-87c4-7749a0cf10db.docs.buildwithfern.com/dynamo/dev

[devin-ai-integration]

❌ Dynamo PR CI failed — run 41680495672 (attempt 1) on 143a8375bb

Gate checks: ✅ backend-status-check · ❌ deploy-status-check · ❌ dynamo-status-check

Framework	Snapshot
vLLM	❌ 1

Other	Jobs
dynamo-runtime	❌ 1 ✅ 6

Failure details

2 jobs failed: the vLLM Snapshot vCluster setup timed out in helm waiting for a DaemonSet (infrastructure), and the dynamo-runtime CPU test job had 5 planner/common tests fail after retries before the runner's custom container implementation died mid-run, so no tracebacks were printed.

❌ vLLM Snapshot Operator Setup: helm "context deadline exceeded" waiting for vcluster-hpm-hostpath-mapper DaemonSet (4/5 ready)

Job: vLLM Snapshot Operator Setup · Failed step: Setup vCluster and operator · Logs: gh run view --job 126086337590 -R ai-dynamo/dynamo --log-failed

ready.go:336: [debug] DaemonSet is not ready: gh-id-41680495672-checkpoint-vllm-dt/vcluster-hpm-hostpath-mapper. 4 out of 5 expected pods are ready
ready.go:336: [debug] DaemonSet is not ready: gh-id-41680495672-checkpoint-vllm-dt/vcluster-hpm-hostpath-mapper. 4 out of 5 expected pods are ready
wait.go:87: [debug] Error received when checking status of resource vcluster-hpm-hostpath-mapper. Error: 'client rate limiter Wait returned an error: context deadline exceeded'
Error: context deadline exceeded
helm.go:86: [debug] context deadline exceeded
##[error]Process completed with exit code 1.

The vCluster helm install waited ~10 minutes for the vcluster-hpm-hostpath-mapper DaemonSet, but one of its 5 pods never became ready, so helm hit its deadline. This happened during K8s cluster setup, before any PR code was deployed. It is an infrastructure (K8s scheduling/pod readiness) timeout, and rerunning the failed jobs is likely enough.

❌ dynamo-runtime / test / sequential cuda13.0, amd64: 5 tests FAILED, then "Executing the custom container implementation failed"

Job: dynamo-runtime / test / sequential cuda13.0, amd64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 126092066722 -R ai-dynamo/dynamo --log-failed

components/src/dynamo/common/tests/http/test_proxy_connections.py::test_one_time_budget_covers_every_redirect_hop FAILED [  9%]
components/src/dynamo/planner/tests/integration/test_fpm_event_delivery.py::test_endpoint_fpm_reaches_planner_and_drives_scale_up FAILED [ 29%]
components/src/dynamo/planner/tests/integration/test_live_retune_e2e.py::test_advisory_records_recommendations_and_never_writes FAILED [ 29%]
components/src/dynamo/planner/tests/integration/test_retune_sweep_workers.py::test_a_dead_worker_fails_only_its_candidate FAILED [ 29%]
components/src/dynamo/planner/tests/unit/test_system_control_api.py::test_planner_control_uses_the_actual_system_listener FAILED [ 64%]
...
components/src/dynamo/profiler/tests/integration/test_profile_sla_dgdr.py::TestAutoBackend::test_no_planner_no_load
##[error]Executing the custom container implementation failed. Please contact your self hosted runner administrator.
...
❌ pre_merge_cpu: step outcome 'failure'
##[error]A pytest stage did not record a passing result. See this step's log for the stage.

In the pre_merge_cpu stage, 5 tests failed on every attempt (3 auto-retries each). At 67%, during test_profile_sla_dgdr.py::TestAutoBackend::test_no_planner_no_load, the runner's container hook (/home/runner/k8s/index.js) stopped the test container. pytest never reached its summary, so the log has no tracebacks or assertion messages for the 5 failures. To get them, run the tests listed here locally, e.g. python3 -m pytest components/src/dynamo/planner/tests/integration/test_live_retune_e2e.py::test_advisory_records_recommendations_and_never_writes -v. The container dying looks like a runner infrastructure problem, but a rerun will not fix the 5 tests if they fail on their own.

For agents

{"pr": 17483, "run_id": 41680495672, "run_attempt": 1, "head_sha": "143a8375bbad500cd6aa659168051fabd51ad444",
 "failures": [
  {"job": "vLLM Snapshot Operator Setup", "job_id": 126086337590, "failed_step": "Setup vCluster and operator", "signature": "helm context deadline exceeded: DaemonSet vcluster-hpm-hostpath-mapper 4/5 pods ready (infra)", "tests": [], "log_cmd": "gh run view --job 126086337590 -R ai-dynamo/dynamo --log-failed"},
  {"job": "dynamo-runtime / test / sequential cuda13.0, amd64", "job_id": 126092066722, "failed_step": "Run CPU-only tests (parallelized)", "signature": "5 pytest FAILED (after retries); then 'Executing the custom container implementation failed' killed the run before the summary", "tests": [
    "components/src/dynamo/common/tests/http/test_proxy_connections.py::test_one_time_budget_covers_every_redirect_hop",
    "components/src/dynamo/planner/tests/integration/test_fpm_event_delivery.py::test_endpoint_fpm_reaches_planner_and_drives_scale_up",
    "components/src/dynamo/planner/tests/integration/test_live_retune_e2e.py::test_advisory_records_recommendations_and_never_writes",
    "components/src/dynamo/planner/tests/integration/test_retune_sweep_workers.py::test_a_dead_worker_fails_only_its_candidate",
    "components/src/dynamo/planner/tests/unit/test_system_control_api.py::test_planner_control_uses_the_actual_system_listener"], "log_cmd": "gh run view --job 126092066722 -R ai-dynamo/dynamo --log-failed"}
 ]}

_{Posted automatically by Devin for run 41680495672. Updated on every full-CI run of this PR.}

Rereviewing as contents has materially changed.

[sttts]

Six P2 findings from the combined live-retune review. The inline comments describe each trigger, production failure path, focused correction and regression coverage.

[sttts]

Approved for the operator and documentation changes at 143a837.

This approval is scoped to those areas. The Planner/tuning findings in my inline review remain open and must be addressed separately before merging.