[tedzhouhk]

Note

Stacked PR, live retune (split from #17562). Merge order: #17483 → #17733 → #17734 → #17735 → #17736 (this PR) → #17562

This PR targets hzhou/live-retune-3-trace, the branch of #17735 (the PR below it), so its diff shows only this layer. #17562 was split into these layers for review. At the split they carried #17562's original diff exactly, plus a 40-line replay check in #17734 that rejects executor: dgdr in offline replay; the review follow-up of 2026-11-02 has since changed several layers.

Overview

This is the fourth layer of live retune (#17562). It adds the live sweep executor, which runs one retune window through DGDR v1beta2 Requests, and the Kubernetes client the live modules use.

run_dgdr_sweep is the sweep_fn that #17562's RetuneController hooks call. Nothing calls it in this layer. A live Planner still rejects retune at startup, and replay still rejects executor: dgdr through #17734's replay check.

Key Changes

About 2.3k lines of implementation and 3.5k lines of tests (1.1k of them the shared fakes). All files are new except two typing changes: contracts.py gains SupportsIsSet, and window_cut.py accepts it as cancel.

• planner/retune/live/dgdr.py: builds and runs the DGDR Requests of one window, and scores the result.
  • It splits the router grid into at most run.max_shards balanced Cartesian shards of at most 100 points each, with planner_config's balanced_shard_parts, the same call P13 (feat(planner): add the live retune configuration and deployment premises #17734) makes, and creates one Request per shard. Each Request carries spec.workload.trace with source.persistentVolumeClaim{claimName, subPath} and no format (the CRD defaults it to Dynamo), the pinned retune.dgdr.engine, the DGD's replica counts and their GPU total as the budget. An off-grid baseline or startup point gets its own single-point Request.
  • One more single-point Request replays a grid point again as a common-random-numbers probe. The two scores must agree within a relative error of 1e-9 (crn_mismatch). There is no runtime-version floor, so the probe alone decides; the Run's runtime image and version are still recorded in the details.
  • It adopts the Requests a previous leader created for the window, matched by DGD uid, window, shard and spec hash. A window whose shards were all adopted as Succeeded is harvested without being cut again.
  • It cuts the window file through feat(planner): cut live retune windows and reconcile the trace claim #17735's cutter and then creates the Requests with Strict field validation. It checks that the API server kept every submitted field. It then checks each Run's status.aisimulate.input against the Request with the same subset check: the input may carry new operator defaults (under engine, router, optimizer or the trace source), but must hold every value the Request asks for, and an expected null (a deleted worker role, an unset SLA or replay speedup) must stay unset. inputOverrides win over the typed trace source, as in the operator, and nothing restores it. The operator's lowering golden from feat(planner): live retune from recorded traces through DGDR v1beta2 #17483 is the oracle for this check.
  • It waits on Runs, reads Candidates, maps their router parameters back to live names, requires identical non-router parameters across the window, and scores them with retune.objective. An open Run's Completed reason and message (ContainerWaiting, TraceSourceMissing, JobRunning) become the shard's status and appear in the timeout and run_pending errors.
  • It never raises and never deletes. Requests to remove go to the SweepRegistry queue. On shutdown or lost leadership it returns at once and keeps every Request for the next leader: the wait between polls blocks on the cancel and abort events, so either ends it immediately.
  • A failed baseline or probe shard fails the window. Any other failed shard only leaves its points unscored.
• planner/retune/live/kube.py: KubernetesRetuneKube, which implements contracts.RetuneKube over the official kubernetes client.
  • It requires an api_client with a loaded configuration; from_kube_api reuses the planner connector's client.
  • Every call carries a request timeout and raises a Kube* error that carries the HTTP status.
  • The DGD (v1beta1) and the DGDR v1beta2 Requests, Runs and Candidates come back as the API server's JSON.
  • Leases are read without model deserialization, so they round-trip unchanged.
• planner/tests/live_retune_fakes.py (moved here from feat(planner): add the live retune configuration and deployment premises #17734): FakeRetuneKube is an in-memory Kubernetes with resourceVersion compare-and-swap, Strict field validation, label selectors and injectable faults. FakeDgdrOperator drives Requests through the DGDR v1beta2 lifecycle the way feat(planner): live retune from recorded traces through DGDR v1beta2 #17483's operator does; its lower_request_input lowers the persistentVolumeClaim trace source like the operator.
• Tests: unit/test_live_retune_dgdr.py covers shard planning and the time budget, the RFC 7386 shape of the Requests, adoption and replacement of attempts, API and webhook errors, Run timeouts, the CRN probe and scoring, the Run input read-back, plus one sweep with the real window cutter. test_request_and_run_input_match_the_operator_golden reads feat(planner): live retune from recorded traces through DGDR v1beta2 #17483's trace_request_lowering.golden.json: build_request must produce its request_spec, the read-back must accept its run_input and still report a restored prefill role or a different trace mount path, and lower_request_input must reproduce run_input. unit/test_live_retune_kube.py covers the client.

dgdr.py imports contracts, planner_config, reconciler.trace_pvc_source (#17735), core.types.RetuneParams (already on main) and retune.sweep (with #17734's fields). kube.py imports contracts and kubernetes. Neither imports a later layer.

Validation

Ran on this layer (d0aedfc17f, before the review follow-up) and compared with the same suites at the stack base 6e8087d359 (#17483's head):

Check	Result
test_live_retune_dgdr.py and test_live_retune_kube.py	104 passed
Planner and replay suites (components/src/dynamo/planner/tests, components/src/dynamo/replay/tests, no xdist)	59 failed, 2698 passed, 2 skipped. The failing tests are the same 59 as at the base (59 failed, 2131 passed).
pre-commit run --files on the 4 changed files	All hooks pass, no files changed
CI mypy (the dynamo-pipeline.yml command, MYPYPATH=components/src:lib/bindings/python/src mypy --explicit-package-bases) on dynamo/planner and dynamo/replay	The same 9 errors as at the base (missing yaml/pandas/nvtx stubs and existing arg-type errors); none is new, and none is in retune/live

The 59 failures come from this host. Its prebuilt bindings lack conversation_affinity (#16854) and the ais-forward-pass feature, and it has no etcd binary. CI builds the bindings from source.

After the review follow-up, the tests ran at the top of the stack (#17562's head 92cee14c88), where this layer's files are unchanged: the planner unit tests (this layer's, the golden test included), the live retune e2e test, the frontend parser builder tests and the common configuration tests gave 2463 passed and 3 skipped (trtllm, sglang and vllm are not installed). pre-commit from 8d098bed48 to 92cee14c88 is clean. Per-layer runs, the replay suite and mypy were not part of that run.

This layer adds library code that nothing reachable calls until #17562. No production code outside this layer references live.dgdr, live.kube, run_dgdr_sweep or KubernetesRetuneKube. retune/controller.py, engine_adapter.py, core/base.py and __main__.py are unchanged. The executor's real-cluster behavior was exercised only in #17562's development-cluster run, on the pre-split branch and before the review follow-up.

Issues and Ongoing Work

• Part of live retune (feat(planner): enable live retune through DGDR v1beta2 #17562), the retune work in feat(planner): engine-owned retune with replay router control #17095 and the runtime reconfiguration DEP DEP (light): Planner runtime reconfiguration with predictor warmup #16835. It relates to DEP DEP: Evolve DGDR for replay-backed multi-candidate search #13092 (DGDR replay-backed search).
• Not verified: the default grid's run time (108 points, 3 shards of 36). feat(planner): enable live retune through DGDR v1beta2 #17562 tracks a smoke measurement to set the defaults.
• The next layer is feat(planner): enable live retune through DGDR v1beta2 #17562, which wires this executor into the live Planner.

Review follow-up (2026-11-02)

Requests and read-back:

• Requests name the trace as source.persistentVolumeClaim{claimName, subPath} and omit format, which the CRD defaults to Dynamo. The read-back finds the trace file through feat(planner): cut live retune windows and reconcile the trace claim #17735's trace_pvc_source.
• The Run input read-back is now a subset check: it tolerates new operator defaults but still catches a missing value or a restored null. It follows feat(planner): live retune from recorded traces through DGDR v1beta2 #17483's rule that inputOverrides win over the typed trace source.
• Added test_request_and_run_input_match_the_operator_golden, which checks the Request, the read-back and the fake operator against feat(planner): live retune from recorded traces through DGDR v1beta2 #17483's lowering golden.

Executor:

• Shard planning and the launch gate call planner_config's shard planner instead of a second copy.
• Removed the CRN runtime-version floor, which nothing set; the Run's runtime image and version are still recorded.
• An open Run's Completed reason and message show up in the shard status and in the timeout and run_pending errors.
• Cancel and abort end the wait between polls at once. A small stop flag replaces the polling Event subclass passed to the window cutter, whose cancel is now typed SupportsIsSet.
• Typed SweepRequest fields are read directly.

Client and tests:

• KubernetesRetuneKube requires its api_client.
• Added live_retune_fakes.py here (moved from feat(planner): add the live retune configuration and deployment premises #17734) and dropped the tests of the fakes themselves.
• Merged feat(planner): cut live retune windows and reconcile the trace claim #17735's review follow-up into this branch.

Second review follow-up (2026-11-02)

• RunNameConflict, RunCreating and CancelingSupersededRun are recorded on the shard, so a deadline reports them instead of a bare timeout.
• generator: direct stays, now with a comment: the planner reads only each Candidate's parameters and score, and direct keeps retune Runs independent of the AIC generator (AISimulate DGD generation rejects candidates without an explicit block size #17581).
• The poll wait is cancel.wait() followed by an abort check, with no helper threads; plan_shards requires parallelism.
• Lease and DGD access in kube.py and the fakes moved to feat(planner): enable live retune through DGDR v1beta2 #17562; update_pod and run_to_completion are gone; the config-load checks P5 already enforces are no longer repeated.

Validation at dfe7edf204: planner unit 1968 passed, including the golden test.

Pull Request Checklist

• I used Dynamo's /pr-checklist skill.

🤖 Generated with Claude Code

[devin-ai-integration]

✅ Dynamo PR CI passed — run 41524037365 (attempt 1) on dfe7edf204

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 41524037365. Updated on every full-CI run of this PR.}

[jthomson04]

Reviewed at 2dce2a5. Three should-fix items inline, plus nits. Naming, adoption after restart, 409/404 handling and the status fields read all match the operator at #17483.

[jthomson04]

Re-reviewed at 444ab6d. All nine comments are addressed. input_mismatches is a subset check now and is tested against the Go golden file; the Python rebuild of the lowering stays (here and in live_retune_fakes.lower_request_input), but the golden pins it. One should-fix inline (the same bare-timeout gap as before, on the Request side), a question and a nit.

Trims:

• The Lease and DGD support in live_retune_fakes.py (get_dgd, the *_lease methods) and in kube.py is used only by #17562; moving it there takes about 300 lines out of this PR. update_pod and run_to_completion aren't used anywhere in the stack.
• dgdr.py:434-484 repeats P5's config-load checks (_validate_dgdr_engine, _validate_dgdr_parallelism): the backend, workers being a mapping, the single preset entry, positive integers. Keep the spec_replicas role and replicas >= 1 checks, which depend on the live DGD, and drop the rest with their test rows.