feat(operator): add LPX integration

[andrewpaprotsky]

Summary

• Add LPX component support to Dynamo for LPU-backed inference workloads.
• Include the LPX API, controller and handoff lifecycle, Dynamo rendering/runtime integration, scheduler and Grove integration, Helm configuration, generated API artifacts, tests, and documentation.
• Add @ai-dynamo/dynamo-lpx-codeowners as a co-owner of the LPX controller and rendering directories, retaining Operator ownership. The LPX team must exist and have repository write access before GitHub can use it for reviews.

Where should the reviewer start?

1. LPX API — component configuration and types.
2. LPX controller — scheduling and lifecycle reconciliation.
3. LPX rendering — runtime resources and configuration.

Related Issues

• Relates to DEP: LPX Component Type Integration #14473 — LPX Component Type Integration.

Validation

The following checks were performed during development:

• Operator Go tests, formatting, vetting, and module checks
• Operator manifest, Go API, and Pydantic generation and validation
• Platform Helm chart tests
• Fern API-documentation generation and linting
• Strict CODEOWNERS coverage, 157 ownership tests, and deterministic generation with LPX co-ownership
• Applicable pre-commit hooks and git diff --check

Known follow-ups

• Native scale across replacement: When component replicas is omitted, a whole-PCS replacement currently loses the externally managed engine count and falls back to one. Preserving that count is deferred to a follow-up.

Summary by CodeRabbit

• New Features

  • Added LPX scheduler integration for graph deployments, including model downloads, workload placement, readiness, scheduling deadlines, and status reporting.
  • Added the LPXGraphDeployment Kubernetes resource and LPX component support with agent and conductor roles.
  • Added configurable model registry integration and optional LPX enablement through Helm.
  • Added validation and access controls for LPX resources and scheduling requests.

• Documentation

  • Added LPX deployment guidance, configuration references, and API documentation.

• Tests

  • Expanded Helm, admission, conversion, and end-to-end coverage for LPX workflows.

[andrewpaprotsky]

@sttts Addressed all five points in 4ad02ea4e: documented experimental stability, gated admission with unchanged-component allowances, conditional CRD installation and controller/watch registration, and early disabled status before workload writes. Existing resources and deletion remain supported. Added regression coverage; operator/envtest and Helm tests pass.

[sttts]

For docs and deploy.

[sttts]

/ok to test f90624b

[sttts]

/ok to test 3104d46

[dmitry-tokarev-nv]

Approving at 3104d46d29. The changes since my approval at cbe156469b meet the bar, and the four earlier findings are fixed. I found no new defect in the parts I read.

Open items: none.

What I ran again at 3104d46d29, and the result of each.

• Round 1, absolute buildId with a configured registry, internal/dynamo/lpx/model_registry.go:129. TestRegistryBuildURL passes 16 of 16. Without that rejection, exactly the 8 rows for absolute references fail.
• Round 1, the LPX checkpoint stanza. Admission now refuses it. My reply on that thread has the measurement.
• Round 1, the podcliquesets/finalizers grant. It is at deploy/helm/charts/platform/components/operator/files/role.yaml:202, with its marker at internal/controller/lpx/controller.go:83. The Grove owner at internal/controller/lpx/pipeline_request.go:179 is still the only owner outside nvidia.com.
• Round 2, CheckPCSGReady coverage. It is back to 100 percent, and the mutant that always reports ready fails 15 subtests. My reply on that thread has the numbers.
• CODEOWNERS. The build_codeowners.py --strict gate passes, and emit_codeowners.py writes a byte-identical file.

I ran the Go probes in the golang:1.26.6 image with envtest assets for Kubernetes 1.30.0.

The DGD CRD: the operator sends 45 percent of the 3 MiB request limit.

crd-apply sends json.Marshal(crd) as a server-side apply patch, at cmd/crd-apply/main.go:109. I repeated its unmarshal and marshal on each file.

tree	file	request body
main at 59ac36782c	3,019,275 bytes	3,058,534 bytes of YAML, 97.23%
3104d46d29	3,025,860 bytes	1,415,460 bytes of JSON, 45.00%

main still sends yaml.Marshal(crd), so this PR lowers the risk. TestCRDApplyInstallsGeneratedSchemas builds the production installer and applies these files to a local API server with its default limits. It passed at this head.

What I read, and what I did not read.

Since cbe156469b the branch has 86 commits of its own. They change 169 files, +12253/-12309. The rest came from 5 merges of main.

I read the full change in the 26 files that ordinary graphs without LPX run through. These files hold the Grove program, grove.go, graph.go, the DGD controller, and the watch predicates. They also hold the pod cache, admission, conversion, the API types, crd-apply, and the operator Helm template. I also read the new LPX gate, the restart rule in CEL and its tests, and Reconcile in internal/controller/lpx/controller.go.

I did not read line by line the rest of the LPX source, about 45 files with 976 new lines. I also did not read the 54 changed test files, the 23 fixtures, or the new internal/controller/lpx/AGENTS.md. LPX stays off by default, and TestSetupDynamoGraphDeploymentWithoutLPXCRDs starts the DGD controller without the LPX CRDs.

CI at this head: Pre Merge passed, and the PR workflow is still running.

Pre Merge run 36475332052 passed. Its operator job ran envtest for internal/controller, internal/webhook/validation, and cmd/crd-apply, and make check finished clean.

The PR workflow, run 36475337054, is still running. The PR diff at f90624ba40 is byte-identical to this head. There, Helm Chart Tests, Operator, Operator Integration, and the vLLM and SGLang snapshot deploy tests passed. The red jobs there do not come from this diff. TRTLLM Snapshot Deploy Test also fails in all 10 post-merge runs on main since 2026-09-24 that ran it. The three GPU test jobs stopped mid-test, two of them at the same second, and this PR changes no backend code.

[dmitry-tokarev-nv]

Approving at f6d6c3e745: the PR still meets the bar. After my approval at 3104d46d29, the only change is this commit, which touches the TRT-LLM checkpoint test and its two CI jobs.

Open items:

• TRTLLM Snapshot Deploy Test in run 36585260018 has not finished. It is the evidence that the agent preload fixes the restore.
• P3: the test value replaces the image's own LD_PRELOAD. tests/deploy/test_dynamocheckpoint.py:186.
• P3: no comment gives the reason that the worker and the agent need the host driver, or the removal condition for the step. .github/workflows/pr.yaml:2057.

Why the agent step matters: at 8c74b2842f, the worker preload alone left the restore hanging.

head and run	worker LD_PRELOAD	snapshot agent	checkpoint	restored worker
3104d46d29, run 36557305687	image default	chart default	did not finish in 12 minutes	not reached
8c74b2842f, run 36571980780, attempt 1	/usr/$LIB/libcuda.so.1	chart default	finished in 70 s	not ready at the 900 s deployment timeout
8c74b2842f, run 36571980780, attempt 2	/usr/$LIB/libcuda.so.1	chart default	finished in 68 s	not ready at the 900 s deployment timeout
f0f8b69, run 36546302715	host libcuda.so.595.58.03 and the two image entries	LD_PRELOAD of the host libcuda.so.595.58.03	finished	ready, and the test passed in 456 s

At 8c74b2842f, the agent log ends at CRIU post-restore: process restored in both attempts. Your driver probe in run 36534810617 shows the same pattern. The helper615_target595_restore case timed out at restore, and matched595 passed all four steps.

This head adds the agent preload. f0f8b69 also put the host library directory ahead of the compat directory in the worker LD_LIBRARY_PATH, and this head does not. The open CI job shows whether the restore needs that too.

What else I checked in f6d6c3e745.

• The commit changes a test and two CI jobs. It changes no operator code, Helm chart, or image, so it cannot hide a missing library in a shipped image.

• The new step runs only in the TRTLLM Snapshot Deploy Test job of pr.yaml and post-merge-ci.yml. No other workflow runs this test. Both files parse, and the step is the same in both.

• In the rc27 image, /usr/$LIB/libcuda.so.1 resolves to /usr/lib/x86_64-linux-gnu/libcuda.so.1. The NVIDIA container toolkit mounts that file from the host driver.

• If that file is absent, the loader ignores the entry and finds libcuda as it does without the preload. It prints this line:

  ERROR: ld.so: object '/usr/$LIB/libdoesnotexist.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
  

• The host driver is the right one. The GPU nodes run driver 595.58.03, and the rc27 image carries the CUDA 13.4 compat driver libcuda.so.615.65.02. Your probe shows that the helper and the process need the same driver.

• I did not run the deploy test, because it needs the GPU cluster. I read the job logs and the pod-log artifacts of the CI runs.

• The rest of the PR did not change. My approval at 3104d46d29 lists what I read and what I did not read, and that list still holds.

CI at this head.

Pre Merge run 36585253526 passed. The PR workflow, run 36585260018, has not finished.

[dmitry-tokarev-nv]

Approving at 4b37faf948: the PR still meets the bar. The only change since my approval at f6d6c3e745 is this commit. It fixes both of my P3 items from that review, so I resolved those two threads.

Open item:

• P3: with this preload, a PTX load segfaults in a worker process that starts after import tensorrt_llm. tests/deploy/test_dynamocheckpoint.py:192.

What changed, and CI at this head: the TRT-LLM checkpoint and the restore both finished.

The commit keeps the two image preloads in the worker LD_PRELOAD and adds the same comment at the three places. The rest of the PR is the same as at f6d6c3e745. My approval at 3104d46d29 lists what I read and what I did not read, and that list still holds.

In run 36596785970, TRTLLM Snapshot Deploy Test passed. The checkpoint finished, the restored worker pod has the nvidia.com/Restored condition with reason RestoreSucceeded, and inference on the restored worker passed. The same job passed at f6d6c3e745 in run 36585260018. Neither run changes the worker LD_LIBRARY_PATH, so the restore does not need the order that my last review left open. The new P3 asks for that order for another reason, the PTX JIT compiler. Other jobs in run 36596785970 have not finished.

[dmitry-tokarev-nv]

Approving at 3546a4316e: the PR still meets the bar, and none of my findings is open. The new worker LD_LIBRARY_PATH fixes my P3 about the PTX JIT compiler, so I resolved that thread.

TRTLLM Snapshot Deploy Test did not run at this head. It is the CI job that checks checkpoint and restore with the new library path.

What changed, what I measured, and why the snapshot job did not run.

The only change since my approval at 4b37faf948 is the worker LD_LIBRARY_PATH in tests/deploy/test_dynamocheckpoint.py. The new commit replaces the old one on the same parent, 3104d46d29. The two heads differ only by the 11 lines added at tests/deploy/test_dynamocheckpoint.py:197-207. The change follows the ask in my P3, so this part of the review checks my own advice.

The new value is the image LD_LIBRARY_PATH with /usr/$LIB added after /opt/dynamo/mpi/lib. On a GPU host with the rc27 runtime image, a child process that loads PTX after import tensorrt_llm passes with the new value. It segfaults with the old value. The new order changes only which driver files load. My reply on the thread has the measurements.

In run 36610307797, three image builds stopped at the 1-hour job limit: frontend / Build multi-arch cpu, planner / Build multi-arch cpu and dynamo-runtime / image / Build multi-arch cuda13.0. The three Snapshot Deploy Tests need the frontend image, so GitHub skipped them. At 4b37faf948, the same three builds took 4 to 7 minutes, and TRTLLM Snapshot Deploy Test passed in run 36596785970.

My approval at 3104d46d29 lists what I read and what I did not read, and that list still holds.

[dmitry-tokarev-nv]

Approving at cf24cb07cb: the PR still meets the bar, and none of my findings is open.

The new commit merges main. It differs from the automatic merge only in the two conflicts, .github/codeowners/areas.yaml and CODEOWNERS. Both take the main side from #15244, and the generated CODEOWNERS matches its source.

TRTLLM Snapshot Deploy Test has not finished at this head yet. It is part of PR run 36630453955, and it is the CI job that tests checkpoint and restore with the new library path.

The merge adds no hand edit outside the two conflicts. Apart from the - lpx line and its generated row, the PR lines did not change.

The merge drops the - lpx line that the PR added to the owner list of the CODEOWNERS rule. The union rule for aggregate lists does not apply to it. #15244 removed that list of every area, so the line has no list to join. The lpx area and its three shared rows are unchanged. On the merge, emit_codeowners.py writes a byte-identical CODEOWNERS, and build_codeowners.py --strict covers 6077 of 6077 files. Each gate fails on a planted edit.

Against the new base, the added and removed lines of the PR are the same as before in every file except these two. Three files merged without a conflict and have changes from both sides: shared_v1beta1.go and its envtest file from #14758, and .github/workflows/post-merge-ci.yml. The #14758 replica rule is inside the ComponentTypeEPP branch, and IsLPX() is true only for ComponentTypeLPX, so the two changes do not meet. The merged post-merge-ci.yml parses, and the host CUDA library step is still in the TRTLLM Snapshot Deploy Test job.

The DGD CRD is the same file as at 3546a4316e. It is 3,025,860 bytes, 96.19% of the 3 MiB request limit. As compact JSON, it is 1,413,575 bytes, 44.94%.

Pre Merge ran on a merge ref with the same tree as this head. There, make check found no uncommitted changes in the generated operator files. The internal/webhook/validation envtests ran with KUBEBUILDER_ASSETS set. The three image builds that stopped at the 1-hour limit at 3546a4316e passed at this head.

My approval at 3104d46d29 lists what I read and what I did not read, and that list still holds.

[dmitry-tokarev-nv]

Approving at 5143224563: the PR still meets the bar, and none of my findings is open. The new commit is an exact revert of 3546a4316e, so the PR no longer changes the three files of that commit.

The commit message says:

Restore default worker and Snapshot-agent CUDA library loading after the CI host driver upgrade.

The GPU nodes of the CI snapshot cluster still report driver 595.58.03. For this reason, expect TRTLLM Snapshot Deploy Test to fail at this head, as it does on main. The test and the two workflows now match main at the merge base, so that failure is not a defect of this PR.

The revert is exact, and the GPU nodes of the snapshot cluster still report driver 595.58.03.

I applied the reverse of 3546a4316e to cf24cb07cb. The result is tree 5fac1f2723, which is the tree of this head. At this head, pr.yaml, post-merge-ci.yml and test_dynamocheckpoint.py are the same blobs as on main at the merge base 0ce693b3b0.

Nothing in the PR still uses what the revert removed. A search of the whole tree for the removed strings finds the same lines at this head as at the merge base. The strings include LD_PRELOAD, libcuda.so.1, /usr/$LIB, snapshot/issues/412 and the name of the removed step.

This head merges with main at 8c226c99bb without a conflict. The codeowners job ran on that merge and passed, with 6080 of 6080 files owned.

The job logs do not print the driver version. A failed checkpoint job saves the node list in its pod-logs-* artifact, as cluster-info/nodes.json. In both failed jobs below, all six GPU nodes have the label nvidia.com/cuda.driver-version.full=595.58.03. Five nodes have nvidia.com/gpu-driver-upgrade-state=upgrade-required, and one has upgrade-failed. In both jobs, the snapshot agent starts the checkpoint, finds the GPU, and then logs nothing until the test times out after 900 s.

job	commit	contains 3546a4316e	result
109617704854	main at 0ce693b3b0	no	failed at 2026-09-29 21:18Z
109630183407	PR at cf24cb07cb	yes	passed at 2026-09-29 21:44Z, in 476.52 s
109756396986	main at 3f46e3d3c5	no	failed at 2026-09-30 05:57Z

The last job started at 05:40Z, after the push of this commit at 05:31Z. TRTLLM Snapshot Deploy Test has not finished at this head yet. When that job fails, the required deploy-status-check also fails.

My approval at 3104d46d29 lists what I read and what I did not read, and that list still holds.