Skip to content

ci(checkpoint): cherry-pick mount shared model-cache PVC in DynamoCheckpoint tests (#12400) #12593

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pvijayakrish merged 1 commit into from
Aug 4, 2026
Merged

ci(checkpoint): cherry-pick mount shared model-cache PVC in DynamoCheckpoint tests (#12400) #12593

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions .github/workflows/pr.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1243,6 +1243,9 @@ jobs:
dockerhub_password: ${{ secrets.DOCKERHUB_ACCESS_TOKEN }}
checkpoint_enabled: 'true'
checkpoint_storage_size: 64Gi
# Same shared model cache as regular deploy tests; empty vars -> HF download.
model_cache_server: ${{ vars.AZURE_MODEL_CACHE_SERVER }}
model_cache_path: ${{ vars.AZURE_MODEL_CACHE_PATH }}

deploy-operator-checkpoint-sglang:
name: SGLang DynamoCheckpoint Operator Setup
Expand Down Expand Up @@ -1274,6 +1277,9 @@ jobs:
dockerhub_password: ${{ secrets.DOCKERHUB_ACCESS_TOKEN }}
checkpoint_enabled: 'true'
checkpoint_storage_size: 64Gi
# Same shared model cache as regular deploy tests; empty vars -> HF download.
model_cache_server: ${{ vars.AZURE_MODEL_CACHE_SERVER }}
model_cache_path: ${{ vars.AZURE_MODEL_CACHE_PATH }}

deploy-operator-checkpoint-trtllm:
name: TRTLLM DynamoCheckpoint Operator Setup
Expand Down Expand Up @@ -1305,6 +1311,9 @@ jobs:
dockerhub_password: ${{ secrets.DOCKERHUB_ACCESS_TOKEN }}
checkpoint_enabled: 'true'
checkpoint_storage_size: 64Gi
# Same shared model cache as regular deploy tests; empty vars -> HF download.
model_cache_server: ${{ vars.AZURE_MODEL_CACHE_SERVER }}
model_cache_path: ${{ vars.AZURE_MODEL_CACHE_PATH }}

deploy-snapshot-agent-checkpoint-vllm:
name: vLLM DynamoCheckpoint Snapshot Agent Setup
Expand Down Expand Up @@ -1341,6 +1350,9 @@ jobs:
dockerhub_password: ${{ secrets.DOCKERHUB_ACCESS_TOKEN }}
checkpoint_enabled: 'true'
checkpoint_storage_size: 64Gi
# Same shared model cache as regular deploy tests; empty vars -> HF download.
model_cache_server: ${{ vars.AZURE_MODEL_CACHE_SERVER }}
model_cache_path: ${{ vars.AZURE_MODEL_CACHE_PATH }}
- name: Connect to vCluster
id: connect-vcluster
uses: ./.github/actions/connect-vcluster
Expand Down Expand Up @@ -1389,6 +1401,9 @@ jobs:
dockerhub_password: ${{ secrets.DOCKERHUB_ACCESS_TOKEN }}
checkpoint_enabled: 'true'
checkpoint_storage_size: 64Gi
# Same shared model cache as regular deploy tests; empty vars -> HF download.
model_cache_server: ${{ vars.AZURE_MODEL_CACHE_SERVER }}
model_cache_path: ${{ vars.AZURE_MODEL_CACHE_PATH }}
- name: Connect to vCluster
id: connect-vcluster
uses: ./.github/actions/connect-vcluster
Expand Down Expand Up @@ -1437,6 +1452,9 @@ jobs:
dockerhub_password: ${{ secrets.DOCKERHUB_ACCESS_TOKEN }}
checkpoint_enabled: 'true'
checkpoint_storage_size: 64Gi
# Same shared model cache as regular deploy tests; empty vars -> HF download.
model_cache_server: ${{ vars.AZURE_MODEL_CACHE_SERVER }}
model_cache_path: ${{ vars.AZURE_MODEL_CACHE_PATH }}
- name: Connect to vCluster
id: connect-vcluster
uses: ./.github/actions/connect-vcluster
Expand Down Expand Up @@ -1486,6 +1504,9 @@ jobs:
dockerhub_password: ${{ secrets.DOCKERHUB_ACCESS_TOKEN }}
checkpoint_enabled: 'true'
checkpoint_storage_size: 64Gi
# Same shared model cache as regular deploy tests; empty vars -> HF download.
model_cache_server: ${{ vars.AZURE_MODEL_CACHE_SERVER }}
model_cache_path: ${{ vars.AZURE_MODEL_CACHE_PATH }}
- name: Connect to vCluster
id: connect-vcluster
uses: ./.github/actions/connect-vcluster
Expand All @@ -1511,6 +1532,9 @@ jobs:
image: ${{ secrets.AZURE_ACR_HOSTNAME }}/ai-dynamo/dynamo:${{ needs.snapshot-placeholder-vllm.outputs.image_tag }}
test_name: checkpoint_dgd_restore_vllm
test_file: tests/deploy/test_dynamocheckpoint.py
# Mount the shared model cache only when both endpoint vars are set
# (matches the PV/PVC creation gate); otherwise workers download from HF.
model_cache_pvc: ${{ vars.AZURE_MODEL_CACHE_SERVER != '' && vars.AZURE_MODEL_CACHE_PATH != '' && 'model-cache' || '' }}
Comment on lines +1535 to +1537

@devin-ai-integration devin-ai-integration Bot Aug 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Shared cache mount is read-only NFS while HF_HOME points at it

mount_model_cache_pvc (tests/utils/managed_deployment.py:494-512) sets HF_HOME to the mount path, and the PV created by .github/actions/setup-dynamo-operator/action.yml:337-377 is an NFS export shared across CI runs. If the HuggingFace client needs to write locks/metadata under HF_HOME (typical when a model is not fully present in the cache), checkpoint pods will attempt writes into the shared export. Worth confirming the export is writable or that the cache always contains Qwen3-0.6B, otherwise the checkpoint jobs may fail differently than the regular deploy tests, which have longer soak history with this mount.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

extra_pytest_args: >-
-m dynamocheckpoint
--checkpoint-backend=vllm
Expand Down Expand Up @@ -1552,6 +1576,9 @@ jobs:
dockerhub_password: ${{ secrets.DOCKERHUB_ACCESS_TOKEN }}
checkpoint_enabled: 'true'
checkpoint_storage_size: 64Gi
# Same shared model cache as regular deploy tests; empty vars -> HF download.
model_cache_server: ${{ vars.AZURE_MODEL_CACHE_SERVER }}
model_cache_path: ${{ vars.AZURE_MODEL_CACHE_PATH }}
- name: Connect to vCluster
id: connect-vcluster
uses: ./.github/actions/connect-vcluster
Expand All @@ -1577,6 +1604,9 @@ jobs:
image: ${{ secrets.AZURE_ACR_HOSTNAME }}/ai-dynamo/dynamo:${{ needs.snapshot-placeholder-sglang.outputs.image_tag }}
test_name: checkpoint_dgd_restore_sglang
test_file: tests/deploy/test_dynamocheckpoint.py
# Mount the shared model cache only when both endpoint vars are set
# (matches the PV/PVC creation gate); otherwise workers download from HF.
model_cache_pvc: ${{ vars.AZURE_MODEL_CACHE_SERVER != '' && vars.AZURE_MODEL_CACHE_PATH != '' && 'model-cache' || '' }}
extra_pytest_args: >-
-m dynamocheckpoint
--checkpoint-backend=sglang
Expand Down Expand Up @@ -1618,6 +1648,9 @@ jobs:
dockerhub_password: ${{ secrets.DOCKERHUB_ACCESS_TOKEN }}
checkpoint_enabled: 'true'
checkpoint_storage_size: 64Gi
# Same shared model cache as regular deploy tests; empty vars -> HF download.
model_cache_server: ${{ vars.AZURE_MODEL_CACHE_SERVER }}
model_cache_path: ${{ vars.AZURE_MODEL_CACHE_PATH }}
- name: Connect to vCluster
id: connect-vcluster
uses: ./.github/actions/connect-vcluster
Expand All @@ -1643,6 +1676,9 @@ jobs:
image: ${{ secrets.AZURE_ACR_HOSTNAME }}/ai-dynamo/dynamo:${{ needs.snapshot-placeholder-trtllm.outputs.image_tag }}
test_name: checkpoint_dgd_restore_trtllm
test_file: tests/deploy/test_dynamocheckpoint.py
# Mount the shared model cache only when both endpoint vars are set
# (matches the PV/PVC creation gate); otherwise workers download from HF.
model_cache_pvc: ${{ vars.AZURE_MODEL_CACHE_SERVER != '' && vars.AZURE_MODEL_CACHE_PATH != '' && 'model-cache' || '' }}
extra_pytest_args: >-
-m dynamocheckpoint
--checkpoint-backend=trtllm
Expand Down
21 changes: 18 additions & 3 deletions tests/deploy/test_dynamocheckpoint.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,9 +168,10 @@ class CheckpointBackendConfig:
"--free-gpu-memory-fraction",
"0.10",
),
# Keep the raw DGD PVC-free: the checkpoint operator mounts
# snapshot-pvc at /checkpoints for checkpoint/restore pods, so HF_HOME
# there preserves model files across restore without a model-cache PVC.
# UCX_TLS is always set. HF_HOME defaults to the snapshot PVC so restore
# pods keep weights without a model-cache PVC; when CI passes
# --model-cache-pvc, _new_checkpoint_spec skips this HF_HOME so the
# shared cache mount can own it (same as regular deploy tests).
env=(("UCX_TLS", "tcp,self"), ("HF_HOME", TRTLLM_HF_HOME)),
# Match the base TRTLLM snapshot recipe and avoid cold-worker/restore
# rollout overlap during initial DGD startup.
Expand Down Expand Up @@ -233,6 +234,9 @@ def _new_checkpoint_spec(
namespace: str,
image: str,
frontend_image: str,
*,
model_cache_pvc: str | None = None,
model_cache_mount: str | None = None,
) -> DeploymentSpec:
spec_path = Path(_get_workspace_dir()).joinpath(*backend.manifest)
deployment_spec = DeploymentSpec(str(spec_path))
Expand Down Expand Up @@ -269,6 +273,10 @@ def _new_checkpoint_spec(
if backend.env:
env = container.setdefault("env", [])
for name, value in backend.env:
# Container HF_HOME would shadow the deployment-level value that
# mount_model_cache_pvc sets; skip it when the shared cache is used.
if name == "HF_HOME" and model_cache_pvc:
continue
for item in env:
if item.get("name") == name:
item["value"] = value
Expand All @@ -281,6 +289,11 @@ def _new_checkpoint_spec(
checkpoint["targetContainerName"] = backend.target_container
if backend.checkpoint_startup_policy is not None:
checkpoint["startupPolicy"] = backend.checkpoint_startup_policy

if model_cache_pvc:
mount = model_cache_mount or "/models"
deployment_spec.mount_model_cache_pvc(model_cache_pvc, mount)
Comment on lines +293 to +295

@devin-ai-integration devin-ai-integration Bot Aug 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Shared model cache is never actually attached in checkpoint tests, and TensorRT-LLM loses its model-storage setting

The shared model cache is attached using an older manifest format (mount_model_cache_pvc at tests/deploy/test_dynamocheckpoint.py:295) that the newer deployment format used by these tests does not accept, so the cache is silently dropped while the TensorRT-LLM model-storage setting is also removed.
Impact: Checkpoint CI jobs still download models from HuggingFace and the TensorRT-LLM restore test can lose its model files between checkpoint and restore, causing slow or failing runs.

Schema mismatch: mount_model_cache_pvc emits v1alpha1-only fields into a v1beta1 DGD

_new_checkpoint_spec loads examples/backends/<fw>/deploy/v1beta1/agg.yaml (see tests/deploy/test_dynamocheckpoint.py:237 and backend.manifest), so DeploymentSpec._schema is v1beta1.

DeploymentSpec.mount_model_cache_pvc (tests/utils/managed_deployment.py:494-512) writes:

  • spec.pvcs — exists only in the v1alpha1 CRD (deploy/operator/config/crd/bases/nvidia.com_dynamographdeployments.yaml:309, inside the v1alpha1 section); the v1beta1 section has no pvcs.
  • spec.envs — v1beta1 uses spec.env.
  • per-service volumeMounts: [{name, mountPoint}] via ServiceSpec.add_pvc_mount (tests/utils/managed_deployment.py:184-188); v1beta1 explicitly replaced per-component envs/volumeMounts with podTemplate (see the v1beta1 component description in the CRD).

Unknown fields are pruned by the CRD structural schema, so the PVC is never referenced or mounted. Meanwhile the new skip at tests/deploy/test_dynamocheckpoint.py:278-279 removes the TRTLLM HF_HOME=<snapshot PVC> container env whenever --model-cache-pvc is passed, relying on the deployment-level HF_HOME that is also pruned — leaving the pod with no HF_HOME at all, which the previous comment says is required to preserve model files across restore.

The pre-existing users of this helper (tests/deploy/conftest.py:363-367) operate on examples/backends/*/deploy/*.yaml, which are v1alpha1, which is why the helper works there.

Prompt for agents 
DeploymentSpec.mount_model_cache_pvc in tests/utils/managed_deployment.py only emits v1alpha1 fields (spec.pvcs, spec.envs, per-service volumeMounts with mountPoint). The DynamoCheckpoint tests build their spec from examples/backends/<framework>/deploy/v1beta1/agg.yaml, so the schema is v1beta1, where those fields do not exist (spec.pvcs is absent, envs/volumeMounts were replaced by podTemplate). As a result the shared model-cache PVC is pruned by the CRD and never mounted, and because _new_checkpoint_spec now skips the TRTLLM container-level HF_HOME whenever --model-cache-pvc is set, the TRTLLM pods end up with no HF_HOME at all, losing the snapshot-PVC model preservation the previous code relied on. Make mount_model_cache_pvc schema-aware: for v1beta1, add a persistentVolumeClaim volume to each component's podTemplate.spec.volumes, a corresponding volumeMounts entry (mountPath) on the main container, and set HF_HOME either via spec.env or per-container env. Alternatively, gate the checkpoint test's cache mounting on the schema and keep the TRTLLM HF_HOME when the mount cannot be applied.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.


return deployment_spec


Expand Down Expand Up @@ -594,6 +607,8 @@ async def test_dgd_checkpoint_restore_deploy(
namespace=namespace,
image=image,
frontend_image=frontend_image,
model_cache_pvc=request.config.getoption("--model-cache-pvc") or None,
model_cache_mount=request.config.getoption("--model-cache-mount") or None,
)

async with ManagedDeployment(
Expand Down
Loading