fix(test): mount model-cache PVC on v1beta1 pod templates - #12419
Closed
galletas1712 wants to merge 2 commits into
Closed
fix(test): mount model-cache PVC on v1beta1 pod templates#12419galletas1712 wants to merge 2 commits into
galletas1712 wants to merge 2 commits into
Conversation
Checkpoint deploy CI was downloading Qwen via ModelExpress/HF on every run while regular deploy tests already used the Azure model-cache PVC. Wire the same provision + mount path through, and honor --model-cache-pvc in the checkpoint test harness (including skipping TRTLLM's container HF_HOME override so it cannot shadow the shared cache). Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
DynamoCheckpoint CI uses v1beta1 DGDs. The shared model-cache helper only wrote legacy volumeMounts, so checkpoint source pods never saw HF_HOME on the Azure cache and fell back to HF downloads. Mount the PVC onto native podTemplate volumes/volumeMounts (and non-main checkpoint targets) so the cache is inherited by operator-generated checkpoint pods. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
Contributor
Author
Contributor
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (4)
WalkthroughShared model-cache server, path, and PVC settings are now propagated through DynamoCheckpoint CI workflows and deployment tests. Deployment utilities support schema-specific PVC mounting, checkpoint target containers, idempotence, and validation for v1beta1 while retaining v1alpha1 behavior. ChangesShared model-cache checkpoint flow
Estimated code review effort: 3 (Moderate) | ~20 minutes Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mount_model_cache_pvconly wrote legacyvolumeMounts, so checkpoint source pods never inherited the Azure model cache.podTemplatevolumes/volumeMounts (including non-main checkpoint target containers) and setHF_HOMEaccordingly.Depends on #12400 (stacked). Does not include the vLLM
os._exitcapture change — that stays in #12184.Validation
managed_deploymentv1beta1 mount pathSummary by CodeRabbit
New Features
Bug Fixes
Tests