Skip to content

fix(container): make vLLM FlashInfer cubin dirs writable - #12770

Merged
bewestphal merged 3 commits into
ai-dynamo:mainfrom
bewestphal:codex/vllm-flashinfer-cubin-permissions
Aug 10, 2026
Merged

fix(container): make vLLM FlashInfer cubin dirs writable#12770
bewestphal merged 3 commits into
ai-dynamo:mainfrom
bewestphal:codex/vllm-flashinfer-cubin-permissions

Conversation

@bewestphal

@bewestphal bewestphal commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • grant group-write permission to directories under flashinfer_cubin/cubins in vLLM runtime images
  • match the existing SGLang runtime behavior for arbitrary OpenShift UIDs in group 0
  • allow FlashInfer TRT-LLM MoE initialization to create its package-local cubin symlinks

Without this, a non-root vLLM worker loading a ModelOpt FP8 MoE checkpoint fails during profile_run() with:

PermissionError: [Errno 13] Permission denied: /usr/local/lib/python3.12/dist-packages/flashinfer_cubin/cubins/flashinfer

Validation

  • rendered the CUDA 13.0 vLLM runtime Dockerfile and verified the permission step
  • git diff --check
  • pre-commit run --files container/templates/vllm_runtime.Dockerfile
  • Ran a e2e DGD deploy before/after that previously required cubin hack which now works without it.

Open in Devin Review

Summary by CodeRabbit

  • Enhancements
    • Improved runtime image setup by ensuring FlashInfer cache directories are accessible when available.

Signed-off-by: Brian Westphal <bwestphal@nvidia.com>
@bewestphal
bewestphal requested a review from a team as a code owner August 6, 2026 20:36

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Open in Devin Review

Comment thread container/templates/vllm_runtime.Dockerfile
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The runtime Dockerfile now conditionally locates the FlashInfer cubin cache and grants group access to its directories. The permission step is skipped when the cache is absent.

Changes

FlashInfer cache permissions

Layer / File(s) Summary
Configure runtime cache permissions
container/templates/vllm_runtime.Dockerfile
The runtime image locates the FlashInfer cubin cache and applies group read, write, and execute permissions to its directories when present.

Estimated code review effort: 2 (Simple) | ~5 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description explains the change and validation, but it omits the required Related Issues section and reviewer starting point. Add the required Related Issues section and identify the file or code area where reviewers should start.
✅ Passed checks (4 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main change: granting write access to vLLM FlashInfer cubin directories.

Comment @coderabbitai help to get the list of available commands.

@bewestphal
bewestphal marked this pull request as draft August 6, 2026 20:55
@nv-anants
nv-anants temporarily deployed to external_collaborator August 7, 2026 16:59 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 3d0ae32

@datadog-official

datadog-official Bot commented Aug 7, 2026

Copy link
Copy Markdown

Pipelines

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 36.73% (-5.58%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 6237556 | Docs | Datadog PR Page | Give us feedback!

@bewestphal
bewestphal marked this pull request as ready for review August 10, 2026 17:41
@bewestphal
bewestphal temporarily deployed to external_collaborator August 10, 2026 17:44 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 6237556

@bewestphal
bewestphal enabled auto-merge (squash) August 10, 2026 20:44
@bewestphal
bewestphal merged commit d263d3d into ai-dynamo:main Aug 10, 2026
224 of 251 checks passed
pvijayakrish pushed a commit that referenced this pull request Aug 11, 2026
…2976)

Signed-off-by: Brian Westphal <bwestphal@nvidia.com>
Co-authored-by: Brian Westphal <14085065+bewestphal@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants