fix(docker): drop --ldconfig arg from enroot NVIDIA hook - #2781
Merged
Conversation
nvidia-container-cli fails to inject GPUs into nested enroot sandboxes with: nvidia-container-cli: ldcache error: process /usr/sbin/ldconfig.real failed with error code: 1 [ERROR] /etc/enroot/hooks.d/98-nvidia.sh exited with return code 1 This reproduces on any container doing nested enroot GPU passthrough under Slurm/pyxis, not just Gym's -- confirmed independently by NVIDIA's computelab team (who hit it in their own Pyxis CI) and reproduced live on cw_dfw with a completely unrelated test image. Root cause: nvidia-container-cli's --ldconfig=@... flag makes it re-exec the target container's ldconfig to regenerate its ld.so.cache, which fails under certain Slurm/pyxis GPU cgroup/device-restriction configurations. Dropping the flag makes it reuse the existing ld.so.cache instead, sidestepping the failure -- the same workaround computelab validated for their own Pyxis test breakage. Verified live on cw_dfw: nvidia-smi now succeeds inside a nested enroot sandbox with this patch applied (H100 GPU visible), where it previously failed at the hook step. Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Contributor
Author
|
/ok to test be6395b |
thomasdhc
approved these changes
Aug 26, 2026
Contributor
Author
|
/ok to test be6395b |
kajalj22
added a commit
that referenced
this pull request
Sep 2, 2026
## Summary - Patches `/etc/enroot/hooks.d/98-nvidia.sh` at build time to drop the `--ldconfig=@...` arg passed to `nvidia-container-cli` - Fixes nested enroot GPU sandboxes failing with: ``` nvidia-container-cli: ldcache error: process /usr/sbin/ldconfig.real failed with error code: 1 [ERROR] /etc/enroot/hooks.d/98-nvidia.sh exited with return code 1 ``` ## Context Nested enroot GPU passthrough under Slurm/pyxis fails under some GPU cgroup configurations because `nvidia-container-cli`'s `--ldconfig=@...` flag makes it re-exec the sandbox's `ldconfig` to regenerate `ld.so.cache`, and that regeneration step crashes. Dropping the flag makes it reuse the existing cache instead, which avoids the failure. Reproduces with any container doing nested enroot GPU passthrough on affected clusters, not something specific to Gym. ## Test plan - [x] Verified live on a cluster hitting this failure: with this patch applied to a running container's hook script, `nvidia-smi` succeeds inside a nested `enroot` sandbox (GPU visible) — previously failed at the hook step with the exact error above - [ ] Verify a full image build applies the patch correctly (`grep` assertion in the Dockerfile fails the build loudly if the hook content changes upstream and the sed no longer matches) - [ ] Re-run the real `gym env start` + `mini_swe_agent_2` + enroot repro from the original bug report against the rebuilt image --------- Signed-off-by: Kajal Jain <kajalj@nvidia.com>
kajalj22
added a commit
that referenced
this pull request
Sep 4, 2026
## Summary - Patches `/etc/enroot/hooks.d/98-nvidia.sh` at build time to drop the `--ldconfig=@...` arg passed to `nvidia-container-cli` - Fixes nested enroot GPU sandboxes failing with: ``` nvidia-container-cli: ldcache error: process /usr/sbin/ldconfig.real failed with error code: 1 [ERROR] /etc/enroot/hooks.d/98-nvidia.sh exited with return code 1 ``` ## Context Nested enroot GPU passthrough under Slurm/pyxis fails under some GPU cgroup configurations because `nvidia-container-cli`'s `--ldconfig=@...` flag makes it re-exec the sandbox's `ldconfig` to regenerate `ld.so.cache`, and that regeneration step crashes. Dropping the flag makes it reuse the existing cache instead, which avoids the failure. Reproduces with any container doing nested enroot GPU passthrough on affected clusters, not something specific to Gym. ## Test plan - [x] Verified live on a cluster hitting this failure: with this patch applied to a running container's hook script, `nvidia-smi` succeeds inside a nested `enroot` sandbox (GPU visible) — previously failed at the hook step with the exact error above - [ ] Verify a full image build applies the patch correctly (`grep` assertion in the Dockerfile fails the build loudly if the hook content changes upstream and the sed no longer matches) - [ ] Re-run the real `gym env start` + `mini_swe_agent_2` + enroot repro from the original bug report against the rebuilt image --------- Signed-off-by: Kajal Jain <kajalj@nvidia.com>
kajalj22
added a commit
that referenced
this pull request
Sep 5, 2026
#2983) ## Summary - Cherry-pick of #2781 from `r0.5.1` into `main`. **Stack position 5 of 5** — all prior positions merged. This is the last one, rebased directly onto `main`. - Patches `/etc/enroot/hooks.d/98-nvidia.sh` at build time to drop the `--ldconfig=@...` arg passed to `nvidia-container-cli`, fixing nested enroot GPU sandboxes failing at the hook step under some GPU cgroup configurations. Signed-off-by: Kajal Jain <kajalj@nvidia.com>
jkyi-nvidia
pushed a commit
that referenced
this pull request
Sep 11, 2026
#2983) ## Summary - Cherry-pick of #2781 from `r0.5.1` into `main`. **Stack position 5 of 5** — all prior positions merged. This is the last one, rebased directly onto `main`. - Patches `/etc/enroot/hooks.d/98-nvidia.sh` at build time to drop the `--ldconfig=@...` arg passed to `nvidia-container-cli`, fixing nested enroot GPU sandboxes failing at the hook step under some GPU cgroup configurations. Signed-off-by: Kajal Jain <kajalj@nvidia.com> Signed-off-by: Junkeun Yi <jkyi@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/etc/enroot/hooks.d/98-nvidia.shat build time to drop the--ldconfig=@...arg passed tonvidia-container-cliContext
Nested enroot GPU passthrough under Slurm/pyxis fails under some GPU cgroup configurations because
nvidia-container-cli's--ldconfig=@...flag makes it re-exec the sandbox'sldconfigto regenerateld.so.cache, and that regeneration step crashes. Dropping the flag makes it reuse the existing cache instead, which avoids the failure. Reproduces with any container doing nested enroot GPU passthrough on affected clusters, not something specific to Gym.Test plan
nvidia-smisucceeds inside a nestedenrootsandbox (GPU visible) — previously failed at the hook step with the exact error abovegrepassertion in the Dockerfile fails the build loudly if the hook content changes upstream and the sed no longer matches)gym env start+mini_swe_agent_2+ enroot repro from the original bug report against the rebuilt image