[https://nvbugs/6115560][fix] catch OSError in config_file_lock for NFS compatibility by sara4dev · Pull Request #11960 · NVIDIA/TensorRT-LLM

sara4dev · 2026-03-05T20:18:32Z

Summary

config_file_lock() in tensorrt_llm/_torch/model_config.py crashes when HF_MODULES_CACHE resides on an NFS-mounted filesystem. On NFS, filelock operations can raise OSError with errno ENOLCK (No locks available) or ESTALE (Stale file handle) instead of PermissionError. Since the current exception handler only catches PermissionError and filelock.Timeout, these NFS-specific errors bypass the existing tempdir fallback and crash the process.

This is particularly impactful in multi-node GPU clusters where a shared NFS cache is standard practice — every pod that tries to load a model config concurrently hits this crash.

Changes

Add OSError to both exception handlers in config_file_lock():

Primary lock attempt (line 51): except (PermissionError, filelock.Timeout) → except (PermissionError, OSError, filelock.Timeout) — triggers the tempdir fallback for NFS errors
Tempdir fallback (line 66): except (PermissionError) → except (PermissionError, OSError) — handles the unlikely case where tempdir also fails

Since PermissionError is a subclass of OSError, catching OSError technically covers both, but keeping PermissionError explicit preserves the original intent and readability.

Root Cause

NFSv3 uses the Network Lock Manager (NLM) protocol for file locking, which is unreliable for cross-node flock()/fcntl() operations. When filelock.FileLock attempts to acquire a lock on an NFS path:

Lock acquisition can fail with OSError: [Errno 37] No locks available (ENOLCK)
Lock release can fail with OSError: [Errno 116] Stale file handle (ESTALE)

The existing fallback to /tmp (local ephemeral storage) is the correct behavior for this case — it just wasn't being triggered.

Reproduction

# On any NFS-mounted path, run from multiple nodes concurrently:
import filelock
lock = filelock.FileLock("/nfs-mount/test.lock")
with lock:  # Raises OSError: [Errno 37] No locks available
    pass

Workaround (for users on affected versions)

Set HF_MODULES_CACHE=/tmp/hf_modules as an environment variable to redirect the lock file to local storage.

Fixes #11958

Made with Cursor

Summary by CodeRabbit

Bug Fixes
- Enhanced file locking behavior to catch additional OS and permission-related errors. Added temporary directory-based fallback locking with timeout handling.

coderabbitai · 2026-03-05T20:22:53Z

📝 Walkthrough

Walkthrough

Expanded exception handling in the config_file_lock() function to catch OSError alongside existing PermissionError and filelock.Timeout exceptions. Added a fallback lock mechanism using a temporary directory-based lock file with timeout handling and logging.

Changes

Cohort / File(s)	Summary
Lock Error Handling Improvements `tensorrt_llm/_torch/model_config.py`	Extended exception handling to catch `OSError` (NFS lock failures). Added fallback mechanism: on primary lock failure, create and use a temporary `_remote_code.lock` file via `FileLock`. Implemented timeout handling with warning log if tempdir lock cannot be acquired. Updated comments to document OS/permission issues and NFS locking failures as fallback triggers.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description follows the template structure with Summary, Changes, Root Cause, Reproduction, and Workaround sections. It provides sufficient context about the issue, the NFS-specific problem, and the solution.
Linked Issues check	✅ Passed	The PR modifications directly address issue `#11958` by catching OSError in both the primary lock attempt and tempdir fallback path, enabling the existing tempdir fallback for NFS-specific errors as required.
Out of Scope Changes check	✅ Passed	All changes in tensorrt_llm/_torch/model_config.py are directly related to fixing the NFS compatibility issue in config_file_lock. No out-of-scope modifications detected.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Title check	✅ Passed	The title clearly and specifically identifies the main change: catching OSError in config_file_lock for NFS compatibility, which directly matches the primary objective of the pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tensorrt_llm/_torch/model_config.py`:
- Around line 51-54: The contextmanager config_file_lock currently contains
multiple yield points and broad try/excepts that can catch exceptions raised by
the with-block; refactor it to a single-yield pattern by performing lock
acquisition and any related filesystem error handling before yielding, then
yield exactly once to the caller, and perform lock release/cleanup in a finally
block after the yield; ensure only the lock acquisition logic (e.g.,
filelock.FileLock.acquire and fallback to tempfile logic) is wrapped in
try/except for PermissionError/OSError/filelock.Timeout, and remove any
try/except that spans the yield so exceptions from the with-block bubble up
normally.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 46c0c039-a585-4aad-aff3-13f66709b223

📥 Commits

Reviewing files that changed from the base of the PR and between 4786834 and 72a8665.

📒 Files selected for processing (1)

tensorrt_llm/_torch/model_config.py

pengbowang-nv · 2026-03-06T07:26:32Z

Hi @chang-l could you also take a look at this one? Thanks!

Also to @sara4dev maybe we should check for errno as OSError is a broad one. In addition, you may need to finish DCO before merge (see github failed check for details).

chienchunhung · 2026-03-10T16:14:41Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-03-10T16:20:59Z

PR_Github #38459 [ run ] triggered by Bot. Commit: 72a8665 Link to invocation

tensorrt-cicd · 2026-03-10T20:12:23Z

PR_Github #38459 [ run ] completed with state ABORTED. Commit: 72a8665
LLM/main/L0_MergeRequest_PR #29815 (Blue Ocean) completed with status: ABORTED

Link to invocation

chienchunhung · 2026-03-10T20:49:41Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-03-10T20:56:59Z

PR_Github #38492 [ run ] triggered by Bot. Commit: 72a8665 Link to invocation

tensorrt-cicd · 2026-03-11T00:58:39Z

PR_Github #38492 [ run ] completed with state SUCCESS. Commit: 72a8665
/LLM/main/L0_MergeRequest_PR pipeline #29842 completed with status: 'FAILURE'

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

chienchunhung

The PR looks good to me. PS: We might need to clean up something in the follow-up PR.

chienchunhung · 2026-03-11T15:56:34Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-03-11T16:03:17Z

PR_Github #38606 [ run ] triggered by Bot. Commit: 72a8665 Link to invocation

tensorrt-cicd · 2026-03-11T17:34:45Z

PR_Github #38606 [ run ] completed with state SUCCESS. Commit: 72a8665
/LLM/main/L0_MergeRequest_PR pipeline #29942 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

chang-l · 2026-05-20T06:27:03Z

/bot run

tensorrt-cicd · 2026-05-20T06:33:36Z

PR_Github #49372 [ run ] triggered by Bot. Commit: 76fad15 Link to invocation

tensorrt-cicd · 2026-05-20T11:03:37Z

PR_Github #49372 [ run ] completed with state SUCCESS. Commit: 76fad15
/LLM/main/L0_MergeRequest_PR pipeline #39025 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chang-l · 2026-05-20T16:18:58Z

@sara4dev @pengbowang-nv — fresh CI (L0_MergeRequest_PR #39025) confirms OSError is too broad in its current placement. 29 failures in unittest._torch.executor.test_pytorch_model_engine, all with RuntimeError: generator didn't stop after throw().

Representative stack trace (e.g. test_warmup):

huggingface_hub.errors.RepositoryNotFoundError: 404 Client Error for url
    https://huggingface.co/dummy/resolve/main/config.json
tensorrt_llm/_torch/model_config.py:79 in config_file_lock
    yield        ← second yield (tempdir-fallback path)

The test calls from_pretrained("dummy") to exercise the 404 path. HF raises RepositoryNotFoundError (a subclass of OSError) inside the with config_file_lock(): body. The new broader catch swallows that, enters the tempdir fallback, and yields a second time — violating @contextmanager's throw() semantics.

Two minimal fix options, both keep the bug-fix intent for ENOLCK/ESTALE:

A. Narrow to specific errnos (matches the original review suggestion):

except OSError as e:
    if e.errno not in {errno.ENOLCK, errno.ESTALE, errno.EACCES, errno.EPERM}:
        raise
    # ... tempdir fallback

B. Scope the broad catch to lock.acquire() only, not the yielded body:

try:
    lock.acquire(timeout=timeout)
except (PermissionError, OSError, filelock.Timeout):
    # tempdir fallback
    ...
else:
    try:
        yield
    finally:
        lock.release()

I think either resolves the failing tests – @sara4dev can you please take a look?

…rnos in config_file_lock Catching broad OSError in config_file_lock swallowed legitimate non-lock OSErrors raised inside the yielded body (notably HuggingFace's RepositoryNotFoundError, an OSError subclass), which broke 29 tests in unittest._torch.executor.test_pytorch_model_engine with "RuntimeError: generator didn't stop after throw()". Narrow the catch to the errnos actually relevant for lock-infrastructure failures: EACCES, EPERM (PermissionError), ENOLCK, ESTALE (NFS locking). All other OSError subclasses propagate to the caller. Addresses pengbowang-nv review feedback on PR NVIDIA#11960. Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>

chang-l · 2026-05-23T00:08:47Z

Applied Option A from earlier comment since author was unresponsive past Friday EOB. New commit narrows the OSError catch to errno set {EACCES, EPERM, ENOLCK, ESTALE} so non-lock OSErrors (e.g. HF RepositoryNotFoundError) propagate out of the lock body cleanly. Triggering fresh CI.

/bot run

…rnos in config_file_lock Catching broad OSError in config_file_lock swallowed legitimate non-lock OSErrors raised inside the yielded body (notably HuggingFace's RepositoryNotFoundError, an OSError subclass), which broke 29 tests in unittest._torch.executor.test_pytorch_model_engine with "RuntimeError: generator didn't stop after throw()". Narrow the catch to the errnos actually relevant for lock-infrastructure failures: EACCES, EPERM (PermissionError), ENOLCK, ESTALE (NFS locking). All other OSError subclasses propagate to the caller. Addresses pengbowang-nv review feedback on PR NVIDIA#11960. Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>

chang-l · 2026-05-23T00:35:06Z

Rebased with sign-offs to satisfy DCO (force-pushed; content unchanged — verified tree hash equals pre-rebase). New HEAD: 4d86ed2089. Triggering fresh CI.

/bot run

chang-l · 2026-05-23T00:49:15Z

/bot run

tensorrt-cicd · 2026-05-23T00:55:51Z

PR_Github #50010 [ run ] triggered by Bot. Commit: 4d86ed2 Link to invocation

tensorrt-cicd · 2026-05-23T04:40:20Z

PR_Github #50010 [ run ] completed with state FAILURE. Commit: 4d86ed2
/LLM/main/L0_MergeRequest_PR pipeline #39575 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chang-l · 2026-05-23T04:49:26Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-23T04:54:57Z

PR_Github #50023 [ run ] triggered by Bot. Commit: 4d86ed2 Link to invocation

tensorrt-cicd · 2026-05-23T10:19:27Z

PR_Github #50023 [ run ] completed with state FAILURE. Commit: 4d86ed2
/LLM/main/L0_MergeRequest_PR pipeline #39586 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

On NFS-backed HF_MODULES_CACHE paths, filelock operations can raise OSError with errno ENOLCK (No locks available) or ESTALE (Stale file handle) instead of PermissionError. This causes config_file_lock() to crash rather than falling back to the tempdir-based lock. Add OSError to the exception handlers so the existing fallback logic handles NFS locking failures gracefully. Fixes NVIDIA#11958 Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>

…rnos in config_file_lock Catching broad OSError in config_file_lock swallowed legitimate non-lock OSErrors raised inside the yielded body (notably HuggingFace's RepositoryNotFoundError, an OSError subclass), which broke 29 tests in unittest._torch.executor.test_pytorch_model_engine with "RuntimeError: generator didn't stop after throw()". Narrow the catch to the errnos actually relevant for lock-infrastructure failures: EACCES, EPERM (PermissionError), ENOLCK, ESTALE (NFS locking). All other OSError subclasses propagate to the caller. Addresses pengbowang-nv review feedback on PR NVIDIA#11960. Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>

chang-l · 2026-05-27T16:36:07Z

/bot run

tensorrt-cicd · 2026-05-27T16:41:52Z

PR_Github #50582 [ run ] triggered by Bot. Commit: fd6ccbc Link to invocation

tensorrt-cicd · 2026-05-27T21:00:04Z

PR_Github #50582 [ run ] completed with state SUCCESS. Commit: fd6ccbc
/LLM/main/L0_MergeRequest_PR pipeline #40082 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

yufeiwu-nv · 2026-06-10T08:55:34Z

hi @chang-l , option-A lead to deepseek_r1_0528 error in GB300. I create a pr to apply option-B. Do you think it's OK?
Error: filelock._error.Timeout: The file lock '/root/.cache/huggingface/modules/_remote_code.lock' could not be acquired. [executor][RANK 0] Failed to initialize executor on rank 0: The file lock '/root/.cache/huggingface/modules/_remote_code.lock' could not be acquired. [F] [llmapi] MpiSession shutdown timeout, aborting... perf/test_perf.py::test_perf[deepseek_r1_0528_fp4-bench-pytorch-float4-maxbs:1000-maxnt:5000-kv_frac:0.85-input_output_len:5000,500-reqs:2000-ep:4-tp:4-gpus:4]
#15213

sara4dev requested a review from a team as a code owner March 5, 2026 20:18

sara4dev requested review from Naveassaf and leslie-fang25 March 5, 2026 20:18

coderabbitai Bot reviewed Mar 5, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/model_config.py Outdated

svc-trtllm-gh-bot added the Community want to contribute PRs initiated from Community label Mar 5, 2026

pengbowang-nv requested a review from chang-l March 9, 2026 05:40

pengbowang-nv assigned chang-l Mar 9, 2026

schetlur-nv requested a review from chienchunhung March 9, 2026 22:02

chang-l approved these changes Mar 11, 2026

View reviewed changes

chienchunhung approved these changes Mar 11, 2026

View reviewed changes

pengbowang-nv force-pushed the fix/config-file-lock-nfs-oserror branch from 72a8665 to e548b35 Compare March 23, 2026 06:54

chang-l changed the title ~~fix: catch OSError in config_file_lock for NFS compatibility~~ [https://nvbugs/6115560][fix] catch OSError in config_file_lock for NFS compatibility May 20, 2026

chang-l force-pushed the fix/config-file-lock-nfs-oserror branch from 3003656 to 4d86ed2 Compare May 23, 2026 00:34

sara4dev and others added 2 commits May 27, 2026 09:13

chang-l force-pushed the fix/config-file-lock-nfs-oserror branch from 4d86ed2 to fd6ccbc Compare May 27, 2026 16:35

chang-l merged commit 59d4369 into NVIDIA:main May 28, 2026
7 checks passed

yufeiwu-nv mentioned this pull request Jun 10, 2026

[None][chore] Fix lock_infra_error #15213

Merged

1 task

Uh oh!

Conversation

sara4dev commented Mar 5, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Root Cause

Reproduction

Workaround (for users on affected versions)

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pengbowang-nv commented Mar 6, 2026

Uh oh!

chienchunhung commented Mar 10, 2026

Uh oh!

tensorrt-cicd commented Mar 10, 2026

Uh oh!

tensorrt-cicd commented Mar 10, 2026

Uh oh!

chienchunhung commented Mar 10, 2026

Uh oh!

tensorrt-cicd commented Mar 10, 2026

Uh oh!

tensorrt-cicd commented Mar 11, 2026

Uh oh!

chienchunhung left a comment

Choose a reason for hiding this comment

Uh oh!

chienchunhung commented Mar 11, 2026

Uh oh!

tensorrt-cicd commented Mar 11, 2026

Uh oh!

tensorrt-cicd commented Mar 11, 2026

Uh oh!

chang-l commented May 20, 2026

Uh oh!

tensorrt-cicd commented May 20, 2026

Uh oh!

tensorrt-cicd commented May 20, 2026

Uh oh!

chang-l commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chang-l commented May 23, 2026

Uh oh!

chang-l commented May 23, 2026

Uh oh!

chang-l commented May 23, 2026

Uh oh!

tensorrt-cicd commented May 23, 2026

Uh oh!

tensorrt-cicd commented May 23, 2026

Uh oh!

chang-l commented May 23, 2026

Uh oh!

tensorrt-cicd commented May 23, 2026

Uh oh!

tensorrt-cicd commented May 23, 2026

Uh oh!

chang-l commented May 27, 2026

Uh oh!

tensorrt-cicd commented May 27, 2026

Uh oh!

tensorrt-cicd commented May 27, 2026

Uh oh!

Uh oh!

yufeiwu-nv commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

sara4dev commented Mar 5, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Mar 5, 2026 •

edited

Loading

chang-l commented May 20, 2026 •

edited

Loading