Repository navigation
fix: handle race condition in ensure_symlink for multi-process TP workers #2981
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -238,6 +238,9 @@ def ensure_symlink( | |||||
| This is used to map C++ include paths (e.g. | ||||||
| ``CUBIN_DIR/flashinfer/trtllm/batched_gemm/trtllmGen_bmm_export``) to the | ||||||
| canonical artifact directory where ``get_artifact()`` stores downloaded files. | ||||||
|
|
||||||
| This function is safe to call concurrently from multiple processes (e.g. | ||||||
| tensor-parallel workers) that may race to create the same symlink. | ||||||
|
Comment on lines
+242
to
+243
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The docstring states that this function is safe to call concurrently, but the cleanup logic for stale links (lines 251-254) still contains a race condition. If multiple processes attempt to |
||||||
| """ | ||||||
| link = pathlib.Path(link) | ||||||
| target = pathlib.Path(target) | ||||||
|
|
@@ -250,7 +253,13 @@ def ensure_symlink( | |||||
| else: | ||||||
| shutil.rmtree(link) | ||||||
| link.parent.mkdir(parents=True, exist_ok=True) | ||||||
| link.symlink_to(target) | ||||||
| try: | ||||||
| link.symlink_to(target) | ||||||
| except FileExistsError: | ||||||
| # Another process created the symlink between our check and create. | ||||||
| if link.is_symlink() and link.resolve() == target.resolve(): | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Comparing
Suggested change
|
||||||
| return | ||||||
| raise | ||||||
|
|
||||||
|
|
||||||
| def verify_symlinked_headers( | ||||||
|
|
||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The claim that this function is safe to call concurrently is not entirely accurate because the cleanup logic (lines 247-254) still contains TOCTOU (Time-of-Check to Time-of-Use) race conditions. If multiple processes attempt to remove a stale file or directory at the same time, one will likely fail with a
FileNotFoundErrorwhen callinglink.unlink()orshutil.rmtree(link). To make this truly safe for concurrent workers, these cleanup operations should be wrapped in atry-except FileNotFoundErrorblock or usemissing_ok=Trueforunlink(available in Python 3.8+).