Adapt delayed delete mechanism #4392
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adienes
pushed a commit
to JuliaLang/julia
that referenced
this pull request
Oct 1, 2025
… in-use DLL (#59635) On Windows, during precompilation of package `A`, a DLL file is generated and may replace the existing one. If `A` is already loaded in the julia session however, the corresponding soon-to-be-obsolete DLL cannot be removed while julia is running. Currently, this problem is solved by moving the old DLL in a special "julia_delayed_deletes" folder, which will be cleaned in a later julia session by `Pkg.gc()`. However, moving an in-use DLL is only permitted on the same drive, and the "julia_delayed_deletes" is located in the `tempdir`, a.k.a. on a fixed drive, say `C:`. Thus, if the `DEPOT_PATH` points to a ".julia" in another drive, say `D:`, any precompilation occuring on an already loaded package will fail. This is what happens in #59589, actually resulting in an infinite loop that bloats up both memory and disk. @staticfloat had actually predicted that such an issue could occur in #53456 (comment). This PR fixes #59589 by changing the "delayed deleting" mechanism: instead of moving the old DLLs to a fixed folder, they are renamed in their initial folder and recorded in a list stored at a fixed location. Upon `Pkg.gc()`, the listed obsolete files will be deleted (JuliaLang/Pkg.jl#4392). It also prevents any similar infinite loops by cutting the `rm -> mv -> rm` cycle into `rm -> rename`. ~I also removed the private argument `allow_delayed_delete` from `rm`, which is only called in by [Pkg](https://github.com/JuliaLang/Pkg.jl/blob/7344e2656475261a83a6bd37d9d4cc1e7dcf5f0d/src/API.jl#L1127) but does not appear to be useful.~ EDIT: current state is #59635 (comment) --------- Co-authored-by: Jameson Nash <[email protected]> Co-authored-by: Elliot Saba <[email protected]>
Member
|
This is in phase with what ended up being merged, right? |
KristofferC
pushed a commit
to JuliaLang/julia
that referenced
this pull request
Oct 10, 2025
… in-use DLL (#59635) On Windows, during precompilation of package `A`, a DLL file is generated and may replace the existing one. If `A` is already loaded in the julia session however, the corresponding soon-to-be-obsolete DLL cannot be removed while julia is running. Currently, this problem is solved by moving the old DLL in a special "julia_delayed_deletes" folder, which will be cleaned in a later julia session by `Pkg.gc()`. However, moving an in-use DLL is only permitted on the same drive, and the "julia_delayed_deletes" is located in the `tempdir`, a.k.a. on a fixed drive, say `C:`. Thus, if the `DEPOT_PATH` points to a ".julia" in another drive, say `D:`, any precompilation occuring on an already loaded package will fail. This is what happens in #59589, actually resulting in an infinite loop that bloats up both memory and disk. @staticfloat had actually predicted that such an issue could occur in #53456 (comment). This PR fixes #59589 by changing the "delayed deleting" mechanism: instead of moving the old DLLs to a fixed folder, they are renamed in their initial folder and recorded in a list stored at a fixed location. Upon `Pkg.gc()`, the listed obsolete files will be deleted (JuliaLang/Pkg.jl#4392). It also prevents any similar infinite loops by cutting the `rm -> mv -> rm` cycle into `rm -> rename`. ~I also removed the private argument `allow_delayed_delete` from `rm`, which is only called in by [Pkg](https://github.com/JuliaLang/Pkg.jl/blob/7344e2656475261a83a6bd37d9d4cc1e7dcf5f0d/src/API.jl#L1127) but does not appear to be useful.~ EDIT: current state is #59635 (comment) --------- Co-authored-by: Jameson Nash <[email protected]> Co-authored-by: Elliot Saba <[email protected]> (cherry picked from commit 3e2a4ed)
KristofferC
pushed a commit
to JuliaLang/julia
that referenced
this pull request
Oct 12, 2025
… in-use DLL (#59635) On Windows, during precompilation of package `A`, a DLL file is generated and may replace the existing one. If `A` is already loaded in the julia session however, the corresponding soon-to-be-obsolete DLL cannot be removed while julia is running. Currently, this problem is solved by moving the old DLL in a special "julia_delayed_deletes" folder, which will be cleaned in a later julia session by `Pkg.gc()`. However, moving an in-use DLL is only permitted on the same drive, and the "julia_delayed_deletes" is located in the `tempdir`, a.k.a. on a fixed drive, say `C:`. Thus, if the `DEPOT_PATH` points to a ".julia" in another drive, say `D:`, any precompilation occuring on an already loaded package will fail. This is what happens in #59589, actually resulting in an infinite loop that bloats up both memory and disk. @staticfloat had actually predicted that such an issue could occur in #53456 (comment). This PR fixes #59589 by changing the "delayed deleting" mechanism: instead of moving the old DLLs to a fixed folder, they are renamed in their initial folder and recorded in a list stored at a fixed location. Upon `Pkg.gc()`, the listed obsolete files will be deleted (JuliaLang/Pkg.jl#4392). It also prevents any similar infinite loops by cutting the `rm -> mv -> rm` cycle into `rm -> rename`. ~I also removed the private argument `allow_delayed_delete` from `rm`, which is only called in by [Pkg](https://github.com/JuliaLang/Pkg.jl/blob/7344e2656475261a83a6bd37d9d4cc1e7dcf5f0d/src/API.jl#L1127) but does not appear to be useful.~ EDIT: current state is #59635 (comment) --------- Co-authored-by: Jameson Nash <[email protected]> Co-authored-by: Elliot Saba <[email protected]> (cherry picked from commit 3e2a4ed)
Member
Author
|
Yes, this looks fine. I don't understand the CI crash though. |
Member
not sure what is up with that.. I'll try rerun it. |
KristofferC
pushed a commit
to JuliaLang/julia
that referenced
this pull request
Oct 14, 2025
… in-use DLL (#59635) On Windows, during precompilation of package `A`, a DLL file is generated and may replace the existing one. If `A` is already loaded in the julia session however, the corresponding soon-to-be-obsolete DLL cannot be removed while julia is running. Currently, this problem is solved by moving the old DLL in a special "julia_delayed_deletes" folder, which will be cleaned in a later julia session by `Pkg.gc()`. However, moving an in-use DLL is only permitted on the same drive, and the "julia_delayed_deletes" is located in the `tempdir`, a.k.a. on a fixed drive, say `C:`. Thus, if the `DEPOT_PATH` points to a ".julia" in another drive, say `D:`, any precompilation occuring on an already loaded package will fail. This is what happens in #59589, actually resulting in an infinite loop that bloats up both memory and disk. @staticfloat had actually predicted that such an issue could occur in #53456 (comment). This PR fixes #59589 by changing the "delayed deleting" mechanism: instead of moving the old DLLs to a fixed folder, they are renamed in their initial folder and recorded in a list stored at a fixed location. Upon `Pkg.gc()`, the listed obsolete files will be deleted (JuliaLang/Pkg.jl#4392). It also prevents any similar infinite loops by cutting the `rm -> mv -> rm` cycle into `rm -> rename`. ~I also removed the private argument `allow_delayed_delete` from `rm`, which is only called in by [Pkg](https://github.com/JuliaLang/Pkg.jl/blob/7344e2656475261a83a6bd37d9d4cc1e7dcf5f0d/src/API.jl#L1127) but does not appear to be useful.~ EDIT: current state is #59635 (comment) --------- Co-authored-by: Jameson Nash <[email protected]> Co-authored-by: Elliot Saba <[email protected]> (cherry picked from commit 3e2a4ed)
KristofferC
pushed a commit
that referenced
this pull request
Oct 15, 2025
(cherry picked from commit 93a960c)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the companion PR to JuliaLang/julia#59635, and should be merged only if the julia PR is.
During
Pkg.gc(), instead of deleting the files stored in thedelayed_delete_dir(), the proposed mechanism consists in deleting the files listed in thedelayed_delete_list().