orc::InProcessMemoryMapper makes too many memory mappings

[gbaraldi]

First seen in JuliaLang/julia#49745, but to summarize, InProcessMemoryMapper calls mprotect many many times on mmaped memory, so many times that for a non trivial workload we go past /proc/sys/vm/max_map_count leading to a ENOMEM error.

I haven't created a non julia reproducer but it should be possible.

[llvmbot]

@llvm/issue-subscribers-orcjit

[lhames]

Within the InProcessMemoryMapper we make no attempt to keep permissions contiguous.

I think the OS should be able to coalesce adjacent mappings. If that's the case then a simple next step would be to allocate text pages from the bottom of the address space and data pages from the top. We'll get some fragmentation due to the mix of RW- and R-- data pages, but text pages should all coalesce. After that we could look at reserving permission-based bins within the reserved address space. E.g. if we used 1Mb bins (where all pages within each bin share permissions) then a 4Gb reservation should be able to be covered with 4096 mappings.

[llvmbot]

Hi!

This issue may be a good introductory issue for people new to working on LLVM. If you would like to work on this issue, your first steps are:

1. Assign the issue to you.
2. Fix the issue locally.
3. Run the test suite locally.
   3.1) Remember that the subdirectories under test/ create fine-grained testing targets, so you can
   e.g. use make check-clang-ast to only run Clang's AST tests.
4. Create a git commit
5. Run git clang-format HEAD~1 to format your changes.
6. Submit the patch to Phabricator.
   6.1) Detailed instructions can be found here

For more instructions on how to submit a patch to LLVM, see our documentation.

If you have any further questions about this issue, don't hesitate to ask via a comment on this Github issue.

@llvm/issue-subscribers-good-first-issue

[gbaraldi]

Something like that, I was thinking of doing something like 4mb chunks that you write text from top to the bottom and data from bottom to top.
I haven't checked if the OS coalesces the adjacent mappings but if yes then it shouldn't be too unreasonable

[llvmbot]

@llvm/issue-subscribers-julialang

[Sh0g0-1758]

Hi @gbaraldi! I am new to LLVM and would like to work on this issue. Can you please assign it to me.

[vchuravy]

@Sh0g0-1758 feel free to work on it, but it won't be officially assigned to you. If you come up with a solution you can open a PR and mention this issue