[v1.9.x] stop closing opened libs

[samskalicky]

Description

stop closing opened libs. fixes #20411 using portions already merged into master/2.0 in #19016.

According to [1]

The dlclose() function shall inform the system that the object referenced by a handle returned from a previous dlopen() invocation is no longer needed by the application.

The use of dlclose() reflects a statement of intent on the part of the process, but does not create any requirement upon the implementation, such as removal of the code or symbols referenced by handle.

So calling dlclose is just a suggestion to the OS that it could close the library. But the OS may not do anything (ie. and wait until the process exits to clean up any opened handles). Im thinking we might as well just let it be cleaned up anyway rather than calling dlclose at all.

As suggested in [2]

If you only ever open one library, use it throughout your program, then calling dlclose just before you exit is probably not essential, but if you open a lot of libraries (e.g. using some sort of plugin in a long-running program that can/will use many different plugins, the program may run out of virtual address space if you don't call dlclose.

(All shared libraries are closed on exit anyway, so leaving it open at exit should not be an issue)

So if a user was loading/unloading a bunch of libraries at runtime they might want to close some they dont need anymore.

However, since MXNet registers components from the library (ie. operators, partitioners, graph passes, etc) and has NO capability to unregister them the references to the loaded library will live at least as long as MXNet (ie. libmxnet.so) does. MXNet already has quite a few components (or its dependencies like TVM/NNVM) that live forever and require the OS to clean them up when the process exits. So its not possible to close libmxnet.so and assume everything is cleaned up. We might as well do the same thing for custom libraries.

This PR removes the code to close opened libraries in the destructor of the LibraryInitializer class with the caveat that MXNet will still have pointers to that library, so closing would not be recommended until process exit anyway.

[mxnet-bot]

Hey @samskalicky , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

• To trigger all jobs: @mxnet-bot run ci [all]
• To trigger specific jobs: @mxnet-bot run ci [job1, job2]

---

CI supported jobs: [windows-cpu, unix-gpu, centos-cpu, edge, windows-gpu, unix-cpu, clang, miscellaneous, centos-gpu, website, sanity]

---

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

[samskalicky]

Manual build on Mac runs the gemm_lib example in extensions without segfault

[szha]

This is a C API change. Is it required for the fix?

[samskalicky]

Thanks for the review @szha! No this change isnt required, but without it we cannot return a handle to the opened library. Its not required per-se since opening the same library using ctypes or some other API will return the same handle anyway (with linux doing the mapping). So we can remove this change if that is important, either way is fine to me.

[szha]

C API is part of the API to maintain semantic versioning for. Let's not include the API change then.

[mseth10]

Thanks for the PR. The destructor change looks good.

What's the benefit of returning handle to opened library, if it's not recommended that users close them? Is there any other use case?

[samskalicky]

Thanks @szha @mseth10 ive made the changes to not return the handle. Now this PR is effectively just not closing opened libraries on exit (and some misc comment updates)

[mseth10]

LGTM