Plugins should set rpath so that they can load their dependencies

[ovidiusm]

What?

Set rpath to $ORIGIN (plugin dir) and $ORIGIN/.. (nixl lib dir) for all plugins

Why?

Some of the plugins do not load in the TRT-LLM image because LD_LIBRARY_PATH is not set to include the nixl directory; and some also need the plugin directory. But LD_LIBRARY_PATH should not be needed. Only the main nixl libs must be loadable; then when they dlopen() the plugins, the dependencies should be resolved automatically.

We set both current dir and parent dir because some plugins depend on other plugins; and all plugins depend on the nixl libs from the parent dir.

[github-actions]

👋 Hi ovidiusm! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[yosefe]

i wonder which nixl libs are needed by plugins? i would expect the main nixl libs would already be loaded in the program when we dlopen() the plugin file?

[ovidiusm]

i wonder which nixl libs are needed by plugins? i would expect the main nixl libs would already be loaded in the program when we dlopen() the plugin file?

The exact issue I have seen was with a plugin trying to load another plugin:

ldd /opt/nvidia/nvda_nixl/lib/x86_64-linux-gnu/plugins/libplugin_UCX_MO.so 
        linux-vdso.so.1 (0x0000800000129000)
libplugin_UCX.so => not found
        libnixl_build.so => /opt/nvidia/nvda_nixl/lib/x86_64-linux-gnu/libnixl_build.so (0x00007ffff7d66000)
        libucx_utils.so => /opt/nvidia/nvda_nixl/lib/x86_64-linux-gnu/libucx_utils.so (0x00007ffff7c9c000)
        libserdes.so => /opt/nvidia/nvda_nixl/lib/x86_64-linux-gnu/libserdes.so (0x00007ffff7c90000)
        libcudart.so.12 => /usr/local/cuda/lib64/libcudart.so.12 (0x00007ffff7800000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffff7582000)
        libm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x00007ffff7ba5000)
        libgcc_s.so.1 => /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffff7b77000)
        libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x00007ffff7370000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ffff7fc5000)
        libnixl_common.so => /opt/nvidia/nvda_nixl/lib/x86_64-linux-gnu/libnixl_common.so (0x00007ffff71ca000)
        libucp.so.0 => /opt/hpcx/ucx/lib/libucp.so.0 (0x00007ffff70d4000)
        libucs.so.0 => /opt/hpcx/ucx/lib/libucs.so.0 (0x00007ffff7b02000)
        libdl.so.2 => /usr/lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffff7afd000)
        libpthread.so.0 => /usr/lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffff7af8000)
        librt.so.1 => /usr/lib/x86_64-linux-gnu/librt.so.1 (0x00007ffff7af3000)
        libuct.so.0 => /opt/hpcx/ucx/lib/./libuct.so.0 (0x00007ffff7093000)
        libucm.so.0 => /opt/hpcx/ucx/lib/./libucm.so.0 (0x00007ffff7ad3000)

For this we need to add $ORIGIN to rpath.

Regarding $ORIGIN/.. : It might be possible to have libnixl loading a plugin that then tries to load another nixl library from the parent directory that is not already loaded in the process. I have not seen this fail so far. But it does not hurt to add $ORIGIN/.. since that should be the first location in which the library is looked up.

[yosefe]

IMO it should be only $ORIGIN/.. and plugin should not load other plugins, but depend on a common lib instead. Can we check why UCX_MO depends on UCX plugin and not only on libucx_utils.so?

[ovidiusm]

IMO it should be only $ORIGIN/.. and plugin should not load other plugins, but depend on a common lib instead. Can we check why UCX_MO depends on UCX plugin and not only on libucx_utils.so?

Good question.

UCX MO was built to create a separate UCX plugin instance per GPU:

https://github.com/ai-dynamo/nixl/blob/main/src/plugins/ucx_mo/ucx_mo_backend.cpp#L192

// Initialize required number of engines
    for (uint32_t i = 0; i < getEngCnt(); i++) {
        auto e = std::make_unique<nixlUcxEngine>(init_params);
        if (e->getInitErr()) {
            this->initErr = true;
            // TODO: Log error
            return;
        }
        engines.push_back(std::move(e));
    }

https://github.com/ai-dynamo/nixl/blob/main/src/plugins/ucx_mo/ucx_mo_backend.cpp#L119

int
nixlUcxMoEngine::setEngCnt(uint32_t num_host)
{
    _gpuCnt = _getNumVramDevices();
    _engineCnt = (_gpuCnt > num_host) ? _gpuCnt : num_host;
    return 0;
}

It's a very thin wrapper around the other plugin. IIRC it was made because of a limitation where we could not work with multiple GPUs at the same time from the same plugin.

Maybe we can use $ORIGIN/.. and make an exception just for UCX MO, what do you think?

[yosefe]

Maybe we can use $ORIGIN/.. and make an exception just for UCX MO, what do you think?

Yes, was just going to suggest to make ucx_mo use also $ORIGIN as exception + add FIXME comment

[ovidiusm]

Done.

[ovidiusm]

/build

[yosefe]

/build