UCX not building with NUMA support, impacting RTD builds

[charlesbluca]

Opening this issue to put this in writing somewhere in this repo instead of Slack 🙂

@pentschev and I noticed that the latest version of UCX isn't being built with NUMA support. It seems like this is an issue with UCX's configure script that can be resolved by not passing --enable-numa, and should be resolved in a later release.

This impacts our RTD builds in that currently, we are building from branch-0.20 because there is no libnuma dependency to manage. However, when this bug is fixed this will probably become a problem again. Some solutions for this:

• Switching back to the readthedocs branch in this repo, which is kept up to date with the latest development branch but uses a no-libnuma version of UCX for the docs (not sure how long we can continue building UCX-Py from source with this method since UCX is pinned to 1.8.0)
• Use apt to install libnuma in the RTD build, which depends on Build: allow to install packages with apt readthedocs/readthedocs.org#8065

Hopefully we can use this issue to keep track of the NUMA support bug and potential solutions for when it is fixed.

[pentschev]

Another -- and to some extent preferable -- alternative may be to have a resolution for openucx/ucx#4570 .

[jakirkham]

Thanks for raising this Charles! 😄

[charlesbluca]

Looks like a fix is in the work for the NUMA support issue openucx/ucx#6782. Depending on when this gets merged / incorporated into a stable release we might want to consider building RTD from the readthedocs branch soon

[jakirkham]

I think that PR was just fixing an issue where libnuma wasn't used even though libnuma was available. IOW a library detection issue during configure time. It doesn't address making libnuma runtime detectable, which is the issue Peter linked above.

In any event RTD recently added changes that will allow installing apt packages during the doc build. This will be supported when they redeploy next Tuesday afternoon. Have a PR ( #734 ) up that will use these changes to install libnuma on RTD. So this should be resolved soon.

Independently we likely will want to build new ucx packages with the upstream patch (assuming we can backport it without too much difficulty).

[pentschev]

Independently we likely will want to build new ucx packages with the upstream patch (assuming we can backport it without too much difficulty).

Let's hold on that until we're ready to support UCX 1.10/1.11, hopefully in the next few weeks.

[jakirkham]

The RTD PR ( #734 ) has been tested (works!) and has been merged

[pentschev]

This is resolved now, thanks @charlesbluca and @jakirkham for the work on RTD!