{tools}[foss/2024a] modin v0.32.0 by lcniel · Pull Request #21667 · easybuilders/easybuild-easyconfigs

lcniel · 2024-10-15T19:26:38Z

(created using eb --new-pr)

Requires:

easybuild/easyconfigs/m/modin/modin-0.32.0-foss-2024a.eb

Micket · 2024-10-17T09:19:51Z

easybuild/easyconfigs/m/modin/modin-0.32.0-foss-2024a.eb

+        'checksums': ['9f68557add5e92617d006eb98bfd93d298e7d4d9932a0cd48a5e38ee4f30a134'],
+    }),
+    ('unidist', '0.7.1', {
+        'source_tmpl': '%(namelower)s-%(version)s-cp312-cp312-manylinux_2_17_%(arch)s.manylinux2014_%(arch)s.whl',


Any particular reason to prefer the wheel for this one? We usually go for the source.

I was also tempted to suggest breaking this out into it's own package, but i see so far modin is the only usecase, so maybe not..? Things might change though, so i wouldn't be against putting this into it's own thing, and just have modin depend on it instead of mpi+dask+ray
https://unidist.readthedocs.io/en/latest/using_unidist/index.html#libraries-powered-by-unidist

I'm also confused as to why we need the extras here specific to mpi? it supposed dask, ray, mpi, and we have all of those as dependencies, so presumably, this should be used here?

unidist automatically detects which execution backends are installed and uses that for scheduling computation.

wouldn't mpi4py be enough then?

There was some error related to copying/finding files during Cython compilation when I tried to build from source. I have absolutely no idea how one would go about solving it, it was trying to do some "include source_file.cpp" thing and not finding source_file.cpp even though that's clearly in that directory...

About the pip install extras, I'm just following the requirements in the modin installation files which specify "unidist[mpi]". I guess that only amounts to a dependency check?

Also, Modin basically supports all three frameworks, but unidist seems to be specifically for MPI, from their setup.py:

https://github.com/modin-project/modin/blob/main/setup.py

Their readme also says that you must set both MODIN_BACKEND and UNIDIST_BACKEND to MPI if you use unidist. So it's a bit of a special thing.

Micket · 2024-10-17T09:21:03Z

easybuild/easyconfigs/m/modin/modin-0.32.0-foss-2024a.eb

+sanity_check_commands = ['python -c "import modin"']
+


this is already done by default for all extensions

Micket · 2024-10-17T11:44:16Z

For reference: issue with builds stem from modin-project/unidist#428

lcniel · 2024-10-17T15:18:29Z

Test report by @lcniel
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
vera22-2 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, Python 3.6.8
See https://gist.github.com/lcniel/f74cb8d96277931b4193d3a7730cdab8 for a full test report.

lcniel · 2024-10-17T15:19:01Z

Test report by @lcniel
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
vera22-2 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, Python 3.6.8
See https://gist.github.com/lcniel/7a32ad8e9219a9e596effbc33aa7327b for a full test report.

lcniel · 2024-10-17T15:19:28Z

Test report by @lcniel
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
vera22-2 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, Python 3.6.8
See https://gist.github.com/lcniel/d21bc09a2b697ad1dea2d4b1b778d3b2 for a full test report.

lcniel · 2024-10-17T15:29:24Z

Test report by @lcniel
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
vera30-1 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, Python 3.6.8
See https://gist.github.com/lcniel/1f0584fa312034f8d6e0375c87ffcf3c for a full test report.

lcniel · 2024-10-17T15:30:20Z

Test report by @lcniel
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
vera22-2 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, Python 3.6.8
See https://gist.github.com/lcniel/36ad19489e639867276b780859a4c341 for a full test report.

lcniel · 2024-10-17T15:30:51Z

Test report by @lcniel
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
vera22-2 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, Python 3.6.8
See https://gist.github.com/lcniel/3affc0936894a0a8886a15f680c70860 for a full test report.

lcniel · 2024-10-17T15:31:19Z

Test report by @lcniel
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
vera22-2 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, Python 3.6.8
See https://gist.github.com/lcniel/42e4dcc36028e86dbb5757369085c4eb for a full test report.

lcniel · 2024-10-17T16:51:17Z

Test report by @lcniel
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
vera41-3 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, Python 3.6.8
See https://gist.github.com/lcniel/8252ad97f9e01975f1ffca8a83373968 for a full test report.

lcniel · 2024-10-17T17:02:16Z

OK, the failure now is from ray-project/ray#7724 when running the test suite.

Long story short, ray relies on AF_UNIX sockets which have hard limits on path length, so it cannot handle complicated tempdir paths. It is a baffling design decision to build the software in this way, because the only way to fix this temporarily would be to overwrite TMPDIR with some short path during the test, which can hardly be done dynamically within an easyconfig. So I'm going to have to remove the test for the Ray backend.

lcniel · 2024-10-17T17:19:51Z

Test report by @lcniel
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
vera41-3 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, Python 3.6.8
See https://gist.github.com/lcniel/b8e1f6677fbaa0f0f7b69922321a9fce for a full test report.

lcniel · 2024-10-17T19:59:26Z

Test report by @lcniel
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
vera41-3 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, Python 3.6.8
See https://gist.github.com/lcniel/0f959a8bb68e495f216171bbcf223ebe for a full test report.

lcniel · 2024-10-17T20:24:19Z

Test report by @lcniel
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
vera26-2 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, Python 3.6.8
See https://gist.github.com/lcniel/e6158ec92c422e1a4486e77901b79c19 for a full test report.

lcniel · 2024-10-17T20:39:59Z

Test report by @lcniel
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
vera11-2 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, Python 3.6.8
See https://gist.github.com/lcniel/cc157d6cf1055fdb0baf2954421460b1 for a full test report.

lcniel · 2024-10-17T20:53:21Z

Test report by @lcniel
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
vera12-3 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, Python 3.6.8
See https://gist.github.com/lcniel/5f214d8c2a3bb1461f14adacbc5592d8 for a full test report.

lcniel · 2024-10-17T21:00:10Z

Test report by @lcniel
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
vera12-3 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, Python 3.6.8
See https://gist.github.com/lcniel/55c5d3390995dee49d2255e3f0e571e5 for a full test report.

lcniel · 2024-10-17T21:14:59Z

OK, so my conclusion is that Unidist does not work at the moment, I tried it in several contexts and it just gets stuck. Dask works fine, Ray works if you don't have a complicated tempdir path specified.

lcniel · 2024-10-17T21:17:37Z

Test report by @lcniel
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
vera18-4 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, Python 3.6.8
See https://gist.github.com/lcniel/3843b0b52d22f769aaf7505abbf96a7d for a full test report.

smoors · 2024-10-18T10:15:24Z

easybuild/easyconfigs/m/modin/modin-0.32.0-foss-2024a.eb

+use_pip = True
+
+exts_list = [
+    # modin has another backend, "unidist", for MPI, but it seems broken currently.


can you add a bit more info why it's broken?
what's the error message?
maybe open a github issue that you can link to?

There is no error message, trying to build it and running tests with unidist as the backend, or even just importing it and trying to use anything, e.g. modin.pandas.DataFrame(), results in all your nodes running some kind of unidist init code and firing up all the nodes to 50%. I could not find any documentation of this behaviour.

I have tried the usual MPI remedies to attempt to debug it but I can't get it to work at all. Not sure if anyone else wants to dig into it, but I have no idea how one would go about debugging it.

Updated to add a little more info, but there is really not much that I can say about it.

I think I found a way to resolve it. Maybe!

@smoors I think it's fixed.

great, thanks a lot!

…asyconfigs into 20241015192634_new_pr_modin0320

YarShev · 2024-10-21T15:50:30Z

For reference: issue with builds stem from modin-project/unidist#428

We just released unidist 0.7.2, which adds missing files in the PyPI tarball. Thanks for raising the issue.

lcniel · 2024-10-21T20:57:36Z

Test report by @lcniel
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
vera02-2 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz, Python 3.6.8
See https://gist.github.com/lcniel/6c4866a015654cf2b7c705ee97a8d5a7 for a full test report.

lcniel · 2024-10-22T12:52:15Z

Test report by @lcniel
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
vera-r08-05 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz, Python 3.6.8
See https://gist.github.com/lcniel/1b461fec19725db550a6ffccb960b683 for a full test report.

lcniel · 2024-10-22T12:55:58Z

OK, I figured out a way to make unidist work by trial and error as well as looking at the way tests are run in the unidist repo: https://github.com/modin-project/unidist/blob/master/.github/workflows/ci.yml

Looks a little bit hackish to me (some implementation choices like using multiprocessing.cpu_count() to get the number of tasks are a bit questionable as well) but I guess they wrote it that way for a reason.

I also had to change UCX_TLS: export UCX_TLS=dc,ud,sm,self, but I didn't put this in the easyconfig as it might be related to our local configuration.

boegel · 2024-10-25T13:04:09Z

@boegelbot please test @ generoso

boegelbot · 2024-10-25T13:05:10Z

@boegel: Request for testing this PR well received on login1

PR test command 'EB_PR=21667 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs /opt/software/slurm/bin/sbatch --job-name test_PR_21667 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 14577

Test results coming soon (I hope)...

Details

- notification for comment with ID 2437725407 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-10-25T13:09:57Z

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
cns1 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/27f94917c02f28e29ef68bf4df815ee6 for a full test report.

boegel · 2024-10-25T13:19:09Z

@boegelbot please test @ jsc-zen3

boegelbot · 2024-10-25T13:20:09Z

@boegel: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=21667 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_21667 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 5161

Test results coming soon (I hope)...

Details

- notification for comment with ID 2437760280 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-10-25T13:23:41Z

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/3a4a1f24a458df85c49fd3778011d6ec for a full test report.

smoors

lgtm

smoors · 2024-11-03T17:58:27Z

Going in, thanks @lcniel!

lcniel added 2 commits October 15, 2024 19:26

adding easyconfigs: modin-0.32.0-foss-2024a.eb

96675c9

updated description and link

5b12232

Micket added the new label Oct 16, 2024

Micket added this to the release after 4.9.4 milestone Oct 17, 2024

Micket reviewed Oct 17, 2024

View reviewed changes

easybuild/easyconfigs/m/modin/modin-0.32.0-foss-2024a.eb Outdated Show resolved Hide resolved

Update easybuild/easyconfigs/m/modin/modin-0.32.0-foss-2024a.eb

df0f3a4

Micket reviewed Oct 17, 2024

View reviewed changes

Update modin-0.32.0-foss-2024a.eb

37217fb

lcniel added 2 commits October 17, 2024 14:38

enable some tests

0840096

checksums

646ab34

test suite

fc5c77d

test suite

da0530e

cut down on test suite

e6198a1

lcniel added 2 commits October 17, 2024 20:26

cut down on test suite

d31ae4f

cut down on test suite

88ec16c

removed unidist due to non-functionality

3296c1b

confirmed ray non-functional tests

54574e3

removed mpi4py dep

9eef90a

smoors reviewed Oct 18, 2024

View reviewed changes

lcniel added 2 commits October 18, 2024 12:20

added unidist problem info

0918cff

Merge branch 'develop' of https://github.com/easybuilders/easybuild-e…

cc3ea2d

…asyconfigs into 20241015192634_new_pr_modin0320

try adding unidist again

e2784ca

lcniel added 2 commits October 21, 2024 20:59

fix a typo

e26dbce

fix unidist test?

0653330

lcniel requested review from Micket and smoors October 28, 2024 21:59

smoors approved these changes Nov 3, 2024

View reviewed changes

smoors merged commit d99fd3d into easybuilders:develop Nov 3, 2024

boegel modified the milestones: release after 4.9.4, 5.0.0 Mar 18, 2025

Conversation

lcniel commented Oct 15, 2024 • edited by Micket Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Micket Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lcniel Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Micket commented Oct 17, 2024

Uh oh!

lcniel commented Oct 17, 2024

Uh oh!

lcniel commented Oct 17, 2024

Uh oh!

lcniel commented Oct 17, 2024

Uh oh!

lcniel commented Oct 17, 2024

Uh oh!

lcniel commented Oct 17, 2024

Uh oh!

lcniel commented Oct 17, 2024

Uh oh!

lcniel commented Oct 17, 2024

Uh oh!

lcniel commented Oct 17, 2024

Uh oh!

lcniel commented Oct 17, 2024

Uh oh!

lcniel commented Oct 17, 2024

Uh oh!

lcniel commented Oct 17, 2024

Uh oh!

lcniel commented Oct 17, 2024

Uh oh!

lcniel commented Oct 17, 2024

Uh oh!

lcniel commented Oct 17, 2024

Uh oh!

lcniel commented Oct 17, 2024

Uh oh!

lcniel commented Oct 17, 2024

Uh oh!

lcniel commented Oct 17, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lcniel Oct 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

YarShev commented Oct 21, 2024

Uh oh!

lcniel commented Oct 21, 2024

Uh oh!

lcniel commented Oct 22, 2024

Uh oh!

lcniel commented Oct 22, 2024

Uh oh!

boegel commented Oct 25, 2024

Uh oh!

boegelbot commented Oct 25, 2024

lcniel commented Oct 15, 2024 •

edited by Micket

Loading

Micket Oct 17, 2024 •

edited

Loading

lcniel Oct 17, 2024 •

edited

Loading

lcniel Oct 18, 2024 •

edited

Loading