fix link order in PETSc easyblock for SCOTCH >= 7.x by boegel · Pull Request #3069 · easybuilders/easybuild-easyblocks

boegel · 2024-01-05T20:32:05Z

(created using eb --new-pr)

When building petsc4py on top of PETSc/3.17.4-foss-2022a, I ran into the following problem during the from petsc4py import PETSc sanity check:

ImportError: /software/PETSc/3.17.4-foss-2022a/lib/libpetsc.so.3.17: undefined symbol: SCOTCH_dgraphOrderTreeDist

This is due to a change in the linking order of SCOTCH libraries to order them alphabetically, which was introduced in #2796 (on my request 🤦 ).
It's vital that libptscotch (which provides SCOTCH_dgraphOrderTreeDist) is linked in after libptscotchparmetisv3 (which requires SCOTCH_dgraphOrderTreeDist).

boegel · 2024-01-05T20:37:46Z

@boegelbot please test @ generoso
CORE_CNT=16
EB_ARGS="PETSc-3.15.1-intel-2021a.eb PETSc-3.17.4-foss-2022a.eb PETSc-3.18.4-intel-2021b.eb PETSc-3.19.2-foss-2022b.eb"

boegelbot · 2024-01-05T20:39:09Z

@boegel: Request for testing this PR well received on login1

PR test command 'EB_PR=3069 EB_ARGS="PETSc-3.15.1-intel-2021a.eb PETSc-3.17.4-foss-2022a.eb PETSc-3.18.4-intel-2021b.eb PETSc-3.19.2-foss-2022b.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --job-name test_PR_3069 --ntasks="16" ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 12516

Test results coming soon (I hope)...

Details

- notification for comment with ID 1879230009 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

migueldiascosta · 2024-01-05T23:58:35Z

easybuild/easyblocks/p/petsc.py

-                                   'libptscotcherr.a', 'libptscotchparmetisv3.a', 'libscotch.a',
-                                   'libscotcherr.a']
+                # which is the reason for this new code;
+                # note: order matters here, don't sort these alphabetically!


but it's much easier for human eyes! :p

lgtm

boegel · 2024-01-06T00:17:35Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS PETSc-3.17.4-foss-2022a.eb
SUCCESS PETSc-3.19.2-foss-2022b.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
node3123.skitty.os - Linux RHEL 8.8, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (skylake_avx512), Python 3.6.8
See https://gist.github.com/boegel/73b19fe05e6b65080d4d2c8e79424ddb for a full test report.

boegelbot · 2024-01-06T01:53:03Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS PETSc-3.15.1-intel-2021a.eb
SUCCESS PETSc-3.17.4-foss-2022a.eb
FAIL (build issue) PETSc-3.18.4-intel-2021b.eb (partial log available at https://gist.github.com/boegelbot/b6a6e337a2d0cad5f7363845b7cb8bd6)
SUCCESS PETSc-3.19.2-foss-2022b.eb

Build succeeded for 3 out of 4 (4 easyconfigs in total)
cnx1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/c9761e8065636243c8b2f277a49746ad for a full test report.

migueldiascosta · 2024-01-06T06:51:40Z

Hm, another issue with Intel MPI on generoso?

boegel · 2024-01-06T08:21:28Z

Hm, another issue with Intel MPI on generoso?

Hmm, yeah, maybe... I wouldn't let this block this PR though, since PETSc-3.18.4-intel-2021b.eb is actually not affected by the changes being made at all (since it uses SCOTCH 6.x < 7.x)

boegel · 2024-01-06T08:39:45Z

@boegelbot please test @ jsc-zen2
CORE_CNT=8
EB_ARGS="PETSc-3.15.1-intel-2021a.eb PETSc-3.17.4-foss-2022a.eb PETSc-3.18.4-intel-2021b.eb PETSc-3.19.2-foss-2022b.eb"

boegelbot · 2024-01-06T08:42:08Z

@boegel: Request for testing this PR well received on jsczen2l1.int.jsc-zen2.easybuild-test.cluster

PR test command 'EB_PR=3069 EB_ARGS="PETSc-3.15.1-intel-2021a.eb PETSc-3.17.4-foss-2022a.eb PETSc-3.18.4-intel-2021b.eb PETSc-3.19.2-foss-2022b.eb" EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --mem-per-cpu=4000M --job-name test_PR_3069 --ntasks="8" ~/boegelbot/eb_from_pr_upload_jsc-zen2.sh' executed!

exit code: 0
output:

Submitted batch job 4017

Test results coming soon (I hope)...

Details

- notification for comment with ID 1879595628 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegel · 2024-01-06T08:52:09Z

More info from configure.log for failure with PETSc-3.18.4-intel-2021b.eb on generoso:

Details

          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
 -------------------------------------------------------------------------------
 Timeout: Unable to run MPI program with /project/boegelbot/Rocky8/haswell/software/impi/2021.4.0-intel-compilers-2021.4.0/mpi/2021.4.0/bin/mpiexec -n 1
     (1) make sure this is the correct program to run MPI jobs
     (2) your network may be misconfigured; see https://petsc.org/release/faq/#mpi-network-misconfigure
     (3) you may have VPN running whose network settings may not play nice with MPI
 *******************************************************************************
   File "/tmp/boegelbot/PETSc/3.18.4/intel-2021b/petsc-3.18.4/config/configure.py", line 461, in petsc_configure
     framework.configure(out = sys.stdout)
   File "/tmp/boegelbot/PETSc/3.18.4/intel-2021b/petsc-3.18.4/config/BuildSystem/config/framework.py", line 1412, in configure
     self.processChildren()
   File "/tmp/boegelbot/PETSc/3.18.4/intel-2021b/petsc-3.18.4/config/BuildSystem/config/framework.py", line 1400, in processChildren
     self.serialEvaluation(self.childGraph)
   File "/tmp/boegelbot/PETSc/3.18.4/intel-2021b/petsc-3.18.4/config/BuildSystem/config/framework.py", line 1375, in serialEvaluation
     child.configure()
   File "/tmp/boegelbot/PETSc/3.18.4/intel-2021b/petsc-3.18.4/config/BuildSystem/config/package.py", line 1222, in configure
     self.executeTest(self.configureLibrary)
   File "/tmp/boegelbot/PETSc/3.18.4/intel-2021b/petsc-3.18.4/config/BuildSystem/config/base.py", line 138, in executeTest
     ret = test(*args,**kargs)
   File "/tmp/boegelbot/PETSc/3.18.4/intel-2021b/petsc-3.18.4/config/BuildSystem/config/packages/MPI.py", line 881, in configureLibrary
     self.executeTest(self.configureMPIEXEC)
   File "/tmp/boegelbot/PETSc/3.18.4/intel-2021b/petsc-3.18.4/config/BuildSystem/config/base.py", line 138, in executeTest
     ret = test(*args,**kargs)
   File "/tmp/boegelbot/PETSc/3.18.4/intel-2021b/petsc-3.18.4/config/BuildSystem/config/packages/MPI.py", line 377, in configureMPIEXEC
     raise RuntimeError('Timeout: %s' % error_message)

looks like a very (very) basic MPI test: https://gitlab.com/petsc/petsc/-/blob/80f88c66aac8f9d1abe2df64bcd926bcfcc2a09a/config/BuildSystem/config/packages/MPI.py#L380

I can reproduce this issue easily when running in an interactive job started with srun -c 16 --time 600 --pty /bin/bash.

After running for x in $(env | grep SLURM| cut -f1 -d=); do unset $x; done to unset all $SLURM_* environment variables, configure works fine.

boegelbot · 2024-01-06T20:53:53Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS PETSc-3.17.4-foss-2022a.eb
SUCCESS PETSc-3.18.4-intel-2021b.eb
SUCCESS PETSc-3.19.2-foss-2022b.eb
FAIL (build issue) SciPy-bundle-2021.05-intel-2021a.eb (partial log available at https://gist.github.com/boegelbot/0847dbaaad8b81bc69fda90dbf22803b)
SUCCESS Boost-1.76.0-intel-compilers-2021.2.0.eb
SUCCESS SCOTCH-6.1.0-iimpi-2021a.eb
SUCCESS MUMPS-5.4.0-intel-2021a-metis.eb
SUCCESS SuiteSparse-5.10.1-intel-2021a-METIS-5.1.0.eb
SUCCESS Hypre-2.21.0-intel-2021a.eb
FAIL (build issue) PETSc-3.15.1-intel-2021a.eb (partial log available at https://gist.github.com/boegelbot/db1551377c3d403a87d78ba535f37e5a)

Build succeeded for 8 out of 10 (4 easyconfigs in total)
jsczen2c1.int.jsc-zen2.easybuild-test.cluster - Linux Rocky Linux 8.5, x86_64, AMD EPYC 7742 64-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/boegelbot/b900abf19ea35a00a92f5668dbce4ac0 for a full test report.

migueldiascosta

lgtm, and failures in the test reports are not related nor affected by these changes

migueldiascosta · 2024-01-10T06:44:29Z

Going in, thanks @boegel!

migueldiascosta · 2024-01-10T06:47:39Z

@boegel SciPy-bundle/2021.05-intel-2021a at jsc-zen2 seems to be broken though, which will likely affect other tests

fix link order in PETSc easyblock for SCOTCH >= 7.x

f7c3e01

boegel added the bug fix label Jan 5, 2024

boegel added this to the release after 4.9.0 milestone Jan 5, 2024

boegel mentioned this pull request Jan 5, 2024

{bio}foss/2022a] CellRank v2.0.2, scVelo v0.2.5, slepc4py v3.17.2, petsc4py v3.17.4 easybuilders/easybuild-easyconfigs#19515

Merged

migueldiascosta reviewed Jan 5, 2024

View reviewed changes

easybuilders deleted a comment from boegelbot Jan 6, 2024

migueldiascosta approved these changes Jan 10, 2024

View reviewed changes

migueldiascosta merged commit c68d093 into easybuilders:develop Jan 10, 2024

boegel deleted the 20240105213202_new_pr_petsc branch February 11, 2024 13:37

Conversation

boegel commented Jan 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegel commented Jan 5, 2024

Uh oh!

boegelbot commented Jan 5, 2024

Uh oh!

migueldiascosta Jan 5, 2024

Choose a reason for hiding this comment

Uh oh!

boegel Jan 6, 2024

Choose a reason for hiding this comment

Uh oh!

boegel commented Jan 6, 2024

Overview of tested easyconfigs (in order)

Uh oh!

boegelbot commented Jan 6, 2024

Overview of tested easyconfigs (in order)

Uh oh!

migueldiascosta commented Jan 6, 2024

Uh oh!

boegel commented Jan 6, 2024

Uh oh!

boegel commented Jan 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegelbot commented Jan 6, 2024

Uh oh!

boegel commented Jan 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegelbot commented Jan 6, 2024

Overview of tested easyconfigs (in order)

Uh oh!

migueldiascosta left a comment

Choose a reason for hiding this comment

Uh oh!

migueldiascosta commented Jan 10, 2024

Uh oh!

migueldiascosta commented Jan 10, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

boegel commented Jan 5, 2024 •

edited

Loading

boegel commented Jan 6, 2024 •

edited

Loading

boegel commented Jan 6, 2024 •

edited

Loading