Skip to content

{vis}[GCCcore/11.3.0] OpenEXR v3.1.5, libwebp v1.2.4, OpenCV v4.6.0 ... w/ Python 3.10.4#16276

Merged
casparvl merged 21 commits intoeasybuilders:developfrom
sara-nl:20220921124503_new_pr_OpenEXR315
Oct 14, 2022
Merged

{vis}[GCCcore/11.3.0] OpenEXR v3.1.5, libwebp v1.2.4, OpenCV v4.6.0 ... w/ Python 3.10.4#16276
casparvl merged 21 commits intoeasybuilders:developfrom
sara-nl:20220921124503_new_pr_OpenEXR315

Conversation

@robogast
Copy link
Copy Markdown
Contributor

(created using eb --new-pr)

@casparvl
Copy link
Copy Markdown
Contributor

Some context for the reviewer on the use of FFmpeg 5.0.1:

Based on #15918 that was already the FFmpeg that we intended to use in this toolchain generation. Note that also e.g. #16162 and #15924 use FFmpeg 5.0.1.

However, #15938, which was merged recently, uses an older FFmpeg, specifically because ParaView was said to not be compatible with FFmpeg 5.0.1. My proposal is that we put this exception in the CI and stick with 5.0.1 as the main version for FFmpeg in this toolchain. I've never done this before though, I think it requires a framework change, right?

@branfosj
Copy link
Copy Markdown
Member

Some context for the reviewer on the use of FFmpeg 5.0.1:

Based on #15918 that was already the FFmpeg that we intended to use in this toolchain generation. Note that also e.g. #16162 and #15924 use FFmpeg 5.0.1.

However, #15938, which was merged recently, uses an older FFmpeg, specifically because ParaView was said to not be compatible with FFmpeg 5.0.1. My proposal is that we put this exception in the CI and stick with 5.0.1 as the main version for FFmpeg in this toolchain. I've never done this before though, I think it requires a framework change, right?

Exceptions are added to https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/test/easyconfigs/easyconfigs.py#L487 - something like the following - note that it requires specifying the exception for both ParaView and applications that depend on it - i.e. OpenFOAM:

            # ParaView 5.10.x requires FFmpeg 4.x
            'FFmpeg': [(r'4\.4\.', [r'ParaView-5\.10\.', 'OpenFOAM-v2206'])],

@casparvl
Copy link
Copy Markdown
Contributor

@boegelbot please test @ generoso

@boegelbot
Copy link
Copy Markdown
Collaborator

@casparvl: Request for testing this PR well received on login1

PR test command 'EB_PR=16276 EB_ARGS= /opt/software/slurm/bin/sbatch --job-name test_PR_16276 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 9173

Test results coming soon (I hope)...

Details

- notification for comment with ID 1253937777 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Copy Markdown
Collaborator

Test report by @boegelbot
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in total)
cns1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/f670bdf082214b9f0e71038c8920ac23 for a full test report.

@casparvl
Copy link
Copy Markdown
Contributor

Test report by @casparvl
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in total)
software1.lisa.surfsara.nl - Linux debian 10.13, x86_64, Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz, Python 3.7.3
See https://gist.github.com/0a65a8f936f99ce9d733b9916562e0d1 for a full test report.

@casparvl
Copy link
Copy Markdown
Contributor

Test report by @casparvl
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in total)
tcn1 - Linux RHEL 8.4, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/20413b700e0f01b60b3c8518a24c8583 for a full test report.

@casparvl casparvl added this to the next release (4.6.2?) milestone Sep 23, 2022
@boegelbot
Copy link
Copy Markdown
Collaborator

@robogast: Tests failed in GitHub Actions, see https://github.com/easybuilders/easybuild-easyconfigs/actions/runs/3112484305
Last 100 lines of output from first failing test suite run:

    EasyConfigTest.parse_all_easyconfigs()
  File "test/easyconfigs/easyconfigs.py", line 202, in parse_all_easyconfigs
    cls._parsed_easyconfigs.extend(process_easyconfig(spec))
  File "/opt/hostedtoolcache/Python/2.7.18/x64/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 2071, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: 'Failed to process easyconfig /home/runner/work/easybuild-easyconfigs/easybuild-easyconfigs/easybuild/easyconfigs/l/libwebp/libwebp-1.2.4-GCCcore-11.3.0.eb: Parsing easyconfig file failed: invalid syntax (<string>, line 32)'

======================================================================
ERROR: test_pr_python_packages (test.easyconfigs.easyconfigs.EasyConfigTest)
Several checks for easyconfigs that install (bundles of) Python packages.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/easyconfigs/easyconfigs.py", line 972, in test_pr_python_packages
    for ec in self.changed_ecs:
  File "test/easyconfigs/easyconfigs.py", line 296, in changed_ecs
    self._get_changed_easyconfigs()
  File "test/easyconfigs/easyconfigs.py", line 234, in _get_changed_easyconfigs
    for ec in self.parsed_easyconfigs:
  File "test/easyconfigs/easyconfigs.py", line 271, in parsed_easyconfigs
    EasyConfigTest.parse_all_easyconfigs()
  File "test/easyconfigs/easyconfigs.py", line 202, in parse_all_easyconfigs
    cls._parsed_easyconfigs.extend(process_easyconfig(spec))
  File "/opt/hostedtoolcache/Python/2.7.18/x64/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 2071, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: 'Failed to process easyconfig /home/runner/work/easybuild-easyconfigs/easybuild-easyconfigs/easybuild/easyconfigs/l/libwebp/libwebp-1.2.4-GCCcore-11.3.0.eb: Parsing easyconfig file failed: invalid syntax (<string>, line 32)'

======================================================================
ERROR: test_pr_sanity_check_paths (test.easyconfigs.easyconfigs.EasyConfigTest)
Make sure a custom sanity_check_paths value is specified for easyconfigs that use a generic easyblock.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/easyconfigs/easyconfigs.py", line 1099, in test_pr_sanity_check_paths
    for ec in self.changed_ecs:
  File "test/easyconfigs/easyconfigs.py", line 296, in changed_ecs
    self._get_changed_easyconfigs()
  File "test/easyconfigs/easyconfigs.py", line 234, in _get_changed_easyconfigs
    for ec in self.parsed_easyconfigs:
  File "test/easyconfigs/easyconfigs.py", line 271, in parsed_easyconfigs
    EasyConfigTest.parse_all_easyconfigs()
  File "test/easyconfigs/easyconfigs.py", line 202, in parse_all_easyconfigs
    cls._parsed_easyconfigs.extend(process_easyconfig(spec))
  File "/opt/hostedtoolcache/Python/2.7.18/x64/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 2071, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: 'Failed to process easyconfig /home/runner/work/easybuild-easyconfigs/easybuild-easyconfigs/easybuild/easyconfigs/l/libwebp/libwebp-1.2.4-GCCcore-11.3.0.eb: Parsing easyconfig file failed: invalid syntax (<string>, line 32)'

======================================================================
ERROR: test_pr_sha256_checksums (test.easyconfigs.easyconfigs.EasyConfigTest)
Make sure changed easyconfigs have SHA256 checksums in place.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/easyconfigs/easyconfigs.py", line 933, in test_pr_sha256_checksums
    for ec in self.changed_ecs:
  File "test/easyconfigs/easyconfigs.py", line 296, in changed_ecs
    self._get_changed_easyconfigs()
  File "test/easyconfigs/easyconfigs.py", line 234, in _get_changed_easyconfigs
    for ec in self.parsed_easyconfigs:
  File "test/easyconfigs/easyconfigs.py", line 271, in parsed_easyconfigs
    EasyConfigTest.parse_all_easyconfigs()
  File "test/easyconfigs/easyconfigs.py", line 202, in parse_all_easyconfigs
    cls._parsed_easyconfigs.extend(process_easyconfig(spec))
  File "/opt/hostedtoolcache/Python/2.7.18/x64/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 2071, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: 'Failed to process easyconfig /home/runner/work/easybuild-easyconfigs/easybuild-easyconfigs/easybuild/easyconfigs/l/libwebp/libwebp-1.2.4-GCCcore-11.3.0.eb: Parsing easyconfig file failed: invalid syntax (<string>, line 32)'

======================================================================
ERROR: test_r_libs_site_env_var (test.easyconfigs.easyconfigs.EasyConfigTest)
Make sure $R_LIBS_SITE is being updated, rather than $R_LIBS.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/easyconfigs/easyconfigs.py", line 887, in test_r_libs_site_env_var
    for ec in self.parsed_easyconfigs:
  File "test/easyconfigs/easyconfigs.py", line 271, in parsed_easyconfigs
    EasyConfigTest.parse_all_easyconfigs()
  File "test/easyconfigs/easyconfigs.py", line 202, in parse_all_easyconfigs
    cls._parsed_easyconfigs.extend(process_easyconfig(spec))
  File "/opt/hostedtoolcache/Python/2.7.18/x64/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 2071, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: 'Failed to process easyconfig /home/runner/work/easybuild-easyconfigs/easybuild-easyconfigs/easybuild/easyconfigs/l/libwebp/libwebp-1.2.4-GCCcore-11.3.0.eb: Parsing easyconfig file failed: invalid syntax (<string>, line 32)'

======================================================================
ERROR: test_sanity_check_paths (test.easyconfigs.easyconfigs.EasyConfigTest)
Make sure specified sanity check paths adher to the requirements.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/easyconfigs/easyconfigs.py", line 871, in test_sanity_check_paths
    for ec in self.parsed_easyconfigs:
  File "test/easyconfigs/easyconfigs.py", line 271, in parsed_easyconfigs
    EasyConfigTest.parse_all_easyconfigs()
  File "test/easyconfigs/easyconfigs.py", line 202, in parse_all_easyconfigs
    cls._parsed_easyconfigs.extend(process_easyconfig(spec))
  File "/opt/hostedtoolcache/Python/2.7.18/x64/lib/python2.7/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 2071, in process_easyconfig
    raise EasyBuildError("Failed to process easyconfig %s: %s", spec, err.msg)
EasyBuildError: 'Failed to process easyconfig /home/runner/work/easybuild-easyconfigs/easybuild-easyconfigs/easybuild/easyconfigs/l/libwebp/libwebp-1.2.4-GCCcore-11.3.0.eb: Parsing easyconfig file failed: invalid syntax (<string>, line 32)'

----------------------------------------------------------------------
Ran 15369 tests in 486.979s

FAILED (errors=11)
ERROR: Not all tests were successful

bleep, bloop, I'm just a bot (boegelbot v20200716.01)
Please talk to my owner @boegel if you notice you me acting stupid),
or submit a pull request to https://github.com/boegel/boegelbot fix the problem.

@casparvl
Copy link
Copy Markdown
Contributor

casparvl commented Sep 30, 2022

In the conf-call of https://github.com/easybuilders/easybuild/wiki/Conference-call-notes-20220928 we discussed the FFmpeg version to use in 2022a. The decision was to stick with 4.4.2. Main reasons are to maintain compatibility with the version used in Paraview (which is not yet compatible with 5.X) and to avoid running into other software in the future that would need 4.4.2 (which would create more incompatible modules in the 2022a toolchain). Once 5.X is matured a bit, we can make the jump there in a subsequent toolchain.

@robogast Could you change the FFmpeg version used in this PR?

@robogast
Copy link
Copy Markdown
Contributor Author

robogast commented Oct 4, 2022

@casparvl done

@casparvl
Copy link
Copy Markdown
Contributor

casparvl commented Oct 5, 2022

@boegelbot please test @ generoso

@boegelbot
Copy link
Copy Markdown
Collaborator

@casparvl: Request for testing this PR well received on login1

PR test command 'EB_PR=16276 EB_ARGS= /opt/software/slurm/bin/sbatch --job-name test_PR_16276 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 9242

Test results coming soon (I hope)...

Details

- notification for comment with ID 1268561318 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@robogast
Copy link
Copy Markdown
Contributor Author

Test report by @robogast
FAILED
Build succeeded for 4 out of 5 (5 easyconfigs in total)
gcn2 - Linux RHEL 8.4, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz, 4 x NVIDIA NVIDIA A100-SXM4-40GB, 515.43.04, Python 3.6.8
See https://gist.github.com/de29bdb3d0b01a38df418938022e859a for a full test report.

@casparvl
Copy link
Copy Markdown
Contributor

Just checked with my colleague @robogast , the failure in OpenCV was due to the optarch we use on our system - the OpenCV EasyBlock doesn't support anything other than 'host' or 'generic' optimization by default (see here). We'll rerun the test with an optarch that does pass :)

@casparvl
Copy link
Copy Markdown
Contributor

@boegelbot please test @ generoso

@boegelbot
Copy link
Copy Markdown
Collaborator

@casparvl: Request for testing this PR well received on login1

PR test command 'EB_PR=16276 EB_ARGS= /opt/software/slurm/bin/sbatch --job-name test_PR_16276 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 9272

Test results coming soon (I hope)...

Details

- notification for comment with ID 1277591286 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Copy Markdown
Collaborator

Test report by @boegelbot
FAILED
Build succeeded for 2 out of 5 (5 easyconfigs in total)
cns2 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/7e47ce14f05ed263ccea0a051967350f for a full test report.

@casparvl
Copy link
Copy Markdown
Contributor

@boegelbot please test @ generoso

@boegelbot
Copy link
Copy Markdown
Collaborator

@casparvl: Request for testing this PR well received on login1

PR test command 'EB_PR=16276 EB_ARGS= /opt/software/slurm/bin/sbatch --job-name test_PR_16276 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 9285

Test results coming soon (I hope)...

Details

- notification for comment with ID 1279078386 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@casparvl
Copy link
Copy Markdown
Contributor

Test report by @casparvl
SUCCESS
Build succeeded for 5 out of 5 (5 easyconfigs in total)
software1.lisa.surfsara.nl - Linux debian 10.13, x86_64, Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz, Python 3.7.3
See https://gist.github.com/bdac2593b362312d186cc1a35981d50d for a full test report.

@boegelbot
Copy link
Copy Markdown
Collaborator

Test report by @boegelbot
SUCCESS
Build succeeded for 5 out of 5 (5 easyconfigs in total)
cns3 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/a93ae7ab9793615575ebec08b51c0a08 for a full test report.

@casparvl
Copy link
Copy Markdown
Contributor

Test report by @casparvl
SUCCESS
Build succeeded for 5 out of 5 (5 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.4, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/1a9c717022a5117ce2c40c0e33856f08 for a full test report.

@casparvl
Copy link
Copy Markdown
Contributor

Test report by @casparvl
SUCCESS
Build succeeded for 5 out of 5 (5 easyconfigs in total)
software1.lisa.surfsara.nl - Linux debian 10.13, x86_64, Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz, Python 3.7.3
See https://gist.github.com/2d36869f3d63bfc7b5f8b86a5328838a for a full test report.

casparvl
casparvl previously approved these changes Oct 14, 2022
Copy link
Copy Markdown
Contributor

@Micket Micket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@casparvl
Copy link
Copy Markdown
Contributor

Test report by @casparvl
SUCCESS
Build succeeded for 5 out of 5 (5 easyconfigs in total)
software1.lisa.surfsara.nl - Linux debian 10.13, x86_64, Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz, Python 3.7.3
See https://gist.github.com/dd1784682941a9e2f005bf245f51f1e5 for a full test report.

Copy link
Copy Markdown
Contributor

@casparvl casparvl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm!

@casparvl
Copy link
Copy Markdown
Contributor

Going in, thanks @robogast!

@casparvl casparvl merged commit f1b930b into easybuilders:develop Oct 14, 2022
@Micket
Copy link
Copy Markdown
Contributor

Micket commented Oct 14, 2022

Test report by @Micket
SUCCESS
Build succeeded for 16 out of 16 (5 easyconfigs in total)
vera-c2 - Linux Rocky Linux 8.5, x86_64, Intel Xeon Processor (Skylake), Python 3.6.8
See https://gist.github.com/3152184dcd4397e2b001c427da921e77 for a full test report.

@easybuilders easybuilders deleted a comment from boegelbot Oct 19, 2022
@branfosj
Copy link
Copy Markdown
Member

@boegelbot please test @ jsc-zen2
EB_ARGS='--optarch="" OpenCV-4.6.0-foss-2022a-contrib.eb'

@boegelbot
Copy link
Copy Markdown
Collaborator

@branfosj: Request for testing this PR well received on jsczen2l1.int.jsc-zen2.easybuild-test.cluster

PR test command 'EB_PR=16276 EB_ARGS="--optarch="" OpenCV-4.6.0-foss-2022a-contrib.eb" /opt/software/slurm/bin/sbatch --job-name test_PR_16276 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen2.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 1735

Test results coming soon (I hope)...

Details

- notification for comment with ID 1293393397 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Copy Markdown
Collaborator

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
jsczen2c1.int.jsc-zen2.easybuild-test.cluster - Linux Rocky Linux 8.5, x86_64, AMD EPYC 7742 64-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/fceac11496b356e961200324079b2253 for a full test report.

@paulmelis
Copy link
Copy Markdown
Contributor

Only seeing this now:

However, #15938, which was merged recently, uses an older FFmpeg, specifically because ParaView was #15938 (comment) with FFmpeg 5.0.1. My proposal is that we put this exception in the CI and stick with 5.0.1 as the main version for FFmpeg in this toolchain. I've never done this before though, I think it requires a framework change, right?

There seems to be a patch available to get ParaView to compile with FFmpeg 5.x, see https://gitlab.kitware.com/vtk/vtk/-/merge_requests/8863.patch. At least, that's the one that is used (plus a few unrelated ones) to build the Arch package for ParaView 5.10.1 against FFmpeg 5.1.2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants