add patch for GCCcore 11.1.0 + 11.2.0 to fix AVX2 bug by Flamefire · Pull Request #17135 · easybuilders/easybuild-easyconfigs

Flamefire · 2023-01-17T16:09:08Z

(created using eb --new-pr)

This fixes a serious bug when using AVX-2 intrinsics such as done by XNNPACK, a dependency of PyTorch and TensorFlow for quantization. It yields wrong results as the order of operands in the AVX vector is wrong when using an affected intrinsic and due to aliasing issues further bugs are possible based on "undefined behavior", i.e.: "everything" is possible.

See e.g. pytorch/pytorch#92246 for an actual bug caused by this and https://stackoverflow.com/a/72837992/1930508 for the post that led me to the solution in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99754

Flamefire · 2023-01-17T17:22:57Z

Test report by @Flamefire
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
taurusi8029 - Linux CentOS Linux 7.9.2009, x86_64, AMD EPYC 7352 24-Core Processor (zen2), 8 x NVIDIA NVIDIA A100-SXM4-40GB, 470.57.02, Python 2.7.5
See https://gist.github.com/59135df94c81dda7e568cef480c74405 for a full test report.

boegel · 2023-01-17T19:19:51Z

@boegelbot please test @ generoso
CORE_CNT=16
EB_ARGS="--installpath /tmp/$USER/pr17135"

boegelbot · 2023-01-17T19:20:08Z

@boegel: Request for testing this PR well received on login1

PR test command 'EB_PR=17135 EB_ARGS="--installpath /tmp/$USER/pr17135" EB_CONTAINER= /opt/software/slurm/bin/sbatch --job-name test_PR_17135 --ntasks="16" ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 10009

Test results coming soon (I hope)...

Details

- notification for comment with ID 1385929660 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2023-01-17T20:44:00Z

Test report by @boegelbot
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
cnx2 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/d429a0c1ef2c655439bfabd0741acd16 for a full test report.

boegel · 2023-01-17T20:49:48Z

@boegelbot please test @ jsc-zen2
CORE_CNT=16
EB_ARGS="--installpath /tmp/$USER/pr17135"

boegelbot · 2023-01-17T20:52:13Z

@boegel: Request for testing this PR well received on jsczen2l1.int.jsc-zen2.easybuild-test.cluster

PR test command 'EB_PR=17135 EB_ARGS="--installpath /tmp/$USER/pr17135" /opt/software/slurm/bin/sbatch --mem-per-cpu=4000M --job-name test_PR_17135 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen2.sh' executed!

exit code: 0
output:

Submitted batch job 2064

Test results coming soon (I hope)...

Details

- notification for comment with ID 1386033358 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2023-01-17T22:06:23Z

Test report by @boegelbot
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
jsczen2c1.int.jsc-zen2.easybuild-test.cluster - Linux Rocky Linux 8.5, x86_64, AMD EPYC 7742 64-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/2c944a092fd8dc026e87a582b97dfafa for a full test report.

Flamefire · 2023-01-18T03:10:59Z

Test report by @Flamefire
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
taurusa12 - Linux CentOS Linux 7.7.1908, x86_64, Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz (broadwell), 3 x NVIDIA GeForce GTX 1080 Ti, 460.32.03, Python 2.7.5
See https://gist.github.com/81b3e929f703166a945134aafcd92026 for a full test report.

boegel · 2023-01-18T11:43:27Z

Test report by @boegel
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
node3102.skitty.os - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (skylake_avx512), Python 3.6.8
See https://gist.github.com/1ebf34a4fd18ccbfaa198093c5af51ba for a full test report.

boegel

lgtm

boegel · 2023-01-18T11:55:11Z

Going in, thanks @Flamefire!

boegel · 2023-01-18T11:55:17Z

@boegelbot please test @ generoso
CORE_CNT=16

boegelbot · 2023-01-18T12:00:14Z

@boegel: Request for testing this PR well received on login1

PR test command 'EB_PR=17135 EB_ARGS= EB_CONTAINER= /opt/software/slurm/bin/sbatch --job-name test_PR_17135 --ntasks="16" ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 10022

Test results coming soon (I hope)...

Details

- notification for comment with ID 1386934278 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

verdurin · 2023-01-18T12:33:50Z

Test report by @verdurin
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
easybuild-c7.novalocal - Linux CentOS Linux 7.9.2009, x86_64, Intel Xeon Processor (Skylake, IBRS), Python 3.6.8
See https://gist.github.com/22e6c63afbe9373ad7e22024d74edd0f for a full test report.

boegelbot · 2023-01-18T13:25:14Z

Test report by @boegelbot
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
cnx1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz (haswell), Python 3.6.8
See https://gist.github.com/422199c50550bfc3dd5256e4538d33a2 for a full test report.

boegel · 2023-01-18T14:12:39Z

@boegelbot please test @ jsc-zen2
CORE_CNT=16

boegelbot · 2023-01-18T14:17:09Z

@boegel: Request for testing this PR well received on jsczen2l1.int.jsc-zen2.easybuild-test.cluster

PR test command 'EB_PR=17135 EB_ARGS= /opt/software/slurm/bin/sbatch --mem-per-cpu=4000M --job-name test_PR_17135 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen2.sh' executed!

exit code: 0
output:

Submitted batch job 2071

Test results coming soon (I hope)...

Details

- notification for comment with ID 1387138137 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

Flamefire · 2023-01-18T14:26:35Z

Test report by @Flamefire
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
taurusml27 - Linux RHEL 7.6, POWER, 8335-GTX (power9le), 6 x NVIDIA Tesla V100-SXM2-32GB, 440.64.00, Python 2.7.5
See https://gist.github.com/58329cafaeec06d03459706e79e50c5c for a full test report.

boegelbot · 2023-01-18T15:31:45Z

Test report by @boegelbot
SUCCESS
Build succeeded for 2 out of 2 (2 easyconfigs in total)
jsczen2c1.int.jsc-zen2.easybuild-test.cluster - Linux Rocky Linux 8.5, x86_64, AMD EPYC 7742 64-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/9637d697cb401fa49b08a80ceb8fbb5e for a full test report.

Fix AVX2 bug in GCC 11

0e3727b

Flamefire mentioned this pull request Jan 17, 2023

{devel}[foss/2021b] PyTorch v1.11.0 w/ Python 3.9.6 + CUDA 11.4.1 #16385

Closed

boegel changed the title ~~Fix AVX2 bug in GCC 11~~ add patch for GCCcore 11.1.0 + 11.2.0 to fix AVX2 bug Jan 17, 2023

boegel added the bug fix label Jan 17, 2023

boegel added this to the next release (4.7.1?) milestone Jan 17, 2023

boegel approved these changes Jan 18, 2023

View reviewed changes

boegel merged commit 8222a69 into easybuilders:develop Jan 18, 2023

Flamefire deleted the 20230117170900_new_pr_GCCcore1110 branch January 18, 2023 11:59

This was referenced Feb 3, 2023

{ai}[foss/2021b] PyTorch v1.12.1 w/ Python 3.9.6 w/ CUDA 11.4.1 #17154

Closed

{ai}[foss/2021b] PyTorch v1.12.1 w/ Python 3.9.6 #17153

Merged

Conversation

Flamefire commented Jan 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Flamefire commented Jan 17, 2023

Uh oh!

boegel commented Jan 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegelbot commented Jan 17, 2023

Uh oh!

boegelbot commented Jan 17, 2023

Uh oh!

boegel commented Jan 17, 2023

Uh oh!

boegelbot commented Jan 17, 2023

Uh oh!

boegelbot commented Jan 17, 2023

Uh oh!

Flamefire commented Jan 18, 2023

Uh oh!

boegel commented Jan 18, 2023

Uh oh!

boegel left a comment

Choose a reason for hiding this comment

Uh oh!

boegel commented Jan 18, 2023

Uh oh!

boegel commented Jan 18, 2023

Uh oh!

boegelbot commented Jan 18, 2023

Uh oh!

verdurin commented Jan 18, 2023

Uh oh!

boegelbot commented Jan 18, 2023

Uh oh!

boegel commented Jan 18, 2023

Uh oh!

boegelbot commented Jan 18, 2023

Uh oh!

Flamefire commented Jan 18, 2023

Uh oh!

boegelbot commented Jan 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Flamefire commented Jan 17, 2023 •

edited

Loading

boegel commented Jan 17, 2023 •

edited

Loading