Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

misc/cgo: intermittent failures in misc/cgo/testsanitizers on ppc64le builders #54645

Closed
laboger opened this issue Aug 24, 2022 · 3 comments
Closed

misc/cgo: intermittent failures in misc/cgo/testsanitizers on ppc64le builders #54645

laboger opened this issue Aug 24, 2022 · 3 comments
Labels
arch-ppc64x Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Go1.20

Comments

@laboger
Copy link
Contributor

laboger commented Aug 24, 2022 •
edited by rsc
Loading 

#!watchflakes
post <- pkg == "misc/cgo/testsanitizers" && goarch == "ppc64le"

The environment is ppc64le with investigation on various distros (thus various gcc versions). The builders have Debian buster which uses gcc 8.3.

Previously the testsanitizers tests were disabled on ppc64le due to the same random failures mentioned in #45040.

The test was re-enabled in https://go-review.googlesource.com/c/go/+/424534 with the thought that some signal fixes and other changes since then that might have fixed the problem. However the intermittent failures continue. Those intermittent failures are not easily reproducible on other machines and seem to be dependent on the gcc version and distro since some distros don't provide libtsan.a which the test requires.

Now that we have access to the builder machine and I have gomote access I can investigate further to understand why this is failing. One interesting note is that the randomize_va_space is 2 on the builders, indicating high ASLR randomization.

After some investigation, I found that the failure in this test is related to this issue that happened with the race detector: #35547. In that issue, it was noted that there can be failures on ppc64le with the thread sanitizer when ASLR is enabled. The race option uses code from LLVM to use the thread sanitizer, and that code disables ASLR when on ppc64e. Originally the seccomp permissions for the Docker container didn't allow the use of the syscall to disable ASLR. That problem has since been fixed.

The builders have gcc 8.3. Code was added in gcc-9 to do the same check as LLVM does, if running on ppc64le with TSAN then disable ASLR. That means the test should work if gcc is >=9 but could get random errors for <= gcc 8.

I'm going to add a check to this test when on ppc64le to verify the gcc version is >= 9, otherwise skip it and see if that will avoid the intermittent error.
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/425355 mentions this issue: misc/cgo: determine compiler version for tsan tests on ppc64le

@laboger
Copy link
Contributor Author

laboger commented Aug 24, 2022

This change should fix the problem in terms of running it only on systems where it makes sense. However there is also the question of whether the ppc64le builders should be updated to a distro with a newer gcc.

@seankhliao seankhliao added Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. arch-ppc64x labels Aug 24, 2022
gopherbot pushed a commit that referenced this issue Aug 25, 2022
@laboger
misc/cgo/testsanitizers: determine compiler version for tsan tests on…
d4ff25a 
… ppc64le

Some tests in misc/cgo/testsanitizers had been disabled on ppc64le
until recently, due to an intermittent error in the tsan tests,
with the goal of trying to understand the failure.

After further investigation, I found that the code for tsan within
gcc does not work consistently when ASLR is enabled on ppc64le. A
fix for that problem was integrated in gcc 9.

This adds a check to testsanitizers to determine the gcc compiler
version on ppc64le and skip the test if the version is too old.

A similar check is needed for asan too.

Updates #54645

Change-Id: I70717d1aa9e967cf1e871566e72b3862b91fea3f
Reviewed-on: https://go-review.googlesource.com/c/go/+/425355
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Ian Lance Taylor <[email protected]>
Reviewed-by: Archana Ravindar <[email protected]>
Reviewed-by: Cherry Mui <[email protected]>
Run-TryBot: Lynn Boger <[email protected]>
@pmur pmur mentioned this issue Aug 25, 2022
Closed
@seankhliao seankhliao added this to the Go1.20 milestone Aug 27, 2022
@laboger laboger mentioned this issue Aug 29, 2022
Closed
rajbarik pushed a commit to rajbarik/go that referenced this issue Sep 1, 2022
@laboger
misc/cgo/testsanitizers: determine compiler version for tsan tests on…
328dabd 
… ppc64le

Some tests in misc/cgo/testsanitizers had been disabled on ppc64le
until recently, due to an intermittent error in the tsan tests,
with the goal of trying to understand the failure.

After further investigation, I found that the code for tsan within
gcc does not work consistently when ASLR is enabled on ppc64le. A
fix for that problem was integrated in gcc 9.

This adds a check to testsanitizers to determine the gcc compiler
version on ppc64le and skip the test if the version is too old.

A similar check is needed for asan too.

Updates golang#54645

Change-Id: I70717d1aa9e967cf1e871566e72b3862b91fea3f
Reviewed-on: https://go-review.googlesource.com/c/go/+/425355
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Ian Lance Taylor <[email protected]>
Reviewed-by: Archana Ravindar <[email protected]>
Reviewed-by: Cherry Mui <[email protected]>
Run-TryBot: Lynn Boger <[email protected]>
@bcmills bcmills mentioned this issue Sep 20, 2022
Closed
@rsc rsc added this to Test Flakes Sep 21, 2022
@rsc rsc moved this to Active in Test Flakes Sep 21, 2022
@rsc rsc closed this as completed Sep 21, 2022
Repository owner moved this from Active to Done in Test Flakes Sep 21, 2022
@gopherbot gopherbot reopened this Sep 21, 2022
@gopherbot
Copy link
Contributor

Found new matching dashboard test flakes for:

#!watchflakes
post <- pkg == "misc/cgo/testsanitizers" && goarch == "ppc64le"
2022-08-19 20:04 linux-ppc64le-power9osu go@2cc6983a misc/cgo/testsanitizers.TestShared (log) 
--- FAIL: TestShared (0.00s)
    cshared_test.go:52: skipping msan_shared test on linux/ppc64le; -msan option is not supported.
    --- FAIL: TestShared/tsan_shared (4.36s)
        cshared_test.go:82: /workdir/tmp/TestShared1626997536/tsan_shared exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x03a9691c0000-0x03a9692c0000
2022-08-19 20:04 linux-ppc64le-power9osu go@2cc6983a misc/cgo/testsanitizers.TestTSAN (log) 
--- FAIL: TestTSAN (40.50s)
    --- FAIL: TestTSAN/tsan (1.88s)
        tsan_test.go:53: /workdir/tmp/TestTSAN976278364/tsan exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x72d878000000-0x72d878100000
    --- FAIL: TestTSAN/tsan7 (1.93s)
        tsan_test.go:53: /workdir/tmp/TestTSAN3034476637/tsan7 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x72b520800000-0x72b520900000
    --- FAIL: TestTSAN/tsan12 (2.04s)
        tsan_test.go:53: /workdir/tmp/TestTSAN2934132569/tsan12 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x747feef00000-0x747fef000000
    --- FAIL: TestTSAN/tsan11 (1.98s)
        tsan_test.go:53: /workdir/tmp/TestTSAN19486457/tsan11 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x7bc25f400000-0x7bc25f500000
    --- FAIL: TestTSAN/tsan4 (2.08s)
        tsan_test.go:53: /workdir/tmp/TestTSAN2410919835/tsan4 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x7c1769300000-0x7c1769400000
    --- FAIL: TestTSAN/tsan9 (2.22s)
        tsan_test.go:53: /workdir/tmp/TestTSAN1870835556/tsan9 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x787f5a200000-0x787f5a300000
    --- FAIL: TestTSAN/tsan6 (1.95s)
...
    --- FAIL: TestTSAN/tsan2 (1.74s)
        tsan_test.go:53: /workdir/tmp/TestTSAN3428070092/tsan2 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x77c8d6600000-0x77c8d6700000
    --- FAIL: TestTSAN/tsan3 (1.91s)
        tsan_test.go:53: /workdir/tmp/TestTSAN4111085149/tsan3 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x78620df00000-0x78620e000000
    --- FAIL: TestTSAN/tsan5 (2.17s)
        tsan_test.go:53: /workdir/tmp/TestTSAN3885609494/tsan5 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x782ee5a00000-0x782ee5b00000
go tool dist: Failed: exit status 1
2022-08-20 00:18 linux-ppc64le-buildlet go@28afa5b1 misc/cgo/testsanitizers.TestShared (log) 
--- FAIL: TestShared (0.00s)
    cshared_test.go:52: skipping msan_shared test on linux/ppc64le; -msan option is not supported.
    --- FAIL: TestShared/tsan_shared (2.77s)
        cshared_test.go:82: /workdir/tmp/TestShared173076323/tsan_shared exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x073448ea0000-0x073448fa0000
2022-08-20 00:18 linux-ppc64le-buildlet go@28afa5b1 misc/cgo/testsanitizers.TestTSAN (log) 
--- FAIL: TestTSAN (24.60s)
    --- FAIL: TestTSAN/tsan (1.20s)
        tsan_test.go:53: /workdir/tmp/TestTSAN2884843627/tsan exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x7e4298200000-0x7e4298300000
    --- FAIL: TestTSAN/tsan3 (1.21s)
        tsan_test.go:53: /workdir/tmp/TestTSAN1032707586/tsan3 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x77c997e00000-0x77c997f00000
    --- FAIL: TestTSAN/tsan7 (1.24s)
        tsan_test.go:53: /workdir/tmp/TestTSAN2754844792/tsan7 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x74afcd400000-0x74afcd500000
    --- FAIL: TestTSAN/tsan4 (1.34s)
        tsan_test.go:53: /workdir/tmp/TestTSAN855020177/tsan4 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x748020700000-0x748020800000
    --- FAIL: TestTSAN/tsan10 (1.25s)
        tsan_test.go:53: /workdir/tmp/TestTSAN1158362804/tsan10 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x73a386400000-0x73a386500000
    --- FAIL: TestTSAN/tsan11 (1.42s)
        tsan_test.go:53: /workdir/tmp/TestTSAN4281520402/tsan11 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x7bf06dd00000-0x7bf06de00000
    --- FAIL: TestTSAN/tsan12 (1.43s)
...
    --- FAIL: TestTSAN/tsan2 (1.21s)
        tsan_test.go:53: /workdir/tmp/TestTSAN140148249/tsan2 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x79b708300000-0x79b708400000
    --- FAIL: TestTSAN/tsan8 (1.40s)
        tsan_test.go:53: /workdir/tmp/TestTSAN3508573628/tsan8 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x72adce000000-0x72adce100000
    --- FAIL: TestTSAN/tsan5 (1.42s)
        tsan_test.go:53: /workdir/tmp/TestTSAN23713737/tsan5 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x7b2440600000-0x7b2440700000
go tool dist: Failed: exit status 1
2022-08-24 12:12 linux-ppc64le-buildlet go@b5a9459c misc/cgo/testsanitizers.TestShared (log) 
--- FAIL: TestShared (0.00s)
    cshared_test.go:52: skipping msan_shared test on linux/ppc64le; -msan option is not supported.
    --- FAIL: TestShared/tsan_shared (2.64s)
        cshared_test.go:82: /workdir/tmp/TestShared848610845/tsan_shared exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x0d4223380000-0x0d4223480000
2022-08-24 12:12 linux-ppc64le-buildlet go@b5a9459c misc/cgo/testsanitizers.TestTSAN (log) 
--- FAIL: TestTSAN (25.26s)
    --- FAIL: TestTSAN/tsan10 (1.28s)
        tsan_test.go:53: /workdir/tmp/TestTSAN1253881176/tsan10 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x73db2eb00000-0x73db2ec00000
    --- FAIL: TestTSAN/tsan6 (1.28s)
        tsan_test.go:53: /workdir/tmp/TestTSAN2520017262/tsan6 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x7e6058900000-0x7e6058a00000
    --- FAIL: TestTSAN/tsan7 (1.30s)
        tsan_test.go:53: /workdir/tmp/TestTSAN2125904041/tsan7 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x7aa0e9a00000-0x7aa0e9b00000
    --- FAIL: TestTSAN/tsan11 (1.31s)
        tsan_test.go:53: /workdir/tmp/TestTSAN712299795/tsan11 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x74b428600000-0x74b428700000
    --- FAIL: TestTSAN/tsan12 (1.36s)
        tsan_test.go:53: /workdir/tmp/TestTSAN75467294/tsan12 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x71f29c600000-0x71f29c700000
    --- FAIL: TestTSAN/tsan4 (1.38s)
        tsan_test.go:53: /workdir/tmp/TestTSAN648519663/tsan4 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x7c2a5f900000-0x7c2a5fa00000
    --- FAIL: TestTSAN/tsan5 (1.57s)
...
    --- FAIL: TestTSAN/tsan2 (1.20s)
        tsan_test.go:53: /workdir/tmp/TestTSAN123276469/tsan2 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x73d9edd00000-0x73d9ede00000
    --- FAIL: TestTSAN/tsan3 (1.26s)
        tsan_test.go:53: /workdir/tmp/TestTSAN2342781266/tsan3 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x78bf9c900000-0x78bf9ca00000
    --- FAIL: TestTSAN/tsan9 (1.41s)
        tsan_test.go:53: /workdir/tmp/TestTSAN2129967694/tsan9 exited with exit status 66
            FATAL: ThreadSanitizer: unexpected memory mapping 0x7bb146000000-0x7bb146100000
go tool dist: Failed: exit status 1

— watchflakes

@rsc rsc closed this as completed Sep 21, 2022
@golang golang locked and limited conversation to collaborators Sep 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-ppc64x Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
Test Flakes
Archived in project
Milestone
Go1.20
Development

No branches or pull requests
4 participants
@rsc @gopherbot @laboger @seankhliao