Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky CI tests FAIL: //src/test/java/com/google/devtools/build/lib/actions:ActionsTests #21471

Closed
sgowroji opened this issue Feb 22, 2024 · 4 comments
Closed

Flaky CI tests FAIL: //src/test/java/com/google/devtools/build/lib/actions:ActionsTests #21471

sgowroji opened this issue Feb 22, 2024 · 4 comments
Assignees
@tjgq
Labels
flaky test P2 We'll consider working on this in future. (Assignee optional) team-Local-Exec Issues and PRs for the Execution (Local) team type: bug

Comments

@sgowroji
Copy link
Member

sgowroji commented Feb 22, 2024 •
edited
Loading

CI: https://buildkite.com/bazel/bazel-bazel/builds/26954#018dd082-ea46-4543-8c3f-ae773a71ecd9

Platform: Ubuntu

Logs:

FAIL: //src/test/java/com/google/devtools/build/lib/actions:ActionsTests (see /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/execroot/_main/bazel-out/k8-fastbuild/testlogs/src/test/java/com/google/devtools/build/lib/actions/ActionsTests/test.log)
FAILED: //src/test/java/com/google/devtools/build/lib/actions:ActionsTests (Summary)
      /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/execroot/_main/bazel-out/k8-fastbuild/testlogs/src/test/java/com/google/devtools/build/lib/actions/ActionsTests/test.log
      /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/execroot/_main/bazel-out/k8-fastbuild/testlogs/src/test/java/com/google/devtools/build/lib/actions/ActionsTests/test_attempts/attempt_1.log
      /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/execroot/_main/bazel-out/k8-fastbuild/testlogs/src/test/java/com/google/devtools/build/lib/actions/ActionsTests/test_attempts/attempt_2.log
(12:15:02) [12,068 / 12,174] 693 / 756 tests, 1 failed; 30 actions, 13 running; last test: .../google/devtools/build/lib/actions:ActionsTests

CC Greenteam @fweikert @meteorcloudy
@sgowroji sgowroji added type: bug untriaged team-Local-Exec Issues and PRs for the Execution (Local) team flaky test labels Feb 22, 2024
@meteorcloudy
Copy link
Member

@tjgq Can you help triage this one?

@joeleba joeleba added P2 We'll consider working on this in future. (Assignee optional) and removed untriaged labels Feb 27, 2024
@Wyverald
Copy link
Member

@bazel-io flag 7.1.1

@bazel-io bazel-io added the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Mar 13, 2024
@Wyverald
Copy link
Member

@bazel-io fork 7.1.1

@bazel-io bazel-io removed the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Mar 13, 2024
@bazel-io bazel-io mentioned this issue Mar 13, 2024
Closed
@Wyverald
Copy link
Member

(forking so that we deflake the release branch as well. release-7.2.0 will likely be based off release-7.1.1)

bazel-io pushed a commit to bazel-io/bazel that referenced this issue Mar 13, 2024
@tjgq @bazel-io
Also inject a failure for createWritableDirectory when testing that A…
b44847f 
…ctionOutputDirectoryHelper propagates exceptions.

Failure to do causes a flaky test failure when simulating an IOException while the `knownDirectories` cache is disabled. The reason is that `ActionOutputDirectoryHelper#forceCreateDirectoryAndParents` calls either `createDirectory` or `createWritableDirectory`, depending on whether the parent directory is in the cache or not. However, "disabling" the cache sets its size to zero, which doesn't prevent insertion; rather, it causes inserted entries to be almost immediately deleted by a background thread. Thus, if only one of `createDirectory` and `createWritableDirectory` throws, the test outcome depends on how fast the background thread runs.

(We could make the zero-sized cache an actual no-op, but it's unlikely that anyone would want to set it to zero outside of test code, so it's not worth the trouble.)

Fixes bazelbuild#21471.

PiperOrigin-RevId: 615513608
Change-Id: Id2247596c6af0e5d5142072de8309227a1d1cbd1
@bazel-io bazel-io mentioned this issue Mar 13, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Mar 13, 2024
@bazel-io @tjgq
[7.1.1] Also inject a failure for createWritableDirectory when testin…
7c7c5c3 
…g that ActionOutputDirectoryHelper propagates exceptions. (#21683)

Failure to do causes a flaky test failure when simulating an IOException
while the `knownDirectories` cache is disabled. The reason is that
`ActionOutputDirectoryHelper#forceCreateDirectoryAndParents` calls
either `createDirectory` or `createWritableDirectory`, depending on
whether the parent directory is in the cache or not. However,
"disabling" the cache sets its size to zero, which doesn't prevent
insertion; rather, it causes inserted entries to be almost immediately
deleted by a background thread. Thus, if only one of `createDirectory`
and `createWritableDirectory` throws, the test outcome depends on how
fast the background thread runs.

(We could make the zero-sized cache an actual no-op, but it's unlikely
that anyone would want to set it to zero outside of test code, so it's
not worth the trouble.)

Fixes #21471.

Commit
f924338

PiperOrigin-RevId: 615513608
Change-Id: Id2247596c6af0e5d5142072de8309227a1d1cbd1

Co-authored-by: Googler <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tjgq tjgq

Labels
flaky test P2 We'll consider working on this in future. (Assignee optional) team-Local-Exec Issues and PRs for the Execution (Local) team type: bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@Wyverald @meteorcloudy @tjgq @joeleba @bazel-io @sgowroji