Skip to content

HDDS-13169. Intermittent failure in testSnapshotOperationsNotBlockedDuringCompaction #8553

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adoroszlai merged 6 commits into from
Jun 5, 2025
Merged

HDDS-13169. Intermittent failure in testSnapshotOperationsNotBlockedDuringCompaction #8553

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8a67ea9
HDDS-13169. Intermittent failure in testSnapshotOperationsNotBlockedD…
peterxcli Jun 3, 2025
ee9c260
Modify junit.sh to run 'clean verify' for subsequent iterations in te…
peterxcli Jun 3, 2025
0cf95b3
Update intermittent-test-check.yml and junit.sh to use TARGET_DIR for…
peterxcli Jun 3, 2025
2373cb9
Refactor intermittent-test-check.yml and junit.sh to replace TARGET_D…
peterxcli Jun 4, 2025
df64f50
Correct output log paths in intermittent-test-check.yml
peterxcli Jun 4, 2025
028f77e
ensure text processing for grep commands in intermittent-test-check.yml
peterxcli Jun 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions hadoop-ozone/dev-support/checks/junit.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,13 @@ for i in $(seq 1 ${ITERATIONS}); do
mkdir -p "${REPORT_DIR}"
fi

mvn ${MAVEN_OPTIONS} -Dmaven-surefire-plugin.argLineAccessArgs="${OZONE_MODULE_ACCESS_ARGS}" "$@" verify \
| tee "${REPORT_DIR}/output.log"
if [[ $i -eq 1 ]]; then
mvn ${MAVEN_OPTIONS} -Dmaven-surefire-plugin.argLineAccessArgs="${OZONE_MODULE_ACCESS_ARGS}" "$@" verify \
| tee "${REPORT_DIR}/output.log"
else
mvn ${MAVEN_OPTIONS} -Dmaven-surefire-plugin.argLineAccessArgs="${OZONE_MODULE_ACCESS_ARGS}" "$@" clean verify \
| tee "${REPORT_DIR}/output.log"
fi

@adoroszlai adoroszlai Jun 3, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be done unconditionally.

  • Project may be already built if OZONE_REPO_CACHED == false, in that case even first iteration can run into the error.
  • If project is not yet built, clean does not make much difference.

@peterxcli peterxcli Jun 3, 2025

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to replace target with a different name. If the module isn't specified, the target directory will get deleted.

I noticed this issue in the following workflow run:
https://github.com/peterxcli/ozone/actions/runs/15413751760/job/43371703642#step:6:20

@adoroszlai adoroszlai Jun 3, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, clean will remove $REPORT_DIR. We should clean separately, before the verify step.

@peterxcli peterxcli Jun 3, 2025 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The _post_process.sh script collect the result form the target dir too, if previous result is cleaned then we might ignore some error.

I think we can use another directory to collect the test result: 0cf95b3

@peterxcli peterxcli Jun 3, 2025 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing it: https://github.com/peterxcli/ozone/actions/runs/15422920154

irc=$?

# shellcheck source=hadoop-ozone/dev-support/checks/_mvn_unit_report.sh
Expand Down
6 changes: 6 additions & 0 deletions ...ne/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestSnapshotCache.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -384,6 +384,7 @@ void testSnapshotOperationsNotBlockedDuringCompaction() throws IOException, Inte
final Semaphore compactionLock = new Semaphore(1);
final AtomicBoolean table1Compacting = new AtomicBoolean(false);
final AtomicBoolean table1CompactedFinish = new AtomicBoolean(false);
final AtomicBoolean table2CompactedFinish = new AtomicBoolean(false);
org.apache.hadoop.hdds.utils.db.DBStore store1 = snapshot1.get().getMetadataManager().getStore();
doAnswer(invocation -> {
table1Compacting.set(true);
Expand All @@ -392,6 +393,10 @@ void testSnapshotOperationsNotBlockedDuringCompaction() throws IOException, Inte
table1CompactedFinish.set(true);
return null;
}).when(store1).compactTable("table1");
doAnswer(invocation -> {
table2CompactedFinish.set(true);
return null;
}).when(store1).compactTable("table2");
compactionLock.acquire();

final UUID dbKey2 = UUID.randomUUID();
Expand All @@ -411,6 +416,7 @@ void testSnapshotOperationsNotBlockedDuringCompaction() throws IOException, Inte
assertFalse(table1CompactedFinish.get());
compactionLock.release();
GenericTestUtils.waitFor(() -> table1CompactedFinish.get(), 50, 3000);
GenericTestUtils.waitFor(() -> table2CompactedFinish.get(), 50, 3000);

verify(store1, times(1)).compactTable("table1");
verify(store1, times(1)).compactTable("table2");
Expand Down