Skip to content

HDDS-12594. Optimize replica checksum verifier #8151

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adoroszlai merged 4 commits into from
Mar 29, 2025
Merged

HDDS-12594. Optimize replica checksum verifier #8151

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d6f9666
add integration test for checksums
ptlrs Mar 21, 2025
99a9212
HDDS-12594. Optimize replica checksum verifier
ptlrs Mar 25, 2025
3271d9f
Delete hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/…
ptlrs Mar 25, 2025
033b08d
HDDS-12661. fix findbugs error
ptlrs Mar 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions hadoop-ozone/dist/src/main/smoketest/debug/ozone-debug-corrupt-block.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Test ozone debug checksums with corrupt block replica
Set Test Variable ${DIR} ${directory}

${count_files} = Count Files In Directory ${directory}
Should Be Equal As Integers ${count_files} 7
Should Be Equal As Integers ${count_files} 1

${json} = Read Replicas Manifest
${md5sum} = Execute md5sum ${TEMP_DIR}/${TESTFILE} | awk '{print $1}'
Expand All @@ -41,9 +41,6 @@ Test ozone debug checksums with corrupt block replica
${datanode} = Set Variable ${json}[blocks][0][replicas][${replica}][hostname]

IF '${datanode}' == '${CORRUPT_DATANODE}'
Verify Corrupt Replica ${json} ${replica} ${md5sum}
Should Contain ${json}[blocks][0][replicas][${replica}][exception] Checksum mismatch
ELSE
Verify Healthy Replica ${json} ${replica} ${md5sum}
END
END
9 changes: 1 addition & 8 deletions hadoop-ozone/dist/src/main/smoketest/debug/ozone-debug-dead-datanode.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,4 @@ Test ozone debug checksums with one datanode DEAD
Set Test Variable ${DIR} ${directory}

${count_files} = Count Files In Directory ${directory}
Should Be Equal As Integers ${count_files} 5

${json} = Read Replicas Manifest
${md5sum} = Execute md5sum ${TEMP_DIR}/${TESTFILE} | awk '{print $1}'

FOR ${replica} IN RANGE 2
Verify Healthy Replica ${json} ${replica} ${md5sum}
END
Should Be Equal As Integers ${count_files} 1
4 changes: 1 addition & 3 deletions hadoop-ozone/dist/src/main/smoketest/debug/ozone-debug-stale-datanode.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Test ozone debug checksums with one datanode STALE
Set Test Variable ${DIR} ${directory}

${count_files} = Count Files In Directory ${directory}
Should Be Equal As Integers ${count_files} 7
Should Be Equal As Integers ${count_files} 1

${json} = Read Replicas Manifest
${md5sum} = Execute md5sum ${TEMP_DIR}/${TESTFILE} | awk '{print $1}'
Expand All @@ -42,7 +42,5 @@ Test ozone debug checksums with one datanode STALE

IF '${datanode}' == '${STALE_DATANODE}'
Verify Stale Replica ${json} ${replica}
ELSE
Verify Healthy Replica ${json} ${replica} ${md5sum}
END
END
31 changes: 7 additions & 24 deletions hadoop-ozone/dist/src/main/smoketest/debug/ozone-debug-tests-ec3-2.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,52 +52,35 @@ Create EC key
Create EC key 1048576 1
${directory} = Execute replicas verify checksums CLI tool
${count_files} = Count Files In Directory ${directory}
Should Be Equal As Integers ${count_files} 6
${sum_size} = Evaluate 1048576 * 3
Verify Healthy EC Replica ${directory} 1 ${sum_size}
Should Be Equal As Integers ${count_files} 1

2 data blocks
Create EC key 1048576 2
${directory} = Execute replicas verify checksums CLI tool
${sum_size} = Evaluate 1048576 * 4
${count_files} = Count Files In Directory ${directory}
Should Be Equal As Integers ${count_files} 6
Verify Healthy EC Replica ${directory} 1 ${sum_size}
Should Be Equal As Integers ${count_files} 1

3 data blocks
Create EC key 1048576 3
${directory} = Execute replicas verify checksums CLI tool
${sum_size} = Evaluate 1048576 * 5
${count_files} = Count Files In Directory ${directory}
Should Be Equal As Integers ${count_files} 6
Verify Healthy EC Replica ${directory} 1 ${sum_size}

Should Be Equal As Integers ${count_files} 1

3 data blocks and partial stripe
Create EC key 1000000 4
${directory} = Execute replicas verify checksums CLI tool
${count_files} = Count Files In Directory ${directory}
${sum_size} = Evaluate 1048576 * 5
${sum_size_last_stripe} = Evaluate ((1000000 * 4) % 1048576) * 3
Should Be Equal As Integers ${count_files} 11
Verify Healthy EC Replica ${directory} 1 ${sum_size}
Verify Healthy EC Replica ${directory} 2 ${sum_size_last_stripe}
Should Be Equal As Integers ${count_files} 1

4 data blocks and partial stripe
Create EC key 1000000 5
${directory} = Execute replicas verify checksums CLI tool
${count_files} = Count Files In Directory ${directory}
${sum_size} = Evaluate 1048576 * 5
${sum_size_last_stripe} = Evaluate 1048576 * 3 + ((1000000 * 5) % 1048576)
Should Be Equal As Integers ${count_files} 11
Verify Healthy EC Replica ${directory} 1 ${sum_size}
Verify Healthy EC Replica ${directory} 2 ${sum_size_last_stripe}
Should Be Equal As Integers ${count_files} 1

6 data blocks
Create EC key 1048576 6
${directory} = Execute replicas verify checksums CLI tool
${count_files} = Count Files In Directory ${directory}
${sum_size} = Evaluate 1048576 * 5
Should Be Equal As Integers ${count_files} 11
FOR ${block} IN RANGE 1 3
Verify Healthy EC Replica ${directory} ${block} ${sum_size}
END
Should Be Equal As Integers ${count_files} 1
36 changes: 9 additions & 27 deletions hadoop-ozone/dist/src/main/smoketest/debug/ozone-debug-tests-ec6-3.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,66 +47,48 @@ Create EC key
Create EC key 1048576 1
${directory} = Execute replicas verify checksums CLI tool
${count_files} = Count Files In Directory ${directory}
Should Be Equal As Integers ${count_files} 10
${sum_size} = Evaluate 1048576 * 4
Verify Healthy EC Replica ${directory} 1 ${sum_size}
Should Be Equal As Integers ${count_files} 1

2 data blocks
Create EC key 1048576 2
${directory} = Execute replicas verify checksums CLI tool
${sum_size} = Evaluate 1048576 * 5
${count_files} = Count Files In Directory ${directory}
Should Be Equal As Integers ${count_files} 10
Verify Healthy EC Replica ${directory} 1 ${sum_size}
Should Be Equal As Integers ${count_files} 1

3 data blocks
Create EC key 1048576 3
${directory} = Execute replicas verify checksums CLI tool
${sum_size} = Evaluate 1048576 * 6
${count_files} = Count Files In Directory ${directory}
Should Be Equal As Integers ${count_files} 10
Verify Healthy EC Replica ${directory} 1 ${sum_size}

Should Be Equal As Integers ${count_files} 1

4 data blocks
Create EC key 1048576 4
${directory} = Execute replicas verify checksums CLI tool
${count_files} = Count Files In Directory ${directory}
${sum_size} = Evaluate 1048576 * 7
Should Be Equal As Integers ${count_files} 10
Verify Healthy EC Replica ${directory} 1 ${sum_size}
Should Be Equal As Integers ${count_files} 1

5 data blocks
Create EC key 1048576 5
${directory} = Execute replicas verify checksums CLI tool
${count_files} = Count Files In Directory ${directory}
${sum_size} = Evaluate 1048576 * 8
Should Be Equal As Integers ${count_files} 10
Verify Healthy EC Replica ${directory} 1 ${sum_size}
Should Be Equal As Integers ${count_files} 1

6 data blocks
Create EC key 1048576 6
${directory} = Execute replicas verify checksums CLI tool
${count_files} = Count Files In Directory ${directory}
${sum_size} = Evaluate 1048576 * 9
Should Be Equal As Integers ${count_files} 10
Verify Healthy EC Replica ${directory} 1 ${sum_size}
Should Be Equal As Integers ${count_files} 1

6 data blocks and partial stripe
Create EC key 1000000 7
${directory} = Execute replicas verify checksums CLI tool
${count_files} = Count Files In Directory ${directory}
${sum_size} = Evaluate 1048576 * 9
${sum_size_last_stripe} = Evaluate ((1000000 * 7) % 1048576) * 4
Should Be Equal As Integers ${count_files} 19
Verify Healthy EC Replica ${directory} 1 ${sum_size}
Verify Healthy EC Replica ${directory} 2 ${sum_size_last_stripe}
Should Be Equal As Integers ${count_files} 1

7 data blocks and partial stripe
Create EC key 1000000 8
${directory} = Execute replicas verify checksums CLI tool
${count_files} = Count Files In Directory ${directory}
${sum_size} = Evaluate 1048576 * 9
${sum_size_last_stripe} = Evaluate 1048576 * 4 + ((1000000 * 8) % 1048576)
Should Be Equal As Integers ${count_files} 19
Verify Healthy EC Replica ${directory} 1 ${sum_size}
Verify Healthy EC Replica ${directory} 2 ${sum_size_last_stripe}
Should Be Equal As Integers ${count_files} 1
9 changes: 1 addition & 8 deletions hadoop-ozone/dist/src/main/smoketest/debug/ozone-debug-tests.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,14 +41,7 @@ Test ozone debug read-replicas
Set Test Variable ${DIR} ${directory}

${count_files} = Count Files In Directory ${directory}
Should Be Equal As Integers ${count_files} 7

${json} = Read Replicas Manifest
${md5sum} = Execute md5sum ${TEMP_DIR}/${TESTFILE} | awk '{print $1}'

FOR ${replica} IN RANGE 3
Verify Healthy Replica ${json} ${replica} ${md5sum}
END
Should Be Equal As Integers ${count_files} 1


Test ozone debug version
Expand Down
6 changes: 0 additions & 6 deletions hadoop-ozone/dist/src/main/smoketest/debug/ozone-debug.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,15 +87,9 @@ Verify Stale Replica
[arguments] ${json} ${replica}

FOR ${block} IN RANGE 2
${n} = Evaluate ${block} + 1
${datanode} = Set Variable ${json}[blocks][${block}][replicas][${replica}][hostname]
${filename} = Set Variable ${DIR}/${TESTFILE}_block${n}_${datanode}

IF '${datanode}' == '${STALE_DATANODE}'
File Should Be Empty ${filename}
Should Contain ${json}[blocks][${block}][replicas][${replica}][exception] UNAVAILABLE
ELSE
${filesize} = Get File Size ${filename}
Should Be Equal As Integers ${json}[blocks][${block}][length] ${filesize}
END
END
Loading