Skip to content

HDDS-10985. EC Reconstruction failed because the size of currentChunks was not equal to checksumBlockDataChunks. #7009

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sodonnel merged 7 commits into from
Sep 11, 2024
Merged

HDDS-10985. EC Reconstruction failed because the size of currentChunks was not equal to checksumBlockDataChunks. #7009

Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
67d3d59
HDDS-10985. EC Reconstruction failed because the size of currentChunk…
slfan1989 Sep 5, 2024
d4cfed3
Merge branch 'apache:master' into HDDS-10985
slfan1989 Sep 6, 2024
2fcb565
HDDS-10985. EC Reconstruction failed because the size of currentChunk…
slfan1989 Sep 10, 2024
08223cc
HDDS-10985. EC Reconstruction failed because the size of currentChunk…
slfan1989 Sep 10, 2024
3b0849e
HDDS-10985. Fix CheckStyle.
slfan1989 Sep 10, 2024
6be8ba6
HDDS-10985. Fix Some Code.
slfan1989 Sep 10, 2024
92ec7a8
Merge branch 'apache:master' into HDDS-10985
slfan1989 Sep 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 27 additions & 1 deletion hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ECBlockOutputStream.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,8 +142,16 @@ ContainerCommandResponseProto> executePutBlock(boolean close,
}

if (checksumBlockData != null) {
List<ChunkInfo> currentChunks = getContainerBlockData().getChunksList();

// We need to determine the size of the chunklist
// for verification based on the size of the blockgrouplength.
int chunkSize = calcEffectiveChunkSize(blockData, totalNodes, blockGroupLength);
List<ChunkInfo> checksumBlockDataChunks = checksumBlockData.getChunks();
if (chunkSize > 0) {
checksumBlockDataChunks = checksumBlockData.getChunks().subList(0, chunkSize);
}

List<ChunkInfo> currentChunks = getContainerBlockData().getChunksList();

Preconditions.checkArgument(
currentChunks.size() == checksumBlockDataChunks.size(),
Expand Down Expand Up @@ -177,6 +185,24 @@ ContainerCommandResponseProto> executePutBlock(boolean close,
return executePutBlock(close, force, blockGroupLength);
}

private int calcEffectiveChunkSize(BlockData[] blockGroup,
Copy link
Contributor

@sodonnel sodonnel Sep 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this calculation really needed? We know the block size we are recovering (blockGroupLength) which is earlier calculated as the min(put_block_size) across the stripe.

Can we then simply divide blockGroupLength / chunkSize and round up to give the number of chunks, and from there know the number of checksum chunks we need to recover?

Copy link
Contributor Author

@slfan1989 slfan1989 Sep 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for reviewing the code! The calculation method you suggested is feasible, and the code will become more concise. I will try to make the modifications.

int replicaCount, long blockGroupLength) {
Preconditions.checkState(blockGroup.length == replicaCount);
int effectiveChunkSize = 0;
for (int i = 0; i < replicaCount; i++) {
if (blockGroup[i] == null) {
continue;
}
String lenStr = blockGroup[i].getMetadata().
get(OzoneConsts.BLOCK_GROUP_LEN_KEY_IN_PUT_BLOCK);
long calcBlockGroupLength = Long.parseLong(lenStr);
if (blockGroupLength == calcBlockGroupLength) {
effectiveChunkSize = Math.max(effectiveChunkSize, blockGroup[i].getChunks().size());
}
}
return effectiveChunkSize;
}

public CompletableFuture<ContainerProtos.
ContainerCommandResponseProto> executePutBlock(boolean close,
boolean force, long blockGroupLength, ByteString checksum)
Expand Down