Skip to content

Commit

Permalink
Fix the overflow issue when loading the large dictionary into the buf…
Browse files Browse the repository at this point in the history
…fer (apache#6476)

Currently, we allow to generate the large dictionary (>2GB) while loading
segment fails due to overflow. This pr fixes the issue.
  • Loading branch information
Seunghyun Lee authored Jan 25, 2021
1 parent e209230 commit e5bf05b
Showing 1 changed file with 3 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ private void checkKeyNotPresent(IndexKey key) {
}
}

private void validateMagicMarker(PinotDataBuffer buffer, int startOffset) {
private void validateMagicMarker(PinotDataBuffer buffer, long startOffset) {
long actualMarkerValue = buffer.getLong(startOffset);
if (actualMarkerValue != MAGIC_MARKER) {
LOGGER.error("Missing magic marker in index file: {} at position: {}", indexFile, startOffset);
Expand Down Expand Up @@ -292,10 +292,10 @@ private void mapAndSliceFile(SortedMap<Long, IndexEntry> startOffsets, List<Long
}
allocBuffers.add(buffer);

int prevSlicePoint = 0;
long prevSlicePoint = 0;
for (Long fileOffset : offsetAccum) {
IndexEntry entry = startOffsets.get(fileOffset);
int endSlicePoint = prevSlicePoint + (int) entry.size;
long endSlicePoint = prevSlicePoint + entry.size;
validateMagicMarker(buffer, prevSlicePoint);
PinotDataBuffer viewBuffer = buffer.view(prevSlicePoint + MAGIC_MARKER_SIZE_BYTES, endSlicePoint);
entry.buffer = viewBuffer;
Expand Down

0 comments on commit e5bf05b

Please sign in to comment.