Skip to content

HBASE-26340 - fix RegionSizeCalculator getLEngth to bytes instead of … #3872

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Apache9 merged 2 commits into from
Dec 18, 2021
Merged

HBASE-26340 - fix RegionSizeCalculator getLEngth to bytes instead of … #3872

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d9dcbd9
HBASE-26340 - fix RegionSizeCalculator getLEngth to bytes instead of …
nkalmar Nov 22, 2021
8ccf300
HBASE-26340 - convert from bytes to MB with min value instead of bool…
nkalmar Dec 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1431,6 +1431,8 @@ RegionLoad createRegionLoad(final HRegion r, RegionLoad.Builder regionLoadBldr,
int maxCompactedStoreFileRefCount = 0;
int storeUncompressedSizeMB = 0;
int storefileSizeMB = 0;
//HBASE-26340 Fix false "0" size under 1MB
boolean nonEmptyStoreExist = false;
int memstoreSizeMB = (int) (r.getMemStoreDataSize() / 1024 / 1024);
long storefileIndexSizeKB = 0;
int rootLevelIndexSizeKB = 0;
Expand All @@ -1449,6 +1451,10 @@ RegionLoad createRegionLoad(final HRegion r, RegionLoad.Builder regionLoadBldr,
currentMaxCompactedStoreFileRefCount);
storeUncompressedSizeMB += (int) (store.getStoreSizeUncompressed() / 1024 / 1024);
storefileSizeMB += (int) (store.getStorefilesSize() / 1024 / 1024);
//HBASE-26340 Fix false "0" size under 1MB
if(store.getStorefilesSize() > 0) {
nonEmptyStoreExist = true;
}
//TODO: storefileIndexSizeKB is same with rootLevelIndexSizeKB?
storefileIndexSizeKB += store.getStorefilesRootLevelIndexSize() / 1024;
CompactionProgress progress = store.getCompactionProgress();
Expand All @@ -1460,6 +1466,10 @@ RegionLoad createRegionLoad(final HRegion r, RegionLoad.Builder regionLoadBldr,
totalStaticIndexSizeKB += (int) (store.getTotalStaticIndexSize() / 1024);
totalStaticBloomSizeKB += (int) (store.getTotalStaticBloomSize() / 1024);
}
//HBASE-26340 Fix false "0" size under 1MB
if(storefileSizeMB < 1 && nonEmptyStoreExist) {

@Apache9 Apache9 Nov 24, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to apply this trick to other fields as well? Just introduce a round method

long round(long size, long unit) {
  // normally return size / unit, if size is smaller than unit but greater than 0, return 1
}

And all the size should be calculated by this method

@nkalmar nkalmar Dec 6, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

storefileSizeMB is integer, it is either 0 or 1 here (well, or any other positive integer, but staying at the less then 1MB example), and we go through all the storelist above, if we do the rounding there, we keep adding 1MB every time there is like 10KB of data, that would result in a false size as well. We need "nonEmptyStoreExist" to tell us if there were storefile anywhere, or else we'll just see 0 here and cant round up to 1 bc it was either really empty or size didn't reach the 1MB

@nkalmar nkalmar Dec 6, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could use a new variable "long storefileSizeMBTemp" and create a function "int round(long size, int unit)" but it just seems kind of a special case to create a function for it.

@Apache9 Apache9 Dec 6, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At region server side we should know the accurate size in bytes, so when calculating we could just cumulate the size in bytes, and at last we convert them to size in KB or size in MB or what ever unit. WDYT?

@nkalmar nkalmar Dec 7, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That works, sure, at least I think we are thinking the same when I said above we could use long to track "storefileSize(MB)(Temp)" - no need for Temp though just remove MB - but at the end we set this as MB just like before. I'll do the change today.

storefileSizeMB = 1;
}

HDFSBlocksDistribution hdfsBd = r.getHDFSBlocksDistribution();
float dataLocality = hdfsBd.getBlockLocalityIndex(serverName.getHostname());
Expand Down