Skip to content

HDDS-5449 Recon namespace summary 'du' information should return replicated size of a key #2489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
smengcl merged 8 commits into from
Aug 11, 2021
Merged

HDDS-5449 Recon namespace summary 'du' information should return replicated size of a key #2489

smengcl merged 8 commits into from
Aug 11, 2021
+492 −37

Conversation

@yuangu002
Copy link
Contributor

@yuangu002 yuangu002 commented Aug 3, 2021

What changes were proposed in this pull request?

In addition to actual key size, Recon 'du' should also return the replicated size of a key.
For example, for a 3 way replicated key of size = 1MB, Recon 'du' should return '1MB' as the size of the key, and '3MB' as the replicated size if the key is fully replicated.

[Refactoring]

  1. DU response now shows a "structure"
  2. DU key response shouldn't have subpaths

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5449

How was this patch tested?

Since this is an improvement/refactoring feature on top of HDDS-5378, no additional unit test is implemented. I tested remotely on the cluster endpoint with the following configs:
Screen Shot 2021-08-03 at 1 46 36 PM
All keys are three-way replicated except key3

[Root DU]
http://localhost:9888/api/v1/namespace/du?path=/&replica=true
root du -rep

[Volume DU]
http://localhost:9888/api/v1/namespace/du?path=/vol&replica=true
vol du -rep

[Bucket DU] &files=true enabled
http://localhost:9888/api/v1/namespace/du?path=/vol/bucket&files=true&replica=true
bucket du -rep

[Directory DU] &files=true enabled
http://localhost:9888/api/v1/namespace/du?path=/vol/bucket/dir1&files=true&replica=true
dir1 du -rep
(Key3 is one-way replicated.)

[Nested Directory DU]
http://localhost:9888/api/v1/namespace/du?path=/vol/bucket/dir1/dir2&files=true&replica=true
dir2 du -rep

[Key DU]
http://localhost:9888/api/v1/namespace/du?path=/vol/bucket/dir1/dir2/dir3/key6&files=true&replica=true
key du -rep

@yuangu002
1. Add DU data size with replica and QueryParam options for showing D…
22cdbd5 
…U and listing keys as subpath. 2. Refactored DU response to reflect a hierarchy.
@yuangu002
Copy link
Contributor Author

yuangu002 commented Aug 3, 2021 •
edited
Loading

cc @smengcl @avijayanhwx. Please review.

@yuangu002 yuangu002 changed the title Recon namespace summary 'du' information should return replicated size of a key HDDS-5449 Recon namespace summary 'du' information should return replicated size of a key Aug 3, 2021
avijayanhwx
avijayanhwx reviewed Aug 4, 2021
View reviewed changes
Copy link
Contributor

@avijayanhwx avijayanhwx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @yuangu002. A few comments inline.

smengcl
smengcl reviewed Aug 5, 2021
View reviewed changes
@yuangu002
Copy link
Contributor Author

yuangu002 commented Aug 7, 2021 •
edited
Loading

[Fix]

  1. Addressed comments.
  2. Optimized listKeys by seek() and DFS. No full iteration on FileTable except root level
  3. Added a unit test for the case that a key spanning across multiple blocks and some of containers are under/over-replicated

[Future works]

  1. https://github.com/yuangu002/ozone/blob/HDDS-5449/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NSSummaryEndpoint.java#L228
  2. https://github.com/yuangu002/ozone/blob/HDDS-5449/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NSSummaryEndpoint.java#L907
  3. Change numOfFiles in NSSummary from int to long

smengcl
smengcl approved these changes Aug 10, 2021
View reviewed changes
Copy link
Contributor

@smengcl smengcl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Latest changes lgtm.

Pending CI.

@smengcl smengcl merged commit 4625061 into apache:master Aug 11, 2021
@smengcl
Copy link
Contributor

smengcl commented Aug 11, 2021

Merged. Thanks @yuangu002 for the patch. Thanks @avijayanhwx for the review.

errose28 added a commit to errose28/ozone that referenced this pull request Aug 12, 2021
@errose28
Merge branch 'master' into HDDS-4183
bbd88d4 
* master:
  HDDS-5358. Incorrect cache entry invalidation causes intermittent failure in testGetS3SecretAndRevokeS3Secret (apache#2518)
  HDDS-5608. Fix wrong command in ugrade doc (apache#2524)
  HDDS-5000. Run CI checks selectively (apache#2479)
  HDDS-4929. Select target datanodes and containers to move for Container Balancer (apache#2441)
  HDDS-5283. getStorageSize cast to int can cause issue (apache#2303)
  HDDS-5449 Recon namespace summary 'du' information should return replicated size of a key (apache#2489)
  HDDS-5558. vUnit invocation unit() may produce NPE (apache#2513)
  HDDS-5531. For Link Buckets avoid showing metadata. (apache#2502)
  HDDS-5549. Add 1.1 to supported versions in security policy (apache#2519)
  HDDS-5555. remove pipeline manager v1 code (apache#2511)
  HDDS-5546.OM Service ID change causes OM startup failure. (apache#2512)
  HDDS-5360. DN failed to process all delete block commands in one heartbeat interval (apache#2420)
  HDDS-5021. dev-support Dockerfile is badly outdated (apache#2480)
errose28 added a commit to errose28/ozone that referenced this pull request Aug 12, 2021
@errose28
Merge branch 'master' into HDDS-5601
7473adf 
* master:
  HDDS-5358. Incorrect cache entry invalidation causes intermittent failure in testGetS3SecretAndRevokeS3Secret (apache#2518)
  HDDS-5608. Fix wrong command in ugrade doc (apache#2524)
  HDDS-5000. Run CI checks selectively (apache#2479)
  HDDS-4929. Select target datanodes and containers to move for Container Balancer (apache#2441)
  HDDS-5283. getStorageSize cast to int can cause issue (apache#2303)
  HDDS-5449 Recon namespace summary 'du' information should return replicated size of a key (apache#2489)
  HDDS-5558. vUnit invocation unit() may produce NPE (apache#2513)
  HDDS-5531. For Link Buckets avoid showing metadata. (apache#2502)
  HDDS-5549. Add 1.1 to supported versions in security policy (apache#2519)
  HDDS-5555. remove pipeline manager v1 code (apache#2511)
  HDDS-5546.OM Service ID change causes OM startup failure. (apache#2512)
  HDDS-5360. DN failed to process all delete block commands in one heartbeat interval (apache#2420)
  HDDS-5021. dev-support Dockerfile is badly outdated (apache#2480)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@avijayanhwx avijayanhwx avijayanhwx left review comments

@smengcl smengcl smengcl approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yuangu002 @smengcl @avijayanhwx