Skip to content

HDDS-4254. Bucket space: add usedBytes and update it when create and delete key. #1431

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
ChenSammi merged 3 commits into from
Sep 23, 2020
Merged

HDDS-4254. Bucket space: add usedBytes and update it when create and delete key. #1431

ChenSammi merged 3 commits into from
Sep 23, 2020

Conversation

@captainzmc
Copy link
Member

@captainzmc captainzmc commented Sep 17, 2020

What changes were proposed in this pull request?

In addition, the current Quota setting does not take effect. HDDS-541 gives all the work needed to perfect Quota.
This PR is a subtask of HDDS-541.
First, we increase usedBytes of Bucket. Later, we will judge whether the Bucket can be written based on this when we write the key.
Volume has implemented increase usedBytes when write, and this is based on HDDS-4053.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-4254

How was this patch tested?

added UT

@captainzmc
Copy link
Member Author

captainzmc commented Sep 17, 2020

Here are the freon test results:
Using 100 threads to write data to different buckets under the same volume, adding quota Usage has little impact on performance. (LongAdder disperses the concurrency of a single cell to each cell, improving concurrency efficiency compared to atomicLong). In my test, I used three virtual machines, each with a key size of 10K.
image

@captainzmc captainzmc closed this Sep 17, 2020
@captainzmc captainzmc reopened this Sep 17, 2020
@captainzmc
Copy link
Member Author

captainzmc commented Sep 17, 2020

hi @ChenSammi @xiaoyuyao, this PR is base on #1296. Can you help to review this?

ChenSammi
ChenSammi reviewed Sep 22, 2020
View reviewed changes
...e-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMAllocateBlockResponse.java Outdated
omMetadataManager.getBucketTable().putWithBatch(batchOperation,
omMetadataManager.getBucketKey(omVolumeArgs.getVolume(),
omBucketInfo.getBucketName()),
omBucketInfo);
Copy link
Contributor

@ChenSammi ChenSammi Sep 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the last two lines can be merged into one line.

ChenSammi
ChenSammi reviewed Sep 22, 2020
View reviewed changes
...ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCommitResponse.java Outdated
omMetadataManager.getBucketTable().putWithBatch(batchOperation,
omMetadataManager.getBucketKey(omVolumeArgs.getVolume(),
omBucketInfo.getBucketName()),
omBucketInfo);
Copy link
Contributor

@ChenSammi ChenSammi Sep 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above.

ChenSammi
ChenSammi reviewed Sep 22, 2020
View reviewed changes
...ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCreateResponse.java Outdated
omMetadataManager.getBucketTable().putWithBatch(batchOperation,
omMetadataManager.getBucketKey(omVolumeArgs.getVolume(),
omBucketInfo.getBucketName()),
omBucketInfo);
Copy link
Contributor

@ChenSammi ChenSammi Sep 22, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the same. please double check the rest files.

@captainzmc
Copy link
Member Author

captainzmc commented Sep 23, 2020

Thanks @ChenSammi for the review. Review issues has been fixed.

@ChenSammi
Copy link
Contributor

ChenSammi commented Sep 23, 2020

LGTM, +1.

@ChenSammi ChenSammi merged commit a78a4b7 into apache:master Sep 23, 2020
errose28 added a commit to errose28/ozone that referenced this pull request Sep 28, 2020
@errose28
Merge branch 'master' into HDDS-4122-remove-code-consolidation
82e70a6 
* master:
  HDDS-4102. Normalize Keypath for lookupKey. (apache#1328)
  HDDS-4263. ReplicatiomManager shouldn't consider origin node Id for CLOSED containers. (apache#1438)
  HDDS-4282. Improve the emptyDir syntax (apache#1450)
  HDDS-4194. Create a script to check AWS S3 compatibility (apache#1383)
  HDDS-4270. Add more reusable byteman scripts to debug ofs/o3fs performance (apache#1443)
  HDDS-2660. Create insight point for datanode container protocol (apache#1272)
  HDDS-3297. Enable TestOzoneClientKeyGenerator. (apache#1442)
  HDDS-4324. Add important comment to ListVolumes logic (apache#1417)
  HDDS-4236. Move "Om*Codec.java" to new project hadoop-ozone/interface-storage (apache#1424)
  HDDS-4254. Bucket space: add usedBytes and update it when create and delete key. (apache#1431)
  HDDS-2766. security/SecuringDataNodes.md (apache#1175)
  HDDS-4206. Attempt pipeline creation more frequently in acceptance tests (apache#1389)
  HDDS-4233. Interrupted exeception printed out from DatanodeStateMachine (apache#1416)
  HDDS-3947: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs (apache#1385)
  HDDS-3102. ozone getconf command should use the GenericCli parent class (apache#1410)
  HDDS-3981. Add more debug level log to XceiverClientGrpc for debug purpose (apache#1214)
  HDDS-4255. Remove unused Ant and Jdiff dependency versions (apache#1433)
  HDDS-4247. Fixed log4j usage in some places (apache#1426)
  HDDS-4241. Support HADOOP_TOKEN_FILE_LOCATION for Ozone token CLI. (apache#1422)
errose28 added a commit to errose28/ozone that referenced this pull request Sep 28, 2020
@errose28
Merge branch 'HDDS-4122-remove-code-consolidation' into HDDS-4122
3cb1e2c 
* HDDS-4122-remove-code-consolidation: (21 commits)
  Restore files that had deduplicated code from master
  Revert other delete request/response files back to their original states on master
  HDDS-4102. Normalize Keypath for lookupKey. (apache#1328)
  HDDS-4263. ReplicatiomManager shouldn't consider origin node Id for CLOSED containers. (apache#1438)
  HDDS-4282. Improve the emptyDir syntax (apache#1450)
  HDDS-4194. Create a script to check AWS S3 compatibility (apache#1383)
  HDDS-4270. Add more reusable byteman scripts to debug ofs/o3fs performance (apache#1443)
  HDDS-2660. Create insight point for datanode container protocol (apache#1272)
  HDDS-3297. Enable TestOzoneClientKeyGenerator. (apache#1442)
  HDDS-4324. Add important comment to ListVolumes logic (apache#1417)
  HDDS-4236. Move "Om*Codec.java" to new project hadoop-ozone/interface-storage (apache#1424)
  HDDS-4254. Bucket space: add usedBytes and update it when create and delete key. (apache#1431)
  HDDS-2766. security/SecuringDataNodes.md (apache#1175)
  HDDS-4206. Attempt pipeline creation more frequently in acceptance tests (apache#1389)
  HDDS-4233. Interrupted exeception printed out from DatanodeStateMachine (apache#1416)
  HDDS-3947: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs (apache#1385)
  HDDS-3102. ozone getconf command should use the GenericCli parent class (apache#1410)
  HDDS-3981. Add more debug level log to XceiverClientGrpc for debug purpose (apache#1214)
  HDDS-4255. Remove unused Ant and Jdiff dependency versions (apache#1433)
  HDDS-4247. Fixed log4j usage in some places (apache#1426)
  ...
@captainzmc captainzmc mentioned this pull request Sep 30, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ChenSammi ChenSammi ChenSammi left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@captainzmc @ChenSammi