HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE. #1337

captainzmc · 2020-08-19T08:25:58Z

What changes were proposed in this pull request?

Currently MAX_QUOTA_IN_BYTES is 1EB, which is not very exact. Later we need use this to determine whether quota is enabled or not.
We need to change it to long.max_value to be consistent with HDFS.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-4129

How was this patch tested?

Existing UT

adoroszlai · 2020-08-24T15:24:53Z

hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java

What about volumes created before this change with the previous max. value? Will those be treated by new version as if quota had been enabled for them?

Thanks for @adoroszlai 's discussion. Yes, the previous volume will enable quota. So I created JIRA HDDS-4106, which supports user Clearing spaceQuota.

Suggest add new logic during OM startup to reset existing Volume max quota value to mitigate the compatibility issue.

I double checked HDFS implementation，it doesn't use LONG.MAX_VALUE to verify whether quota is enabled or not. It use "-1" which is more reasonable for a set/unset check.

Thanks sammi for the discussion. It is indeed uncommon to use the maximum value to determine whether to enable quota, and it is more common to use -1 to determine. Later I will confirm whether to enable quota with -1 in HDDS-3727. The MAX_QUOTA_IN_BYTES here will be used to determine if the parameter values for setquota and updatequota are reasonable in HDDS-3725.

bharatviswa504 · 2020-08-24T16:12:57Z

Closing this as HDDS-4089 planning to fix this issue to make ACL commands work with the external authorizer.

captainzmc · 2020-08-25T08:07:36Z

Closing this as HDDS-4089 planning to fix this issue to make ACL commands work with the external authorizer.

Hi @bharatviswa504 , I think this change has nothing to do with ACL commands. Is it something I don't understand?

adoroszlai · 2020-08-25T08:47:27Z

Closing this as HDDS-4089 planning to fix this issue to make ACL commands work with the external authorizer.

Hi @bharatviswa504 , I think this change has nothing to do with ACL commands. Is it something I don't understand?

@captainzmc I think you can ignore it, the comment was intended for another PR: #1271 (comment)

bharatviswa504 · 2020-08-26T04:16:52Z

Sorry, @captainzmc I have posted the comment on the wrong PR. I was trying to search where I have posted, not able to figure out. You can ignore my comment.

captainzmc · 2020-09-04T06:36:39Z

Hi @ChenSammi , I had added the initialization and change the old volume quota default value to -1 when OM starts. Could you help review this PR again?

ChenSammi · 2020-09-04T08:48:32Z

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java

Can we add a UT for this function?

Yes, the corresponding UT has been added.

ChenSammi · 2020-09-04T08:49:23Z

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java

Why omVolumeArgs.getQuotaInBytes() == OzoneConsts.MAX_QUOTA_IN_BYTES check is required here?

This judgment is redundant, will delete it.

ChenSammi · 2020-09-04T08:49:52Z

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java

duplicate ","

ChenSammi · 2020-09-04T08:52:17Z

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java

There is one risk here, if user set the quota to 1EB specifically, we cann't reset it to -1 on next OM restart.

Currently, the way to determine the old quota is by seeing whether his quota is 1 EB. So 1EB is still a special value. At present, OZone's cluster size is not large, far from the upper limit of 1EB.
So can we ignore this case? In addition, I adjust the LOG level from INFO to Warn (I can change it to ERROR if needed) to alert the user that the corresponding quota has been changed.

captainzmc · 2020-09-07T08:53:03Z

Thanks for Sammi's review, fixed review issues. @ChenSammi Could you help take another look?

This reverts commit c955f26.

This reverts commit daae0e2.

ChenSammi · 2020-09-11T07:12:20Z

+1. After a thoroughly thinking, we choose a conservative way.

Thanks @captainzmc for the contribution and @adoroszlai for the review.

* master: (47 commits) HDDS-4104. Provide a way to get the default value and key of java-based-configuration easily (apache#1369) HDDS-4250. Fix wrong logger name (apache#1429) HDDS-4244. Container deleted wrong replica cause mis-replicated. (apache#1423) HDDS-4053. Volume space: add quotaUsageInBytes and update it when write and delete key. (apache#1296) HDDS-4210. ResolveBucket during checkAcls fails. (apache#1398) HDDS-4075. Retry request on different OM on AccessControlException (apache#1303) HDDS-4166. Documentation index page redirects to the wrong address (apache#1372) HDDS-4039. Reduce the number of fields in hdds.proto to improve performance (apache#1289) HDDS-4155. Directory and filename can end up with same name in a path. (apache#1361) HDDS-3927. Rename Ozone OM,DN,SCM runtime options to conform to naming conventions (apache#1401) HDDS-4119. Improve performance of the BufferPool management of Ozone client (apache#1336) HDDS-4217.Remove test TestOzoneContainerRatis (apache#1408) HDDS-4218.Remove test TestRatisManager (apache#1409) HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE. (apache#1337) HDDS-4228: add field 'num' to ALLOCATE_BLOCK of scm audit log. (apache#1413) HDDS-4196. Add an endpoint in Recon to query Prometheus (apache#1390) HDDS-4211. [OFS] Better owner and group display for listing Ozone volumes and buckets (apache#1397) HDDS-4150. recon.api.TestEndpoints test is flaky (apache#1396) HDDS-4170 - Fix typo in method description. (apache#1406) HDDS-4064. Show container verbose info with verbose option (apache#1290) ...

…ponse * HDDS-4122-quota-attempt2: (51 commits) Remove redundant check status calls in children of AbstractOMOpenKeyDeleteRequest Remove unused inports and fix super constructor calls Move common volume byte usage update code to AbstractOMKeyDeleteResponse Add volume quota update to open key delete response, and group duplicate code HDDS-4104. Provide a way to get the default value and key of java-based-configuration easily (apache#1369) HDDS-4250. Fix wrong logger name (apache#1429) HDDS-4244. Container deleted wrong replica cause mis-replicated. (apache#1423) HDDS-4053. Volume space: add quotaUsageInBytes and update it when write and delete key. (apache#1296) HDDS-4210. ResolveBucket during checkAcls fails. (apache#1398) HDDS-4075. Retry request on different OM on AccessControlException (apache#1303) HDDS-4166. Documentation index page redirects to the wrong address (apache#1372) HDDS-4039. Reduce the number of fields in hdds.proto to improve performance (apache#1289) HDDS-4155. Directory and filename can end up with same name in a path. (apache#1361) HDDS-3927. Rename Ozone OM,DN,SCM runtime options to conform to naming conventions (apache#1401) HDDS-4119. Improve performance of the BufferPool management of Ozone client (apache#1336) HDDS-4217.Remove test TestOzoneContainerRatis (apache#1408) HDDS-4218.Remove test TestRatisManager (apache#1409) HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE. (apache#1337) HDDS-4228: add field 'num' to ALLOCATE_BLOCK of scm audit log. (apache#1413) HDDS-4196. Add an endpoint in Recon to query Prometheus (apache#1390) ...

adoroszlai reviewed Aug 24, 2020

View reviewed changes

ChenSammi reviewed Sep 4, 2020

View reviewed changes

captainzmc added 3 commits September 4, 2020 17:12

change max quota in bytes to long max value.

7e8f4f6

initialize old volume quota.

daae0e2

fix review issues.

c955f26

captainzmc force-pushed the change-max-quota branch from 3f1fd51 to c955f26 Compare September 4, 2020 09:28

captainzmc added 2 commits September 10, 2020 17:33

Revert "fix review issues."

a64e251

This reverts commit c955f26.

Revert "initialize old volume quota."

1613758

This reverts commit daae0e2.

captainzmc closed this Sep 10, 2020

captainzmc reopened this Sep 10, 2020

ChenSammi merged commit 04ac1ef into apache:master Sep 11, 2020

llemec pushed a commit to llemec/hadoop-ozone that referenced this pull request Sep 15, 2020

HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE. (apache#1337)

324d6a6

HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE. #1337

HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE. #1337

Uh oh!

Conversation

captainzmc commented Aug 19, 2020

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChenSammi Aug 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

captainzmc Aug 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bharatviswa504 commented Aug 24, 2020

Uh oh!

captainzmc commented Aug 25, 2020

Uh oh!

adoroszlai commented Aug 25, 2020

Uh oh!

bharatviswa504 commented Aug 26, 2020

Uh oh!

captainzmc commented Sep 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

captainzmc Sep 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

captainzmc commented Sep 7, 2020

Uh oh!

ChenSammi commented Sep 11, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ChenSammi Aug 26, 2020 •

edited

Loading

captainzmc Aug 27, 2020 •

edited

Loading

captainzmc commented Sep 4, 2020 •

edited

Loading

captainzmc Sep 4, 2020 •

edited

Loading