Skip to content

HDDS-4562. Old bucket needs to be accessible after the cluster was upgraded to the Quota version. #1677

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bharatviswa504 merged 2 commits into from
Dec 23, 2020
Merged

HDDS-4562. Old bucket needs to be accessible after the cluster was upgraded to the Quota version. #1677

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bc0ec8b
fix bucket quotaInBytes.
captainzmc Dec 8, 2020
554c9a8
fix review issues and change default to -2.
captainzmc Dec 16, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions hadoop-hdds/docs/content/feature/Quota.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,16 @@ So far, we know that Ozone allows users to create volumes, buckets, and keys. A
1. Storage Space level quota
Copy link
Contributor

@linyiqun linyiqun Dec 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The namespace quota introduction can also be added. We could file a new JIRA for tacking this.

Copy link
Contributor

@amaliujia amaliujia Dec 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can add namespace quota to this documentation: https://issues.apache.org/jira/browse/HDDS-4594. Before adding to documentation, I will first finish namespace support on bucket: https://issues.apache.org/jira/browse/HDDS-4277


Administrators should be able to define how much storage space a Volume or Bucket can use. The following Settings for Storage space quota are currently supported：

a. By default, the quota for volume and bucket is not enabled.

b. When volume quota is enabled, the total size of bucket quota cannot exceed volume.
Copy link
Contributor

@bharatviswa504 bharatviswa504 Dec 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor:
When volume quota is enabled, the total quota of the buckets, cannot exceed the volume quota.


c. Bucket quota can be set separately without enabling Volume quota. The size of bucket quota is unrestricted at this point.

d. Volume quota is not currently supported separately, and volume quota takes effect only if bucket quota is set. Because ozone only check the usedBytes of the bucket when we write the key.

e. If the cluster is upgraded from old version less than 1.1.0, use of quota on older volumes and buckets is not recommended. Since the old key is not counted to the bucket's usedBytes, the quota setting is inaccurate at this point.
Copy link
Contributor

@linyiqun linyiqun Dec 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can make this check in OM side and throw exception once user try to set quota on older version volumes/buckets.

Copy link
Member Author

@captainzmc captainzmc Dec 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, we have not agreed on whether bucket can be enabled for the old quota. At present, we only put forward the suggestion in the document very briefly. If the requirement is clear, we can issue Jira and solve it.

Copy link
Contributor

@linyiqun linyiqun Dec 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense to me.

Copy link
Contributor

@bharatviswa504 bharatviswa504 Dec 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also add documentation about the clear space quota and the behavior?

Copy link
Member Author

@captainzmc captainzmc Dec 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. To prevent conflicts, I have updated the usage and behavior of clear space quota in this PR.


## Client usage
### Storage Space level quota
Expand Down
6 changes: 6 additions & 0 deletions hadoop-hdds/docs/content/feature/Quota.zh.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,16 @@ menu:
1. Storage space级别配额

管理员应该能够定义一个Volume或Bucket可以使用多少存储空间。目前支持以下storage space quota的设置:

a. 默认情况下volume和bucket的quota不启用。

b. 当volume quota启用时，bucket quota的总大小不能超过volume。

c. 可以在不启用volume quota的情况下单独给bucket设置quota。此时bucket quota的大小是不受限制的。

d. 目前不支持单独设置volume quota，只有在设置了bucket quota的情况下volume quota才会生效。因为ozone在写入key时只检查bucket的usedBytes。

e. 如果集群从小于1.1.0的旧版本升级而来，则不建议在旧volume和bucket上使用配额。由于旧的key没有计算到bucket的usedBytes中，所以此时配额设置是不准确的。

## 客户端用法
### Storage space级别配额
Expand Down
6 changes: 3 additions & 3 deletions hadoop-ozone/dist/src/main/compose/upgrade/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,9 +65,9 @@ source "${COMPOSE_DIR}/../testlib.sh"
export OZONE_KEEP_RESULTS=true
start_docker_env
execute_robot_test scm -v PREFIX:pre freon/validate.robot
# TODO can be enabled with HDDS-4562:
# execute_robot_test scm -v PREFIX:post freon/generate.robot
# execute_robot_test scm -v PREFIX:post freon/validate.robot
# test write key to old bucket after upgrade
execute_robot_test scm -v PREFIX:post freon/generate.robot
execute_robot_test scm -v PREFIX:post freon/validate.robot
stop_docker_env

generate_report
6 changes: 3 additions & 3 deletions hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -363,7 +363,7 @@ message VolumeInfo {
optional uint64 objectID = 8;
optional uint64 updateID = 9;
optional uint64 modificationTime = 10;
optional uint64 quotaInNamespace = 11;
optional sint64 quotaInNamespace = 11 [default = -1];
optional uint64 usedNamespace = 12;
}

Expand Down Expand Up @@ -503,8 +503,8 @@ message BucketInfo {
optional string sourceVolume = 12;
optional string sourceBucket = 13;
optional uint64 usedBytes = 14;
optional uint64 quotaInBytes = 15;
optional uint64 quotaInNamespace = 16;
optional sint64 quotaInBytes = 15 [default = -1];
optional sint64 quotaInNamespace = 16 [default = -1];
}

enum StorageTypeProto {
Expand Down