Skip to content

Conversation

@ayushtkn
Copy link
Member

@ayushtkn ayushtkn commented Nov 22, 2020

What changes were proposed in this pull request?

Added a hidden option --quota, to work same as --space-quota, if --space-quota isn't specified. Made hidden so as to prevent new users from using it.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-4492

How was this patch tested?

Added UT and on docker cluster ::

Usage: ozone sh volume create [-hV] [--count-quota=<quotaInCounts>]
                              [--space-quota=<quotaInBytes>] [-u=<ownerName>]
                              <value>
Creates a volume for the specified user
      <value>              URI of the volume.
                           Ozone URI could start with o3:// or without prefix.
                             URI may contain the host/serviceId  and port of
                             the OM server. Both are optional. If they are not
                             specified it will be identified from the config
                             files.
      --count-quota=<quotaInCounts>
                           For volume this parameter represents the number of
                             buckets, and for buckets represents the number of
                             keys (eg. 5)
  -h, --help               Show this help message and exit.
      --space-quota=<quotaInBytes>
                           The maximum space quota can be used (eg. 1GB)
  -u, --user=<ownerName>   Owner of the volume
  -V, --version            Print version information and exit.
bash-4.2$ ozone sh volume create vol1 --quota 10BYTES
bash-4.2$ ozone sh volume info vol1
{
  "metadata" : { },
  "name" : "vol1",
  "admin" : "hadoop",
  "owner" : "hadoop",
  "quotaInBytes" : 10,
  "quotaInCounts" : -1,
  "creationTime" : "2020-11-22T17:42:57.206Z",
  "modificationTime" : "2020-11-22T17:43:32.772081Z",
  "acls" : [ {
    "type" : "USER",
    "name" : "hadoop",
    "aclScope" : "ACCESS",
    "aclList" : [ "ALL" ]
  }, {
    "type" : "GROUP",
    "name" : "users",
    "aclScope" : "ACCESS",
    "aclList" : [ "ALL" ]
  } ],
  "usedBytes" : 0
}
bash-4.2$ 


Usage: ozone sh bucket create [-ghV] [--count-quota=<quotaInCounts>]
                              [-k=<bekName>] [--space-quota=<quotaInBytes>]
                              <value>
creates a bucket in a given volume
      <value>         URI of the volume/bucket.
                      Ozone URI could start with o3:// or without prefix. URI
                        may contain the host/serviceId  and port of the OM
                        server. Both are optional. If they are not specified it
                        will be identified from the config files.
      --count-quota=<quotaInCounts>
                      For volume this parameter represents the number of
                        buckets, and for buckets represents the number of keys
                        (eg. 5)
  -g, --enforcegdpr   if true, indicates GDPR enforced bucket,
                        false/unspecified indicates otherwise
  -h, --help          Show this help message and exit.
  -k, --bucketkey=<bekName>
                      bucket encryption key name
      --space-quota=<quotaInBytes>
                      The maximum space quota can be used (eg. 1GB)
  -V, --version       Print version information and exit.
bash-4.2$ ozone sh bucket create vol1/buck1 --quota 5BYTES
bash-4.2$ ozone sh bucket info vol1/buck1               
{
  "metadata" : { },
  "volumeName" : "vol1",
  "name" : "buck1",
  "storageType" : "DISK",
  "versioning" : false,
  "usedBytes" : 0,
  "creationTime" : "2020-11-22T17:45:14.969Z",
  "modificationTime" : "2020-11-22T17:45:14.969Z",
  "encryptionKeyName" : null,
  "sourceVolume" : null,
  "sourceBucket" : null,
  "quotaInBytes" : 5,
  "quotaInCounts" : -1
}
bash-4.2$

Copy link
Contributor

@adoroszlai adoroszlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ayushtkn for working on this. Backward compatibility is addressed by this change.

Have you considered adding --quota as an alias for --space-quota, instead of as a hidden option? In addition to requiring less code (both for option definition and in accessor method), it also prevents usage of both options in the same command. With the current patch one can specify eg. --space-quota 10GB --quota 1GB and the latter is ignored.

@ayushtkn
Copy link
Member Author

Thanx @adoroszlai for the review.

Have you considered adding --quota as an alias for --space-quota

I tried that, But for commands there was an option to provide an alias but for @CommandLine.Option I couldn't find any. One way was to add it in the names array but the value will be visible.
Is there some way to add an alias for options as well, so that it isn't visible.
or do you suggest we add it as part of name array only like this -
@CommandLine.Option(names = {"--space-quota","--quota"},

@adoroszlai
Copy link
Contributor

or do you suggest we add it as part of name array only like this -
@CommandLine.Option(names = {"--space-quota","--quota"},

Yes, sorry for the confusion about aliases. Command aliases are also visible I think.

One way was to add it in the names array but the value will be visible.
Is there some way to add an alias for options as well, so that it isn't visible.

I don't think it's a problem that --quota would appear in help output as an alternative to --space-quota. It's not a secret. ;)

@ayushtkn
Copy link
Member Author

Thanx @adoroszlai I have changed the code to add --quota to the names array.

Copy link
Contributor

@adoroszlai adoroszlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ayushtkn for updating the patch.

Copy link
Contributor

@avijayanhwx avijayanhwx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @ayushtkn. LGTM +1.

@avijayanhwx
Copy link
Contributor

Thanks for the review @adoroszlai!

@avijayanhwx avijayanhwx merged commit a9ff68a into apache:master Nov 24, 2020
errose28 added a commit to errose28/ozone that referenced this pull request Dec 1, 2020
* HDDS-3698-upgrade:
  HDDS-4429. Create unit test for SimpleContainerDownloader. (apache#1551)
  HDDS-4461. Reuse compiled binaries in acceptance test (apache#1588)
  HDDS-4511: Avoiding StaleNodeHandler to take effect in TestDeleteWithSlowFollower. (apache#1625)
  HDDS-4510. SCM can avoid creating RetriableDatanodeEventWatcher for deletion command ACK (apache#1626)
  HDDS-3363. Intermittent failure in testContainerImportExport (apache#1618)
  HDDS-4370. Datanode deletion service can avoid storing deleted blocks. (apache#1620)
  HDDS-4512. Remove unused netty3 transitive dependency (apache#1627)
  HDDS-4481. With HA OM can send deletion blocks to SCM multiple times. (apache#1608)
  HDDS-4487. SCM can avoid using RETRIABLE_DATANODE_COMMAND for datanode deletion commands. (apache#1621)
  HDDS-4471. GrpcOutputStream length can overflow (apache#1617)
  HDDS-4308. Fix issue with quota update (apache#1489)
  HDDS-4392. [DOC] Add Recon architecture to docs (apache#1602)
  HDDS-4501. Reload OM State fail should terminate OM for any exceptions. (apache#1622)
  HDDS-4492. CLI flag --quota should default to 'spaceQuota' to preserve backward compatibility. (apache#1609)
  HDDS-3689. Add various profiles to MiniOzoneChaosCluster to run different modes. (apache#1420)
  HDDS-4497. Recon File Size Count task throws SQL Exception. (apache#1612)
errose28 added a commit to errose28/ozone that referenced this pull request Dec 1, 2020
* HDDS-3698-upgrade:
  HDDS-4429. Create unit test for SimpleContainerDownloader. (apache#1551)
  HDDS-4461. Reuse compiled binaries in acceptance test (apache#1588)
  HDDS-4511: Avoiding StaleNodeHandler to take effect in TestDeleteWithSlowFollower. (apache#1625)
  HDDS-4510. SCM can avoid creating RetriableDatanodeEventWatcher for deletion command ACK (apache#1626)
  HDDS-3363. Intermittent failure in testContainerImportExport (apache#1618)
  HDDS-4370. Datanode deletion service can avoid storing deleted blocks. (apache#1620)
  HDDS-4512. Remove unused netty3 transitive dependency (apache#1627)
  HDDS-4481. With HA OM can send deletion blocks to SCM multiple times. (apache#1608)
  HDDS-4487. SCM can avoid using RETRIABLE_DATANODE_COMMAND for datanode deletion commands. (apache#1621)
  HDDS-4471. GrpcOutputStream length can overflow (apache#1617)
  HDDS-4308. Fix issue with quota update (apache#1489)
  HDDS-4392. [DOC] Add Recon architecture to docs (apache#1602)
  HDDS-4501. Reload OM State fail should terminate OM for any exceptions. (apache#1622)
  HDDS-4492. CLI flag --quota should default to 'spaceQuota' to preserve backward compatibility. (apache#1609)
  HDDS-3689. Add various profiles to MiniOzoneChaosCluster to run different modes. (apache#1420)
  HDDS-4497. Recon File Size Count task throws SQL Exception. (apache#1612)
errose28 added a commit to errose28/ozone that referenced this pull request Jan 5, 2021
* master: (40 commits)
  HDDS-4473. Reduce number of sortDatanodes RPC calls (apache#1610)
  HDDS-4485. [DOC] add the authentication rules of the Ozone Ranger. (apache#1603)
  HDDS-4528. Upgrade slf4j to 1.7.30 (apache#1639)
  HDDS-4424. Update README with information how to report security issues (apache#1548)
  HDDS-4484. Use RaftServerImpl isLeader instead of periodic leader update logic in OM and isLeaderReady for read/write requests (apache#1638)
  HDDS-4429. Create unit test for SimpleContainerDownloader. (apache#1551)
  HDDS-4461. Reuse compiled binaries in acceptance test (apache#1588)
  HDDS-4511: Avoiding StaleNodeHandler to take effect in TestDeleteWithSlowFollower. (apache#1625)
  HDDS-4510. SCM can avoid creating RetriableDatanodeEventWatcher for deletion command ACK (apache#1626)
  HDDS-3363. Intermittent failure in testContainerImportExport (apache#1618)
  HDDS-4370. Datanode deletion service can avoid storing deleted blocks. (apache#1620)
  HDDS-4512. Remove unused netty3 transitive dependency (apache#1627)
  HDDS-4481. With HA OM can send deletion blocks to SCM multiple times. (apache#1608)
  HDDS-4487. SCM can avoid using RETRIABLE_DATANODE_COMMAND for datanode deletion commands. (apache#1621)
  HDDS-4471. GrpcOutputStream length can overflow (apache#1617)
  HDDS-4308. Fix issue with quota update (apache#1489)
  HDDS-4392. [DOC] Add Recon architecture to docs (apache#1602)
  HDDS-4501. Reload OM State fail should terminate OM for any exceptions. (apache#1622)
  HDDS-4492. CLI flag --quota should default to 'spaceQuota' to preserve backward compatibility. (apache#1609)
  HDDS-3689. Add various profiles to MiniOzoneChaosCluster to run different modes. (apache#1420)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants