Skip to content

HDDS-5236. Require block token for more operations #2254

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
swagle merged 12 commits into from
May 24, 2021
Merged

HDDS-5236. Require block token for more operations #2254

swagle merged 12 commits into from
May 24, 2021

Conversation

@adoroszlai
Copy link
Contributor

@adoroszlai adoroszlai commented May 17, 2021

What changes were proposed in this pull request?

Require block token in datanode for the following operations:

  • CompactChunk (currently unsupported operation)
  • DeleteBlock
  • DeleteChunk
  • GetCommittedBlockLength
  • ListChunk (currently unsupported operation)

Require container token for ListBlock (currently unsupported operation).

Do not require container token for ListContainer (as it does not have container ID).

https://issues.apache.org/jira/browse/HDDS-5236

How was this patch tested?

Updated existing integration test for secure TestSecureContainerServer to include more operations (only the supported ones).

https://github.com/adoroszlai/hadoop-ozone/actions/runs/849515305

@adoroszlai adoroszlai self-assigned this May 17, 2021
@adoroszlai adoroszlai requested review from elek, fapifta and xiaoyuyao May 17, 2021 16:44
xiaoyuyao
xiaoyuyao reviewed May 19, 2021
View reviewed changes
Copy link
Contributor

@xiaoyuyao xiaoyuyao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @adoroszlai for working on this. Patch LGTM overall. A few comments added inline...

xiaoyuyao
xiaoyuyao reviewed May 20, 2021
View reviewed changes
...p-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/token/BlockTokenVerifier.java
accessMode = READ;
} else if (cmd.getCmdType() == DeleteBlock ||
cmd.getCmdType() == DeleteChunk) {
accessMode = DELETE;
Copy link
Contributor

@xiaoyuyao xiaoyuyao May 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not find any place where we add block tokens with DELETE mode? Do you plan to add those in follow up JIRAs? I assume those will be used for some debug CLI as the deletion done async via Hadoop RPC.

Copy link
Contributor Author

@adoroszlai adoroszlai May 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @xiaoyuyao for flagging this. Indeed, further change is needed in SCM.

Copy link
Contributor Author

@adoroszlai adoroszlai May 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not plan to work on debug CLI. Maybe someone will find that useful to take initiative. But checking for the token still prevents user from sending such command without auth.

Regarding the further change I mentioned, I opened HDDS-5264 and submitted another PR. It turned out to be quite separate from this one, so on second thought I wanted to avoid mixing the two.

Copy link
Contributor

@xiaoyuyao xiaoyuyao May 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bq. Regarding the further change I mentioned, I opened HDDS-5264 and submitted another PR. It turned out to be quite separate from this one, so on second thought I wanted to avoid mixing the two.

Sounds good to me. +1.

xiaoyuyao
xiaoyuyao reviewed May 20, 2021
View reviewed changes
Copy link
Contributor

@xiaoyuyao xiaoyuyao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just have one more question: how do we plan to add the DELETE mode?

@adoroszlai adoroszlai marked this pull request as draft May 21, 2021 07:35
@adoroszlai adoroszlai marked this pull request as ready for review May 22, 2021 05:19
xiaoyuyao
xiaoyuyao approved these changes May 24, 2021
View reviewed changes
Copy link
Contributor

@xiaoyuyao xiaoyuyao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, +1.

...p-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/token/BlockTokenVerifier.java
accessMode = READ;
} else if (cmd.getCmdType() == DeleteBlock ||
cmd.getCmdType() == DeleteChunk) {
accessMode = DELETE;
Copy link
Contributor

@xiaoyuyao xiaoyuyao May 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bq. Regarding the further change I mentioned, I opened HDDS-5264 and submitted another PR. It turned out to be quite separate from this one, so on second thought I wanted to avoid mixing the two.

Sounds good to me. +1.

@swagle
Copy link
Contributor

swagle commented May 24, 2021

Thanks for the review @xiaoyuyao merging this.

@swagle swagle merged commit 355096b into apache:master May 24, 2021
@adoroszlai
Copy link
Contributor Author

adoroszlai commented May 25, 2021

Thanks @xiaoyuyao for the reviews, @swagle for merging this.

@adoroszlai adoroszlai deleted the HDDS-5236 branch May 25, 2021 20:03
errose28 added a commit to errose28/ozone that referenced this pull request Jun 1, 2021
@errose28
Merge remote-tracking branch 'upstream/master' into HDDS-3698-nonroll…
1d18f47 
…ing-upgrade-master-merge

* upstream/master: (76 commits)
  HDDS-5280. Make XceiverClientManager creation when necessary in ContainerOperationClient (apache#2289)
  HDDS-5272. Make ozonefs.robot execution repeatable (apache#2280)
  HDDS-5123. Use the pre-created apache/ozone-testkrb5 image during secure acceptance tests (apache#2165)
  HDDS-4993. Add guardrail for reserved buffer size when DN reads a chunk (apache#2058)
  HDDS-4936. Change ozone groupId from org.apache.hadoop to org.apache.ozone (apache#2018)
  HDDS-4043. allow deletion from Trash directory without -skipTrash option (apache#2110)
  HDDS-4927. Determine over and under utilized datanodes in Container Balancer. (apache#2230)
  HDDS-5273. Handle unsecure cluster convert to secure cluster for SCM. (apache#2281)
  HDDS-5158. Add documentation for SCM HA Security. (apache#2205)
  HDDS-5275. Datanode Report Publisher publishes one extra report after DN shutdown (apache#2283)
  HDDS-5241. SCM UI should have leader/follower and Primordial SCM information (apache#2260)
  HDDS-5219. Limit number of bad volumes by dfs.datanode.failed.volumes.tolerated. (apache#2243)
  HDDS-5252. PipelinePlacementPolicy filter out datanodes with not enough space. (apache#2271)
  HDDS-5191. Increase default pvc storage size (apache#2219)
  HDDS-5073. Use ReplicationConfig on client side  (apache#2136)
  HDDS-5250. Build integration tests with Maven cache (apache#2269)
  HDDS-5236. Require block token for more operations (apache#2254)
  HDDS-5266 Misspelt words in S3MultipartUploadCommitPartRequest.java line 202 (apache#2279)
  HDDS-5249. Race Condition between Full and Incremental Container Reports (apache#2268)
  HDDS-5142. Make generic streaming client/service for container re-replication, data read, scm/om snapshot download (apache#2256)
  ...

Conflicts:
	hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java
	hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java
	hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
	hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/MockNodeManager.java
	hadoop-ozone/dist/src/main/compose/testlib.sh
	hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManager.java
	hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto
	hadoop-ozone/ozone-manager/pom.xml
	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/utils/OzoneManagerRatisUtils.java
	hadoop-ozone/s3gateway/pom.xml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiaoyuyao xiaoyuyao xiaoyuyao approved these changes

@elek elek Awaiting requested review from elek

@fapifta fapifta Awaiting requested review from fapifta

Assignees

@adoroszlai adoroszlai

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@adoroszlai @swagle @xiaoyuyao