HDDS-1299. Support TokenIssuer interface for running jobs with OzoneFileSystem. #627
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-compose.yaml
Outdated
Show resolved
Hide resolved
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-image/docker-krb5/README.md
Outdated
Show resolved
Hide resolved
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
ajayydv
reviewed
Mar 21, 2019
Contributor
There was a problem hiding this comment.
Shall we file a jira for this targeting 0.5?
Contributor
Author
There was a problem hiding this comment.
We already have one: https://issues.apache.org/jira/browse/HDDS-738
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-config
Outdated
Show resolved
Hide resolved
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-config
Outdated
Show resolved
Hide resolved
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-config
Outdated
Show resolved
Hide resolved
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-config
Outdated
Show resolved
Hide resolved
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-image/docker-krb5/Dockerfile-krb5
Outdated
Show resolved
Hide resolved
Contributor
There was a problem hiding this comment.
Seems issuing token even if renewer is not passed seems to be legit case.
Contributor
Author
There was a problem hiding this comment.
I don't think null renewer will be a legit case. HDFS (DistrubutedFIleSystem/DFSClient) returns null in this case, which matches the behavior implemented for Ozone.
Contributor
There was a problem hiding this comment.
What if a use case don't have a requirement for renewing the token. In that case we should allow empty/null renewer.
Contributor
Author
There was a problem hiding this comment.
Only block token does not require renew, but it will not be retrieved from here. So I think we will not allow empty/null renewer like HDFS.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
Outdated
Show resolved
Hide resolved
…ileSystem. Contributed by Xiaoyu Yao.
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Contributor
|
+1 pending jenkins. |
|
💔 -1 overall
This message was automatically generated. |
Contributor
|
Given this has broken things, can I ask a process question here, specifically regarding Yetus's v-1
There are lots of ways to test for DTs being ussued: The
You can even verify that MR cluster launch collects the tokens: org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob Please. write tests. Apart from the MR job one its straightforward, and I've just given you all the clases to start with. Without tests, when people like myself make an incompatible change which breaks Ozone's DT support, we're just going to shrug and say "well, you should have written the tests" |
Contributor
Author
|
Thanks Steve for the heads up. Ozone DT via HCFS requires Kerberos setup for various component to test e2e. There is no easy way to test inside unit test as OM/SCM/DN all login with different principles.
We have follow up JIRA to add system test.
Before committing the changes, I spend a few days to get the manual steps with various testing documented here:
%your_hadoop_rep%/hadoop-ozone/dist/src/main/compose/ozonesecure-mr/README.md
This will be the basis of system tests being added, where we may use different Hadoop-versions like 3.2 and 2.7 to test the combabilities.
The test will be very similar to the one below just slightly different security setup.
%your_hadoop_rep%/hadoop/hadoop-ozone/dist/src/main/smoketest/security/ozone-secure.robot
Bq. Without tests, when people like myself make an incompatible change which breaks Ozone's DT support, we're just going to shrug and say "well, you should have written the tests"
Agree, test will be added. I have a WIP patch for DT system tests but there is a refactor of Ozone smoke test in HDDS-1255<#632>. I will post my test after that.
HDDS-1299 based on HADOOP-14445 are all Hadoop-3 only changes, it brings Ozone DT support. Before that, Ozone does not support DT. There is a pending HADOOP-14445 patch for Hadoop-2.
We could build support for Hadoop-2 after that in a separate ozone branch.
Thanks,
Xiaoyu
From: Steve Loughran <[email protected]>
Reply-To: apache/hadoop <[email protected]>
Date: Monday, March 25, 2019 at 8:58 AM
To: apache/hadoop <[email protected]>
Cc: Xiaoyu Yao <[email protected]>, State change <[email protected]>
Subject: Re: [apache/hadoop] HDDS-1299. Support TokenIssuer interface for running jobs with OzoneFileSystem. (#627)
Given this has broken things, can I ask a process question here, specifically regarding Yetus's v-1
The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
There are lots of ways to test for DTs being ussued: The hdfs fs command, the hadoop dtuils, the actualyh M
* Marshalling/unmarshalling of tokens: org.apache.hadoop.fs.s3a.auth.delegation.TestS3ADelegationTokenSupport
* org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens
* org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationInFileystem
* org.apache.hadoop.fs.azurebfs.extensions.ITestAbfsDelegationTokens
You can even verify that MR cluster launch collects the tokens: org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob
Please. write tests. Apart from the MR job one its straightforward, and I've just given you all the clases to start with.
Without tests, when people like myself make an incompatible change which breaks Ozone's DT support, we're just going to shrug and say "well, you should have written the tests"
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub<#627 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AGto0K4e1tFGFPgU-8npgy5vjk8phZnUks5vaO-3gaJpZM4b9Vx2>.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.