Skip to content

HADOOP-15710. ABFS checkException to map 403 to AccessDeniedException.#2648

Merged
steveloughran merged 4 commits intoapache:trunkfrom
steveloughran:abfs/HADOOP-15710-access-denied
Feb 2, 2021
Merged

HADOOP-15710. ABFS checkException to map 403 to AccessDeniedException.#2648
steveloughran merged 4 commits intoapache:trunkfrom
steveloughran:abfs/HADOOP-15710-access-denied

Conversation

@steveloughran
Copy link
Copy Markdown
Contributor

This maps both 401 and 403 to AccessDeniedException.

The test does not work, because even though I'm removing permissions
from the file and parent dir, it's still opening.

Tested: Azure cardiff

This maps both 401 and 403 to AccessDeniedException.

The test does not work, because even though I'm removing permissions
from the file and parent dir, it's still opening.

Change-Id: Ieb65eb6bb057085b312fb1582eaf65c55ef2e634
Copy link
Copy Markdown
Contributor

@mehakmeet mehakmeet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

production code looks good but the test might need another commit as mentioned by you above.

Change-Id: I5fa1a1f5da7f581bd22571f2ce1f1c8e5b896398
@apache apache deleted a comment from hadoop-yetus Jan 28, 2021
@steveloughran
Copy link
Copy Markdown
Contributor Author

Updated with the other place this exception is looked for. But I don't have the auth setup to run those tests (I need to spend another few hours trying to get them).

The new test is failing, that is, I can remove permissions from a file and then read the results. That shouldn't happen unless those permissions aren't actually being enforced on my store

Change-Id: I55f45d7bdb2861b21a96ae19699da64438921abf
@steveloughran
Copy link
Copy Markdown
Contributor Author

logs from failure of latest code

2021-01-28 16:35:26,888 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(293)) - HttpRequest: Flush: 200,,,cid=b0cf224d-20d0-4862-a1c3-aa381ca2b7f4,rid=8e1b4c7b-501f-0008-3f93-f5417a000000,sent=0,recv=0,PUT,https://stevelukwest.dfs.core.windows.net/abfs-testcontainer-fdfe14a4-f838-406b-ba15-e21d1a438504/user/stevel/testPermissionDenied/file?action=flush&retainUncommittedData=false&position=27&close=true&timeout=90
2021-01-28 16:35:26,890 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:execute(185)) - First execution of REST operation - SetPermissions
2021-01-28 16:35:26,890 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(229)) - Signing request with shared key
2021-01-28 16:35:26,962 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(293)) - HttpRequest: SetPermissions: 200,,,cid=e7617275-e7a0-4d55-8ec0-b846e5cbfd38,rid=8e1b4c7c-501f-0008-4093-f5417a000000,sent=0,recv=0,PUT,https://stevelukwest.dfs.core.windows.net/abfs-testcontainer-fdfe14a4-f838-406b-ba15-e21d1a438504/user/stevel/testPermissionDenied/file?action=setAccessControl&timeout=90
2021-01-28 16:35:26,963 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:execute(185)) - First execution of REST operation - SetPermissions
2021-01-28 16:35:26,964 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(229)) - Signing request with shared key
2021-01-28 16:35:26,988 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(293)) - HttpRequest: SetPermissions: 200,,,cid=aed82242-8582-482b-9592-c72dfb94a05b,rid=8e1b4c7d-501f-0008-4193-f5417a000000,sent=0,recv=0,PUT,https://stevelukwest.dfs.core.windows.net/abfs-testcontainer-fdfe14a4-f838-406b-ba15-e21d1a438504/user/stevel/testPermissionDenied?action=setAccessControl&timeout=90
2021-01-28 16:35:26,995 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:execute(185)) - First execution of REST operation - GetPathStatus
2021-01-28 16:35:26,996 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(229)) - Signing request with shared key
2021-01-28 16:35:27,014 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(293)) - HttpRequest: GetPathStatus: 200,,,cid=f1e022fb-f730-475b-a959-9b630e06e56b,rid=8e1b4c7e-501f-0008-4293-f5417a000000,sent=0,recv=0,HEAD,https://stevelukwest.dfs.core.windows.net/abfs-testcontainer-fdfe14a4-f838-406b-ba15-e21d1a438504/user/stevel/testPermissionDenied/file?upn=false&action=getStatus&timeout=90
2021-01-28 16:35:27,024 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:execute(185)) - First execution of REST operation - GetPathStatus
2021-01-28 16:35:27,024 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(229)) - Signing request with shared key
2021-01-28 16:35:27,046 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(293)) - HttpRequest: GetPathStatus: 200,,,cid=e25824c9-400c-43cf-90a2-0c8a13994e89,rid=8e1b4c7f-501f-0008-4393-f5417a000000,sent=0,recv=0,HEAD,https://stevelukwest.dfs.core.windows.net/abfs-testcontainer-fdfe14a4-f838-406b-ba15-e21d1a438504/user/stevel/testPermissionDenied/file?upn=false&action=getStatus&timeout=90
2021-01-28 16:35:27,165 DEBUG [ABFS-prefetch-6]: services.AbfsClient (AbfsRestOperation.java:execute(185)) - First execution of REST operation - ReadFile
2021-01-28 16:35:27,165 DEBUG [ABFS-prefetch-6]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(229)) - Signing request with shared key
2021-01-28 16:35:27,463 DEBUG [ABFS-prefetch-6]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(293)) - HttpRequest: ReadFile: 206,,,cid=b734c4ec-9dc9-4705-9904-bdd2148d1467,rid=8e1b4c80-501f-0008-4493-f5417a000000,sent=0,recv=27,GET,https://stevelukwest.dfs.core.windows.net/abfs-testcontainer-fdfe14a4-f838-406b-ba15-e21d1a438504/user/stevel/testPermissionDenied/file?timeout=90
2021-01-28 16:35:27,464 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:execute(185)) - First execution of REST operation - DeleteFileSystem
2021-01-28 16:35:27,464 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(229)) - Signing request with shared key
2021-01-28 16:35:27,506 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(293)) - HttpRequest: DeleteFileSystem: 202,,,cid=bcff9a9c-7949-4b3d-adb9-fef36e04348b,rid=8e1b4c82-501f-0008-4593-f5417a000000,sent=0,recv=0,DELETE,https://stevelukwest.dfs.core.windows.net/abfs-testcontainer-fdfe14a4-f838-406b-ba15-e21d1a438504?resource=filesystem&timeout=90
2021-01-28 16:35:27,515 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:execute(185)) - First execution of REST operation - GetFileSystemProperties
2021-01-28 16:35:27,516 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(229)) - Signing request with shared key
2021-01-28 16:35:27,539 DEBUG [JUnit-testPermissionDenied]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(293)) - HttpRequest: GetFileSystemProperties: 404,,,cid=6c7b2ba3-8ea1-4b03-9b67-708e7c07ee06,rid=8e1b4c83-501f-0008-4693-f5417a000000,sent=0,recv=0,HEAD,https://stevelukwest.dfs.core.windows.net/abfs-testcontainer-fdfe14a4-f838-406b-ba15-e21d1a438504?resource=filesystem&timeout=90

java.lang.AssertionError: Expected a java.nio.file.AccessDeniedException to be thrown, but got the result: : "This should not be readable"

	at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
	at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
	at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:453)
	at org.apache.hadoop.fs.azurebfs.ITestAbfsRestOperationException.testPermissionDenied(ITestAbfsRestOperationException.java:146)

@hadoop-yetus
Copy link
Copy Markdown

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 38s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 0m 0s test4tests The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 32m 31s trunk passed
+1 💚 compile 0m 38s trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 compile 0m 33s trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 checkstyle 0m 26s trunk passed
+1 💚 mvnsite 0m 38s trunk passed
+1 💚 shadedclient 14m 21s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 31s trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 0m 29s trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+0 🆗 spotbugs 1m 0s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 0m 58s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 28s the patch passed
+1 💚 compile 0m 30s the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javac 0m 30s the patch passed
+1 💚 compile 0m 25s the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 javac 0m 25s the patch passed
+1 💚 checkstyle 0m 17s the patch passed
+1 💚 mvnsite 0m 27s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 12m 53s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 26s the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 0m 24s the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 findbugs 0m 58s the patch passed
_ Other Tests _
+1 💚 unit 2m 5s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 32s The patch does not generate ASF License warnings.
73m 4s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2648/3/artifact/out/Dockerfile
GITHUB PR #2648
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux dc01cabbc469 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / f8769e0
Default Java Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2648/3/testReport/
Max. process+thread count 539 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2648/3/console
versions git=2.25.1 maven=3.6.3 findbugs=4.0.6
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@apache apache deleted a comment from hadoop-yetus Jan 28, 2021
@mehakmeet
Copy link
Copy Markdown
Contributor

mehakmeet commented Jan 29, 2021

Yes, pretty weird that you can still read the file with no permissions, maybe add

LOG.info(fs.getFileStatus(path).getPermission());

to see if the permissions are actually getting set or not too.

Edit: I added this before intercept() and got "---------" returned.
Edit2: We are the Super User and would be able to access the file with any Permission setting. Thanks for the clarification @steveloughran

Needed to make the translation method static and package scoped

Change-Id: I1120bac9d271f224ef117d49ce82fd61e91e8149
@hadoop-yetus
Copy link
Copy Markdown

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 11s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 0m 0s test4tests The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 32m 40s trunk passed
+1 💚 compile 0m 39s trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 compile 0m 34s trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 checkstyle 0m 27s trunk passed
+1 💚 mvnsite 0m 38s trunk passed
+1 💚 shadedclient 14m 25s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 32s trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 0m 29s trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+0 🆗 spotbugs 0m 59s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 0m 56s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 28s the patch passed
+1 💚 compile 0m 29s the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javac 0m 29s the patch passed
+1 💚 compile 0m 26s the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 javac 0m 26s the patch passed
+1 💚 checkstyle 0m 18s the patch passed
+1 💚 mvnsite 0m 30s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 13m 7s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 26s the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 0m 25s the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚 findbugs 0m 58s the patch passed
_ Other Tests _
+1 💚 unit 2m 0s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 34s The patch does not generate ASF License warnings.
74m 28s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2648/4/artifact/out/Dockerfile
GITHUB PR #2648
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux c2dcc9365af3 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / fa15594
Default Java Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2648/4/testReport/
Max. process+thread count 601 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2648/4/console
versions git=2.25.1 maven=3.6.3 findbugs=4.0.6
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Copy link
Copy Markdown
Contributor

@mehakmeet mehakmeet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@mukund-thakur
Copy link
Copy Markdown
Contributor

UT's is good option here. +1.
BTW likewise we added test for others, we can add one more for HTTP_UNAUTHORIZED translation as well.

@steveloughran
Copy link
Copy Markdown
Contributor Author

thanks. merging.

@steveloughran steveloughran merged commit f37bf65 into apache:trunk Feb 2, 2021
asfgit pushed a commit that referenced this pull request Feb 2, 2021
#2648)

When 403 is returned from an ABFS HTTP call, an AccessDeniedException is raised.
The exception text is unchanged, for any application string matching on the getMessage() contents.

Contributed by Steve Loughran.

Change-Id: I519d50ccd657968fd8ee72d132518099de901e15
@steveloughran steveloughran deleted the abfs/HADOOP-15710-access-denied branch October 15, 2021 19:48
jojochuang pushed a commit to jojochuang/hadoop that referenced this pull request May 23, 2023
…iedException. (apache#2648)

When 403 is returned from an ABFS HTTP call, an AccessDeniedException is raised.
The exception text is unchanged, for any application string matching on the getMessage() contents.

Contributed by Steve Loughran.

Change-Id: I519d50ccd657968fd8ee72d132518099de901e15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants