HADOOP-16505. Add ability to register custom signer with AWS SignerFactory #1280

viczsaurav · 2019-08-12T15:40:38Z

Currently, the AWS SignerFactory restricts the class of Signer algorithms that can be used.
We require an ability to register a custom Signer. The SignerFactory supports this functionality through its registerSigner method.

By providing a fully qualified classname to the existing parameter fs.s3a.signing-algorithm, the custom signer can be registered.

bgaborg · 2019-08-16T08:28:04Z

...tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ASigningAlgorithmOverride.java

+  }
+
+  @Test
+  public void testCustomSignerOverride() throws AssertionError {


Please use a testing principle in your test: Arrange, Act, Assert - http://wiki.c2.com/?ArrangeActAssert

...tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ASigningAlgorithmOverride.java

bgaborg · 2019-08-16T08:29:22Z

...tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ASigningAlgorithmOverride.java

+  @Test
+  public void testCustomSignerOverride() throws AssertionError {
+    LOG.debug("Inside createConfiguration...");
+    assertTrue(assertIsCustomSignerLoaded());


This does not need to be factored out in a separate method.

Refactored the code

...tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ASigningAlgorithmOverride.java

hadoop-yetus · 2019-08-16T14:32:04Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	67	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1075	trunk passed
+1	compile	31	trunk passed
+1	checkstyle	19	trunk passed
+1	mvnsite	35	trunk passed
+1	shadedclient	706	branch has no errors when building and testing our client artifacts.
+1	javadoc	23	trunk passed
0	spotbugs	52	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	51	trunk passed
-0	patch	70	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1	mvninstall	35	the patch passed
+1	compile	26	the patch passed
+1	javac	26	the patch passed
-0	checkstyle	18	hadoop-tools/hadoop-aws: The patch generated 2 new + 7 unchanged - 1 fixed = 9 total (was 8)
+1	mvnsite	27	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	746	patch has no errors when building and testing our client artifacts.
+1	javadoc	22	the patch passed
+1	findbugs	59	the patch passed
		_ Other Tests _
+1	unit	70	hadoop-aws in the patch passed.
+1	asflicense	28	The patch does not generate ASF License warnings.
		3113

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/1/artifact/out/Dockerfile
GITHUB PR	#1280
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux f8497e9eeb77 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `9b8359b`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/1/testReport/
Max. process+thread count	412 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-08-20T12:47:42Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	41	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1447	trunk passed
+1	compile	35	trunk passed
+1	checkstyle	25	trunk passed
+1	mvnsite	39	trunk passed
+1	shadedclient	808	branch has no errors when building and testing our client artifacts.
+1	javadoc	23	trunk passed
0	spotbugs	57	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	54	trunk passed
-0	patch	76	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1	mvninstall	64	the patch passed
+1	compile	27	the patch passed
+1	javac	27	the patch passed
-0	checkstyle	21	hadoop-tools/hadoop-aws: The patch generated 3 new + 7 unchanged - 1 fixed = 10 total (was 8)
+1	mvnsite	35	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	815	patch has no errors when building and testing our client artifacts.
+1	javadoc	22	the patch passed
+1	findbugs	71	the patch passed
		_ Other Tests _
+1	unit	70	hadoop-aws in the patch passed.
+1	asflicense	30	The patch does not generate ASF License warnings.
		3711

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/2/artifact/out/Dockerfile
GITHUB PR	#1280
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux e6e698e23c1e 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 6244502
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/2/testReport/
Max. process+thread count	351 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-08-22T12:03:27Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	37	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1250	trunk passed
+1	compile	31	trunk passed
+1	checkstyle	23	trunk passed
+1	mvnsite	38	trunk passed
+1	shadedclient	782	branch has no errors when building and testing our client artifacts.
+1	javadoc	25	trunk passed
0	spotbugs	57	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	55	trunk passed
-0	patch	76	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1	mvninstall	35	the patch passed
+1	compile	27	the patch passed
+1	javac	27	the patch passed
-0	checkstyle	19	hadoop-tools/hadoop-aws: The patch generated 3 new + 7 unchanged - 1 fixed = 10 total (was 8)
+1	mvnsite	34	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	857	patch has no errors when building and testing our client artifacts.
+1	javadoc	26	the patch passed
+1	findbugs	74	the patch passed
		_ Other Tests _
+1	unit	70	hadoop-aws in the patch passed.
+1	asflicense	30	The patch does not generate ASF License warnings.
		3490

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/3/artifact/out/Dockerfile
GITHUB PR	#1280
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux a376409a21ec 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `ee7c261`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/3/testReport/
Max. process+thread count	340 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-08-27T08:58:25Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	130	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1293	trunk passed
+1	compile	31	trunk passed
+1	checkstyle	23	trunk passed
+1	mvnsite	36	trunk passed
+1	shadedclient	816	branch has no errors when building and testing our client artifacts.
+1	javadoc	26	trunk passed
0	spotbugs	61	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	59	trunk passed
-0	patch	83	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1	mvninstall	36	the patch passed
+1	compile	28	the patch passed
+1	javac	27	the patch passed
-0	checkstyle	19	hadoop-tools/hadoop-aws: The patch generated 3 new + 7 unchanged - 1 fixed = 10 total (was 8)
+1	mvnsite	31	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	839	patch has no errors when building and testing our client artifacts.
+1	javadoc	27	the patch passed
+1	findbugs	68	the patch passed
		_ Other Tests _
+1	unit	77	hadoop-aws in the patch passed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		3657

Subsystem	Report/Notes
Docker	Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/4/artifact/out/Dockerfile
GITHUB PR	#1280
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 2dd52d56ba17 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `3329257`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/4/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/4/testReport/
Max. process+thread count	332 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/4/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-09-03T00:36:24Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	33	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1230	trunk passed
+1	compile	32	trunk passed
+1	checkstyle	44	trunk passed
+1	mvnsite	36	trunk passed
+1	shadedclient	807	branch has no errors when building and testing our client artifacts.
+1	javadoc	27	trunk passed
0	spotbugs	63	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	61	trunk passed
-0	patch	83	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1	mvninstall	34	the patch passed
+1	compile	27	the patch passed
+1	javac	27	the patch passed
-0	checkstyle	19	hadoop-tools/hadoop-aws: The patch generated 3 new + 7 unchanged - 1 fixed = 10 total (was 8)
+1	mvnsite	34	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	854	patch has no errors when building and testing our client artifacts.
+1	javadoc	25	the patch passed
+1	findbugs	61	the patch passed
		_ Other Tests _
+1	unit	92	hadoop-aws in the patch passed.
+1	asflicense	34	The patch does not generate ASF License warnings.
		3510

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/5/artifact/out/Dockerfile
GITHUB PR	#1280
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux e49c3ef3ed52 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `915cbc9`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/5/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/5/testReport/
Max. process+thread count	340 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1280/5/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

steveloughran · 2019-10-03T09:22:03Z

closing this as its been superceded by the other patches from Sid

Can you experiment with the current trunk branch and verify it meets your needs? If it doesn't -now is the time to make the appropriate changes

Saurav Verma added 12 commits August 12, 2019 17:36

hadoop-16505-Adding-custom-signer

c5a6a94

hadoop-16505-Adding-custom-signer

cee9853

Added tests

5657e67

removed the core-site property which brike other tests

dd8b5ff

renamed Signer class

0ecef36

Removing Tests for now

1f1322c

removing signer override factory

67c867e

working tests

be258a1

formatting + test remove

f789374

formatting + test remove

f33079f

formatting + test remove

6dd0635

Adding back tests

cb29c6b

bgaborg requested review from bgaborg and steveloughran August 14, 2019 13:44

bgaborg changed the title ~~Hadoop 16505- Add ability to register custom signer with AWS SignerFactory~~ HADOOP-16505. Add ability to register custom signer with AWS SignerFactory Aug 14, 2019

Merge branch 'trunk' into HADOOP-16505

0a6756b

bgaborg requested changes Aug 16, 2019

View reviewed changes

Saurav Verma added 2 commits August 19, 2019 11:25

Merge branch 'trunk' into HADOOP-16505

cc21be9

refactor test code

f4a2538

steveloughran removed their request for review August 19, 2019 12:35

steveloughran closed this Oct 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-16505. Add ability to register custom signer with AWS SignerFactory #1280

HADOOP-16505. Add ability to register custom signer with AWS SignerFactory #1280

Uh oh!

viczsaurav commented Aug 12, 2019 •

edited

Loading

Uh oh!

bgaborg Aug 16, 2019

Uh oh!

Uh oh!

bgaborg Aug 16, 2019

Uh oh!

viczsaurav Aug 21, 2019

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Aug 16, 2019

Uh oh!

hadoop-yetus commented Aug 20, 2019

Uh oh!

hadoop-yetus commented Aug 22, 2019

Uh oh!

hadoop-yetus commented Aug 27, 2019

Uh oh!

hadoop-yetus commented Sep 3, 2019

Uh oh!

steveloughran commented Oct 3, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HADOOP-16505. Add ability to register custom signer with AWS SignerFactory #1280

HADOOP-16505. Add ability to register custom signer with AWS SignerFactory #1280

Uh oh!

Conversation

viczsaurav commented Aug 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bgaborg Aug 16, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bgaborg Aug 16, 2019

Choose a reason for hiding this comment

Uh oh!

viczsaurav Aug 21, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Aug 16, 2019

Uh oh!

hadoop-yetus commented Aug 20, 2019

Uh oh!

hadoop-yetus commented Aug 22, 2019

Uh oh!

hadoop-yetus commented Aug 27, 2019

Uh oh!

hadoop-yetus commented Sep 3, 2019

Uh oh!

steveloughran commented Oct 3, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

viczsaurav commented Aug 12, 2019 •

edited

Loading