HADOOP-16117 Update AWS SDK to 1.11.563

[steveloughran]

Contributed by Steve Loughran.

Change-Id: Iff125bf3e2f409ae7fd0d05440f4f7ba38c24331

[steveloughran]

Testing

• full s3a tests run in auth+nonauth, scale against s3 ireland
• test output reviewed

TODO:

• switch to v4 auth only, rerun all,
• do those FS commands to run on the CLI

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	33	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
0	mvndep	68	Maven dependency ordering for branch
+1	mvninstall	1041	trunk passed
+1	compile	1028	trunk passed
+1	mvnsite	837	trunk passed
+1	shadedclient	3649	branch has no errors when building and testing our client artifacts.
+1	javadoc	334	trunk passed
		_ Patch Compile Tests _
0	mvndep	34	Maven dependency ordering for patch
+1	mvninstall	1025	the patch passed
+1	compile	978	the patch passed
+1	javac	978	the patch passed
+1	mvnsite	817	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	633	patch has no errors when building and testing our client artifacts.
+1	javadoc	340	the patch passed
		_ Other Tests _
-1	unit	8824	root in the patch failed.
+1	asflicense	45	The patch does not generate ASF License warnings.
		16844

Reason	Tests
Failed junit tests	hadoop.hdfs.web.TestWebHdfsTimeouts

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-818/2/artifact/out/Dockerfile
GITHUB PR	#818
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname	Linux e0d54062f2f2 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 52128e3
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-818/2/artifact/out/patch-unit-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-818/2/testReport/
Max. process+thread count	4541 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-818/2/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	35	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
0	mvndep	66	Maven dependency ordering for branch
+1	mvninstall	1041	trunk passed
+1	compile	1014	trunk passed
+1	mvnsite	850	trunk passed
+1	shadedclient	3619	branch has no errors when building and testing our client artifacts.
+1	javadoc	329	trunk passed
		_ Patch Compile Tests _
0	mvndep	30	Maven dependency ordering for patch
+1	mvninstall	1026	the patch passed
+1	compile	986	the patch passed
+1	javac	986	the patch passed
+1	mvnsite	803	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	615	patch has no errors when building and testing our client artifacts.
+1	javadoc	323	the patch passed
		_ Other Tests _
-1	unit	954	root in the patch failed.
+1	asflicense	39	The patch does not generate ASF License warnings.
		8889

Reason	Tests
Failed junit tests	hadoop.util.TestDiskChecker

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-818/3/artifact/out/Dockerfile
GITHUB PR	#818
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname	Linux b39b6c1814a6 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 1ae062c
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-818/3/artifact/out/patch-unit-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-818/3/testReport/
Max. process+thread count	1347 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-818/3/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[steveloughran]

Latest patch: tested s3 ireland DDB+ scale. Followed SDK update protocol in docs and added some more commands for the runbook: expunge and hdfs fetchdt. Couldn't use hadoop dtutil on the CLI as I couldn't use a -D option to set the delegation tokens in that command.

I've been using this with the HADOOP-15663 PR to create on-demand DDB tables (#879) and I have not encountered problems. IMO, this update is ready to go with trunk for broader testing

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	32	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
0	mvndep	65	Maven dependency ordering for branch
+1	mvninstall	1042	trunk passed
+1	compile	1014	trunk passed
+1	mvnsite	844	trunk passed
+1	shadedclient	3636	branch has no errors when building and testing our client artifacts.
+1	javadoc	342	trunk passed
		_ Patch Compile Tests _
0	mvndep	23	Maven dependency ordering for patch
+1	mvninstall	1073	the patch passed
+1	compile	965	the patch passed
+1	javac	965	the patch passed
+1	mvnsite	828	the patch passed
-1	whitespace	0	The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	668	patch has no errors when building and testing our client artifacts.
+1	javadoc	379	the patch passed
		_ Other Tests _
-1	unit	8185	root in the patch failed.
+1	asflicense	46	The patch does not generate ASF License warnings.
		16342

Reason	Tests
Failed junit tests	hadoop.hdfs.TestMultipleNNPortQOP
	hadoop.hdfs.web.TestWebHdfsTimeouts

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-818/4/artifact/out/Dockerfile
GITHUB PR	#818
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname	Linux 69a492540875 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 3b1c257
Default Java	1.8.0_212
whitespace	https://builds.apache.org/job/hadoop-multibranch/job/PR-818/4/artifact/out/whitespace-eol.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-818/4/artifact/out/patch-unit-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-818/4/testReport/
Max. process+thread count	4427 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-tools/hadoop-aws . U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-818/4/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[ajfabbri]

+1 commit to trunk and continue integration testing.

[steveloughran]

merged into trunk