HADOOP-17343. Upgrade aws-java-sdk to 1.11.892 #2429

dongjoon-hyun · 2020-11-03T17:14:13Z

This PR aims to upgrade aws-java-sdk from 1.1.563 (May 30, 2019) to 1.11.892 (Nov 01, 2020).

hadoop-yetus · 2020-11-04T03:13:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	33m 48s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 1s		Maven dependency ordering for branch
+1 💚	mvninstall	20m 26s		trunk passed
+1 💚	compile	20m 6s		trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	compile	17m 25s		trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚	mvnsite	25m 18s		trunk passed
+1 💚	shadedclient	15m 28s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 11s		trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	javadoc	7m 27s		trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 44s		Maven dependency ordering for patch
+1 💚	mvninstall	20m 10s		the patch passed
+1 💚	compile	19m 36s		the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	javac	19m 36s		the patch passed
+1 💚	compile	17m 23s		the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚	javac	17m 23s		the patch passed
+1 💚	mvnsite	20m 11s		the patch passed
+1 💚	shellcheck	0m 0s		There were no new shellcheck issues.
+1 💚	shelldocs	0m 18s		There were no new shelldocs issues.
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	shadedclient	15m 7s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 7s		the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	javadoc	7m 24s		the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
			_ Other Tests _
-1 ❌	unit	321m 54s	/patch-unit-root.txt	root in the patch failed.
+0 🆗	asflicense	0m 44s		ASF License check generated no output?
		596m 59s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestFileChecksumCompositeCrc
	hadoop.hdfs.TestFileChecksum
	hadoop.hdfs.server.namenode.TestNameNodeMXBean
	hadoop.security.TestLdapGroupsMapping
	hadoop.yarn.client.api.impl.TestAMRMClient
	hadoop.yarn.server.nodemanager.TestNodeManagerShutdown
	hadoop.yarn.server.nodemanager.TestNodeManagerResync
	hadoop.yarn.server.nodemanager.TestDeletionService
	hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch
	hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
	hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
	hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
	hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2429/2/artifact/out/Dockerfile
GITHUB PR	#2429
Optional Tests	dupname asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname	Linux b6e1678215a7 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e580280`
Default Java	Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2429/2/testReport/
Max. process+thread count	3617 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2429/2/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-11-04T08:20:48Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	33m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 50s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 49s		trunk passed
+1 💚	compile	21m 40s		trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	compile	18m 18s		trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚	mvnsite	26m 5s		trunk passed
+1 💚	shadedclient	16m 47s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 20s		trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	javadoc	7m 35s		trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 40s		Maven dependency ordering for patch
+1 💚	mvninstall	23m 34s		the patch passed
+1 💚	compile	22m 27s		the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	javac	22m 27s		the patch passed
+1 💚	compile	19m 24s		the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚	javac	19m 25s		the patch passed
+1 💚	mvnsite	20m 46s		the patch passed
+1 💚	shellcheck	0m 0s		There were no new shellcheck issues.
+1 💚	shelldocs	0m 14s		There were no new shelldocs issues.
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	shadedclient	16m 41s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 19s		the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚	javadoc	7m 38s		the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
			_ Other Tests _
-1 ❌	unit	612m 12s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 27s		The patch does not generate ASF License warnings.
		905m 27s

Reason	Tests
Failed junit tests	hadoop.security.TestLdapGroupsMapping
	hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
	hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
	hadoop.fs.azure.TestNativeAzureFileSystemMocked
	hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
	hadoop.fs.azure.TestOutOfBandAzureBlobOperations
	hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
	hadoop.fs.azure.TestBlobMetadata
	hadoop.fs.azure.TestWasbFsck
	hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
	hadoop.yarn.server.nodemanager.TestNodeManagerShutdown
	hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
	hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch
	hadoop.yarn.server.nodemanager.TestNodeManagerResync
	hadoop.yarn.server.nodemanager.TestNodeManagerReboot
	hadoop.yarn.server.nodemanager.TestDeletionService
	hadoop.yarn.applications.distributedshell.TestDistributedShell
	hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
	hadoop.hdfs.TestFileChecksumCompositeCrc
	hadoop.hdfs.TestFileChecksum
	hadoop.hdfs.server.datanode.TestBPOfferService
	hadoop.hdfs.TestMultipleNNPortQOP

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2429/1/artifact/out/Dockerfile
GITHUB PR	#2429
Optional Tests	dupname asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname	Linux 0d9a8487b24d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e580280`
Default Java	Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2429/1/testReport/
Max. process+thread count	2814 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2429/1/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

liuml07 · 2020-11-04T17:46:33Z

Hi @dongjoon-hyun , did you have a look at https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/testing.html#Qualifying_an_AWS_SDK_Update? Thanks,

dongjoon-hyun · 2020-11-05T05:01:58Z

Oh, thank you so much for the guidance, @liuml07 ! I'll take a look.

dongjoon-hyun · 2020-11-05T05:03:17Z

Given the documentation, I'll close this PR for now.

dongjoon-hyun · 2020-11-05T05:05:14Z

cc @sunchao and @dbtsai

steveloughran · 2020-11-12T12:26:12Z

no, no, don't run away. All you have to do is follow the instructions. the fact that you are set up to build and test spark with is even better, as you'll get better downstream coverage.

AWS upgrades are always a bit of work as sometimes there are performance regressions and error messages related to us calling abort() on streams, even the shaded releases often accidentally export libraries from their poms, and if we have tests which expect specific errors, they can regress

But that doesn't mean we don't need these updates.

please, restore the patch and try out the process. It's primarily one of a bit of extra due diligence above just a "do the ITests pass"

steveloughran · 2020-11-12T12:27:24Z

(I've been thinking about it being time to upgrade for a while, certainly before any 3.3.1 release)

dongjoon-hyun · 2020-11-12T21:50:20Z

Sorry, @steveloughran . It's beyond what I can provide for now. I believe someone-else can do if he needs it.

no, no, don't run away. All you have to do is follow the instructions.

steveloughran · 2020-11-13T13:09:10Z

ok. I'll try and find the time fo rit

steveloughran · 2020-11-17T14:48:38Z

see #2468

dongjoon-hyun added 2 commits November 3, 2020 09:10

Upgrade aws-java-sdk to 1.11.892

a45ade8

Update LICENSE-binary

5732288

dongjoon-hyun changed the title ~~Upgrade aws-java-sdk to 1.11.892~~ HADOOP-17343. Upgrade aws-java-sdk to 1.11.892 Nov 3, 2020

dongjoon-hyun closed this Nov 5, 2020

dongjoon-hyun deleted the HADOOP-AWS branch November 5, 2020 05:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-17343. Upgrade aws-java-sdk to 1.11.892 #2429

HADOOP-17343. Upgrade aws-java-sdk to 1.11.892 #2429

Uh oh!

dongjoon-hyun commented Nov 3, 2020

Uh oh!

hadoop-yetus commented Nov 4, 2020

Uh oh!

hadoop-yetus commented Nov 4, 2020

Uh oh!

liuml07 commented Nov 4, 2020

Uh oh!

dongjoon-hyun commented Nov 5, 2020

Uh oh!

dongjoon-hyun commented Nov 5, 2020

Uh oh!

dongjoon-hyun commented Nov 5, 2020

Uh oh!

steveloughran commented Nov 12, 2020

Uh oh!

steveloughran commented Nov 12, 2020

Uh oh!

dongjoon-hyun commented Nov 12, 2020 •

edited

Loading

Uh oh!

steveloughran commented Nov 13, 2020

Uh oh!

steveloughran commented Nov 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HADOOP-17343. Upgrade aws-java-sdk to 1.11.892 #2429

HADOOP-17343. Upgrade aws-java-sdk to 1.11.892 #2429

Uh oh!

Conversation

dongjoon-hyun commented Nov 3, 2020

Uh oh!

hadoop-yetus commented Nov 4, 2020

Uh oh!

hadoop-yetus commented Nov 4, 2020

Uh oh!

liuml07 commented Nov 4, 2020

Uh oh!

dongjoon-hyun commented Nov 5, 2020

Uh oh!

dongjoon-hyun commented Nov 5, 2020

Uh oh!

dongjoon-hyun commented Nov 5, 2020

Uh oh!

steveloughran commented Nov 12, 2020

Uh oh!

steveloughran commented Nov 12, 2020

Uh oh!

dongjoon-hyun commented Nov 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steveloughran commented Nov 13, 2020

Uh oh!

steveloughran commented Nov 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dongjoon-hyun commented Nov 12, 2020 •

edited

Loading