HADOOP-17871. S3A CSE: minor tuning #3412

mehakmeet · 2021-09-09T12:07:54Z

Region: ap-south-1
mvn clean verify -Dparallel-tests -DtestsThreadCount=4 -Dscale

UT:

[INFO] Results:
[INFO] 
[WARNING] Tests run: 575, Failures: 0, Errors: 0, Skipped: 5

IT:

[INFO] Results:
[INFO] 
[ERROR] Tests run: 1478, Failures: 0, Errors: 2, Skipped: 466

IT-scale:

[INFO] 
[ERROR] Tests run: 151, Failures: 0, Errors: 1, Skipped: 28

CSE-test:

[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.17 s - in org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms

mehakmeet · 2021-09-09T12:08:09Z

CC: @steveloughran @mukund-thakur

steveloughran

production code looks OK, apart from one query.

I'm worried about complications in testing.

I'm likely to have a test setup with KMS encryption,using the old values

  <property>
    <name>fs.s3a.bucket.hwdev-steve-london.server-side-encryption-algorithm</name>
    <value>SSE-KMS</value>
  </property>
  <property>
    <name>fs.s3a.bucket.stevel-london.server-side-encryption.key</name>
    <value>${london.kms.key}</value>
  </property>

I was thinking about how best I could retain the settings and still
test consistently across different versions.

I think I'm going to have to

Set both sets of options, explicitly
Comment both sets out when I want to disable it

What would be good is for all test cases which call removeBaseAndBucketOverrides() to remove encryption options to still remove the old options. This will stop any problems surfacing where the old option is still so the FS instance will be encrypted.

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestSSEConfiguration.java

...rc/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ITestSessionDelegationInFileystem.java

...hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3AHugeFilesSSECDiskBlocks.java

hadoop-yetus · 2021-09-09T15:37:36Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 59s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 19 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 27s		Maven dependency ordering for branch
+1 💚	mvninstall	23m 7s		trunk passed
+1 💚	compile	23m 6s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	19m 37s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	3m 51s		trunk passed
+1 💚	mvnsite	2m 22s		trunk passed
+1 💚	javadoc	1m 32s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 11s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 39s		trunk passed
+1 💚	shadedclient	17m 7s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 31s		the patch passed
+1 💚	compile	22m 17s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	22m 17s		the patch passed
+1 💚	compile	19m 38s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	19m 38s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	3m 49s	/results-checkstyle-root.txt	root: The patch generated 6 new + 170 unchanged - 5 fixed = 176 total (was 175)
+1 💚	mvnsite	2m 23s		the patch passed
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	javadoc	1m 31s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 14s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	4m 1s		the patch passed
+1 💚	shadedclient	17m 21s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	16m 51s		hadoop-common in the patch passed.
+1 💚	unit	2m 33s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 51s		The patch does not generate ASF License warnings.
		208m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/1/artifact/out/Dockerfile
GITHUB PR	#3412
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint
uname	Linux c7e848c98062 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `1516d40`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/1/testReport/
Max. process+thread count	3098 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java

...op-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AClientSideEncryption.java

steveloughran

all good except for some minor test tuning

...hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java

steveloughran · 2021-09-13T10:25:41Z

...hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java

-        fileSystem.initialize(fsURI, conf);
-        throw new Exception("Do not reach here");
-      });
+      Configuration conf = super.createConfiguration();


nit; if we generate that error "unknown server side algorithm", should we change to "unknown s3 encryption algorithm"

Seems like we are already returning "Unknown encryption algorithm " exception string in case of an unknown encryption algorithm. This test was just not running and hence, never failed.

...hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java

mehakmeet · 2021-09-13T11:02:49Z

@steveloughran, Just realized something in the test ITestS3AEncryptionAlgorithmValidation.java that it is tagged @Ignore, and doesn't actually run. That is why the string to be compared was outdated as well and would fail if not for the @Ignore tag. Guess we still make the changes?

steveloughran · 2021-09-13T12:56:11Z

it looks like the test has never run. That is, it went in broken and has stayed that way.

I don't see why we need to keep maintaining disabled tests. Either we turn it on (more grief) or we just remove entirely.

For now: I'll go with your PR as is, so we can backport it for 3.3.2; but really we can/should delete the entire test suite.

Funny: we'd never noticed until now.

mehakmeet · 2021-09-13T13:01:07Z

Yes, it's been like that since the test was created. Okay, let's maybe merge this in after yetus, and I can cherry pick into 3.3 with the rest of the commits.

hadoop-yetus · 2021-09-13T13:04:54Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 43s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 19 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 55s		Maven dependency ordering for branch
+1 💚	mvninstall	20m 24s		trunk passed
+1 💚	compile	21m 18s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	18m 29s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	3m 44s		trunk passed
+1 💚	mvnsite	2m 36s		trunk passed
+1 💚	javadoc	1m 49s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 28s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 47s		trunk passed
+1 💚	shadedclient	15m 33s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 30s		the patch passed
+1 💚	compile	20m 36s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
-1 ❌	javac	20m 36s	/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt	root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 14 new + 1917 unchanged - 0 fixed = 1931 total (was 1917)
+1 💚	compile	18m 29s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
-1 ❌	javac	18m 29s	/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt	root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~~20.04-b10 generated 14 new + 1790 unchanged - 0 fixed = 1804 total (was 1790)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	3m 32s	/results-checkstyle-root.txt	root: The patch generated 13 new + 137 unchanged - 38 fixed = 150 total (was 175)
+1 💚	mvnsite	2m 35s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	1m 45s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 28s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	4m 13s		the patch passed
+1 💚	shadedclient	15m 1s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 10s		hadoop-common in the patch passed.
+1 💚	unit	2m 31s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 59s		The patch does not generate ASF License warnings.
		198m 46s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/2/artifact/out/Dockerfile
GITHUB PR	#3412
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint
uname	Linux e331a69896b5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `05e2636`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/2/testReport/
Max. process+thread count	2430 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

mehakmeet · 2021-09-13T13:17:36Z

@steveloughran can we ignore javac errors, which are regarding old keys being used in removeBucketOverides since they are deprecated now? some of the javac errors are not from this patch also.

Checkstyle errors are in that disabled test. Guess, the indentation is still wrong somehow.

mehakmeet · 2021-09-13T13:25:57Z

In the last Yetus run, checkstyle says the correct indentation is level 6 not 8 and in this one, it says it should be 8, 10, or 12 and not 6.

hadoop-yetus · 2021-09-13T15:48:01Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 0s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 19 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 55s		Maven dependency ordering for branch
+1 💚	mvninstall	27m 1s		trunk passed
+1 💚	compile	29m 42s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	24m 21s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	4m 28s		trunk passed
+1 💚	mvnsite	2m 57s		trunk passed
+1 💚	javadoc	1m 54s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 36s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	4m 42s		trunk passed
+1 💚	shadedclient	18m 41s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 56s		the patch passed
+1 💚	compile	28m 24s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
-1 ❌	javac	28m 24s	/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt	root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 14 new + 1917 unchanged - 0 fixed = 1931 total (was 1917)
+1 💚	compile	24m 21s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
-1 ❌	javac	24m 21s	/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt	root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~~20.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~~20.04-b10 generated 14 new + 1790 unchanged - 0 fixed = 1804 total (was 1790)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 21s	/results-checkstyle-root.txt	root: The patch generated 13 new + 137 unchanged - 38 fixed = 150 total (was 175)
+1 💚	mvnsite	2m 53s		the patch passed
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	javadoc	1m 49s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 38s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	5m 5s		the patch passed
+1 💚	shadedclient	19m 3s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	19m 45s		hadoop-common in the patch passed.
+1 💚	unit	2m 57s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 58s		The patch does not generate ASF License warnings.
		248m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/3/artifact/out/Dockerfile
GITHUB PR	#3412
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint
uname	Linux 89f76a61c5a6 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `34944b9`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/3/testReport/
Max. process+thread count	1352 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran

+1 for the last patch; will wait for yetus to approve as the final due diligence

hadoop-yetus · 2021-09-14T14:53:17Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 54s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 19 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 50s		Maven dependency ordering for branch
+1 💚	mvninstall	20m 9s		trunk passed
+1 💚	compile	21m 17s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	18m 25s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	3m 43s		trunk passed
+1 💚	mvnsite	2m 37s		trunk passed
+1 💚	javadoc	1m 46s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 29s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 50s		trunk passed
+1 💚	shadedclient	14m 48s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 31s		the patch passed
+1 💚	compile	20m 42s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	20m 42s		the patch passed
+1 💚	compile	18m 34s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	18m 34s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	3m 34s	/results-checkstyle-root.txt	root: The patch generated 13 new + 137 unchanged - 38 fixed = 150 total (was 175)
+1 💚	mvnsite	2m 35s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	1m 46s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 27s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	4m 10s		the patch passed
+1 💚	shadedclient	15m 13s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 12s		hadoop-common in the patch passed.
+1 💚	unit	2m 25s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		197m 49s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/4/artifact/out/Dockerfile
GITHUB PR	#3412
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint
uname	Linux 13bce504b68a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `57964fc`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/4/testReport/
Max. process+thread count	2804 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3412/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

This migrates the fs.s3a-server-side encryption configuration options to a name which covers client-side encryption too. fs.s3a.server-side-encryption-algorithm becomes fs.s3a.encryption.algorithm fs.s3a.server-side-encryption.key becomes fs.s3a.encryption.key The existing keys remain valid, simply deprecated and remapped to the new values. If you want server-side encryption options to be picked up regardless of hadoop versions, use the old keys. (the old key also works for CSE, though as no version of Hadoop with CSE support has shipped without this remapping, it's less relevant) Contributed by: Mehakmeet Singh

This migrates the fs.s3a-server-side encryption configuration options to a name which covers client-side encryption too. fs.s3a.server-side-encryption-algorithm becomes fs.s3a.encryption.algorithm fs.s3a.server-side-encryption.key becomes fs.s3a.encryption.key The existing keys remain valid, simply deprecated and remapped to the new values. If you want server-side encryption options to be picked up regardless of hadoop versions, use the old keys. (the old key also works for CSE, though as no version of Hadoop with CSE support has shipped without this remapping, it's less relevant) Contributed by: Mehakmeet Singh Change-Id: I51804b21b287dbce18864f0a6ad17126aba2b281

This migrates the fs.s3a-server-side encryption configuration options to a name which covers client-side encryption too. fs.s3a.server-side-encryption-algorithm becomes fs.s3a.encryption.algorithm fs.s3a.server-side-encryption.key becomes fs.s3a.encryption.key The existing keys remain valid, simply deprecated and remapped to the new values. If you want server-side encryption options to be picked up regardless of hadoop versions, use the old keys. (the old key also works for CSE, though as no version of Hadoop with CSE support has shipped without this remapping, it's less relevant) Contributed by: Mehakmeet Singh

HADOOP-17195. S3A CSE: minor tuning

1516d40

mehakmeet changed the title ~~HADOOP-17195. S3A CSE: minor tuning~~ HADOOP-17871. S3A CSE: minor tuning Sep 9, 2021

steveloughran requested changes Sep 9, 2021

View reviewed changes

mukund-thakur reviewed Sep 13, 2021

View reviewed changes

HADOOP-17871. review comments

05e2636

steveloughran requested changes Sep 13, 2021

View reviewed changes

HADOOP-17871. test fix

34944b9

HADOOP-17871. suppress deprecation warnings

57964fc

steveloughran approved these changes Sep 14, 2021

View reviewed changes

steveloughran merged commit c54bf19 into apache:trunk Sep 15, 2021

HADOOP-17871. S3A CSE: minor tuning #3412

HADOOP-17871. S3A CSE: minor tuning #3412

Uh oh!

Conversation

mehakmeet commented Sep 9, 2021

UT:

IT:

IT-scale:

CSE-test:

Uh oh!

mehakmeet commented Sep 9, 2021

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Sep 9, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

steveloughran Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

mehakmeet Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mehakmeet commented Sep 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steveloughran commented Sep 13, 2021

Uh oh!

mehakmeet commented Sep 13, 2021

Uh oh!

hadoop-yetus commented Sep 13, 2021

Uh oh!

mehakmeet commented Sep 13, 2021

Uh oh!

mehakmeet commented Sep 13, 2021

Uh oh!

hadoop-yetus commented Sep 13, 2021

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Sep 14, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mehakmeet commented Sep 13, 2021 •

edited

Loading