HADOOP-17844. Upgrade JSON smart to 2.4.7

[prasad-acit]

HADOOP-17844. Upgrade JSON smart to 2.4.7

[aajisaka]

+1 pending Jenkins

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 47s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 32s		trunk passed
+1 💚	compile	24m 49s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	19m 25s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	mvnsite	25m 13s		trunk passed
+1 💚	javadoc	7m 45s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	7m 52s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	shadedclient	31m 14s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	3m 26s		Maven dependency ordering for patch
+1 💚	mvninstall	22m 20s		the patch passed
+1 💚	compile	22m 13s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	22m 13s		the patch passed
+1 💚	compile	19m 16s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	19m 16s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	21m 2s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	7m 42s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	7m 50s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	shadedclient	32m 6s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	1022m 47s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 28s		The patch does not generate ASF License warnings.
		1287m 51s

Reason	Tests
Failed junit tests	hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure
	hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor
	hadoop.hdfs.server.namenode.TestDecommissioningStatus
	hadoop.tools.dynamometer.TestDynamometerInfra
	hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3299/1/artifact/out/Dockerfile
GITHUB PR	#3299
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml shellcheck shelldocs
uname	Linux 91f2d78e2876 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / dbe1207
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3299/1/testReport/
Max. process+thread count	2484 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3299/1/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[prasad-acit]

Thank you @aajisaka for quick review and feedback.
Jenkins got completed and no new failures. Also, it's a third party upgrade, no new tests added.

[aajisaka]

+1, the test failures are not related.

[aajisaka]

Thank you @prasad-acit !

[prasad-acit]

Thank you @aajisaka for review & processing it further.

[sangys]

@prasad-acit , can you do more help to check the upgrade? From hadoop-client-runtime.jar version 3.3.2 ~ 3.3.4, the shaded jason-smart still showing version 1.3.2. Which cause vulnerability scan report CVE-2021-31684 issue.

[degant]

@steveloughran as pointed out by @sangys there are still 3 jars as part of hadoop 3.3.4 that include json-smart 1.3.2. As a result hadoop 3.3.4 continues to get flagged for CVE-2021-31684

I used the below to check all versions of json-smart inside all packages:

jar_files=$(find . -iname "*.jar" | xargs -I '{}' sh -c "jar tf '{}' | grep -e 'json-smart' -q --label='{}' && echo '{}' ")
echo "$jar_files" | xargs -I '{}' sh -c "bsdtar -xO -f '{}' 'META-INF/maven/net.minidev/json-smart/pom.properties' | grep -i version && echo '{}'"

which returns

version=1.3.2
./share/hadoop/client/hadoop-client-runtime-3.3.4.jar
version=1.3.2
./share/hadoop/hdfs/lib/nimbus-jose-jwt-9.8.1.jar
version=2.4.7
./share/hadoop/hdfs/lib/json-smart-2.4.7.jar
version=1.3.2
./share/hadoop/common/lib/nimbus-jose-jwt-9.8.1.jar
version=2.4.7
./share/hadoop/common/lib/json-smart-2.4.7.jar

hadoop-client-runtime-3.3.4 and nimbus-jose-jwt-9.8.1 both include the shaded json-smart 1.3.2

[steveloughran]

this is 3.3.5; looks like nimbus-jose has it; the client jar simply includes that

version=2.4.7
./share/hadoop/common/lib/json-smart-2.4.7.jar
version=1.3.2
./share/hadoop/common/lib/nimbus-jose-jwt-9.8.1.jar
version=2.4.7
./share/hadoop/hdfs/lib/json-smart-2.4.7.jar
version=1.3.2
./share/hadoop/hdfs/lib/nimbus-jose-jwt-9.8.1.jar
version=1.3.2
./share/hadoop/client/hadoop-client-runtime-3.3.5.jar

@degant feel free to start work on a patch to upgrade nimbus-jose-jwt. If you could show that this introduced any server-side vulnerabilities, that would help.