Rev: HADOOP-18178 & HADOOP-18033 to Fix Class conflicts in downstreams

[ayushtkn]

Reverting the original commits upgrading Jackson, Fixed the conflicts in revert.

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[pjfanning]

There have been public releases of Hadoop with the jackson upgrades - eg https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common/3.3.3

Is there anything that can be done in the downstream projects to try to fix their builds to use the up to date and CVE free Jackson version used in Hadoop v3.3.3?

[ayushtkn]

@pjfanning Yes, that CVE fixed in a released version is a problem. But downstream projects don't have an option I think. There are two dependencies coming in and conflicting, since they have same classes, Jersey upgrade can be a solution at Hadoop, but that also leads to incompatible changes(Our initial assumptions & past experiences).
Bunch of details here:
#4461
and in the end here:
https://issues.apache.org/jira/browse/HADOOP-18033

It leads to issues with Spark, Tez, Hive & kyuubi(apache/kyuubi#2904), The Tez jira and other details are also linked in HADOOP-18033.
Do let me know your thoughts? Plan is to put in the release notes and flag it may be in the release announcement and so, and re-work the Jackson upgrade along with Jersey without blocking any release lines

[pjfanning]

What I don't get is what has rs-api got to do with jackson? Any idea why HADOOP-18033 made the rs-api changes?

[pjfanning]

The Tez issue is not really described - ie no comments on the PR or the jira issue. It looks like it could be a mockito issue - apache/tez#213 (comment)

[ayushtkn]

You checked the second-last build, It indeed was a Mockito issue, we discussed internally and upgraded mockito and that got sorted, Check the last build result, answered here:
apache/tez#213 (comment)

Any idea why HADOOP-18033 made the rs-api changes?

There is a comment telling that, it is required:
https://issues.apache.org/jira/browse/HADOOP-18033?focusedCommentId=17454049&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17454049

I guess may be a requirement for Jackson? I haven't checked if it was a false claim or mis-judgement, but I think some tests were failing due to this, the build results are gone, so I can't see what actually was the problem

[pjfanning]

rs-api is not used by the main jackson jars but could be used by the jackson-jaxrs module

In Tez build, would it be possible to exclude the rs-api jar?

For apache/kyuubi#2904 would it be feasible to just concentrate on why org.apache.hadoop.yarn.util.timeline.TimelineUtils.(TimelineUtils.java:60) is trying to use a non-existent class? Could there be an issue in the hadoop build or could kyuubi be using hadoop jars that are mismatched version wise?

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 1s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 18s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 25s		trunk passed
+1 💚	compile	27m 43s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	26m 15s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 50s		trunk passed
+1 💚	mvnsite	23m 25s		trunk passed
+1 💚	javadoc	9m 35s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	8m 11s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+0 🆗	spotbugs	0m 27s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 29s		branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 27s		branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 26s		branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	64m 40s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 44s		Maven dependency ordering for patch
+1 💚	mvninstall	45m 21s		the patch passed
+1 💚	compile	26m 46s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	26m 46s		the patch passed
+1 💚	compile	21m 36s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
-1 ❌	javac	21m 36s	/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt	root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu120.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu120.04-b07 generated 1 new + 2677 unchanged - 2 fixed = 2678 total (was 2679)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 22s		the patch passed
+1 💚	mvnsite	19m 30s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	8m 23s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	7m 29s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+0 🆗	spotbugs	0m 25s		hadoop-project has no data from spotbugs
+0 🆗	spotbugs	0m 33s		hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests has no data from spotbugs
+0 🆗	spotbugs	0m 30s		hadoop-client-modules/hadoop-client-runtime has no data from spotbugs
+0 🆗	spotbugs	0m 31s		hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs
+1 💚	shadedclient	56m 51s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1019m 48s		root in the patch passed.
+1 💚	asflicense	2m 4s		The patch does not generate ASF License warnings.
		1476m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4544/1/artifact/out/Dockerfile
GITHUB PR	#4544
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs
uname	Linux a4533a1e3c55 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 4f0321e
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4544/1/testReport/
Max. process+thread count	3135 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4544/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[pjfanning]

@ayushtkn my PR failed (#4547) - it seems even jackson 2.12 causes issues - it looks like your PR is a better bet

I would still ask for one change to your PR, if that's ok? The jackson versions could be set to 2.9.10 and jackson-databind to 2.9.10.8

• https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind -- note how many CVEs are fixed in v2.9.10.8

If you are looking to use a new Jira issue for your change, feel free to use HADOOP-18332 and modify as you see fit.