HADOOP-18687. Remove json-smart dependency #5549

pjfanning · 2023-04-12T21:15:38Z

complete removal of json-smart jar

hadoop-yetus · 2023-04-13T14:08:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	16m 13s		Maven dependency ordering for branch
+1 💚	mvninstall	26m 4s		trunk passed
+1 💚	compile	23m 7s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	20m 26s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	mvnsite	25m 24s		trunk passed
+1 💚	javadoc	8m 2s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	7m 16s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	shadedclient	34m 20s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
+1 💚	mvninstall	22m 0s		the patch passed
+1 💚	compile	22m 32s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	22m 32s		the patch passed
+1 💚	compile	20m 28s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	javac	20m 28s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	20m 6s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	8m 7s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	7m 20s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	shadedclient	36m 20s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	738m 19s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 32s		The patch does not generate ASF License warnings.
		1011m 34s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestDirectoryScanner
	hadoop.mapreduce.v2.TestMRJobs
	hadoop.mapreduce.v2.TestMRJobsWithProfiler
	hadoop.mapreduce.v2.TestUberAM

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5549/1/artifact/out/Dockerfile
GITHUB PR	#5549
JIRA Issue	HADOOP-18687
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname	Linux fca934463446 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e69ac0b`
Default Java	Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5549/1/testReport/
Max. process+thread count	3572 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5549/1/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-04-14T20:29:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	16m 12s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 30s		trunk passed
+1 💚	compile	23m 6s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	20m 29s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	mvnsite	25m 31s		trunk passed
+1 💚	javadoc	8m 2s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	7m 16s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	shadedclient	31m 36s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for patch
+1 💚	mvninstall	22m 13s		the patch passed
+1 💚	compile	22m 38s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	22m 38s		the patch passed
+1 💚	compile	20m 27s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	javac	20m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	24m 33s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	8m 15s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	7m 17s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	shadedclient	36m 19s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	738m 41s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 36s		The patch does not generate ASF License warnings.
		1013m 21s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestDirectoryScanner
	hadoop.mapreduce.v2.TestMRJobs
	hadoop.mapreduce.v2.TestMRJobsWithProfiler
	hadoop.mapreduce.v2.TestUberAM

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5549/2/artifact/out/Dockerfile
GITHUB PR	#5549
JIRA Issue	HADOOP-18687
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname	Linux e56c583cf682 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `fc03294`
Default Java	Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5549/2/testReport/
Max. process+thread count	3203 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5549/2/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn

LGTM

pjfanning · 2023-04-19T15:49:58Z

@ayushtkn I'm not sure yet about this change. Those test failures appear to be consistent issues.

ayushtkn · 2023-04-19T16:05:37Z

@pjfanning no worries, I am sure about them, they aren't related. They are failing in the daily build as well. Need to chase them, not getting time unfortunately

The daily build test result. I think all are same as in this PR
https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1199/testReport/

pjfanning · 2023-04-19T17:11:01Z

Thanks @ayushtkn for clarifying. I ran the MR tests locally but they just timed out and I wasn't really sure how to proceed.

degant · 2023-04-19T18:21:03Z

Do you also plan to upgrade nimbus-jose-jwt to a newer version since the current version shades json-smart 1.3.2 which gets flagged by scanners for both CVE-2021-31684 and CVE-2023-1370?

nimbus-jose-jwt also dropped the json-smart dependency completely with nimbus-jose-jwt 9.24 and replaces it with Gson 2.9.1 (shaded) as seen in the commit history here: https://bitbucket.org/connect2id/nimbus-jose-jwt/commits/tag/9.24

So upgrading nimbus-jose-jwt from 9.8.1 to >9.24.4 could help completely get rid of json-smart from scanners and address HADOOP-18687. Related discussion here: #3299

pjfanning · 2023-04-19T18:44:20Z

Could you raise a separate Jira for nimbus?

degant · 2023-04-19T19:01:41Z

I don't have an account there and not sure how easy it is to get one

ayushtkn · 2023-04-19T19:18:08Z

Merged Thanx @pjfanning for the contribution.

@degant feel free to create a new ticket for nimbus. You can request to create a jira account by filling this form

https://selfserve.apache.org/jira-account.html

pjfanning · 2023-04-19T19:20:11Z

@degant I created https://issues.apache.org/jira/browse/HADOOP-18711

degant · 2023-04-19T19:33:46Z

Thanks for filing it! I went ahead and requested an account on jira

ayushtkn · 2023-04-20T05:06:03Z

For some strange reason, seems the builds have failing on compilation post this
#5575 (comment)
#5483 (comment)

Might not be related, but since it is failing post this only, have reverted this. Would require a rebased PR again or if I find there is something else, can commit it again directly

…ted by PJ Fanning." This reverts commit b6c0ec7.

ayushtkn · 2023-04-20T06:23:36Z

I have triggered those builds again post reverting this. @pjfanning can you check once. I think it may be this commit is conflicting with 0d1b4a3556d24641c14bbfc7ae1b985d4a998649, the PR ran build on 0185afafeac26a447b6138b2d74a6f5ed0051d0b which is prior to this one. Just my guess because errors mentioned SBOM

ayushtkn · 2023-04-20T10:39:29Z

Context-> #5575 (comment)
Not sure @pjfanning , I triggered those builds again for the HDFS ticket and they failed, the moment I reverted and triggered again it passed. If you say, I can directly re commit the same commit again?

Raised an Infra ticket as well: INFRA-24480

pjfanning · 2023-04-20T10:52:03Z

@ayushtkn can we try again tomorrow perhaps? I have a couple of other PRs that are rebuilding that were affected by a similar issue to this when they first ran.

ayushtkn · 2023-04-20T11:28:24Z

Sure, I will commit it tomorrow, Infra confirmed that the issue was on their part only. Sorry, for creating unnecessary noise.

pjfanning · 2023-04-20T12:22:45Z

If you're happy it was an infra issue, feel free to reapply this commit whenever suits (today even) - I can rebase the other PRs after this is added back

…J Fanning. Signed-off-by: Ayush Saxena <[email protected]>

ayushtkn · 2023-04-20T13:11:52Z

I have pushed again 🤞

…d by PJ Fanning. Signed-off-by: Ayush Saxena <[email protected]>

Contains * HADOOP-18687. hadoop-auth: remove unnecessary dependency on json-smart (#5524) Contributed by Michiel de Jong * HADOOP-18687. Remove json-smart dependency. (#5549). Contributed by PJ Fanning.

…d by PJ Fanning. Signed-off-by: Ayush Saxena <[email protected]>

…ntributed by PJ Fanning." This reverts commit b6c0ec7.

remove json-smart dependency

fc03294

pjfanning force-pushed the remove-json-smart branch from e69ac0b to fc03294 Compare April 14, 2023 03:34

steveloughran changed the title ~~HADOOP-18687: remove json-smart dependency~~ HADOOP-18687. Remove json-smart dependency Apr 14, 2023

ayushtkn approved these changes Apr 19, 2023

View reviewed changes

ayushtkn merged commit b6c0ec7 into apache:trunk Apr 19, 2023

pjfanning deleted the remove-json-smart branch April 19, 2023 19:19

asfgit pushed a commit that referenced this pull request Apr 20, 2023

Revert "HADOOP-18687. Remove json-smart dependency. (#5549). Contribu…

9e3d5c7

…ted by PJ Fanning." This reverts commit b6c0ec7.

asfgit pushed a commit that referenced this pull request Apr 20, 2023

HADOOP-18687. Remove json-smart dependency. (#5549). Contributed by P…

0918c87

…J Fanning. Signed-off-by: Ayush Saxena <[email protected]>

rohit-kb pushed a commit to rohit-kb/hadoop that referenced this pull request May 5, 2023

HADOOP-18687. Remove json-smart dependency. (apache#5549). Contribute…

b5558ad

…d by PJ Fanning. Signed-off-by: Ayush Saxena <[email protected]>

ferdelyi pushed a commit to ferdelyi/hadoop that referenced this pull request May 26, 2023

HADOOP-18687. Remove json-smart dependency. (apache#5549). Contribute…

dc1c889

…d by PJ Fanning. Signed-off-by: Ayush Saxena <[email protected]>

ferdelyi pushed a commit to ferdelyi/hadoop that referenced this pull request May 26, 2023

Revert "HADOOP-18687. Remove json-smart dependency. (apache#5549). Co…

bb14458

…ntributed by PJ Fanning." This reverts commit b6c0ec7.

HADOOP-18687. Remove json-smart dependency #5549

HADOOP-18687. Remove json-smart dependency #5549

Uh oh!

Conversation

pjfanning commented Apr 12, 2023

Uh oh!

hadoop-yetus commented Apr 13, 2023

Uh oh!

hadoop-yetus commented Apr 14, 2023

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

pjfanning commented Apr 19, 2023

Uh oh!

ayushtkn commented Apr 19, 2023

Uh oh!

pjfanning commented Apr 19, 2023

Uh oh!

degant commented Apr 19, 2023

Uh oh!

pjfanning commented Apr 19, 2023

Uh oh!

degant commented Apr 19, 2023

Uh oh!

ayushtkn commented Apr 19, 2023

Uh oh!

pjfanning commented Apr 19, 2023

Uh oh!

degant commented Apr 19, 2023

Uh oh!

ayushtkn commented Apr 20, 2023

Uh oh!

ayushtkn commented Apr 20, 2023

Uh oh!

ayushtkn commented Apr 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pjfanning commented Apr 20, 2023

Uh oh!

ayushtkn commented Apr 20, 2023

Uh oh!

pjfanning commented Apr 20, 2023

Uh oh!

ayushtkn commented Apr 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ayushtkn commented Apr 20, 2023 •

edited

Loading