TEZ-4419: Upgrade node and yarn version and fix npm security issues in Tez UI module #215

amanraj2520 · 2022-06-06T11:58:33Z

Upgrade node and yarn version and fix npm security issues in Tez UI module
Track this issue:
https://issues.apache.org/jira/browse/TEZ-4419

The RFC documentation which adds selective dependency resolution in the description : https://github.com/yarnpkg/rfcs/blob/master/implemented/0000-selective-versions-resolutions.md

…fix the vulnerability [TEZ-4423][CVE-2021-44906] Upgrade minimist version from 0.0.8 to 1.2.6 to fix the vulnerability [TEZ-4422][CVE-2021-43138] Upgrade async from 2.3.0 to 2.6.4 to fix the vulnerability

… 4.1.2 to fix vulnerability

….1.2. The cryptiles 4.1.2 version requires node version>=8.9.0

… cryptiles 4.1.2 version and upgrade lodash.merge version to 4.6.2 to fix vulnerability

tez-yetus · 2022-06-06T14:15:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	17m 36s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	6m 40s	Maven dependency ordering for branch
+1 💚	mvninstall	10m 38s	master passed
+1 💚	compile	3m 39s	master passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	3m 30s	master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javadoc	2m 55s	master passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 21s	master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 16s	Maven dependency ordering for patch
+1 💚	mvninstall	5m 42s	the patch passed
+1 💚	compile	3m 31s	the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	3m 31s	the patch passed
+1 💚	compile	3m 13s	the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	3m 13s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 3s	The patch has no ill-formed XML file.
+1 💚	javadoc	2m 46s	the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 24s	the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
		_ Other Tests _
+1 💚	unit	1m 57s	tez-ui in the patch passed.
+1 💚	unit	66m 51s	root in the patch passed.
+1 💚	asflicense	1m 25s	The patch does not generate ASF License warnings.
		136m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-215/1/artifact/out/Dockerfile
GITHUB PR	#215
JIRA Issue	TEZ-4419
Optional Tests	dupname asflicense javac javadoc unit xml compile
uname	Linux 5aeefad56750 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `cf9e3ff`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-215/1/testReport/
Max. process+thread count	2089 (vs. ulimit of 5500)
modules	C: tez-ui . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-215/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

guptanikhil007 · 2022-06-06T14:36:12Z

@abstractdog @rbalamohan
Can you please help with the review?

guptanikhil007 · 2022-06-06T14:37:26Z

pom.xml

    <protoc.path>${env.PROTOC_PATH}</protoc.path>
    <scm.url>scm:git:https://gitbox.apache.org/repos/asf/tez.git</scm.url>
-    <frontend-maven-plugin.version>1.4</frontend-maven-plugin.version>
+    <frontend-maven-plugin.version>1.8.0</frontend-maven-plugin.version>


Please change this to 1.12.1

When we change the version to anything more than 1.8.0, this is the error in the build pipeline. The plugin com.github.eirslett:frontend-maven-plugin:1.12.1 requires Maven version 3.6.0 @guptanikhil007

Cool, then let's go with 1.8.0 only

guptanikhil007 · 2022-06-06T14:40:30Z

tez-ui/pom.xml

            <configuration>
              <nodeVersion>${nodeVersion}</nodeVersion>
-              <yarnVersion>v0.21.3</yarnVersion>
+              <yarnVersion>v1.6.0</yarnVersion>


Please mention the RFC which adds selective dependency resolution in the description.

tez-yetus · 2022-06-06T18:59:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 31s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	6m 29s	Maven dependency ordering for branch
+1 💚	mvninstall	10m 36s	master passed
+1 💚	compile	3m 38s	master passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	3m 26s	master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javadoc	2m 56s	master passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 19s	master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 17s	Maven dependency ordering for patch
+1 💚	mvninstall	5m 41s	the patch passed
+1 💚	compile	3m 26s	the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	3m 26s	the patch passed
+1 💚	compile	3m 13s	the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	3m 13s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	javadoc	2m 45s	the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 22s	the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
		_ Other Tests _
+1 💚	unit	1m 57s	tez-ui in the patch passed.
+1 💚	unit	66m 0s	root in the patch passed.
+1 💚	asflicense	1m 26s	The patch does not generate ASF License warnings.
		119m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-215/2/artifact/out/Dockerfile
GITHUB PR	#215
JIRA Issue	TEZ-4419
Optional Tests	dupname asflicense javac javadoc unit xml compile
uname	Linux 87e1cf18bf86 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `cf9e3ff`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-215/2/testReport/
Max. process+thread count	1383 (vs. ulimit of 5500)
modules	C: tez-ui . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-215/2/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

slachiewicz · 2022-06-06T19:49:15Z

tez-ui/pom.xml

    <webappDir>src/main/webapp</webappDir>

-    <nodeVersion>v5.12.0</nodeVersion>
+    <nodeVersion>v8.9.0</nodeVersion>


maybe we can try with v16 ?
https://nodejs.org/en/about/releases/

@slachiewicz When I update the node version to 16.15.1, this is the error that I get :
yarn run v1.6.0
(node:2298) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use node --trace-deprecation ... to show where the warning was created)
$ TMPDIR=tmp node/node ./node_modules/ember-cli/bin/ember build -prod

exports.dir = path.resolve(os.tmpDir());
^

TypeError: os.tmpDir is not a function
at Object. (/home/tez-ui/src/main/webapp/node_modules/temp/lib/temp.js:273:45)
at Module._compile (node:internal/modules/cjs/loader:1105:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
at Module.load (node:internal/modules/cjs/loader:981:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Module.require (node:internal/modules/cjs/loader:1005:19)
at require (node:internal/modules/cjs/helpers:102:18)
at Object. (/home/amanraj2520/Repositories/wildfire-tez/tez-ui/src/main/webapp/node_modules/ember-cli/lib/tasks/install-blueprint.js:7:21)
at Module._compile (node:internal/modules/cjs/loader:1105:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

What I think this error means is that we need to replace all the places of occurrence of new Buffer() to either Buffer.alloc() or Buffer.from(). Let me know you thoughts

Probably stack should be upgraded. I commented on Node's version because currently 16 is LTS version. I don't have experience with frontend tools.

Yes thats true, but I think it will be another big change. I think we should make this change first and then work on upgrading the stack. Any thoughts ? @abstractdog @slachiewicz

abstractdog · 2022-06-06T20:11:21Z

thanks guys for taking care of security issues in Tez!
I can see that TEZ-4419 is an umbrella with lots of subtasks
if we're tracking fixes on separate jiras, we might want to fix them in separate PRs/commits too, if possible, can you please do this accordingly?
I'm adding contributor rights to all of you in jira to tez project, feel free to assign tickets to yourselves

abstractdog · 2022-11-17T09:51:28Z

TEZ-4419 is resolved, I think we can close this PR
please reopen if I'm wrong

amanraj2520 added 5 commits June 6, 2022 16:07

[TEZ-4424][CVE-2021-3918] Upgrade json-schema from 0.2.3 to 0.4.0 to …

3f4a25f

…fix the vulnerability [TEZ-4423][CVE-2021-44906] Upgrade minimist version from 0.0.8 to 1.2.6 to fix the vulnerability [TEZ-4422][CVE-2021-43138] Upgrade async from 2.3.0 to 2.6.4 to fix the vulnerability

[TEZ-4425][WS-2020-0345] Upgrade jsonpointer version from 4.0.1 to 4.1.0

c87c5cb

[TEZ-4426] [CVE-2018-1000620] Upgrade cryptiles version from 2.0.5 to…

36e6467

… 4.1.2 to fix vulnerability

Upgraded node version to 8.9.0 to fix the build issue for cryptiles 4…

664c59a

….1.2. The cryptiles 4.1.2 version requires node version>=8.9.0

[TEZ-4427] [CVE-2019-10744] Upgraded node version to 8.9.0 to support…

b728441

… cryptiles 4.1.2 version and upgrade lodash.merge version to 4.6.2 to fix vulnerability

guptanikhil007 reviewed Jun 6, 2022

View reviewed changes

Upgraded the version of frontend maven plugin to 1.12.1

bc0c2db

slachiewicz reviewed Jun 6, 2022

View reviewed changes

abstractdog closed this Nov 17, 2022

TEZ-4419: Upgrade node and yarn version and fix npm security issues in Tez UI module #215

TEZ-4419: Upgrade node and yarn version and fix npm security issues in Tez UI module #215

Uh oh!

Conversation

amanraj2520 commented Jun 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tez-yetus commented Jun 6, 2022

Uh oh!

guptanikhil007 commented Jun 6, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tez-yetus commented Jun 6, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abstractdog commented Jun 6, 2022

Uh oh!

abstractdog commented Nov 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

amanraj2520 commented Jun 6, 2022 •

edited

Loading

abstractdog commented Nov 17, 2022 •

edited

Loading