[SPARK-37625] update log4j to 2.15

[3333qwe]

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

[AmplabJenkins]

Can one of the admins verify this patch?

[AngersZhuuuu]

This CVE issue have no impact on spark, we don't need to do anything. I don't think we need to update to log4j2

[dannymeijer]

This CVE issue have no impact on spark, we don't need to do anything. I don't think we need to update to log4j2

@AngersZhuuuu
Could not disagree more. I'm seeing Log4J version 1.2.17 being used in the Spark 3.2 image (build from latest) that we are deploying. This version was EOL way back in 2015 already. This PR needs highest possible priority to ensure that this CVE is definitely not affecting Spark as well as make it future proof for the foreseeable future.
source: https://logging.apache.org/log4j/1.2/

[dnskr]

CVE-2021-44228 affects only log4j2, so current log4j:1.2.17 doesn't need to be updated to fix the issue.
See https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228 for a more details.
However it is recommended to update because of https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17571

[tonycox]

https://nvd.nist.gov/vuln/detail/CVE-2019-17571 impacts on log4j 1.2.*

[n-marion]

This fix seems incomplete based on: https://logging.apache.org/log4j/2.x/manual/migration.html

[bradbm]

CVE-2021-44228 affects only log4j2, so current log4j:1.2.17 doesn't need to be updated to fix the issue. See https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228 for a more details. However it is recommended to update because of https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17571

Unfortunately there is CVE-2021-4104 which is a new similar CVE issued for Log4j 1.x, in addition to CVE-2019-17571.

[daguito81]

Attaching some previous discussion regarding migrating from Log4j 1.x to 2.x
We have this issue from 2015 https://issues.apache.org/jira/browse/SPARK-6305
where a lot of information can be read regarding the problem with dependencies and bumping log4j to 2.x

Regarding CVE-2021-4104 @bradbm stated, supposedly this only affects if you have JMSAppender on your Log4j configuration, which Spark doesn't use by default. If your application uses JMSAppender you can see mitigations here https://access.redhat.com/security/cve/CVE-2021-4104 so you're not vulnerable.

[cchantep]

Even if spark is vulnerable to a very specific case this time, using a log4j version that is no longer maintained by Apache itself is not consistent at all.

If migration issues have been identified since 2015, I'm quite surprise it has been fixed since ...

[attilapiros]

I am afraid you are working parallel on this: #34895

[srowen]

@dannymeijer @cchantep @3333qwe et al - folks, nobody's dumb here. In fact, per above, this has been tried a few times over the years. The short summary is: Spark is easy to update to log4j 2, as it really does not use a concrete logging framework. But that doesn't change what its dependencies (mostly Hadoop) do.

This change is not nearly sufficient, unfortunately, so I'd close it in favor of the duplicate, which is another try and farther along.