[BUILD] Migrate from log4j1 to log4j2 #1769
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
appreciate it for driving this @turboFei |
Member
|
Spark 3.0/3.1/3.2 use log4j12, we need to exclude those deps and bridge it to log4j2 |
pan3793
reviewed
Jan 17, 2022
dev/kyuubi-extension-spark-3-1/src/test/resources/log4j2.properties
Outdated
Show resolved
Hide resolved
pan3793
reviewed
Jan 17, 2022
externals/kyuubi-trino-engine/src/test/resources/log4j2.properties
Outdated
Show resolved
Hide resolved
yaooqinn
reviewed
Jan 17, 2022
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/log/LogDivertAppender.scala
Outdated
Show resolved
Hide resolved
Codecov Report
@@ Coverage Diff @@
## master #1769 +/- ##
============================================
+ Coverage 59.43% 59.46% +0.02%
- Complexity 241 272 +31
============================================
Files 273 277 +4
Lines 13415 13849 +434
Branches 1695 1775 +80
============================================
+ Hits 7973 8235 +262
- Misses 4756 4910 +154
- Partials 686 704 +18
Continue to review full report at Codecov.
|
pan3793
reviewed
Jan 22, 2022
| log4j-api/2.17.1//log4j-api-2.17.1.jar | ||
| log4j-core/2.17.1//log4j-core-2.17.1.jar | ||
| log4j-slf4j-impl/2.17.1//log4j-slf4j-impl-2.17.1.jar | ||
| log4j/1.2.17//log4j-1.2.17.jar |
Member
There was a problem hiding this comment.
We should remove log4j-1.2 from the runtime jar
Member
Author
There was a problem hiding this comment.
sorry, what it is the runtime jar?
Member
There was a problem hiding this comment.
I mean the jars under the $KYUUBI_HOME/jars
Member
There was a problem hiding this comment.
I think log4j-1.2.17.jar should not existed after your change, but why it still here?
Member
Author
There was a problem hiding this comment.
I checked that log4j-1.2.17.jar is not existing in $KYUUBI_HOME/jars.
Member
Author
There was a problem hiding this comment.
[INFO] org.apache.kyuubi:kyuubi-assembly_2.12:pom:1.5.0-SNAPSHOT
[INFO] +- org.apache.kyuubi:kyuubi-common_2.12:jar:1.5.0-SNAPSHOT:compile
[INFO] | +- org.scala-lang:scala-library:jar:2.12.15:compile
[INFO] | +- org.slf4j:slf4j-api:jar:1.7.30:compile
[INFO] | +- org.slf4j:jcl-over-slf4j:jar:1.7.30:compile
[INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.30:compile
[INFO] | | \- log4j:log4j:jar:1.2.17:compile
[INFO] | +- commons-codec:commons-codec:jar:1.15:compile
I do not know why the org.slf4j:slf4j-log4j12:jar:1.7.30 is transferred.
pan3793
reviewed
Jan 23, 2022
pan3793
reviewed
Jan 23, 2022
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/log/LogDivertAppender.scala
Outdated
Show resolved
Hide resolved
pan3793
reviewed
Jan 23, 2022
pan3793
reviewed
Jan 23, 2022
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/log/LogDivertAppender.scala
Outdated
Show resolved
Hide resolved
pan3793
reviewed
Jan 23, 2022
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/log/LogDivertAppender.scala
Outdated
Show resolved
Hide resolved
Member
Author
|
Looks fine now. |
yaooqinn
reviewed
Jan 25, 2022
yaooqinn
approved these changes
Jan 25, 2022
Member
Author
|
thanks, merging to master |
turboFei
added a commit
that referenced
this pull request
Jan 27, 2022
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This is a followup of #1769. Cleanup the log4j properties files and using log4j2.properties for UT. ### _How was this patch tested?_ Passed UT. Closes #1842 from turboFei/remove_log4j1. Closes #1769 b675755 [Fei Wang] unused change 365fbec [Fei Wang] revert license 14c64ec [Fei Wang] exclude log4j 1.2.17 in slf4j-log4j12 034b04d [Fei Wang] recover jcl-over-slf4j and slf4j-api for k8s it 2beae75 [Fei Wang] remove log4j 1.2.17 b00f07b [Fei Wang] remove from license e55fd2e [Fei Wang] remove unused dependencies ab86f02 [Fei Wang] [KYUUBI #1769][FOLLOWUP] Some cleanups for log4j properties Authored-by: Fei Wang <[email protected]> Signed-off-by: Fei Wang <[email protected]>
pan3793
added a commit
to pan3793/kyuubi
that referenced
this pull request
Jan 29, 2022
3 tasks
pan3793
added a commit
to pan3793/kyuubi
that referenced
this pull request
Jan 31, 2022
pan3793
pushed a commit
that referenced
this pull request
Feb 1, 2022
…-jdbc(shaded) modules ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Revert the dependency changes for kyuubi-hive-jdbc and kyuubi-hive-jdbc-shaded modules from #1842 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1854 from turboFei/slf4j-api. Closes #1769 dffe6c5 [Fei Wang] reserve for kyuubi-hive-beeline 2001243 [Fei Wang] [KYUUBI #1769][FOLLOWUP] Revert the dependency change for kyuubi-hive-beeline and kyuubi-hive-jdbc(shaded) modules Authored-by: Fei Wang <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
pan3793
added a commit
to pan3793/kyuubi
that referenced
this pull request
Feb 1, 2022
turboFei
pushed a commit
that referenced
this pull request
Feb 1, 2022
…x detect deps change workflow ### _Why are the changes needed?_ The previous PR did not handle log4j 1.2 deps of `spark-hive`, and the current detect deps change workflow has the dirty cache issue, this PR also fix it. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1856 from pan3793/log4j2. Closes #1769 2785acf [Cheng Pan] nit 58affcb [Cheng Pan] update workflow 301acb6 [Cheng Pan] update check deps script 7611c7e [Cheng Pan] revert exclusion 239e7cc [Cheng Pan] Remove duplicated scalatest e143fec [Cheng Pan] dependencyList a725477 [Cheng Pan] [KYUUBI #1769][FOLLOWUP] Exclude log4j 12 deps from test deps Authored-by: Cheng Pan <[email protected]> Signed-off-by: Fei Wang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
This patch proposes to migrate from log4j1 to log4j2 in Kyuubi.
Refer the spark patch: apache/spark#34895
Does this PR introduce any user-facing change?
Yes. Users may need to rewrite their log4j properties file for log4j2. As of version 2.4, log4j now supports configuration via properties files. Note that the property syntax is not the same as the syntax used in log4j 1, but during the migration I found the syntax is pretty close so the migration should be straightforward.
How was this patch tested?
Passed all existing tests.