[SPARK-51168][BUILD] Upgrade to Hadoop 3.4.2 #51127
Conversation
There was a problem hiding this comment.
Introduced by HADOOP-19348
|
Thank you, @pan3793 . |
|
No surprise so far, waiting for Hadoop 3.4.2 next RC. |
|
Is there any update, @pan3793 ? |
|
still waiting for the next RC |
pan3793
force-pushed
the
hadoop-3.4.2
branch
3 times, most recently
from
e54e614 to
c0d49b6
Compare
|
Thank you for testing RC3, @pan3793 . The result looks good, right? Maybe, could you re-trigger the flaky ones? |
|
@dongjoon-hyun RC3 is almost a repackage of RC2 without code changes. The current test results are good, the K8s IT fails consistently, likely due to being unable to download the package from the maven staging repo (I haven't investigated the root cause) |
|
Ya, I agree with you. I casted +1 for Apache Hadoop 3.4.2 RC3 after testing with Apache ORC 2.3.0-SNAPSHOT, too. |
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
It's published finally. Please update the PR, @pan3793 .
pan3793
changed the title
| aliyun-java-sdk-kms/2.11.0//aliyun-java-sdk-kms-2.11.0.jar | ||
| aliyun-java-sdk-ram/3.1.0//aliyun-java-sdk-ram-3.1.0.jar | ||
| aliyun-sdk-oss/3.13.2//aliyun-sdk-oss-3.13.2.jar | ||
| analyticsaccelerator-s3/1.2.1//analyticsaccelerator-s3-1.2.1.jar |
There was a problem hiding this comment.
this won't be present at the official release artifacts, so we don't need to change the LICENSE/NOTICE files
There was a problem hiding this comment.
What do you mean? If we don't want to change LICENSE/NOTICE, we need to exclude this explicitly.
There was a problem hiding this comment.
My understanding is: the LICENSE/NOTICE should match the content of the artifact.
For the source release tarball, in addition to Spark code itself, the LICENSE/NOTICE only reflects the source code we included from outside of the Spark project.
For the binary release tarball, the LICENSE-binary/NOTICE-binary should only reflect the included Spark and third-party libs, so LICENSE/NOTICE of testing deps and other optional libs is not necessary.
I'm not an expert in this area, the above is what I learned from several incubating projects.
There was a problem hiding this comment.
$ dev/make-distribution.sh -Pyarn -Pkubernetes -Phadoop-3 -Phive -Phive-thriftserver
...
$ ls dist/jars | grep analyticsaccelerator
<no ouptut>
There was a problem hiding this comment.
Now I got your meaning. So, you mean we found a bug of dev/test-dependencies.sh, right? Could you file a bug JIRA issue for this independently?
There was a problem hiding this comment.
dev/test-dependencies.sh also gathers deps from optional modules, for example, the official release tarball does not include the hadoop-cloud module, thus it won't pull those transitive deps.
There was a problem hiding this comment.
Got it~ It makes sense.
|
@dongjoon-hyun thanks for the information, updated. |
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
Outdated
Show resolved
Hide resolved
|
Document is published too. |
|
Merged to master for Apache Spark 4.1.0. Thank you so much, @pan3793 . |
…Scala 2.13.17 ### What changes were proposed in this pull request? This PR aims to regenerate benchmark results after upgrading to Scala 2.13.17. ### Why are the changes needed? Since last update, we change important libraries, not only Scala, but also Hadoop, ORC, ZSTD libraries. This PR aims to make the benchmark result up-to-date as a way to detect any performance regression. - #52509 - #51127 - #52478 - #52591 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52600 from dongjoon-hyun/SPARK-53893. Lead-authored-by: Dongjoon Hyun <[email protected]> Co-authored-by: dongjoon-hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
…Scala 2.13.17 ### What changes were proposed in this pull request? This PR aims to regenerate benchmark results after upgrading to Scala 2.13.17. ### Why are the changes needed? Since last update, we change important libraries, not only Scala, but also Hadoop, ORC, ZSTD libraries. This PR aims to make the benchmark result up-to-date as a way to detect any performance regression. - apache#52509 - apache#51127 - apache#52478 - apache#52591 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52600 from dongjoon-hyun/SPARK-53893. Lead-authored-by: Dongjoon Hyun <[email protected]> Co-authored-by: dongjoon-hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
2 participants
What changes were proposed in this pull request?
Hadoop 3.4.2 Release Note.
Why are the changes needed?
Keep the Hadoop client up to date.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GHA.
Was this patch authored or co-authored using generative AI tooling?
No.