[SPARK-53999][CORE] Native KQueue Transport support on BSD/MacOS #52703

yaooqinn · 2025-10-23T07:32:49Z

What changes were proposed in this pull request?

This PR adds Native KQUEUE via JNI support for transport, such as shuffle, file, and rpc procedures

Why are the changes needed?

Feature parity between Linux and MacOS/BSD platforms

Does this PR introduce any user-facing change?

Yes, a new option for io.mode

How was this patch tested?

new unit tests

Was this patch authored or co-authored using generative AI tooling?

no

LuciferYang · 2025-10-23T12:16:34Z

common/network-common/src/main/java/org/apache/spark/network/util/NettyUtils.java

    return switch (mode) {
      case NIO -> new NioEventLoopGroup(numThreads, threadFactory);
      case EPOLL -> new EpollEventLoopGroup(numThreads, threadFactory);
+      case KQUEUE ->  new KQueueEventLoopGroup(numThreads, threadFactory);


This is a deprecated API usage.
Of course, we can fix it uniformly in a separate pr

sarutak · 2025-10-23T15:15:42Z

ShuffleNettyKQueueSuite doesn't finish on my Mac.

[info] ShuffleNettyKQueueSuite:
[info] - groupByKey without compression (804 milliseconds)
[info] *** Test still running after 6 minutes, 53 seconds: suite name: ShuffleNettyKQueueSuite, test name: shuffle non-zero block size. 
[info] *** Test still running after 11 minutes, 53 seconds: suite name: ShuffleNettyKQueueSuite, test name: shuffle non-zero block size. 
[info] *** Test still running after 16 minutes, 53 seconds: suite name: ShuffleNettyKQueueSuite, test name: shuffle non-zero block size.

@yaooqinn Does this test pass on your Mac?

dongjoon-hyun · 2025-10-23T17:36:31Z

+1, LGTM.

Thank you, @yaooqinn , @LuciferYang , @sarutak . Merged to master for Apache Spark 4.1.0-preview3.

I manually verified on my Mac. It works.

$ sw_vers
ProductName:		macOS
ProductVersion:		26.0.1
BuildVersion:		25A362

$ build/sbt "core/testOnly *.ShuffleNetty*Suite"
...
[info] ShuffleNettyEpollSuite:
[info] - groupByKey without compression [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - shuffle non-zero block size [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - shuffle serializer [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - zero sized blocks [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - zero sized blocks without kryo [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - shuffle on mutable pairs [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - sorting on mutable pairs [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - cogroup using mutable pairs [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - subtract mutable pairs [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - sort with Java non serializable class - Kryo [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - sort with Java non serializable class - Java [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - shuffle with different compression settings (SPARK-3426) [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - [SPARK-4085] rerun map stage if reduce stage cannot find its local shuffle file [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - cannot find its local shuffle file if no execution of the stage and rerun shuffle [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - metrics for shuffle without aggregation [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - metrics for shuffle with aggregation [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - multiple simultaneous attempts for one task (SPARK-8029) [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - SPARK-34541: shuffle can be removed [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - SPARK-36206: shuffle checksum detect disk corruption [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] - SPARK-39771: warn when shuffle block number is too large [disabled on Mac OS X with EPOLL] !!! IGNORED !!!
[info] ShuffleNettyNioSuite:
[info] - groupByKey without compression (1 second, 253 milliseconds)
[info] - shuffle non-zero block size (2 seconds, 705 milliseconds)
[info] - shuffle serializer (2 seconds, 321 milliseconds)
[info] - zero sized blocks (3 seconds, 414 milliseconds)
[info] - zero sized blocks without kryo (3 seconds, 377 milliseconds)
[info] - shuffle on mutable pairs (2 seconds, 205 milliseconds)
[info] - sorting on mutable pairs (2 seconds, 205 milliseconds)
[info] - cogroup using mutable pairs (2 seconds, 117 milliseconds)
[info] - subtract mutable pairs (2 seconds, 24 milliseconds)
[info] - sort with Java non serializable class - Kryo (2 seconds, 298 milliseconds)
[info] - sort with Java non serializable class - Java (1 second, 618 milliseconds)
[info] - shuffle with different compression settings (SPARK-3426) (170 milliseconds)
[info] - [SPARK-4085] rerun map stage if reduce stage cannot find its local shuffle file (276 milliseconds)
[info] - cannot find its local shuffle file if no execution of the stage and rerun shuffle (38 milliseconds)
[info] - metrics for shuffle without aggregation (132 milliseconds)
[info] - metrics for shuffle with aggregation (200 milliseconds)
[info] - multiple simultaneous attempts for one task (SPARK-8029) (31 milliseconds)
[info] - SPARK-34541: shuffle can be removed (65 milliseconds)
[info] - SPARK-36206: shuffle checksum detect disk corruption (3 seconds, 345 milliseconds)
[info] - SPARK-39771: warn when shuffle block number is too large (75 milliseconds)
[info] ShuffleNettyKQueueSuite:
[info] - groupByKey without compression (344 milliseconds)
[info] - shuffle non-zero block size (2 seconds, 198 milliseconds)
[info] - shuffle serializer (2 seconds, 269 milliseconds)
[info] - zero sized blocks (3 seconds, 131 milliseconds)
[info] - zero sized blocks without kryo (3 seconds, 415 milliseconds)
[info] - shuffle on mutable pairs (2 seconds, 382 milliseconds)
[info] - sorting on mutable pairs (2 seconds, 350 milliseconds)
[info] - cogroup using mutable pairs (2 seconds, 159 milliseconds)
[info] - subtract mutable pairs (2 seconds, 257 milliseconds)
[info] - sort with Java non serializable class - Kryo (2 seconds, 289 milliseconds)
[info] - sort with Java non serializable class - Java (1 second, 710 milliseconds)
[info] - shuffle with different compression settings (SPARK-3426) (130 milliseconds)
[info] - [SPARK-4085] rerun map stage if reduce stage cannot find its local shuffle file (283 milliseconds)
[info] - cannot find its local shuffle file if no execution of the stage and rerun shuffle (39 milliseconds)
[info] - metrics for shuffle without aggregation (81 milliseconds)
[info] - metrics for shuffle with aggregation (133 milliseconds)
[info] - multiple simultaneous attempts for one task (SPARK-8029) (25 milliseconds)
[info] - SPARK-34541: shuffle can be removed (36 milliseconds)
[info] - SPARK-36206: shuffle checksum detect disk corruption (3 seconds, 415 milliseconds)
[info] - SPARK-39771: warn when shuffle block number is too large (31 milliseconds)
[info] Run completed in 59 seconds, 880 milliseconds.
[info] Total number of tests run: 40
[info] Suites: completed 3, aborted 0
[info] Tests: succeeded 40, failed 0, canceled 0, ignored 20, pending 0
[info] All tests passed.
[success] Total time: 92 s (0:01:32.0), completed Oct 23, 2025, 10:35:14 AM

sarutak · 2025-10-23T18:58:52Z

I confirmed this test passes on my clean environment.
Sorry for confusing.

dongjoon-hyun · 2025-10-23T19:19:12Z

No problem at all, @sarutak . Thank you for letting us know. 😄

yaooqinn · 2025-10-24T02:02:55Z

Thank you @LuciferYang @sarutak @dongjoon-hyun

### What changes were proposed in this pull request? Support native kqueue transport on BSD/MacOS for `celeborn.<module>.io.mode`. Backport: - apache/spark#52703 - apache/spark#52724 ### Why are the changes needed? Netty provides the following platform specific JNI transports for [native transports](https://netty.io/wiki/native-transports.html): - Linux (since 4.0.16) - MacOS/BSD (since 4.1.11) These JNI transports add features specific to a particular platform, generate less garbage, and generally improve performance when compared to the NIO based transport. ### Does this PR introduce _any_ user-facing change? Change the default value of `celeborn.<module>.io.mode` from `NIO` to `EPOLL` if epoll mode is available, from `NIO` to `KQUEUE` if kqueue mode is available, falling back to `NIO` otherwise. ### How was this patch tested? CI. Closes #3518 from SteNicholas/CELEBORN-2185. Authored-by: SteNicholas <[email protected]> Signed-off-by: SteNicholas <[email protected]>

### What changes were proposed in this pull request? This PR adds Native KQUEUE via JNI support for transport, such as shuffle, file, and rpc procedures ### Why are the changes needed? Feature parity between Linux and MacOS/BSD platforms ### Does this PR introduce _any_ user-facing change? Yes, a new option for io.mode ### How was this patch tested? new unit tests ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#52703 from yaooqinn/SPARK-53999. Authored-by: Kent Yao <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

[SPARK-53999][CORE] Native KQueue Transport support on BSD/MacOS

1e37244

github-actions bot added the CORE label Oct 23, 2025

LuciferYang reviewed Oct 23, 2025

View reviewed changes

LuciferYang approved these changes Oct 23, 2025

View reviewed changes

dongjoon-hyun closed this in c9ae4ab Oct 23, 2025

dongjoon-hyun mentioned this pull request Oct 23, 2025

[SPARK-54009][CORE] Support spark.io.mode.default #52717

Closed

SteNicholas mentioned this pull request Oct 28, 2025

[CELEBORN-2185] Support native kqueue transport on BSD/MacOS apache/celeborn#3518

Closed

yaooqinn deleted the SPARK-53999 branch November 18, 2025 02:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-53999][CORE] Native KQueue Transport support on BSD/MacOS #52703

[SPARK-53999][CORE] Native KQueue Transport support on BSD/MacOS #52703

Uh oh!

yaooqinn commented Oct 23, 2025

Uh oh!

LuciferYang Oct 23, 2025 •

edited

Loading

Uh oh!

sarutak commented Oct 23, 2025 •

edited

Loading

Uh oh!

dongjoon-hyun commented Oct 23, 2025 •

edited

Loading

Uh oh!

sarutak commented Oct 23, 2025

Uh oh!

dongjoon-hyun commented Oct 23, 2025

Uh oh!

yaooqinn commented Oct 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[SPARK-53999][CORE] Native KQueue Transport support on BSD/MacOS #52703

[SPARK-53999][CORE] Native KQueue Transport support on BSD/MacOS #52703

Uh oh!

Conversation

yaooqinn commented Oct 23, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

LuciferYang Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sarutak commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sarutak commented Oct 23, 2025

Uh oh!

dongjoon-hyun commented Oct 23, 2025

Uh oh!

yaooqinn commented Oct 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

LuciferYang Oct 23, 2025 •

edited

Loading

sarutak commented Oct 23, 2025 •

edited

Loading

dongjoon-hyun commented Oct 23, 2025 •

edited

Loading