[SPARK-45522][BUILD][CORE][SQL][UI] Migrate from Jetty 9 to Jetty 10

[HiuKwok]

What changes were proposed in this pull request?

This is an upgrade ticket to bump the Jetty version from 9 to 10.
This PR aims to bring incremental Jetty upgrades to Spark, as Jetty 9 support already reached EOL.

Why are the changes needed?

Jetty 9 is already beyond EOL, which means that we won't receive any security fix onward for Spark.

Does this PR introduce any user-facing change?

No, SNI host check is now defaulted to true on embedded Jetty, hence set it back to false to maintain backward compatibility.
Despite the redirect behaviour changed for trailing /, but modern browser should be able to pick up the 302 status code and perform redirect accordingly, hence there is no impact on user level.

How was this patch tested?

Junit test case.

Was this patch authored or co-authored using generative AI tooling?

No

[HiuKwok]

Checking the build failure

[LuciferYang]

[error] /home/runner/work/spark/spark/core/src/test/scala/org/apache/spark/ui/UISuite.scala:415: File line length exceeds 100 characters

[HiuKwok]

I have fixed the code style and other miscellaneous linter errors, right now I'm investigating the Python test case fail, for PySpark connect.

[LuciferYang]

cc @srowen @dongjoon-hyun @HyukjinKwon @yaooqinn FYI

[srowen]

This is a good step - we should try to get to Jetty 11 next before Spark 4, or even Jetty 12

[HiuKwok]

Resolved all comments.

[srowen]

Merged to master

[pan3793]

@srowen @dongjoon-hyun @LuciferYang @yaooqinn @cloud-fan @HiuKwok

Unfortunately, we identified the compatibility issue on Jetty 11, and it is likely to revert Jetty 11 upgrading, see details in SPARK-48238. If we downgrade Jetty, which version we want to keep, 9 or 10?

I would prefer 9, because I think

1. both of those versions are End of Community Support, but still receiving critical CVE fixes
2. some downstream projects may extend Spark UI tabs, e.g. Apache Kyuubi, Apache Gluten, keep using Jetty 9 as Spark 3 does would reduce the potential compatibility issues
3. Jetty 9 might live longer than Jetty 10

jetty/jetty.project#10103 (comment)

Jetty 9.4 EOL and Jetty 10/11 EOL are only very loosely related.

This is because Java 8 (minimum requirement for Jetty 9.4) currently has longer support from Java vendors than Java 11 (minimum requirement for Jetty 10/11).
Certain vendors support Java 8 until 2030, but Java 11 only until 2026, for example.

For this reason, it may be possible that Jetty 10/11 will go in EOL at a different time than Jetty 9.4, possibly even earlier.

jetty/jetty.project#10485

... specifically that 1 January 2024 will mark the official End of Community Support(EoCS) for Jetty 10 and Jetty 11.

[LuciferYang]

@srowen @dongjoon-hyun @LuciferYang @yaooqinn @cloud-fan @HiuKwok

Unfortunately, we identified the compatibility issue on Jetty 11, and it is likely to revert Jetty 11 upgrading, see details in SPARK-48238. If we downgrade Jetty, which version we want to keep, 9 or 10?

I would prefer 9, because I think

1. both of those versions are End of Community Support, but still receiving critical CVE fixes
2. some downstream projects may extend Spark UI tabs, e.g. Apache Kyuubi, Apache Gluten, keep using Jetty 9 as Spark 3 does would reduce the potential compatibility issues
3. Jetty 9 might live longer than Jetty 10

jetty/jetty.project#10103 (comment)

Jetty 9.4 EOL and Jetty 10/11 EOL are only very loosely related.
This is because Java 8 (minimum requirement for Jetty 9.4) currently has longer support from Java vendors than Java 11 (minimum requirement for Jetty 10/11).
Certain vendors support Java 8 until 2030, but Java 11 only until 2026, for example.
For this reason, it may be possible that Jetty 10/11 will go in EOL at a different time than Jetty 9.4, possibly even earlier.

jetty/jetty.project#10485

... specifically that 1 January 2024 will mark the official End of Community Support(EoCS) for Jetty 10 and Jetty 11.

+1 for 9.x

[HiuKwok]

@srowen @dongjoon-hyun @LuciferYang @yaooqinn @cloud-fan @HiuKwok

Unfortunately, we identified the compatibility issue on Jetty 11, and it is likely to revert Jetty 11 upgrading, see details in SPARK-48238. If we downgrade Jetty, which version we want to keep, 9 or 10?

I would prefer 9, because I think

1. both of those versions are End of Community Support, but still receiving critical CVE fixes
2. some downstream projects may extend Spark UI tabs, e.g. Apache Kyuubi, Apache Gluten, keep using Jetty 9 as Spark 3 does would reduce the potential compatibility issues
3. Jetty 9 might live longer than Jetty 10

jetty/jetty.project#10103 (comment)

Jetty 9.4 EOL and Jetty 10/11 EOL are only very loosely related.
This is because Java 8 (minimum requirement for Jetty 9.4) currently has longer support from Java vendors than Java 11 (minimum requirement for Jetty 10/11).
Certain vendors support Java 8 until 2030, but Java 11 only until 2026, for example.
For this reason, it may be possible that Jetty 10/11 will go in EOL at a different time than Jetty 9.4, possibly even earlier.

jetty/jetty.project#10485

... specifically that 1 January 2024 will mark the official End of Community Support(EoCS) for Jetty 10 and Jetty 11.

Agree.

[cloud-fan]

Thanks for the discussion here! @HiuKwok are you going to do the revert work? If not please let us know and we can help.