Doc: Remove Spark 3 specific wordings in docs #14357

jackylee-ch · 2025-10-17T07:11:06Z

manuzhang · 2025-10-17T08:06:22Z

docs/docs/spark-getting-started.md


 ```sh
-spark-shell --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }}
+spark-shell --packages org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:{{ icebergVersion }}


We can add a variable sparkVersion at https://github.com/apache/iceberg/blob/main/site/mkdocs.yml#L86

manuzhang · 2025-10-17T14:49:44Z

docs/docs/spark-getting-started.md

 !!! info
    <!-- markdown-link-check-disable-next-line -->
-    If you want to include Iceberg in your Spark installation, add the [`iceberg-spark-runtime-3.5_2.12` Jar](https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/{{ icebergVersion }}/iceberg-spark-runtime-3.5_2.12-{{ icebergVersion }}.jar) to Spark's `jars` folder.
+    If you want to include Iceberg in your Spark installation, add the [`iceberg-spark-runtime-{{ sparkVersion }}_{{ scalaVersion }}` Jar](https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-{{ sparkVersion }}_{{ scalaVersion }}/{{ icebergVersion }}/iceberg-spark-runtime-{{ sparkVersion }}_{{ scalaVersion }}-{{ icebergVersion }}.jar) to Spark's `jars` folder.


If sparkVersion always comes together with scalaVersion, I think we can just use one variable.

pan3793 · 2025-10-17T18:15:11Z

docs/docs/spark-procedures.md

 # Spark Procedures

-To use Iceberg in Spark, first configure [Spark catalogs](spark-configuration.md). Stored procedures are only available when using [Iceberg SQL extensions](spark-configuration.md#sql-extensions) in Spark 3.
+To use Iceberg in Spark, first configure [Spark catalogs](spark-configuration.md). Stored procedures are only available when using [Iceberg SQL extensions](spark-configuration.md#sql-extensions) in Spark.


configuring the extension is not necessary for Spark 4.

Oh, yes, we have used Spark Call syntax from Spark 4.0, thanks for point out.

also better to document behavior change for Spark 4 in CALL syntax resolving, see #13106 and SPARK-53523

For SPARK-53523, we may ignore it until Spark 4.1.0 is released and supported in Iceberg.

pan3793 · 2025-10-17T18:16:27Z

site/mkdocs.yml

  flinkVersion: '2.0.0'
  flinkVersionMajor: '2.0'
+  sparkVersion: '4.0'
+  scalaVersion: '2.13'


sparkBinaryVersion and scalaBinaryVersion are more accurate

docs/docs/spark-structured-streaming.md

docs/docs/spark-getting-started.md

manuzhang · 2025-10-20T02:02:22Z

docs/docs/spark-procedures.md


-To use Iceberg in Spark, first configure [Spark catalogs](spark-configuration.md). Stored procedures are only available when using [Iceberg SQL extensions](spark-configuration.md#sql-extensions) in Spark 3.
+To use Iceberg in Spark, first configure [Spark catalogs](spark-configuration.md).  
+For Spark 3.x, stored procedures are only available when using [Iceberg SQL extensions](spark-configuration.md#sql-extensions) in Spark.  


Please share a snapshot of this page after the PR.

huaxingao · 2025-10-21T22:54:55Z

@jackylee-ch Could you please also share a screenshot of spark-getting-started page? Thanks

jackylee-ch · 2025-10-22T04:48:16Z

@jackylee-ch Could you please also share a screenshot of spark-getting-started page? Thanks

Sure, would done this later~

docs/docs/spark-structured-streaming.md

…s_from_docs

kevinjqliu

LGTM, added a few nit comments on not removing the warning about spark 3.0.

Check all the pages locally:

example screenshot:

kevinjqliu · 2025-11-06T17:45:59Z

site/docs/spark-quickstart.md


 <!-- markdown-link-check-disable-next-line -->
-[spark-runtime-jar]: https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/{{ icebergVersion }}/iceberg-spark-runtime-3.5_2.12-{{ icebergVersion }}.jar
+[spark-runtime-jar]: https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-{{ sparkVersionMajor }}/{{ icebergVersion }}/iceberg-spark-runtime-{{ sparkVersionMajor }}-{{ icebergVersion }}.jar


nit: this isnt rendered
same problem on https://iceberg.apache.org/spark-quickstart/#learn-more right now

It seems more reasonable to have the note point to the release at this stage. Exposing the spark-runtime-jar directly might appear a bit abrupt. Can we just remote this line?

agreed, we dont need to expose the jar directly

docs/docs/spark-queries.md

docs/docs/spark-structured-streaming.md

kevinjqliu · 2025-11-06T17:53:19Z

btw if you merge main (to pull in #14267), you can use make serve-dev and render the site

jackylee-ch · 2025-11-10T01:48:42Z

@jackylee-ch Could you please also share a screenshot of spark-getting-started page? Thanks

@huaxingao Sorry for the late response. I just built the doc locally, and here’s the screenshot of the Spark Getting Started page.

jackylee-ch · 2025-11-10T02:20:06Z

LGTM, added a few nit comments on not removing the warning about spark 3.0.

@kevinjqliu Since we only support Spark 3.4 or later, users may not be able to run it with Spark 3.0. Therefore, the warning seems unnecessary to me.
cc @huaxingao @manuzhang

kevinjqliu · 2025-11-10T17:17:02Z

Since we only support Spark 3.4 or later, users may not be able to run it with Spark 3.0. Therefore, the warning seems unnecessary to me.

makes sense, we have versioned docs so this warning still exists in the previous versions

kevinjqliu · 2025-11-10T17:18:22Z

site/docs/spark-quickstart.md


 <!-- markdown-link-check-disable-next-line -->
-[spark-runtime-jar]: https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/{{ icebergVersion }}/iceberg-spark-runtime-3.5_2.12-{{ icebergVersion }}.jar
+[spark-runtime-jar]: https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-{{ sparkVersionMajor }}/{{ icebergVersion }}/iceberg-spark-runtime-{{ sparkVersionMajor }}-{{ icebergVersion }}.jar


Suggested change



[spark-runtime-jar]: https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/{{ icebergVersion }}/iceberg-spark-runtime-3.5_2.12-{{ icebergVersion }}.jar

[spark-runtime-jar]: https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-{{ sparkVersionMajor }}/{{ icebergVersion }}/iceberg-spark-runtime-{{ sparkVersionMajor }}-{{ icebergVersion }}.jar

lets remove this then

kevinjqliu · 2025-11-10T18:12:12Z

Thanks for the PR @jackylee-ch and thanks everyone for the review!

kevinjqliu · 2025-11-10T18:17:04Z

Follow up to remove spark-runtime-jar #14555
I wasn't able to make the change on this PR

jackylee-ch · 2025-11-11T05:10:54Z

Thanks for your help @kevinjqliu. And thanks for your review. @huaxingao @kevinjqliu @manuzhang @pan3793

Doc: Remove Spark 3 specific wordings in docs

eb67961

github-actions bot added the docs label Oct 17, 2025

manuzhang reviewed Oct 17, 2025

View reviewed changes

use sparkVersion and scalaVersion to generate docs

b0d711a

manuzhang reviewed Oct 17, 2025

View reviewed changes

pan3793 reviewed Oct 17, 2025

View reviewed changes

pan3793 reviewed Oct 18, 2025

View reviewed changes

docs/docs/spark-structured-streaming.md Outdated Show resolved Hide resolved

update statements

606d6f1

huaxingao reviewed Oct 18, 2025

View reviewed changes

docs/docs/spark-structured-streaming.md Outdated Show resolved Hide resolved

huaxingao reviewed Oct 18, 2025

View reviewed changes

docs/docs/spark-structured-streaming.md Outdated Show resolved Hide resolved

huaxingao reviewed Oct 18, 2025

View reviewed changes

docs/docs/spark-getting-started.md Outdated Show resolved Hide resolved

manuzhang reviewed Oct 20, 2025

View reviewed changes

update statements

5825e88

jackylee-ch force-pushed the refine_Spark_3_words_from_docs branch from 6003251 to 5825e88 Compare October 20, 2025 02:06

manuzhang approved these changes Oct 27, 2025

View reviewed changes

manuzhang reviewed Oct 27, 2025

View reviewed changes

docs/docs/spark-structured-streaming.md Show resolved Hide resolved

jackylee-ch added 2 commits October 30, 2025 20:20

update commonts

5033945

Merge remote-tracking branch 'upstream/main' into refine_Spark_3_word…

6efed89

…s_from_docs

jackylee-ch force-pushed the refine_Spark_3_words_from_docs branch from 1320d80 to 6efed89 Compare October 30, 2025 12:33

manuzhang approved these changes Oct 30, 2025

View reviewed changes

kevinjqliu approved these changes Nov 6, 2025

View reviewed changes

kevinjqliu reviewed Nov 10, 2025

View reviewed changes

kevinjqliu merged commit 5481507 into apache:main Nov 10, 2025
3 checks passed

kevinjqliu mentioned this pull request Nov 10, 2025

docs: remove hidden spark-runtime-jar link #14555

Merged

Doc: Remove Spark 3 specific wordings in docs #14357

Doc: Remove Spark 3 specific wordings in docs #14357

Uh oh!

Conversation

jackylee-ch commented Oct 17, 2025 • edited by manuzhang Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

huaxingao commented Oct 21, 2025

Uh oh!

jackylee-ch commented Oct 22, 2025

Uh oh!

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kevinjqliu commented Nov 6, 2025

Uh oh!

jackylee-ch commented Nov 10, 2025

Uh oh!

jackylee-ch commented Nov 10, 2025

Uh oh!

kevinjqliu commented Nov 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kevinjqliu commented Nov 10, 2025

Uh oh!

kevinjqliu commented Nov 10, 2025

Uh oh!

jackylee-ch commented Nov 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jackylee-ch commented Oct 17, 2025 •

edited by manuzhang

Loading

pan3793 Oct 17, 2025 •

edited

Loading