Skip to content

Commit

Permalink
remove support for Spark 2.4, 3.1 and 3.2: #955
Browse files Browse the repository at this point in the history
  • Loading branch information
@gschmutz
gschmutz committed Nov 13, 2024
1 parent b36d40d commit fd7939a
Show file tree
Hide file tree
Showing 5 changed files with 7 additions and 47 deletions.
1 change: 1 addition & 0 deletions documentation/changes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,7 @@ The Modern Data Platform version 1.18.0 contains the following bug fixes and enh
* change default value for `TRINO_edition` to `oss`
* NiFi extensions now need to be listed using property `NIFI_extensions`
* `dataverse` port changed to `28394` (used to be `28294` and by taht conflicting with `ckan`)
* Support for Spark versions `2.4`, `3.1` and `3.2` removed

## What's new in 1.17.1

Expand Down
2 changes: 1 addition & 1 deletion documentation/configuration.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -320,7 +320,7 @@ For each service there might be some other settings, such as controlling the num
| `HADOOP_datanodes` | `2 ` | 1.0.0 | number of Hadoop Datanodes | | |
| [**_Apache Spark_**](./services/spark.md)     ![x86-64](./images/x86-64.png) | | | |
| `SPARK_enable` | `false` | 1.0.0 | Generate Spark services | | |
| `SPARK_base_version` | `2.4` | 1.15.0 | Which base version of Spark to use, one of `2.4` or `3.1` or `3.2`. Replaces `SPARK_major_version` from `1.15.0`. |
| `SPARK_base_version` | `2.4` | 1.15.0 | Which base version of Spark to use, one of `3.3` or `3.5` or `3.5`. Replaces `SPARK_major_version` from `1.15.0`. |
| `SPARK_catalog` | `in-memory` | 1.2.0 | the catalog to use for Spark, either use `in-memory` or `hive`. | | |
| `SPARK_workers` | `2` | 1.0.0 | number of Spark Worker nodes | | |
| `SPARK_master_opts` | | 1.10.0 | Configuration properties that apply only to the master in the form "-Dx=y" (default: none). See [here](https://spark.apache.org/docs/latest/spark-standalone.html) list of possible options. |
Expand Down
13 changes: 1 addition & 12 deletions modern-data-platform-stack/generator-config/stack-config.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,6 @@ vars:
HADOOP_version : 3.2.1

#spark
SPARK_2_4_version: 2.4.6
SPARK_3_1_version: 3.1.3
SPARK_3_2_version: 3.2.4
SPARK_3_3_version: 3.3.4
SPARK_3_4_version: 3.4.3
SPARK_3_5_version: 3.5.3
Expand Down Expand Up @@ -265,15 +262,7 @@ vars:
JUPYTER_SCIPY_version: latest
JUPYTER_TENSORFLOW_version: latest
JUPYTER_DATASCIENCE_version: latest
# - 2.4.0 tag 59b402ce701d
# - 2.4.3 tag abdb27a6dfbb
# - 2.4.4 tag 7a0c7325e470
# - 2.4.5 tag f9b134f7bd08
# JUPYTER_ALL_SPARK_version: f9b134f7bd08
# 3.0.1 tag 016833b15ceb
JUPYTER_ALL_SPARK_2_4_version: f9b134f7bd08
JUPYTER_ALL_SPARK_3_1_version: spark-3.1.2
JUPYTER_ALL_SPARK_3_2_version: spark-3.2.1

JUPYTER_ALL_SPARK_3_3_version: spark-3.3.2
JUPYTER_ALL_SPARK_3_4_version: spark-3.4.1
JUPYTER_ALL_SPARK_3_5_version: spark-3.5.3
Expand Down
36 changes: 3 additions & 33 deletions modern-data-platform-stack/generator-config/templates/docker-compose.yml.j2
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,16 +79,7 @@ networks:
{% set __HADOOP_version = HADOOP_version | default('latest') -%}
{% set __SQOOP_version = SQOOP_version | default('latest') -%}

{% if (SPARK_base_version | lower) == '2.4' %}
{% set __SPARK_version = SPARK_2_4_version -%}
{% set __LIVY_version = LIVY_version ~ "-spark" ~ SPARK_2_4_version.split(".")[0] ~ "." ~ SPARK_2_4_version.split(".")[1] -%}
{% elif (SPARK_base_version | lower) == '3.1' %}
{% set __SPARK_version = SPARK_3_1_version -%}
{% set __LIVY_version = LIVY_version ~ "-spark" ~ SPARK_3_1_version.split(".")[0] ~ "." ~ SPARK_3_1_version.split(".")[1] -%}
{% elif (SPARK_base_version | lower) == '3.2' %}
{% set __SPARK_version = SPARK_3_2_version -%}
{% set __LIVY_version = LIVY_version ~ "-spark" ~ SPARK_3_2_version.split(".")[0] ~ "." ~ SPARK_3_2_version.split(".")[1] -%}
{% elif (SPARK_base_version | lower) == '3.3' %}
{% if (SPARK_base_version | lower) == '3.3' %}
{% set __SPARK_version = SPARK_3_3_version -%}
{% set __LIVY_version = LIVY_version ~ "-spark" ~ SPARK_3_3_version.split(".")[0] ~ "." ~ SPARK_3_3_version.split(".")[1] -%}
{% elif (SPARK_base_version | lower) == '3.4' %}
Expand Down Expand Up @@ -244,13 +235,7 @@ networks:
{%elif (JUPYTER_edition | lower) == 'tensorflow' | default(false) %}
{% set __JUPYTER_version = JUPYTER_TENSORFLOW_version | default('latest') -%}
{%elif (JUPYTER_edition | lower) == 'all-spark' | default(false) %}
{% if (SPARK_base_version | lower) == '2.4' %}
{% set __JUPYTER_version = JUPYTER_ALL_SPARK_2_4_version | default('latest') -%}
{% elif (SPARK_base_version | lower) == '3.1' %}
{% set __JUPYTER_version = JUPYTER_ALL_SPARK_3_1_version | default('latest') -%}
{% elif (SPARK_base_version | lower) == '3.2' %}
{% set __JUPYTER_version = JUPYTER_ALL_SPARK_3_2_version | default('latest') -%}
{% elif (SPARK_base_version | lower) == '3.3' %}
{% if (SPARK_base_version | lower) == '3.3' %}
{% set __JUPYTER_version = JUPYTER_ALL_SPARK_3_3_version | default('latest') -%}
{% elif (SPARK_base_version | lower) == '3.4' %}
{% set __JUPYTER_version = JUPYTER_ALL_SPARK_3_4_version | default('latest') -%}
Expand Down Expand Up @@ -1164,22 +1149,7 @@ networks:
{% set ns.sparkMasterUrl = "spark://spark-master:7077" %}
{% endif -%}

{% if (SPARK_base_version | lower) == '2.4' %}
{% set ns.sparkTableFormatJar = ("/opt/bitnami/spark/jars/delta-core_2.12-0.6.1.jar,/opt/bitnami/spark/jars/delta-storage-0.6.1.jar" if SPARK_table_format_type == 'delta' else "/opt/bitnami/spark/jars/iceberg-spark-2-0.13.2.jar" if SPARK_table_format_type == 'iceberg' else "/opt/bitnami/spark/jars/hudi-spark2.4-bundle_2.12-0.13.0" if SPARK_table_format_type == 'hudi') -%}
{% set ns.sparkDeltaDependencies = "io.delta:delta-core_2.12:0.6.1" -%}
{% set ns.sparkIcebergDependencies = "org.apache.iceberg:iceberg-spark-2:0.13.2" -%}
{% set ns.sparkHudiDependencies ="org.apache.hudi:hudi-spark2.4-bundle_2.12:0.13.0" -%}
{% elif (SPARK_base_version | lower) == '3.1' %}
{% set ns.sparkTableFormatJar = ("/opt/bitnami/spark/jars/delta-core_2.12-1.0.1.jar,/opt/bitnami/spark/jars/delta-storage-1.0.1.jar" if SPARK_table_format_type == 'delta' else "/opt/bitnami/spark/jars/iceberg-spark-runtime-3.1_2.12-1.2.1.jar" if SPARK_table_format_type == 'iceberg' else "/opt/bitnami/spark/jars/hudi-spark3.1-bundle_2.12-0.13.0.jar" if SPARK_table_format_type == 'hudi') -%}
{% set ns.sparkDeltaDependencies = "io.delta:delta-core_2.12:1.0.1,io.delta:delta-storage:1.0.1" -%}
{% set ns.sparkIcebergDependencies = "org.apache.iceberg:iceberg-spark-runtime-3.1_2.12:1.3.1" -%}
{% set ns.sparkHudiDependencies ="org.apache.hudi:hudi-spark3.1-bundle_2.12:0.13.0" -%}
{% elif (SPARK_base_version | lower) == '3.2' %}
{% set ns.sparkTableFormatJar = ("/opt/bitnami/spark/jars/delta-core_2.12-2.0.2.jar,/opt/bitnami/spark/jars/delta-storage-2.0.2.jar" if SPARK_table_format_type == 'delta' else "/opt/bitnami/spark/jars/iceberg-spark-runtime-3.2_2.12-1.2.1.jar" if SPARK_table_format_type == 'iceberg' else "/opt/bitnami/spark/jars/hudi-spark3.2-bundle_2.12-0.13.0.jar" if SPARK_table_format_type == 'hudi') -%}
{% set ns.sparkDeltaDependencies = "io.delta:delta-core_2.12:2.0.2,io.delta:delta-storage:2.0.2" -%}
{% set ns.sparkIcebergDependencies = "org.apache.iceberg:iceberg-spark-runtime-3.2_2.12:1.4.3" -%}
{% set ns.sparkHudiDependencies ="org.apache.hudi:hudi-spark3.2-bundle_2.12:0.13.0" -%}
{% elif (SPARK_base_version | lower) == '3.3' %}
{% if (SPARK_base_version | lower) == '3.3' %}
{% set ns.sparkTableFormatJar = ("/opt/bitnami/spark/jars/delta-core_2.12-2.3.0.jar,/opt/bitnami/spark/jars/delta-storage-2.3.0.jar" if SPARK_table_format_type == 'delta' else "/opt/bitnami/spark/jars/iceberg-spark-runtime-3.3_2.12-1.2.1.jar" if SPARK_table_format_type == 'iceberg' else "/opt/bitnami/spark/jars/hudi-spark3.3-bundle_2.12-0.13.0.jar" if SPARK_table_format_type == 'hudi') -%}
{% set ns.sparkDeltaDependencies = "io.delta:delta-core_2.12:2.3.0,io.delta:delta-storage:2.3.0" -%}
{% if NESSIE_enable | default(false) %}
Expand Down
2 changes: 1 addition & 1 deletion modern-data-platform-stack/generator-config/vars/config.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -535,7 +535,7 @@
#
SPARK_enable: false

# either 2.4 or 3.1 or 3.2 or 3.3 or 3.4 or 3.5
# either 3.3 or 3.4 or 3.5
SPARK_base_version: 3.5

# "hive" or "in-memory"
Expand Down

0 comments on commit fd7939a

Please sign in to comment.