Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

post-repository transfer adjustments #4

Merged
merged 1 commit into from
Jul 11, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Instructions and tools for deploying and using the https://github.com/GoogleClou

We include in this project custom Spark images with the Hadoop S3a connector to connect to S3-based object storage. Those images are based on https://catalog.redhat.com/software/containers/ubi8/openjdk-8/5dd6a48dbed8bd164a09589a[ubi8/openjdk-8], and are updated accordingly.

Pre-built images can be found here, https://quay.io/repository/guimou/spark-odh and https://quay.io/repository/guimou/pyspark-odh, but you can also choose to build your own.
Pre-built images can be found here, https://quay.io/repository/opendatahub-contrib/spark and https://quay.io/repository/opendatahub-contrib/pyspark, but you can also choose to build your own.

Available images:

Expand Down
9 changes: 1 addition & 8 deletions spark-history-server/spark-hs-deployment.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
### Adapted from Helm Chart: https://artifacthub.io/packages/helm/spot/spark-history-server
# Source: spark-history-server/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
Expand All @@ -8,21 +7,18 @@ metadata:
app.kubernetes.io/name: spark-history-server
app.kubernetes.io/instance: spark-history-server
---
# Source: spark-history-server/templates/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: spark-history-server-cr
labels:
app.kubernetes.io/name: spark-history-server
app.kubernetes.io/instance: spark-history-server
app.kubernetes.io/managed-by: Helm
rules:
- apiGroups: [""]
resources: ["deployments", "pods"]
verbs: ["*"]
---
# Source: spark-history-server/templates/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
Expand All @@ -36,14 +32,12 @@ roleRef:
name: spark-history-server-cr
apiGroup: rbac.authorization.k8s.io
---
# Source: spark-history-server/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: spark-history-server
labels:
app.kubernetes.io/name: spark-history-server
helm.sh/chart: spark-history-server-1.5.0
app.kubernetes.io/instance: spark-history-server
spec:
type: ClusterIP
Expand Down Expand Up @@ -82,7 +76,6 @@ metadata:
name: spark-history-server
labels:
app.kubernetes.io/name: spark-history-server
helm.sh/chart: spark-history-server-1.5.0
app.kubernetes.io/instance: spark-history-server
spec:
replicas: 1
Expand All @@ -99,7 +92,7 @@ spec:
serviceAccountName: spark-history-server
containers:
- name: spark-history-server
image: "quay.io/guimou/spark-odh:s3.0.1-h3.3.0_v0.0.2"
image: "quay.io/opendatahub-contrib/spark:s3.3.0-h3.3.3_latest"
imagePullPolicy: IfNotPresent
serviceAccount: 'spark-operator-spark'
env:
Expand Down
2 changes: 1 addition & 1 deletion test/spark_app_shakespeare_2.4.6.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ spec:
"spark.metrics.conf.*.source.jvm.class": "org.apache.spark.metrics.source.JvmSource"
"spark.metrics.appStatusSource.enabled": "true"
mainApplicationFile: 'local:///home/wordcount.py'
image: "quay.io/guimou/pyspark-odh:s2.4.6-h3.3.0_latest"
image: "quay.io/opendatahub-contrib/pyspark:s2.4.6-h3.3.0_latest"
imagePullPolicy: Always
volumes:
- name: wordcount
Expand Down
2 changes: 1 addition & 1 deletion test/spark_app_shakespeare_3.0.1.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ spec:
"spark.metrics.conf.*.source.jvm.class": "org.apache.spark.metrics.source.JvmSource"
"spark.metrics.appStatusSource.enabled": "true"
mainApplicationFile: 'local:///home/wordcount.py'
image: "quay.io/guimou/pyspark-odh:s3.0.1-h3.3.0_latest"
image: "quay.io/opendatahub-contrib/pyspark:s3.0.1-h3.3.0_latest"
imagePullPolicy: Always
volumes:
- name: wordcount
Expand Down
2 changes: 1 addition & 1 deletion test/spark_app_shakespeare_3.3.0.history_server.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ spec:
"spark.hadoop.fs.s3a.bucket.YOUR_BUCKET.path.style.access": "true"
"spark.hadoop.fs.s3a.bucket.YOUR_BUCKET.connection.ssl.enabled": "false"
mainApplicationFile: 'local:///home/wordcount.py'
image: "quay.io/guimou/pyspark-odh:s3.3.0-h3.3.3_latest"
image: "quay.io/opendatahub-contrib/pyspark:s3.3.0-h3.3.3_latest"
imagePullPolicy: Always
volumes:
- name: wordcount
Expand Down
2 changes: 1 addition & 1 deletion test/spark_app_shakespeare_3.3.0.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ spec:
"spark.metrics.conf.*.source.jvm.class": "org.apache.spark.metrics.source.JvmSource"
"spark.metrics.appStatusSource.enabled": "true"
mainApplicationFile: 'local:///home/wordcount.py'
image: "quay.io/guimou/pyspark-odh:s3.3.0-h3.3.3_latest"
image: "quay.io/opendatahub-contrib/pyspark:s3.3.0-h3.3.3_latest"
imagePullPolicy: Always
volumes:
- name: wordcount
Expand Down