#356 Spark working with java17

356 update deprecated mda scala reference 356 Update mavenUtils to be compatable with jdk17 356 Create aissemble compatible quarkus bom Also updated the version of pyasn1 to the latest compatible version
boozallen · Sep 27, 2024 · 47245c9 · 47245c9
1 parent 7559a95
commit 47245c9
Show file tree

Hide file tree

Showing 35 changed files with 197 additions and 91 deletions.
diff --git a/bom/aissemble-quarkus-bom/LICENSE.txt b/bom/aissemble-quarkus-bom/LICENSE.txt
@@ -0,0 +1,49 @@
+Booz Allen Public License v1.0
+------------------------------
+
+INTRODUCTION
+The Booz Allen Public License allows government, non-profit academic, other non-profit, and commercial entities access to distinctive, disruptive, and robust code with the goal of Empowering People to Change the World(SM). Products licensed under the Booz Allen Public License are founded on the basis that collective ingenuity can make the largest impact in the community.
+
+DEFINITIONS
+* **Commercial Entity.** “Commercial Entity” means any individual or entity other than a government, non-profit academic, or other non-profit entity.
+* **Derivative.** “Derivative” means any work of authorship in Source Code or Object Code form that results from an addition to, deletion from, or modification of the Source Code of the Product.
+* **License.**  “License” means this Booz Allen Public License.
+* **Object Code.** “Object Code” means the form resulting from transformation or translation of Source Code into machine readable code, including but not limited to, compiled object code.
+* **Originator.** “Originator” means each individual or legal entity that creates, contributes to the creation of, or owns the Product. 
+* **Patent Claims.** “Patent Claims” means any patent claim(s) in any patent to which Originator has a right to grant a license that would be infringed by Your making, using, selling, offering for sale, having made, or importing of the Product, but for the grant of this License. 
+* **Product.** “Product” means the Source Code of the software which the initial Originator made available under this License, and any Derivative of such Source Code. 
+* **Source Code.** “Source Code” means software in human-readable form.
+* **You.** “You” means either an individual or an entity (if you are taking this license on behalf of an entity) that exercises the rights granted under this License.  
+
+LICENSE
+**Government/Non-Profit Academic/Other Non-Profit.**  
+This Section applies if You are not a Commercial Entity.  
+
+* **License.** Subject to the terms and conditions of this License, each Originator hereby grants You a perpetual, worldwide, non-exclusive, royalty-free license to reproduce, display, perform, modify, distribute and otherwise use the Product and Derivatives, in Source Code and Object Code form, in accordance with the terms and conditions of this License in order to support the general public good and for your internal business purposes.
+* **Distribution.** You may distribute to third parties copies of the Product, including any Derivative that You create, in Source Code or Object Code form.  If You distribute copies of the Product, including any Derivative that You create, in Source Code form, such distribution must be under the terms of this License and You must inform recipients of the Source Code that the Product is governed under this License and how they can obtain a copy of this License.  You may distribute to third parties copies of the Product, including any Derivative that You create, in Object Code form, or allow third parties to access or use the Product, including any Derivative that You create, under a license of Your choice.
+* **Commercial Sales.** You may not distribute, or allow third parties to access or use, the Product or any Derivative for a fee, unless You first obtain permission from the Originator. If Booz Allen Hamilton is the Originator, please contact Booz Allen Hamilton at  <[email protected]>.  
+
+**Commercial Entities**.
+This Section applies if You are a Commercial Entity.
+
+* **License.** Subject to the terms and conditions of this License, each Originator hereby grants You a perpetual, worldwide, non-exclusive, royalty-free license to reproduce, display, perform, modify, distribute and otherwise use the Product and Derivatives, in Source Code and Object Code form, in accordance with the terms and conditions of this License for the sole purpose of Your internal business purposes and the provision of services to government, non-profit academic, and other non-profit entities. 
+* **Distribution and Derivatives.** You may distribute to third parties copies of the Product, including any Derivative that You create, in Source Code or Object Code form.  If You distribute copies of the Product, including any Derivative that You create, in Source Code form, such distribution must be under the terms of this License and You must inform recipients of the Source Code that the Product is governed under this License and how they can obtain a copy of this License.  You may distribute to third parties copies of the Product, including any Derivative that You create, in Object Code form, or allow third parties to access or use the Product, including any Derivative that You create, under a license of Your choice, provided that You make available, and inform the recipient of such distribution how they can obtain, a copy of the Source Code thereof, at no charge, and inform the recipient of the Source Code that the Product is governed under this License and how they can obtain a copy of this License.
+* **Commercial Sales.** You may not distribute, or allow third parties to access or use, the Product or any Derivative for a fee, unless You first obtain permission from the Originator. If Booz Allen Hamilton, please contact Booz Allen Hamilton at <[email protected]>.   
+
+
+**Patent Claim(s)**.  
+This Section applies regardless of whether You are a government, non-profit academic, or other non-profit entity or a Commercial Entity. 
+
+* **Patent License.** Subject to the limitations in the Sections above, each Originator hereby grants You a perpetual, worldwide, non-exclusive, royalty-free license under Patent Claims of such Originator to make, use, sell, offer for sale, have made, and import the Product.  The foregoing patent license does not apply (a) to any code that an Originator has removed from the Product, or (b) for infringement caused by Your modifications of the Product or the combination of any Derivative created by You or on Your behalf with other software.      
+
+GENERAL TERMS
+This Section applies regardless of whether You are a government, non-profit academic, or other non-profit entity or a Commercial Entity.
+
+* **Required Notices.** If You distribute the Product or a Derivative, in Object Code or Source Code form, You shall not remove or otherwise modify any proprietary markings or notices contained within or placed upon the Product or any Derivative. Any distribution of the Product or a Derivative, in Object Code or Source Code form, shall contain a clear and conspicuous Originator copyright and license reference in accordance with the below:
+	* *Unmodified Product Notice*: “This software package is licensed under the Booz Allen Public License. Copyright © 20__ [Copyright Holder Name].  All Rights Reserved.”
+	* *Derivative Notice*: “This software package is licensed under the Booz Allen Public License. Portions of this code are Copyright © 20__ [Copyright Holder Name].  All Rights Reserved.”
+* **Compliance with Laws.** You agree that You shall not reproduce, display, perform, modify, distribute and otherwise use the Product in any way that violates applicable law or regulation or infringes or violates the rights of others, including, but not limited to, third party intellectual property, privacy, and publicity rights.
+* **Disclaimer.** You understand that the Product is licensed to You, and not sold. The Product is provided on an “As Is” basis, without any warranties, representations, and guarantees, whether oral or written, express, implied or statutory, with regard to the Product, including without limitation, warranties of merchantability, fitness for a particular purpose, title, non-infringement, non-interference, and warranties arising from course of dealing or usage of trade, to the maximum extent permitted by applicable law.  Originator does not warrant that (i) the Product will meet your needs; (ii) the Product will be error-free or accessible at all times; or (iii) the use or the results of the use of the Product will be correct, accurate, timely, or otherwise reliable. You acknowledge that the Product has not been prepared to meet Your individual requirements, whether or not such requirements have been communicated to Originator. You assume all responsibility for use of the Product.
+* **Limitation of Liability.** Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Originator, or anyone who distributes the Product in accordance with this License, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if informed of the possibility of such damages.
+* **Export Control.** The Product is subject to U.S. export control laws and may be subject to export or import regulations in other countries.  You agree to strictly comply with all such laws and regulations and acknowledges that You are responsible for obtaining such licenses to export, re-export, or import as may be required.
+* **Severability.** If the application of any provision of this License to any particular facts or circumstances shall be held to be invalid or unenforceable, then the validity and enforceability of other provisions of this License shall not in any way be affected or impaired thereby.  
diff --git a/bom/aissemble-quarkus-bom/pom.xml b/bom/aissemble-quarkus-bom/pom.xml
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+
+    <parent>
+        <groupId>com.boozallen.aissemble</groupId>
+        <artifactId>bom</artifactId>
+        <version>1.10.0-SNAPSHOT</version>
+    </parent>
+
+    <artifactId>aissemble-quarkus-bom</artifactId>
+    <packaging>pom</packaging>
+
+    <name>aiSSEMBLE::BOM::Quarkus</name>
+    <description>An aiSSEMBLE compabible version of the quarkus-universe-bom</description>
+
+
+    <dependencyManagement>
+        <dependencies>
+            <dependency>
+                <groupId>io.quarkus</groupId>
+                <artifactId>quarkus-universe-bom</artifactId>
+                <version>${version.quarkus.plugin}</version>
+                <type>pom</type>
+                <scope>import</scope>
+                <exclusions>
+                    <exclusion>
+                        <groupId>io.smallrye.reactive</groupId>
+                        <artifactId>*</artifactId>
+                    </exclusion>
+                </exclusions>
+            </dependency>
+        </dependencies>
+    </dependencyManagement>
+
+</project>
diff --git a/bom/bom-component/pom.xml b/bom/bom-component/pom.xml
@@ -24,13 +24,6 @@
                 <type>pom</type>
                 <scope>import</scope>
             </dependency>
-            <dependency>
-                <groupId>io.quarkus</groupId>
-                <artifactId>quarkus-universe-bom</artifactId>
-                <version>${version.quarkus.plugin}</version>
-                <type>pom</type>
-                <scope>import</scope>
-            </dependency>
             <dependency>
                 <groupId>org.technologybrewery.krausening</groupId>
                 <artifactId>krausening</artifactId>

diff --git a/bom/pom.xml b/bom/pom.xml
@@ -16,6 +16,7 @@
     <modules>
         <module>bom-component</module>
         <module>bom-instantiation</module>
+        <module>aissemble-quarkus-bom</module>
     </modules>
 
 </project>
diff --git a/build-parent/pom.xml b/build-parent/pom.xml
@@ -55,7 +55,7 @@
         <version.plexus.component>2.1.1</version.plexus.component>
         <version.postgresql>42.5.1</version.postgresql>
         <version.resteasy>4.7.7.Final</version.resteasy>
-        <version.vertx>4.2.5</version.vertx>
+        <version.vertx>4.4.6</version.vertx>
         <version.urllib3>^1.26.18</version.urllib3>
         <version.fastapi>&gt;=0.95.0</version.fastapi>
         <version.uvicorn>&gt;=0.18.0</version.uvicorn>
@@ -71,10 +71,11 @@
         <version.jakarta.cdi>4.0.1</version.jakarta.cdi>
         <version.jakarta.inject>2.0.1</version.jakarta.inject>
         <version.jakarta.annotation>3.0.0</version.jakarta.annotation>
+        <version.jakarta.wr.rs>3.1.0</version.jakarta.wr.rs>
         <version.javax.servlet>3.1.0</version.javax.servlet>
 
 <!--        Spark Default Dependencies.  See `spark-*` profiles below for alternative sets -->
-        <version.spark>3.4.0</version.spark>
+        <version.spark>3.5.2</version.spark>
         <version.scala>2.12.13</version.scala>
         <version.delta>2.4.0</version.delta>
         <version.elasticsearch>8.9.0</version.elasticsearch>
@@ -863,6 +864,9 @@
                             <TESTCONTAINERS_RYUK_DISABLED>${TESTCONTAINERS_RYUK_DISABLED}</TESTCONTAINERS_RYUK_DISABLED>
                             <!--<TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX>YOUR MIRROR OR INTENRAL REPO GOES HERE - e.g., nexus.foobar.com/</TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX>-->
                         </environmentVariables>
+                        <!-- Spark needs to use an internal API that has been made inaccessible by default. This
+                        exposes it -->
+                        <argLine>--add-exports java.base/sun.nio.ch=ALL-UNNAMED</argLine>
                     </configuration>
                 </plugin>
                 <plugin>

diff --git a/extensions/extensions-alerting/extensions-alerting-teams/pom.xml b/extensions/extensions-alerting/extensions-alerting-teams/pom.xml
@@ -16,18 +16,11 @@
     <dependencyManagement>
         <dependencies>
             <dependency>
-                <groupId>io.quarkus</groupId>
-                <artifactId>quarkus-universe-bom</artifactId>
-                <version>${version.quarkus.plugin}</version>
+                <groupId>${project.groupId}</groupId>
+                <artifactId>aissemble-quarkus-bom</artifactId>
+                <version>${project.version}</version>
                 <type>pom</type>
                 <scope>import</scope>
-                <exclusions>
-                    <!-- Quarkus 3.6 uses SmallRye Messaging 4.10, which is incompatible with 4.24. This causes issues for in-memory import -->
-                    <exclusion>
-                        <groupId>io.smallrye.reactive</groupId>
-                        <artifactId>*</artifactId>
-                    </exclusion>
-                </exclusions>
             </dependency>
             <dependency>
                 <groupId>io.smallrye.reactive</groupId>
@@ -147,4 +140,4 @@
             </build>
         </profile>
     </profiles>
-</project>
+</project>
diff --git a/extensions/extensions-alerting/pom.xml b/extensions/extensions-alerting/pom.xml
@@ -38,9 +38,9 @@
                 <version>${project.version}</version>
             </dependency>
             <dependency>
-                <groupId>io.quarkus</groupId>
-                <artifactId>quarkus-universe-bom</artifactId>
-                <version>${version.quarkus.plugin}</version>
+                <groupId>${project.groupId}</groupId>
+                <artifactId>aissemble-quarkus-bom</artifactId>
+                <version>${project.version}</version>
                 <type>pom</type>
                 <scope>import</scope>
             </dependency>
@@ -119,4 +119,4 @@
         </dependencies>
     </dependencyManagement>
 
-</project>
+</project>
diff --git a/...sions/extensions-data-delivery/aissemble-extensions-data-delivery-spark-py/pyproject.toml b/...sions/extensions-data-delivery/aissemble-extensions-data-delivery-spark-py/pyproject.toml
@@ -11,7 +11,7 @@ packages = [
 [tool.poetry.dependencies]
 python = ">=3.8"
 krausening = ">=20"
-pyspark = "3.4.0"
+pyspark = "3.5.2"
 pyyaml = "^6.0"
 
 [tool.poetry.group.dev.dependencies]

diff --git a/...rk/src/main/java/com/boozallen/aiops/data/delivery/messaging/PipelineMessagingConfig.java b/...rk/src/main/java/com/boozallen/aiops/data/delivery/messaging/PipelineMessagingConfig.java
@@ -47,7 +47,7 @@ public Map<String, String> getProperties() {
         Map<String, String> map = new HashMap<>();
         for (String name : pipelineProperties.stringPropertyNames()) {
             String reactiveName = mapToReactiveProperty(name);
-            String value = pipelineProperties.getProperty(reactiveName);
+            String value = pipelineProperties.getProperty(name);
             map.put(reactiveName, value);
         }
         return map;

diff --git a/...ions-data-delivery-spark/src/test/resources/krausening/base/pipeline-messaging.properties b/...ions-data-delivery-spark/src/test/resources/krausening/base/pipeline-messaging.properties
@@ -18,3 +18,4 @@ NoChannelTestStep.in.connector=smallrye-kafka
 # these are overridden to in-memory in test/pipeline-messaging.properties
 OverrideTestStep.in.connector=smallrye-kafka
 OverrideTestStep.out.connector=smallrye-kafka
+
diff --git a/...ons-data-delivery-spark/src/test/resources/specifications/messaging-configuration.feature b/...ons-data-delivery-spark/src/test/resources/specifications/messaging-configuration.feature
@@ -9,7 +9,7 @@ Feature: Pipeline messaging steps can be configured
     When messages flow from the external system
     Then the configuration directs the messaging to the step
     And the configuration directs the step result to the external system
-
+#
   Scenario: I can configure a step with only incoming messaging
     Given a step with incoming messaging
     And a pipeline configuration for the step
@@ -35,4 +35,4 @@ Feature: Pipeline messaging steps can be configured
     And a pipeline configuration for the step
     When messages flow from the external system
     Then the configuration directs the messaging to the step
-    And the configuration directs the step result to the external system
+    And the configuration directs the step result to the external system
diff --git a/...ons-data-delivery-spark/src/test/resources/specifications/object-store-validation.feature b/...ons-data-delivery-spark/src/test/resources/specifications/object-store-validation.feature
@@ -1,8 +1,8 @@
 @data-delivery @object-store-validation
-Feature: Object store credentials can be verified for read and write access 
-  I want to use configured credentials to connect to an object store and 
+Feature: Object store credentials can be verified for read and write access
+  I want to use configured credentials to connect to an object store and
   verify read and write access
 
   Scenario: I can retrieve credentials from a Krausening file
     Given a properties file exists
-    Then the credentials are used to verify object store connectivity
+    Then the credentials are used to verify object store connectivity
diff --git a/extensions/extensions-docker/aissemble-spark/src/main/resources/docker/Dockerfile b/extensions/extensions-docker/aissemble-spark/src/main/resources/docker/Dockerfile
@@ -1,7 +1,7 @@
 # Script for creating base Spark Docker image
 #
 ARG SPARK_VERSION
-FROM apache/spark-py:v${SPARK_VERSION}
+FROM apache/spark:${SPARK_VERSION}-scala2.12-java17-python3-ubuntu
 
 LABEL org.opencontainers.image.source="https://github.com/boozallen/aissemble"
 
@@ -14,9 +14,7 @@ RUN mkdir $SPARK_HOME/checkpoint && \
   mkdir $SPARK_HOME/krausening && \
   mkdir $SPARK_HOME/warehouse && \
   mkdir $SPARK_HOME/policies && \
-  mkdir $SPARK_HOME/logs && \
-  useradd --home /opt/spark --group 0 --shell /usr/sbin/nologin --uid 185 spark && \
-  chown -R spark:spark /opt/spark
+  mkdir $SPARK_HOME/logs
 
 # Update repositories and add necessary ones
 RUN apt-get -y update \

diff --git a/extensions/extensions-docker/pom.xml b/extensions/extensions-docker/pom.xml
@@ -69,16 +69,16 @@
         <module>aissemble-spark-operator</module>
         <module>aissemble-vault</module>
         <module>aissemble-versioning</module>
-        <module>aissemble-model-training-api-containers</module>
-        <module>aissemble-data-lineage-http-consumer</module>
+<!--        <module>aissemble-model-training-api-containers</module>-->
+<!--        <module>aissemble-data-lineage-http-consumer</module>-->
         <module>aissemble-pipeline-invocation</module>
-        <module>aissemble-configuration-store</module>
+<!--        <module>aissemble-configuration-store</module>-->
     </modules>
 
     <dependencyManagement>
         <dependencies>
             <dependency>
-                <groupId>com.boozallen.aissemble</groupId>
+                <groupId>${project.groupId}</groupId>
                 <artifactId>bom-component</artifactId>
                 <version>${project.version}</version>
                 <type>pom</type>

diff --git a/extensions/extensions-helm/aissemble-spark-operator-chart/templates/rbac.yaml b/extensions/extensions-helm/aissemble-spark-operator-chart/templates/rbac.yaml
@@ -38,6 +38,7 @@ rules:
   - delete
   - update
   - list
+  - patch
 - apiGroups:
   - extensions
   - networking.k8s.io
@@ -137,4 +138,4 @@ roleRef:
   kind: ClusterRole
   name: {{ include "spark-operator.fullname" (index .Subcharts "spark-operator") }}
   apiGroup: rbac.authorization.k8s.io
-{{- end }}
+{{- end }}
diff --git a/...sions-helm/extensions-helm-spark-infrastructure/aissemble-spark-history-chart/values.yaml b/...sions-helm/extensions-helm-spark-infrastructure/aissemble-spark-history-chart/values.yaml
@@ -7,9 +7,9 @@ deployment:
   labels: {}
   replicas: 1
   image:
-    repository: "apache/spark"
+    repository: "ghcr.io/boozallen/aissemble-spark"
     imagePullPolicy: IfNotPresent
-    tag: "3.5.1"
+    tag: "1.9.0-SNAPSHOT"
   command: ["/opt/spark/sbin/start-history-server.sh"]
   env:
     SPARK_NO_DAEMONIZE: "true"