Skip to content

Deploy Spark as an alternative historian #462

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 31 commits into
base: main
Choose a base branch
from
Draft

Deploy Spark as an alternative historian #462

wants to merge 31 commits into from

Conversation

@fabian-amrc
Copy link

@fabian-amrc fabian-amrc commented Mar 25, 2025 •
edited
Loading

  • Add spark operator to umbrella chart
  • Update MinIO operator and tenant (and add new bucket + remove external dependency)
  • Add Kafka controller to umbrella chart
  • Edit UNS-Ingester to post data to kafka topic as well as store in InfluxDB
  • Add spark application to stream data into the datastore (application created but database configuration still to be worked out)
  • Configure with OpenId

@fabian-amrc fabian-amrc reopened this Mar 25, 2025
@fabian-amrc fabian-amrc marked this pull request as draft March 25, 2025 16:03
fabian-amrc and others added 18 commits March 26, 2025 16:05
@fabian-amrc
Deleted unnecessary spark operator crds
4b8546e
@fabian-amrc
Chang minio ingress to use templates
2493c77 
This is instead of using the values file to specify ingress
@fabian-amrc
Disabled spark application ingress
8efc7c5 
No adequate solution could be found to have the spark applications use the base url (also specified in the values file)
@fabian-amrc
Fix linting error
28f4519
@fabian-amrc
Merge remote-tracking branch 'origin/testing/v4' into fje/data-lake
f33092f
@fabian-amrc
Increase partitions and replication factor for kafka
39f459a 
This increases the speed and resilience of the kafka broker
@fabian-amrc
Create separate historian module for kafka
50df5d7
@fabian-amrc
Revert "Add push to kafka topic from mqtt broker"
cf69857 
This reverts commit b9febbc.
@fabian-amrc
Update example env file for historian-uns-kafka
d7fb562
@fabian-amrc
Fix errors getting process.env variables
f924630
@fabian-amrc
Rename KafkaClient to MqttClient
8b794e0
@fabian-amrc
Add kafka ingress
97d6000
@fabian-amrc
Add kafkaUns to historians values
a0b5bd5
@fabian-amrc
Add kafka kerberos key
8922b39
@fabian-amrc
Add Kafka group uuid
8a46173
@fabian-amrc
Remove unnecessary dependencies from kafka-uns
e0ea057
@fabian-amrc
Fix kafka module naming and description
7af03ed
@fabian-amrc
Add historian-uns-kafka to github workflow
73db45c
@amrc-benmorrow amrc-benmorrow changed the title Fje/data lake Deploy Spark as an alternative historian Apr 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

work in progress

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@fabian-amrc @amrc-benmorrow