Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Document how to update spark jobs that are called from Airflow DAGs #17

Open
adwk67 opened this issue Nov 28, 2023 · 1 comment
Open

Document how to update spark jobs that are called from Airflow DAGs #17

adwk67 opened this issue Nov 28, 2023 · 1 comment

Comments

@adwk67
Copy link
Member

adwk67 commented Nov 28, 2023

When upgrading the airflow demo from 23.7 to 23.11 the spark job wrapped by an existing DAG is not updated if the config map containing its definition is changed. This should be investigated and documented as to how to do this.
@fhennig
Copy link
Contributor

fhennig commented Feb 1, 2024

I'm assuming this is about this job: https://docs.stackable.tech/home/stable/demos/airflow-scheduled-job

The caching issue might also not be related to just the demo, but can happen to regular users too.

@fhennig fhennig transferred this issue from stackabletech/stackablectl Feb 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@adwk67 @fhennig