Adds workflow_dag engine_config column #209
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
saurav-c
changed the title
saurav-c
force-pushed
the
eng-1276-add-backend-support-for-connecting-to-airflow
branch
from
d90961a to
15f5939
Compare
Base automatically changed from
eng-1276-add-backend-support-for-connecting-to-airflow
to
main
July 14, 2022 23:32
saurav-c
force-pushed
the
03-eng-1243-add-engine-config-col-to-workflow-dag
branch
3 times, most recently
from
c985837 to
f1ed136
Compare
saurav-c
changed the title
saurav-c
changed the base branch from
main
to
eng-1392-refactor-python-operator-entrypoints
Base automatically changed from
eng-1392-refactor-python-operator-entrypoints
to
main
July 21, 2022 17:48
saurav-c
force-pushed
the
03-eng-1243-add-engine-config-col-to-workflow-dag
branch
4 times, most recently
from
6345003 to
ef78cd5
Compare
saurav-c
commented
Jul 22, 2022
saurav-c
force-pushed
the
03-eng-1243-add-engine-config-col-to-workflow-dag
branch
2 times, most recently
from
3ca8c68 to
92114b5
Compare
saurav-c
changed the base branch from
main
to
eng-1435-allow-user-to-specify-s3-creds-airflow
likawind
approved these changes
Jul 25, 2022
src/golang/cmd/migrator/versions/000013_add_workflow_dag_engine_config/up_postgres.go
Show resolved
Hide resolved
| @@ -29,7 +29,7 @@ import ( | |||
| ) | |||
|
|
|||
| const ( | |||
| RequiredSchemaVersion = 8 | |||
There was a problem hiding this comment.
wow where is this field used? I'm surprised that nothing has been broken so far
There was a problem hiding this comment.
Well this was more important in the enterprise project. Here once we modify the install_local and packaging scripts, we know that the next release will be running the correct schema version.
We do have a check here https://github.com/aqueducthq/aqueduct/blob/main/src/golang/cmd/server/server/aqueduct_server.go#L118 just to confirm.
Base automatically changed from
eng-1435-allow-user-to-specify-s3-creds-airflow
to
main
July 26, 2022 20:49
saurav-c
force-pushed
the
03-eng-1243-add-engine-config-col-to-workflow-dag
branch
from
92114b5 to
de41446
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Describe your changes and why you are making these changes
This PR adds an
engine_configcolumn to theworkflow_dagtable. This is necessary to store important information for mapping an Aqueduct workflow to the native concepts of the 3rd-party engine the workflow is set to run on, such as Airflow.Related issue number (if any)
ENG-1243
Checklist before requesting a review
python3 scripts/run_linters.py -hfor usage).run_integration_test: Runs integration testsskip_integration_test: Skips integration tests (Should be used when changes are ONLY documentation/UI)Test
I tested this schema change by creating a workflow (and workflow_dag) before applying the schema change. Then, I applied the schema change and confirmed that the new column was created. After this, I tested that the
WorkflowDagReaderwas able to successfully read theengine_configcolumn.