[SDK] Adds consistent parameter validation #158
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kenxu95
commented
Jun 29, 2022
| ), "New spec has a different type." | ||
|
|
||
| self.operators[str(op.id)].spec = spec | ||
| self.operator_by_name[op.name].spec = spec |
There was a problem hiding this comment.
This seemed safer than the blind replace of operators we currently have. Plus, we can add additional validation on the operation.
There was a problem hiding this comment.
What's the main downside of direct replacement?
There was a problem hiding this comment.
It's just bit uglier from the callers side, since editing an operator always requirements a fetch and replace. With this we can just construct the new spec from scratch. Replacing an operator fully might be better in the long term though, not sure.
likawind
reviewed
Jun 29, 2022
sdk/aqueduct/dag.py
Outdated
|
|
||
| if any(not isinstance(name, str) for name in self.parameters): | ||
| raise InvalidUserArgumentException("Parameters must be keyed by strings.") | ||
| check_overwriting_parameters_are_valid(dag, self.parameters) |
There was a problem hiding this comment.
maybe validate_overwriting_parameters ?
likawind
approved these changes
Jun 29, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Describe your changes and why you are making these changes
Both get() and trigger() allow for execution of flows with different parameters. This pipes any user-defined dictionary of overriding parameters through the
check_overwriting_parameters_are_valid()helper, which validates that the new parameter values in the dictionary can be used on the dag.get() uses the dag of the artifact. trigger() now uses the latest run of the flow. If there isn't a run of the flow yet, it'll throw an exception.
Related issue number (if any)
ENG-1144
Checklist before requesting a review