Super pipeline generator #233
Conversation
Signed-off-by: Mohammad Nassar <[email protected]>
Signed-off-by: Mohammad Nassar <[email protected]>
Signed-off-by: Mohammad Nassar <[email protected]>
Signed-off-by: Mohammad Nassar <[email protected]>
Signed-off-by: Mohammad Nassar <[email protected]>
|
@Mohammad-nassar10, @revit13, @roytman is this still relevant? It has been dormant for a while |
|
@blublinsky yes, I updated in the past two days. |
| @@ -0,0 +1,6 @@ | |||
| ## Steps to generate a new super pipeline in KFP v1. | |||
| - The super pipeline is a way to execute several transforms with one pipeline. You can find more details in [multi_transform_pipeline.md](../../doc/multi_transform_pipeline.md) | |||
There was a problem hiding this comment.
Please restrict line length to 80 characters:
How about the following explanation ?
- The super pipeline allows you to execute several transforms within a single pipeline. For more details, refer to multi_transform_pipeline.md
- Create a super_pipeline_definitions.yaml file for the required task. You can refer to the example super_pipeline_definitions.yaml.
- Execute ./run.sh < super_pipeline_definitions.yaml> <destination_directory>. Here,
super_pipeline_definitions.yamlis the super pipeline definition file, that you created above, and destination_directory is the directory where the new super pipeline file will be generated.
Note: The component_spec_path is the path to the kfp_ray_components folder and depends on where the workflow is compiled.
There was a problem hiding this comment.
where do we set component_spec_path ?
There was a problem hiding this comment.
The Readme.md updated.
The user can set the component_spec_path in the super_pipeline_definitions.yaml.
| @@ -0,0 +1,11 @@ | |||
| #!/bin/bash | |||
|
|
|||
| DEF_FILE=$1 | |||
There was a problem hiding this comment.
can you take examples of arguments processing from run.sh in the single pipeline generator
There was a problem hiding this comment.
I changed the implementation to use jinja as in the single pipeline generator.
Signed-off-by: Mohammad Nassar <[email protected]>
Signed-off-by: Mohammad Nassar <[email protected]>
Signed-off-by: Mohammad Nassar <[email protected]>
Why are these changes needed?
Add a tool that helps to generate super pipelines python code.
Related issue number (if any).