Added TRANSFROM_NAME to docker build arg #929
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…f docker file template Signed-off-by: Maroun Touma <[email protected]>
roytman
reviewed
Jan 9, 2025
|
|
||
| # END OF STEPS destined for a data-prep-kit base image | ||
|
|
||
| COPY --chown=dpk:root dpk_${TRANSFORM_NAME}/ dpk_${TRANSFORM_NAME}/ |
There was a problem hiding this comment.
I think it should be --chown=dpk:users
roytman
reviewed
Jan 9, 2025
transforms/Dockerfile.spark.template
Outdated
| # Install project source | ||
|
|
||
| ## Copy the python version of the tansform | ||
| COPY --chown=spark:root dpk_${TRANSFORM_NAME}/ dpk_${TRANSFORM_NAME}/ |
| @@ -0,0 +1,31 @@ | |||
| ARG BASE_IMAGE=docker.io/rayproject/ray:2.24.0-py310 | |||
There was a problem hiding this comment.
we use this BASE_IMAGE mechanism with Ray only, should it be extended to Python and Spark as well?
There was a problem hiding this comment.
@roytman I'd like to defer this to another PR when I do the full cleanup of the Makefile and other CI/CD related artifacts. #930 For this PR, I just want to setup the templates so developers of new transforms don't have to edit the Dockerfiles and simply use the template as is unless they have some complex custom requirements. I will need to do another round on the templates when I do the Makefile here #930
Signed-off-by: Maroun Touma <[email protected]>
roytman
approved these changes
Jan 9, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…f docker file template
Why are these changes needed?
While doing the refactoring of the transforms, we observed that now, most of the Dockerfile have the same scripts except for the module name. This change adds TRANSFORM_NAME build arg and dockerfile templates for python, ray and spark so transforms owner can simply copy/paste the template. In most cases, the transform owner will not need to modify the file unless their transform require additional non-common components that are unique to the run time environment for this transform
Related issue number (if any).
[Feature] Restructure transforms as their own modules #774