[feature] Add support for model artifacts in the "modelcar" format #11584

mprahl · 2025-02-04T19:44:56Z

Feature Area

/area backend

What feature would you like to see?

Currently, a user may import a model as an artifact with the KFP SDK like the following example, but it's limited to S3.

model_source_s3_task = dsl.importer(artifact_uri=model_in_s3, artifact_class=dsl.Model)

It'd be nice to add support for the modelcar format that is supported in KServe.

The suggestion is for the artifact URI passed to dsl.importer to have a prefix of oci://. No changes to the SDK should be needed for this.

The KFP driver would see this and add an init container to the launcher pod which launches the modelcar container image and copies the model to an emptyDir volume. The artifact output would be the local path to this emptyDir volume in the launcher container. The benefit of doing it this way allows for the global image pull secret or image pull secrets assigned to the KFP pipeline runner service account to pull it without specifying additional credentials.

Once OCI Volume Source has been GA for several Kubernetes releases, we may consider leverage that instead of the init container.

What is the use case or pain point?

In order to support fine tuning a model stored in an OCI registry in the modelcar format without first copying to S3, it'd be nice to have native support for a model artifact stored in OCI.

Is there a workaround currently?

The user must copy it to S3 outside of the pipeline or manually pull down the image.

Love this idea? Give it a 👍.

The text was updated successfully, but these errors were encountered:

mprahl · 2025-02-04T19:45:04Z

/assign @mprahl

HumairAK · 2025-02-05T00:09:25Z

I think this is a great proposal @mprahl, thank you. There are various benefits to this approach beyond pull secrets. The linked Kserve page outlines some of these, but I'll paste them here for clarity:

Reduced Startup Times: By avoiding repetitive downloads of large models, startup delays are significantly minimized.

Lower Disk Space Usage: The feature decreases the need for duplicated local storage, conserving disk space.

Enhanced Performance: Modelcars allows for advanced techniques like pre-fetching images and lazy-loading, improving efficiency.

As kserve is an addon component to KF, aligning here just makes sense as well.

cc @franciscojavierarceo fyi

franciscojavierarceo · 2025-02-05T01:37:24Z

Nice, this is great @mprahl !

This allows dsl.import to leverage Modelcar container images in an OCI repository. This works by having any component container have an init container with the image set to the Modelcar container image and the /models directory contents copied to an empty dir volume that is accessible in the component container. This approach has the benefit of leveraging image pull secrets configured on the Kubernetes cluster rather than require separate credentials for importing the artifact. Note that once Kubernetes supports OCI images as volume mounts for several releases, consider replacing the init container with that approach. Resolves: kubeflow#11584 Signed-off-by: mprahl <[email protected]>

This allows dsl.import to leverage Modelcar container images in an OCI repository. This works by having any component container have an init container with the image set to the Modelcar container image and the /models directory contents copied to an emptyDir volume that is accessible in the component container. This approach has the benefit of leveraging image pull secrets configured on the Kubernetes cluster rather than require separate credentials for importing the artifact. Note that once Kubernetes supports OCI images as volume mounts for several releases, consider replacing the init container with that approach. Resolves: kubeflow#11584 Signed-off-by: mprahl <[email protected]>

This allows dsl.import to leverage Modelcar container images in an OCI repository. This works by having an init container prepull the image and then adding a sidecar container when the launcher container is running. The Modelcar container adds a symlink to its /models directory in an emptyDir volume that is accessible by the launcher container. Once the launcher is done running the user code, it stops the Modelcar container images. This approach has the benefit of leveraging image pull secrets configured on the Kubernetes cluster rather than require separate credentials for importing the artifact. Additionally, no data is copied to the emptyDir volume, so the storage cost is just pulling the Modelcar container image on the Kubernetes worker node. Note that once Kubernetes supports OCI images as volume mounts for several releases, consider replacing the init container with that approach. Resolves: kubeflow#11584 Signed-off-by: mprahl <[email protected]>

This allows dsl.import to leverage Modelcar container images in an OCI repository. This works by having an init container prepull the image and then adding a sidecar container when the launcher container is running. The Modelcar container adds a symlink to its /models directory in an emptyDir volume that is accessible by the launcher container. Once the launcher is done running the user code, it stops the Modelcar containers. This approach has the benefit of leveraging image pull secrets configured on the Kubernetes cluster rather than require separate credentials for importing the artifact. Additionally, no data is copied to the emptyDir volume, so the storage cost is just pulling the Modelcar container image on the Kubernetes worker node. Note that once Kubernetes supports OCI images as volume mounts for several releases, consider replacing the init container with that approach. Resolves: kubeflow#11584 Signed-off-by: mprahl <[email protected]>

This allows dsl.import to leverage Modelcar container images in an OCI repository. This works by having an init container prepull the image and then adding a sidecar container when the launcher container is running. The Modelcar container adds a symlink to its /models directory in an emptyDir volume that is accessible by the launcher container. Once the launcher is done running the user code, it stops the Modelcar containers. This approach has the benefit of leveraging image pull secrets configured on the Kubernetes cluster rather than require separate credentials for importing the artifact. Additionally, no data is copied to the emptyDir volume, so the storage cost is just pulling the Modelcar container image on the Kubernetes worker node. Note that once Kubernetes supports OCI images as volume mounts for several releases, consider replacing the init container with that approach. This also adds a new environment variable of PIPELINE_RUN_AS_USER to set the runAsUser on all pods created by Argo Workflows. Resolves: kubeflow#11584 Signed-off-by: mprahl <[email protected]>

This allows dsl.import to leverage Modelcar container images in an OCI repository. This works by having an init container prepull the image and then adding a sidecar container when the launcher container is running. The Modelcar container adds a symlink to its /models directory in an emptyDir volume that is accessible by the launcher container. Once the launcher is done running the user code, it stops the Modelcar containers. This approach has the benefit of leveraging image pull secrets configured on the Kubernetes cluster rather than require separate credentials for importing the artifact. Additionally, no data is copied to the emptyDir volume, so the storage cost is just pulling the Modelcar container image on the Kubernetes worker node. Note that once Kubernetes supports OCI images as volume mounts for several releases, consider replacing the init container with that approach. This also adds a new environment variable of PIPELINE_RUN_AS_USER to set the runAsUser on all Pods created by Argo Workflows. Resolves: kubeflow#11584 Signed-off-by: mprahl <[email protected]>

mprahl added the kind/feature label Feb 4, 2025

google-oss-prow bot added the area/backend label Feb 4, 2025

google-oss-prow bot assigned mprahl Feb 4, 2025

HumairAK added this to the KFP 2.5.0 milestone Feb 5, 2025

HumairAK added this to KFP 2.x Release Feb 5, 2025

HumairAK moved this to In Progress in KFP 2.x Release Feb 5, 2025

mprahl mentioned this issue Feb 5, 2025

WIP: feat(backend): Add support for importing models stored in the Modelcar format (emptyDir copy) #11590

Closed

2 tasks

mprahl mentioned this issue Feb 8, 2025

feat(backend): Add support for importing models stored in the Modelcar format (sidecar) #11606

Merged

2 tasks

google-oss-prow bot closed this as completed in cc1c435 Feb 19, 2025

github-project-automation bot moved this from In Progress to Done in KFP 2.x Release Feb 19, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature] Add support for model artifacts in the "modelcar" format #11584

[feature] Add support for model artifacts in the "modelcar" format #11584

mprahl commented Feb 4, 2025

mprahl commented Feb 4, 2025

HumairAK commented Feb 5, 2025

franciscojavierarceo commented Feb 5, 2025

[feature] Add support for model artifacts in the "modelcar" format #11584

[feature] Add support for model artifacts in the "modelcar" format #11584

Comments

mprahl commented Feb 4, 2025

Feature Area

What feature would you like to see?

What is the use case or pain point?

Is there a workaround currently?

mprahl commented Feb 4, 2025

HumairAK commented Feb 5, 2025

franciscojavierarceo commented Feb 5, 2025