Is there any public Kubeflow Pipeline Registry ?

[haiminh2001]

Hi, I am quite new with Kubeflow Pipeline. I found the concept of pipeline registry, I think such a thing will benefit a lot. But I can not find any public registry that I can download pipelines from other than VertexAI's one. I expect common tasks like downloading data from a S3 storage should be available like BigQuery's one.
My guess about the absence of the registry is that perhaps the community is not enough or the kubeflow pipeline itself is not mature enough?

[milosjava]

As far as I know there is not currently. In my company we have our own set of reusable components that internal users can use. Practically its a library on top of kfp that enables reusability and other stuff as well. Talked with people in other companies and they all seems to have similar tools implemented including the reusable components.
But I like your idea and I believe it could be useful to have something like that.

So lets see how this looks like in the S3 example.

from kfp import dsl, kubernetes
from kfp.dsl import Output, Dataset


@dsl.component(
    base_image="python:3.9",
    packages_to_install=["s3fs==2024.6.0","pandas==2.2.2"])
def from_s3_comp(bucket: str, key: str, output: Output[Dataset]):
    """Fetches data from an S3 bucket and writes it to an output file.

    Args:
        bucket (str): The name of the S3 bucket.
        key (str): The key of the object in the S3 bucket to be fetched.
        output (Output[Dataset]): The output file where the fetched data will be written.
    """
    # pylint: disable=invalid-name
    from s3fs import S3FileSystem

    s3 = S3FileSystem(anon=False)

    with s3.open(f"{bucket}/{key}", mode="rb") as f:
        content = f.read()

    with open(output.path, "wb") as f:
        f.write(content)


def from_s3(
    bucket: str,
    key: str,
):
    comp = from_s3_comp(bucket=bucket, key=key)

    kubernetes.use_secret_as_env(comp,
                                 secret_name='awscreds',
                                 secret_key_to_env={'AWS_ACCESS_KEY_ID': 'AWS_ACCESS_KEY_ID',
                                                    'AWS_SECRET_ACCESS_KEY': 'AWS_SECRET_ACCESS_KEY',
                                                    'AWS_REGION': 'AWS_REGION'})

    return comp

from_s3_comp will create a component with the basic code functionality (maybe that could be part of component repository), while from_s3 in this case is mapping secrets from awscreds to env variables so that from_s3_comp can authenticate. Now imagine that various users would maybe like to add pod labels , pod annotations retry strategy, have different secrets files etc...
If we could make **from_s3 generic enough so everybody can use it then registry could work.
Please let me know what do you think about it.

[haiminh2001]

@milosjava oh yes that is exactly what I meant. A shared library of reusable components is not only reducing tedious tasks, but also helping people who have little knowledge of K8S get used to Kubeflow easier.
Google Cloud has a python package that provides connectors to their data sources such as BigQuery, Dataflow, training templates, etc... I think that package is very convenient, but only works for Google Cloud's services. We can create a version of it but works with popular open sources like S3 storages. In addition, Google Cloud's approach is also similar to your company, a library on top of the KFP, so I think that is the right approach.
Making generic for all use cases is indeed very great, but I think practically impossible, the components may only work 90% the times and it is still good enough.

[milosjava]

@haiminh2001 yes, creating a library will be a challenge due to the all use cases and some specific settings that every organisation has. But we can start with the repo with various components (as one I have shared) at least as examples that could be quickly adjusted for specific cases in other organisations. I hope that could help process of kubeflow adoption in various comapnies. I have some repo where I have planned to put various useful commands and kubeflow related code for future reference so I guess I can use it for now.
In the mean time lets see if anybody else will share some additional info on this topic, maybe there are some other resources that we are currently not aware about.

@rimolive do you have maybe info regarding some repo or location with kubeflow components that can be reused ?

[gregsheremeta]

I'm not very familiar with it, but isn't that the intention of https://github.com/kubeflow/pipelines/tree/master/components ?

[milosjava]

@gregsheremeta yes, that could be a good place. currently it seems pretty non-updated and not very useful so I guess I can create a PR and see what happens.

[haiminh2001]

@gregsheremeta @milosjava I think that the components from kubeflow pipelines github is not very active though, they only serve as examples now, and contributing to a github repository is not so simple. A public repository should be easy to contribute. Therefore, I think perhaps a python SDK library is not the way to go after all. The idea in my head when I opened this discussion is something like a helm repository.
In addition, since kubeflow pipeline v2 has introduced an Intermediate Representation (IR), may be storing the shared components in this manner is more generic and easier to use than a python SDK. Helm Charts are also yaml files and a bunch of values to customize the resources to be deployed, just like kubeflow pipeline IR, I think it should work the same.

[milosjava]

@haiminh2001 if I understood correctly you think that storing components in the IR format would be better then providing python sdk examples? How would you incorporate that into your own pipeline ?

[haiminh2001]

@milosjava You can do something like this:

loaded_comp = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/2.0.0/sdk/python/test_data/components/add_numbers.yaml')

I think a python package is not suitable because it is not possible for many people to contribute at the same time and can potentially cause incompatibility problem with python and kfp sdk version.