This repository has been archived by the owner on May 21, 2024. It is now read-only.

Support python function pipelines #211

Open

lukemarsden opened this issue Aug 11, 2020 · 1 comment

lukemarsden commented Aug 11, 2020 •

edited

Loading

Enable a data scientist to define pachyderm pipeline steps as python functions (e.g. from a jupyter notebook).

Contributor

msteffen commented Jun 16, 2021

To add a bit of color:

I think a python library could use getsource() to get the body of a python function/object, and then commit that to Pachyderm and run it in a standard python container. I believe kubeflow does this for function-based components.
Deps could be downloaded and committed to pachyderm as well)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.