-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
helm chart #3173
Comments
Issue-Label Bot is automatically applying the label Links: app homepage, dashboard and code for this bot. |
Got started on something, is this worth moving forward on? |
I would like to see kubeflow in helm. We use helm for IAC and kubeflow is a massive project with no helm support. |
@naseemkullah kubeflow is moving to kustomize for 0.6. Kustomize provides some support for helm through a helm plugin which would be a way to integrate new components. Your approach is interesting and goes the other way - moving everything to helm |
Thanks for the info @kkasravi |
Is there an update to this? I am also woking with @naseemkullah to create a helm chart for kubeflow. |
@dodizzle even if the kubeflow org doesn't support helm. A community helm chart will be greatly appreciated. |
kubeflow is a collection of components - eg the kustomization.yaml's under https://github.com/kubeflow/manifests are used to generate separate groups of resources. Each kustomize target is deployed separately. if you do a Currently kfctl doesn't provide an option of deploying either using a helm chart or kustomize. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
@dodizzle curious where did the helm chart end up? |
helm is better! |
Why is this closed? |
I think that the resistance to helm is a huge bummer, we use helm + kustomize + flux pretty heavily and would love to deploy kubeflow but the lack of a stable chart and many interleaved components caused us to abandon this project for a different path. Why not helm? Sure It might have some weird spots but it's an incredibly powerful and easy to use tool for automating complex deployments, I think this project would gain alot more traction if it embraced the tooling that the community was asking for. |
Just to restate what @jay-manday said: Helm has its quirks, kustomize has its quirks, <fill-in-tool-name> has its quirks, ..., nothing is perfect. Realistically though, Helm has achieved widespread adoption in the K8s application deployment space, and the packaging format they use for distribution makes it easy to use. Running something like
makes me feel like I'm forced to manually do things that Helm does for free with values, hooks, and CLI flags. I also have no interest in writing production-grade automation code using Bash and Secondly, in order to perform an upgrade, Helm has a well-defined procedure for rollbacks in the event of a failure; kustomize applies (overwrites) so hold onto your butts. 🤷♂️ I've used both Helm and kustomize deployment strategies in production and I will never repeat the latter given the pain I experienced there, but that's just me. I'd use Kubeflow in anger if a Helm chart was available, otherwise I feel like I need to search elsewhere. |
@jay-manday @sonnysideup I hope to release an unofficial Helm chart for Kubeflow soon. It will be more or less based on Kustomize files without the third-party components (Istio, cert-manager, an OIDC provider, ...) |
@sebastien-prudhomme did you already release your helm chart? |
@Legion2 just some components: see there https://github.com/cowboysysop/charts I'm stuck with some problems, one of them is that some components are still using old APIs and are not compatible with K8S 1.22. Fortunately it seems that things are changing since KServe which is remplacing KFServing now has an official Helm chart: https://github.com/kserve/kserve/tree/master/manifests/charts |
@sebastien-prudhomme the KServe Helm chart is not publised and will come with KServe 0.8 version |
There is here a helm chart for kubeflow-pipelines:
https://getindata.com/blog/kubeflow-pipelines-running-5-minutes/ |
@mtparet its not official and uptodate with latest KFP releases |
Indeed there is some lack for latest update but it is a good start for all people wishing an easy way to use kubeflow. |
Also, feel free to check this very simple helm chart for kubeflow pipelines. It is aimed at running smaller experiments locally. I mostly use it for my personal ml projects. https://github.com/Techn0logic/kubeflow-pipelines-standalone |
Please feel free to try and join https://github.com/alauda/kubeflow-chart |
This is was also identified as a priority from our survey outcome right? |
Just to give you some update - I started work on KF Helm Chart and I think it looks promising but I want to finish this PR first: kubeflow/manifests#2544. I was requested to do in-depth auth analysis for the kubeflow/manifests#2544 in respect to changes around istio and to understand how auth is currently working for kubeflow components. Also, I'll need to add some more tests in there... |
@kromanow94 how are you building the kubeflow helm chart? Are you building it from scratch or are you using something like flux or argcd to create helm-release resources? |
@clayrisser building from scratch. The idea is to create a strong base with meaningful and up to standard templates and template functions so they can be used across all the kf components. I'm focusing on single helm chart approach but I'm not using helm chart dependencies as I want to focus on decoupling various out-of-kubeflow components like istio and cert-manager that can be (and most often are) deployed and maintained by platform teams. I'm having in mind that there must exist a standard, example installation like the example kf with kustomize so instructions or scripts or value files for helm charts that are installing istio and other components will definitely be a part of the proposed solution. |
I think a helm chart that deploys helm charts might be an alternative approach. For example . . . @kromanow94 what do you think are the pros/cons to that approach? |
@clayrisser I think such approach might be an alternative. But, as far as I understand, it would depend on In my experience, the business should guide the technology in use. Having this in mind, I believe the approach "The fewer tools are needed to deploy Kubeflow, the better." gives more flexibility to the end user and helps in adoption. But, this is just my opinion and you're very much welcome to challenge that. Best way would be to create a solution based on your proposal ot at least an MVP and then we can compare and discuss. |
I would recommend using an umbrella chart to avoid redundancies and reduce maintenance efforts. For instance, Istio and Cert Manager etc are apps that already have existing Helm charts. These could be enabled through an 'if' statement using gotpl. In my scenario, a cert-manager and Istio are already installed. https://artifacthub.io/packages/helm/cert-manager/cert-manager
You can modify the default values of these charts in the root values.yaml file of the Kubeflow Helm chart. I suggest adding 'enable' variables for each component to toggle them on or off as needed. Thank you for your time! <3 |
@Syntax3rror404 I don't know if this is flexible enough because ideally values will need to be shared between the charts. |
I know, but you can access this vars from the values files directly, or if the subchart generates something, you can also access this vars. For example how I access in my chart a templated value from the bitnami mariadb chart in my main chart: And if its only the value you want, the subchart and your main chart shares the same values file which is located in your main chart. |
Thank you for the insights, it's always nice to hear some ideas, alternatives and opposite approaches. I'm definitely going with the Right now I'm not planning to change my approach. I'm going with a single helm chart but I do put a big stress on having the right conventions and in-chart self-explanatory organization. I think I'm able to bring the benefits of a single helm chart and an umbrella chart together. Let's see... Also, what I like in this approach is that the KF Helm Chart would focus only on the Kubeflow Components and then we can have an Example KF Platform Helm Chart defining only the dependencies. The dependencies could be kubeflow, istio, cert-manager, knative, whatever needed to deploy a ready-to-use platform. This would bring a great flexibility to the enterprise users who usually have all the infra components set and only need the installation tool that does one thing and does it good. Also, I've learned about scenarios where teams administrate multiple KF Installations so giving them the possibility to define a Helm Chart with their per-installation resources and use Kubeflow Helm Chart as a dependency would be great. In a few weeks I should have something to share so we can have a discussion about it on real example. |
@kromanow94 could you share the link? I might be interested in contributing. |
@clayrisser hey, that would be great! There is still some stuff I want to finish before showing an mvp, then I'm planning to share to gather feedback and open for contributions. I think I need two weeks or so. |
Hi, I'm testing the deployment with Kustomize and I think the Helm chart is highly necessary. Is there a time schedule for the first test release? |
Let's try to keep this thread about actual progress or proposals. If anyone has any updates they want to share please do so. PS: @Termibg22, if you're not already aware of deployKF, it will probably give you what you need. Even though it's not exactly the same as a helm chart, it still has values and supports every kind of config you can imagine. |
Tomorrow I should have some more updates. There was a discussion about this Helm Chart in the last KF Manifests WG Meeting and so far it was received positively. Current plan is to bring this subject in tomorrows KF Community Meeting. The idea to discuss it on the Community Meeting comes from the proposition of commitment to this feature from my company. I shared my work there and it was received positively with plan to invest further and share with the community. This is also a reason why I want to postpone sharing current code to align and have a clear perspective. FYI, current idea is to bring this as an alpha feature and it will definitely not replace current kustomize setup, rather bring more options to the community. |
Just chiming in as the three of us above were discussing in the community meeting.
|
Hello everyone, I'm excited to share a preliminary Helm Chart for Kubeflow I've been working on with the community. While it's not a complete solution and some components are still under development, it provides a solid foundation for testing the most widely used features and the Helm Chart itself. I'm looking for feedback and would like to spark a discussion to ensure this approach aligns with the community's needs. To facilitate this, I've created a Helm Chart Repository hosted on GitHub Pages. You can find it here, along with quickstart instructions for easy setup. To see the repository, please visit:
AssumptionsThe primary focus of this Helm Chart is to streamline the installation of Kubeflow components. It's designed as a singular Helm Chart, marrying the benefits of individual charts for each component with the convenience of a unified installation process. Quickstart InstallationHelmThis script will create 'kubeflow' namespace configured with istio injection and install helm releases for each kubeflow dependency and kubeflow itself in the correct order and will wait until the dependencies are ready.
ArgoCD AppsThis script installs ArgoCD if not installed, deploys dependencies and Kubeflow itself through ArgoCD Apps. It deploys the Apps in the correct order and will wait until the dependencies are ready within the script.
ArgoCD App of AppsThis script installs ArgoCD, configures it for health assessment required for using sync-waves for ArgoCD Apps, and deploys an App of Apps manifest for Kubeflow and its dependencies. It uses sync-waves to cover for correct installation order of the dependencies and Kubeflow Components.
If you already have ArgoCD installed and configured with health assessment for ArgoCD Apps, it's enough to execute this command:
To see more about the configuration for ArgoCD App health assessment, please visit: Helm + EKSThis script is for reference only and will not work out of the box. Also, it assumes the script is run from a locally cloned git repository. 'kubeflow' and 'oauth2-proxy' Helm Charts needs to be parameterized for AWS IAM OIDC Issuer to enable access through M2M Service Account Tokens. For required parameterization, please see the following files and replace
Assuming you've cloned this repository and are using it locally, you can parameterize the AWS IAM OIDC Issuer URL with: OIDC_ISSUER_URL=https://oidc.eks..amazonaws.com/id/1234567890
grep -Rl 'https://oidc.eks..amazonaws.com/id/1234567890' --exclude 'quickstart*' | xargs sed "s;https://oidc.eks..amazonaws.com/id/1234567890;\$OIDC_ISSUER_URL;g" -i Verifying the InstallationAfter completing the installation, use the following commands to verify your setup and perform initial configuration:
What's included in the Kubeflow Helm Chart
What's not included
Future of this Helm ChartThis Helm Chart will be developed and supported by myself and the company I work with. We rely on Kubeflow in multiple teams across multiple projects and our business solutions are already greatly leveraged by using Kubeflow. We started an internal initiative focused on sharing expertise and improving the ways of installing Kubeflow in enterprise environment. Kubeflow as a Platform is important to us. I started work on this Helm Chart myself and now seeing how positively it was received in my company and with some Kubeflow community members, I expect it to be continued. My wish is to have this Helm Chart as part of the official Kubeflow project and some discussion across the community points to that it's quite possible. |
@kromanow94 this is looking really good. My project is more focused on bootstrapping and would benefit from having your helm interface to depend on. For my and others' benefit -- when do you recommend others start using this in production? I.e. Do you expect the repo URL/versioning/development to change radically in the near future? +1 on this being part of Kubeflow and in an ideal world not having a hard Istio dependency. |
Hey, sorry for not responding earlier. First I was traveling, then I was sick, then I was traveling again... 😄 I want to give an update on the Helm Chart :).
If all goes well, I expect it to be production ready within a month. Then we have to align with the community if it's going to be merged into
The URL might change if this work will be merged to To also cover for some common doubts I hear from various talks:
|
Hey all, a short update before I start vacationing. We have a new release of the Kubeflow Helm Chart! This version is released as
The example deployment with this helm chart is defined in example/helm with all ArgoCD Apps, Quickstart and Destroy scripts and Values Files for every Helm Release and ArgoCD Apps (they use the same values.yaml files). Quickstart scripts:
|
Hey @kromanow94 this looks really well, good job! I'm a big fan of the chart not including external dependencies likes Istio/cert-manager/Dex/MinIO/etc. and just assume that they are set up by the user! Just a note: CustomResourceDefinitions placed in the Also, won't the test container here always fail? As far as I understand, it will be rejected by Istio since it doesn't include a valid JWT |
/close this belongs to kubeflow/manifests not kubeflow/kubeflow. Please recreate there. |
@juliusvonkohout: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Please check kubeflow/manifests#2730 |
Are there any plans for a helm chart?
Preferably one that would leverage the already existing argo, ambassador, minio and any other component's community charts as dependencies of the kubeflow chart?
Would be great if we could deploy and manage kubeflow with the package manager we all love and know!
My 2 cents...
The text was updated successfully, but these errors were encountered: