support boot addons for pluggable install of other systems

[jstrachan]

we are using a new design process for this issue:
jenkins-x/enhancements#1 details here: https://github.com/jstrachan/enhancements/blob/1/proposals/2/README.md

the user guide for the PoC is here: https://github.com/jstrachan/enhancements/blob/1/proposals/2/docs/README.md
with the boot git config repo here: https://github.com/jenkins-x-labs/boot-helmfile-poc

we currently manually configure in the boot pipeline the install of multiple systems usually in separate namespaces:

• velero (if enabled)
• nginx
• externaldns
• cert-manager
• acme certs chart

ideally those should be modular and optional. e.g it should be really trivial to disable nginx if folks are using, say, istio - without having to hack a pipeline yaml.

we'd also like to make it easy to add a number of other systems in order in separate namespaces independently of the Staging / Production environment namespaces:

• knative enable knative as part of jx boot #6331
• gloo
• istio
• linkerd enable linkerd as an option on jx boot #6330
• flagger

so it'd be nice to have a simple addon model where anyone can add any systems/charts to any namespace at any point in the flow before we setup the dev/staging/production environment in the traditional way.

So some kind of yaml file with an ordered list of releases with a list of namespaces and charts would be great that would be easy to PR against like we've done with the jx add app commands historically, so managed via GitOps etc.

use helmfile?

One possible solution might be to add a single step the boot pipeline to invoke helmfile: https://github.com/roboll/helmfile#configuration

e.g. if there was an addons/helmfile.yaml file in the file system then run helmfile as a step on the file.

It looks like helmfile already supports tillerless + helm3 and lets us define namespaces and orders etc.

We may need to build our own step to wrap helmfile so that we can do some of the things we do with jx step helm apply right now

• should we support access to secrets in vault / local file system via URLs or Parameters injection?
• do we need to support exposing the properties from the jx-requirements.yml as values.yaml files that can be reused inside the helmfile templates?
• do we need to improve access to certs / domain after we've set those up in boot?
• allow the use of version streams to manage versions of things if no version is specified in the helmfile.yaml - like we do with helm charts requirements.yaml file right now (in jx step helm apply)

So if we do need anything other than vanilla helmfile we could maybe do a similar thing to jx step helm apply where we copy the directory where helmfile lives, then generate any secret yamls + jx-requirements.yaml files and then run helmfile in a temporary directory (to avoid accidentally committing any secrets to git). Hopefully we don't need to fork helmfile; but we could just wrap it slightly with a pre-processor?

We may need to introduce phases of helmfile execution to make things simpler to reason about. e.g. we may need to come up with a list of phases and the order in which they run. pre-secrets, pre-domain, post-tls and so forth so that adding a boot addon we can reason about where best in the pipeline to add to the right helmfile.

e.g. the file system could be something like...

addons/
   preDomain/
       helmfile.yaml
   preSecrets/
       helmfile.yaml
   postTLS
       helmfile.yaml
   postSecrets/
       helmfile.yaml

Or whatever. Any empty dirs or missing helmfiles would be ignored.

Then we could have a list of apps which we group as to where they get added based on what they are & whether they need integration with secrets / TLS / domain / certs etc?

We can then have commands like the old jx add app / jx delete app for boot addons which grok where the app needs to go; if its a boot addon it may go in the helmfile areas if its in a specific phase or in the traditional env/ environment if not?

Maybe over time we move more towards this kind of helmfile-like approach for all environments?

I wonder if as a spike we could

• try out helmfile for installing things like gloo / knative / linkerd / istio?
• if that works and we are happy with it, we could look at adding the phases approach for easier app composition
• if thats all looking good try migrate existing things like nginx / cert manager / external dns to the helmfile approach?

[vfarcic]

Even though I'd love to see Helmfile included, I think that we should stop adding more tools for a while. The complexity is already too high and we are having difficulty maintaining even what we have (e.g., we still don't support Helm 3 CLI). For now, I'd limit the solution to "pure" Helm, and work on it only after Helm 3 support is finished. Ultimately, the solution can be as simple as storing the chart in the dev repo. jx add app --repository is working OK.

Major things missing are:

1. The ability to specify a namespace
2. Lack of documentation that would explain where to change the values
3. The ability to add an app without pushing it to Git so that we have the chance to modify the values.
4. The ability to have more than one app based on the same repository.

[vbehar]

another possible organization is to use helmfile's labels:

• each release can have labels (key/value pairs)
• helmfile accept a label selector to filter releases to work on

so instead of relying on multiple directories pre-secrets, pre-domain, post-tls, we could just run helmfile with a label selector of pre-secrets=true and it will apply to all releases which have:

labels:
  pre-secrets: true

[jstrachan]

@vbehar ah thats a great way to figure out ordering thanks!

[jstrachan]

@vfarcic I understand the reluctance to add another tool.

The thing is we'd have to basically write something almost exactly like helmfile itself as there's no way to install separate helm charts easily in separate namespaces in a well defined order with a concept of phases and a nice modular way without hacking the pipeline yaml - which isn't terribly modular. e.g. which line in the pipeline yaml does the jx add addon pick to add the knative addon?

If we want to have a simple CLI to add/remove boot addons, we'll need a simple canonical file format in git to easily add/remove boot addons. So we'd want a simple helmfile-like YAML file where we can list the separate addons + which namespace they are in and what order they are applied in etc.

We could make up a brand new YAML file syntax ourselves and essentially write our own helmfile; or we could just look at reusing helmfile. We should definitely look at both - but to handle boot addons nicely we need a new YAML file really; the current boot process isn't ideal.

BTW really helmfile is just a YAML file for listing helm chart releases in order in namespaces - its also mostly implementation detail inside boot itself. It'd also be completely optional if folks don't wanna include istio/knative/gloo/flagger as part of the boot process.

[jstrachan]

@vfarcic btw helmfile already supports helm3 and you can run helmfile locally on the CLI against local git clone of the dev environment. But this thread is really about what does the dev Git repo source code look like; there's currently no way to have a simple modular representation on disk of modular apps installed by boot into different namespaces in order and respecting boots idea of phases (pre-secrets / pre-TLS etc) - so far helmfile looks a pretty good solution to that - if folks want to use boot addons - but it would be a NOOP if folks don't want to go there

[jstrachan]

@vfarcic btw am not 100% saying we must use helmfile; more we're in discovery right now trying to find the right way forward and it's a good experiment to see how it works + if it suits our needs.

We can easily try a prototype boot config repo where we add helmfile and add some helmfile steps and see how it works and what we think of the general git source code & if it helps us and users have modular boot addons without committing to any significant changes in the jx repos.

Maybe we find when we look at migrating the current charts to boot addons (nginx / cert manager / external-dns in particular) we may find its got some limitations and building a simple similar tool might be easier. Or we may find helmfile gets us where we need to go faster.

One area I'm not 100% sure on is how the pipeline and the helmfile-like thing interact.

e.g. currently in boot we do something like this:

• install a bunch of charts in different namespaces with a defined order
• then we modify the requirements based on ingress: jx step create install values` (to handle things like detecting the domain/ingress)
• install more charts (e.g. external dns/cert manager/vault)
• populate parameters/secrets (jx step create values --name parameters)
• no the traditional install and maybe other charts

so we already have 3 natural places to invoke a helmfile-like thing to install charts in different phases. So maybe thats 3 places where we invoke helmfile with a selector as @vbehar mentions if thats something helmfile can do? or we use 3 separate helmfiles maybe?

e.g. the pipeline with boot apps could be something like this...

jx step helmfile --dir addons/preIngress/helmfile.yaml 

# populate ingress stuff (e.g. default domain from nip.io if not using custom domain)
jx step create install values

# add any charts that need a domain injected
jx step helmfile --dir addons/postIngress/helmfile.yaml 

# populate any missing charts in vault
jx step create values --name parameters

# lets include generated secrets after they have been populated in vault
jx step helmfile --dir addons/postSecrets/helmfile.yaml --secrets

[rawlingsj]

This issue is also part of a design we are putting together, details here https://docs.google.com/document/d/1DL1oEeAZtWFfpJOWk5UVO54D88BnlqZDf5IDaOkGxCk/edit#

[polothy]

FWIW, we are currently playing around with helmfile to deploy our cluster level tooling. The nice thing about it is being able to install charts into various namespaces. It has loads of other nice features as well.

We are currently moving it into a pipeline so JX runs it, keeping with the gitops workflow.

EDIT: ah, another nice thing with helmfile (as mentioned above) is control over the order in which charts are installed and you can even do things like add custom waiting logic (EG: waiting for CRDs to be active, pods to be ready, etc).

[jstrachan]

@polothy thanks for the feedback - glad to hear it.

We've been kinda getting those benefits up to now by using steps in the pipeline; but doing it a more metadata/declarative way will make it easier to make tooling to let folks add/remove addons in a simple UI / CLI way and compose addons across all environments/teams without folks having to actually hack pipelines by hand (which can be a little brittle)

[polothy]

Yes, pre-helmfile, we had a very lengthy pipeline to do multiple helm applies across namespaces. Things were done by separate team members and it ended up being quite messy and inconsistent. Refactoring to helmfile really helped with that.

[jstrachan]

am gonna mark this as fixed as we now have: https://jenkins-x.io/docs/labs/boot/