[wip] multi cluster support #6664

jstrachan · 2020-01-31T11:04:14Z

note this PR depends on the helmfile + apps support: #6660 - once that merges we can rebase this PR (which will then be quite small ;)

for documentation see https://github.com/jenkins-x/enhancements/tree/master/proposals/2/docs

jenkins-x-bot · 2020-01-31T11:04:30Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: jstrachan

If they are not already assigned, you can assign the PR to them by writing /assign @jstrachan in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jstrachan · 2020-01-31T11:04:56Z

/test images

jstrachan · 2020-01-31T11:05:13Z

/build images

houndci-bot · 2020-01-31T16:25:29Z

pkg/cmd/step/verify/step_verify_preinstall.go

+		// lets make sure we have the secrets defined as an env var
+		secretsYaml := os.Getenv("JX_SECRETS_YAML")
+		if secretsYaml == "" {
+			return fmt.Errorf("no $JX_SECRETS_YAML environment variable defined.\nPlease point this at your 'secrets.yaml' file.\nSee https://github.com/jenkins-x/enhancements/blob/master/proposals/2/docs/getting-started.md#setting-up-your-secrets\n")


error strings should not be capitalized or end with punctuation or a newline

houndci-bot · 2020-01-31T16:25:29Z

pkg/cmd/step/verify/step_verify_preinstall.go

+		// lets make sure we have the secrets defined as an env var
+		secretsYaml := os.Getenv("JX_SECRETS_YAML")
+		if secretsYaml == "" {
+			return fmt.Errorf("no $JX_SECRETS_YAML environment variable defined.\nPlease point this at your 'secrets.yaml' file.\nSee https://github.com/jenkins-x/enhancements/blob/master/proposals/2/docs/getting-started.md#setting-up-your-secrets\n")


error strings should not be capitalized or end with punctuation or a newline

polothy · 2020-01-31T22:09:13Z

So, I was reading https://github.com/jenkins-x/enhancements/blob/master/proposals/2/docs/multi-cluster.md and I had a couple of questions:

What are in Staging and Production remote environments? Are those full jx boot repos? Or closer to one of these standard environments, https://github.com/jenkins-x/default-environment-charts
Is it at all possible to have PRs built in the remote environments/clusters? We currently use a pattern to deploy our infrastructure with our applications and building the PR in the remote cluster would allow us to preview infrastructure changes (highly requested from our developers).

Thanks so much for improving multi-cluster support. The proposal looks like a very good improvement over the environment controller! 👍

jstrachan · 2020-01-31T22:24:02Z

yeah, remote Staging & Production are git repos using boot & helmfile, which are cut down to just these apps https://github.com/jenkins-x/default-environment-helmfile/blob/master/jx-apps.yml
Yeah we can do either really. I liked the idea of keeping apps that build images in the Dev cluster; but we can run pipelines in any cluster. It’s currently defined by the Scheduler referenced by the SourceRepository in each cluster

polothy · 2020-01-31T22:25:23Z

Hey @jstrachan - sorry, I cannot figure out how to send feedback on other parts of your proposal, so I hope you don't mind I jot them down here. Questions are as follows:

Do you plan to support hemfile for regular JX environments so we can get Helm 3 support everywhere? EG: a new or updated one of these: https://github.com/jenkins-x/default-environment-charts
From what I understand, the jx-apps.yaml generates the two helmfile (one for apps and one for system)? Not sure I understand the need for the abstraction and generation of the files. EG: does this prevent us from using certain helmfile features? What's gained by not having the two helmfiles and modifying those when you need new apps or system installs?

Thanks! This is all very exciting. Cannot believe you are already taking this way beyond POC, you are a machine! 🚀 💯

polothy · 2020-01-31T22:28:06Z

Yeah we can do either really. I liked the idea of keeping apps that build images in the Dev cluster; but we can run pipelines in any cluster. It’s currently defined by the Scheduler referenced by the SourceRepository in each cluster

Right there with you on building images in dev only. This is really only PRs for the environment, where you bump the version of AppFoo to 1.2.3.

jstrachan · 2020-01-31T22:34:56Z

@polothy thanks! Any feedback anywhere is great ;)

absolutely - would love all environments (local or remote) to use helmfile & helm 3
good question. We could make the jx-apps.yml file be a real helmfile too; though am liking the concise yaml (if you compare jx-apps.yml to the generated helmfile.yaml files you’ll see what I mean ;).

It might be simpler to let the jx-apps.yml to allow an optional list of helmfiles to be applied after the system & apps?

Incidentally the system charts are for dealing with ingress/DNS/certs/TLS. Most apps (the apps folder) need to know what the domain is & the TLS certs etc; so the system charts sort that out - then jx step verify ingress runs to check that is all good, then the apps are installed

jstrachan · 2020-01-31T22:42:43Z

@polothy btw I’d love this...
#6672

actually to implement the diff, we’d want to run the PR pipeline in Staging / Production

rawlingsj · 2020-01-31T22:43:55Z

It might be simpler to let the jx-apps.yml to allow an optional list of helmfiles to be applied after the system & apps?

+1
There are lots of great helmfile features, which as folks get used to it I'm sure they'd want to use. jx-apps.yaml is a really great way to add apps to boot but it is easy for folks just to add their own helmfiles if they wanted to. Under the covers the jx boot pipeline just runs vanilla helmfile sync

jstrachan · 2020-01-31T22:53:44Z

@rawlingsj maybe we could add a final pipeline step of helmfile sync in a helmfiles/ folder with an empty helmfile.yaml file - then folks have a place to add any extra vanilla helmfile stuff without modifying the pipeline?

Mind you it’s trivial to do that yourself via jx create step ;)

rawlingsj · 2020-01-31T22:57:40Z

actually I think having a helmfiles/ dir is a really nice idea, then folks can just drop their own helmfiles in

polothy · 2020-02-01T03:57:47Z

We currently install many cluster wide tools using a helmfile. Just trying to figure out how these files all work and how we can add our own very custom charts and values. I think some we would actually want to add to the system helmfile (or maybe even before system 🤔 ) because they are security related.

Totally right that we can just modify the pipeline and do a helmfile sync wherever we need. I had not checked out the pipeline in the boot POC repo yet. That's also perfectly fine for us.

Starting to get the impression that the jx-apps.yaml is probably more for distributing the JX apps/charts/install and maybe less about our random charts that we just want to add to the cluster. That's totally fine and makes sense for distributing JX and all the things.

---Switching topics here---

I did notice this helmfile and it seems like it isn't being used. It looks like from the pipeline that the system and app helmfiles are generated, then the system helmfile is run, a JX command is run and then the apps helmfile is run.

I was wondering if you are interested in the helmfile hooks? The thought being, you could use the hook to run the JX command like so:

- name: NAME
  namespace: NAMESPACE
  chart: foo/bar
  version: 0.1.0
  atomic: true
  wait: true
  labels:
    initPhase: "true"
  needs:
  - some_other/chart
  hooks:
  - events: ["postsync"]
    showlogs: true
    command: jx
    args:
    - step
    - create
    - install
    - values
    # And so on...

I'm not even close to knowing if that would work for you but, this would keep all the steps in the helmfile and maybe we could use that root helmfile to control the whole sync, like so:

helmDefaults:
  # values, values, values...

helmfiles:
  - helmfiles/my_crazy_init.yaml
  - system/helmfile.yaml
  - apps/helmfile.yaml
  - helmfiles/cluster_tools.yaml
  - helmfiles/rule_the_world.yaml

That way JX can own/manage the lifecycle of the system and app helmfiles and we can shove in our stuff where we need to rather easily. Pipeline is just a single hemlfile sync.

Just bouncing ideas around, like I said, this is hot stuff, I like what you have already and where you are taking it 🍻

polothy · 2020-02-01T03:59:21Z

@polothy btw I’d love this...
#6672

actually to implement the diff, we’d want to run the PR pipeline in Staging / Production

oooOOooOOoo yesssss

Seems like it wouldn't be hard to implement - I just don't anything running yet. Soon though, soon I get to play with the awesome toys.

houndci-bot · 2020-02-01T04:29:10Z

pkg/cmd/step/verify/step_verify_preinstall.go

+	// lets make sure we have the secrets defined as an env var
+	secretsYaml := os.Getenv("JX_SECRETS_YAML")
+	if secretsYaml == "" {
+		return fmt.Errorf("no $JX_SECRETS_YAML environment variable defined.\nPlease point this at your 'secrets.yaml' file.\nSee https://github.com/jenkins-x/enhancements/blob/master/proposals/2/docs/getting-started.md#setting-up-your-secrets\n")


error strings should not be capitalized or end with punctuation or a newline

houndci-bot · 2020-02-01T04:29:10Z

pkg/cmd/step/verify/step_verify_preinstall.go

+	// lets make sure we have the secrets defined as an env var
+	secretsYaml := os.Getenv("JX_SECRETS_YAML")
+	if secretsYaml == "" {
+		return fmt.Errorf("no $JX_SECRETS_YAML environment variable defined.\nPlease point this at your 'secrets.yaml' file.\nSee https://github.com/jenkins-x/enhancements/blob/master/proposals/2/docs/getting-started.md#setting-up-your-secrets\n")


error strings should not be capitalized or end with punctuation or a newline

houndci-bot · 2020-02-01T04:29:10Z

pkg/cmd/step/verify/step_verify_preinstall.go

+	// lets make sure we have the secrets defined as an env var
+	secretsYaml := os.Getenv("JX_SECRETS_YAML")
+	if secretsYaml == "" {
+		return fmt.Errorf("no $JX_SECRETS_YAML environment variable defined.\nPlease point this at your 'secrets.yaml' file.\nSee https://github.com/jenkins-x/enhancements/blob/master/proposals/2/docs/getting-started.md#setting-up-your-secrets\n")


error strings should not be capitalized or end with punctuation or a newline

jstrachan · 2020-02-01T04:29:34Z

/build images

jstrachan · 2020-02-01T05:11:15Z

@polothy its totally fine for folks to do vanilla helmfile stuff; but hopefully we can convince folks to use Jenkins X Apps for lots of common things like nginx/gloo/istio/flagger/prometheus and the like. Mostly as lots of these tools need to be configured with secrets + domain/TLS/certs stuff; so we can figure out that binding between an off-the-shelf helm chart and the jx-requirements.yml which defines common ingress/storage/network stuff so that folks don't have to keep binding charts to their environments. configuring charts is undifferentiated heavy lifting requiring folks to grok all the internals of each charts values.yaml - we can certainly help include default bindings for common charts in the version stream so we can share that stuff. I'm hoping Jenkins X can keep improving the tooling around chart configuration too; so its easier to add popular OSS charts.

In terms of the top level helmfile.yaml thats mostly so you can easily uninstall everything in one simple command: https://github.com/jenkins-x/enhancements/blob/master/proposals/2/docs/getting-started.md#uninstall

incidentally this git repo is a cleaner example of what boot config looks like with helmfile: https://github.com/jenkins-x/jenkins-x-boot-helmfile-config (its the git repo used if you run jx boot --helmfile)

Thanks for the heads up on helm hooks, they look great. We could take advantage of those for sure; though I'm also liking the flexibility we get with pipelines. e.g. by default the pipeline steps are defined in the build pack: https://github.com/jenkins-x-buildpacks/jenkins-x-kubernetes/blob/master/packs/environment-helmfile/pipeline.yaml#L18 so that the default jenkins-x.yml file is pretty much empty until you add a custom step via jx create step which greatly simplifies maintenance (if we need to modify one of the pipeline steps for everyone's boot install). Also we're pretty used to having different pipeline steps for PRs versus release builds which the Jenkins X pipelines sort out nicely for us - along with having a nice composition, inheritence + override model so its easy for folks to disable steps or add custom pre/post steps or replace steps in a simple way.

until we get this PR merged jenkins-x/jx#6664

jstrachan · 2020-02-01T06:23:52Z

/build images

jstrachan · 2020-02-01T06:40:17Z

/build images

as we want to create and manage all k8s resources via helm instead Signed-off-by: James Strachan <[email protected]>

also add validation of the enviroment variable `JX_SECRETS_YAML` to ensure its defined to point to the secrets yaml file Signed-off-by: James Strachan <[email protected]>

so that we handle the public chart museum URL and include it into the `jx-apps.yml` file in the pull request Signed-off-by: James Strachan <[email protected]>

so its easier for folks to populate it. Longer term we can help the user populate it like we do with the current boot approach

rename the `jx-apps.yml` apps.valueFiles to apps.values to match helmfile's naming convention also allow discovery of a `myrepo/mychart`'s yaml file via `mychart/values.yaml` as using `myrepo/mychart/values.yaml` seems to confuse helm into thinking the `myrepo/mychart` folder is a local chart source code

if we have no releases for a helmfile lets add a dummy empty chart to avoid `helmfile sync` failing in the system or apps dir refactored the test cases so they are easier to create

lets lazily create the `jx-requirements.values.yaml.gotmpl` file if it does not exist. This lets us use a simpler PR pipeline on local environments without needing to run the whole `jx step verify preinstall` step. also lets default the `--values` arguments with nice validation if folks don't specify them so we can simplify the OOTB pipelines

which it usually does on helm 2 containers

…pressions e.g. so we can enable helm 3 in a build pack via `PATH=/opt/bin/helm3-bin:$PATH`

if we are using helmfile for dev

…ues running steps and capturing system out

…amespace#3

if we have configured helm 3 then lets use that version to validate helm Signed-off-by: James Strachan <[email protected]>

so that we can reuse the same code in jx alpha / jx 3.x where we default to helm 3 Signed-off-by: James Strachan <[email protected]>

Signed-off-by: James Strachan <[email protected]>

jenkins-x-bot · 2020-03-01T13:40:43Z

@jstrachan: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
bdd	`7b3b393`	link	`/test bdd`
unit	`7b3b393`	link	`/test unit`
integration	`7b3b393`	link	`/test integration`
boot-vault	`7b3b393`	link	`/test boot-vault`
tekton	`7b3b393`	link	`/test tekton`
lint	`7b3b393`	link	`/test lint`

View all Builds for this Pull Request

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

jstrachan · 2020-03-02T15:06:31Z

I'm gonna close this PR for now - we're using this approach for folks to test out multi cluster: https://jenkins-x.io/docs/labs/boot/

jenkins-x-bot added the do-not-merge/work-in-progress label Jan 31, 2020

jenkins-x-bot requested review from rawlingsj and wbrefvem January 31, 2020 11:04

jenkins-x-bot added the size/XXL label Jan 31, 2020

jstrachan force-pushed the multicluster branch from bf1e2d8 to 6946ad5 Compare January 31, 2020 14:08

houndci-bot reviewed Jan 31, 2020

View reviewed changes

jstrachan force-pushed the multicluster branch from d1ccee5 to da1ea5a Compare February 1, 2020 04:29

houndci-bot reviewed Feb 1, 2020

View reviewed changes

jstrachan added a commit to jenkins-x-buildpacks/jenkins-x-kubernetes that referenced this pull request Feb 1, 2020

fix: use a temporary custom image for helmfile pipelines

42016aa

until we get this PR merged jenkins-x/jx#6664

jstrachan and others added 25 commits February 27, 2020 06:23

fix: avoid creating the dev Environment when using helmfile

93db455

as we want to create and manage all k8s resources via helm instead Signed-off-by: James Strachan <[email protected]>

fix: default the namespace from the jx-requirements.yml file

5747dcc

also add validation of the enviroment variable `JX_SECRETS_YAML` to ensure its defined to point to the secrets yaml file Signed-off-by: James Strachan <[email protected]>

fix: promotion for helmfile and multi-cluster

be4c1cd

so that we handle the public chart museum URL and include it into the `jx-apps.yml` file in the pull request Signed-off-by: James Strachan <[email protected]>

fix: lets generate a default secrets YAML file for helmfile

baa6e51

so its easier for folks to populate it. Longer term we can help the user populate it like we do with the current boot approach

fix: helmfile promote to match charts with prefix

78d44f8

fix: avoid creating invalid helmfile

1419770

if we have no releases for a helmfile lets add a dummy empty chart to avoid `helmfile sync` failing in the system or apps dir refactored the test cases so they are easier to create

fix: lets ensure the requirements yaml is created

b43b3b9

fix: handle helm 2 based builder images

a9b1e2f

fix: lets not fail if helm version errors

ed97e29

which it usually does on helm 2 containers

fix: trim the prefix from the version

ea318b5

fix: lets use the helm3 binary inside classic builder images

4f7d509

fix: add the jx repository if its not there for empty helmfiles

c3aec4f

fix: default to requirements environment first

640d141

fix: allow environment variables to reference environment variable ex…

c51a083

…pressions e.g. so we can enable helm 3 in a build pack via `PATH=/opt/bin/helm3-bin:$PATH`

fix: default the cluster registry on GKE

12339c3

fix: lets default to helmfile for staging and production

49dc84b

if we are using helmfile for dev

chore: avoid warn message with custom built jx binaries as causes iss…

27bd17b

…ues running steps and capturing system out

fix: secrets.yaml files being added to generated helmfile

1877cea

chore: use patched namespace gitrepo while waiting for zloeber/helm-n…

734215b

…amespace#3

chore: rebase

a605584

fix: better validation of helm

8ef772f

if we have configured helm 3 then lets use that version to validate helm Signed-off-by: James Strachan <[email protected]>

fix: lets add a conditional compilation flag

bafcb8f

so that we can reuse the same code in jx alpha / jx 3.x where we default to helm 3 Signed-off-by: James Strachan <[email protected]>

fix: lets default to helm 3 in this branch

2e16696

Signed-off-by: James Strachan <[email protected]>

jstrachan force-pushed the multicluster branch from b692131 to 2e16696 Compare February 27, 2020 06:39

fix: allow the secret yaml env var validation to be disabled

7b3b393

Signed-off-by: James Strachan <[email protected]>

jstrachan closed this Mar 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[wip] multi cluster support #6664

[wip] multi cluster support #6664

jstrachan commented Jan 31, 2020

jenkins-x-bot commented Jan 31, 2020

jstrachan commented Jan 31, 2020

jstrachan commented Jan 31, 2020

houndci-bot Jan 31, 2020

houndci-bot Jan 31, 2020

polothy commented Jan 31, 2020

jstrachan commented Jan 31, 2020 •

edited

Loading

polothy commented Jan 31, 2020

polothy commented Jan 31, 2020

jstrachan commented Jan 31, 2020

jstrachan commented Jan 31, 2020 •

edited

Loading

rawlingsj commented Jan 31, 2020

jstrachan commented Jan 31, 2020

rawlingsj commented Jan 31, 2020

polothy commented Feb 1, 2020

polothy commented Feb 1, 2020

houndci-bot Feb 1, 2020

houndci-bot Feb 1, 2020

houndci-bot Feb 1, 2020

jstrachan commented Feb 1, 2020

jstrachan commented Feb 1, 2020

jstrachan commented Feb 1, 2020

jstrachan commented Feb 1, 2020

jenkins-x-bot commented Mar 1, 2020

jstrachan commented Mar 2, 2020

[wip] multi cluster support #6664

[wip] multi cluster support #6664

Conversation

jstrachan commented Jan 31, 2020

jenkins-x-bot commented Jan 31, 2020

jstrachan commented Jan 31, 2020

jstrachan commented Jan 31, 2020

houndci-bot Jan 31, 2020

Choose a reason for hiding this comment

houndci-bot Jan 31, 2020

Choose a reason for hiding this comment

polothy commented Jan 31, 2020

jstrachan commented Jan 31, 2020 • edited Loading

polothy commented Jan 31, 2020

polothy commented Jan 31, 2020

jstrachan commented Jan 31, 2020

jstrachan commented Jan 31, 2020 • edited Loading

rawlingsj commented Jan 31, 2020

jstrachan commented Jan 31, 2020

rawlingsj commented Jan 31, 2020

polothy commented Feb 1, 2020

polothy commented Feb 1, 2020

houndci-bot Feb 1, 2020

Choose a reason for hiding this comment

houndci-bot Feb 1, 2020

Choose a reason for hiding this comment

houndci-bot Feb 1, 2020

Choose a reason for hiding this comment

jstrachan commented Feb 1, 2020

jstrachan commented Feb 1, 2020

jstrachan commented Feb 1, 2020

jstrachan commented Feb 1, 2020

jenkins-x-bot commented Mar 1, 2020

jstrachan commented Mar 2, 2020

jstrachan commented Jan 31, 2020 •

edited

Loading

jstrachan commented Jan 31, 2020 •

edited

Loading