Expose Customization Options for Migration Job in Helm Chart's values.yaml

[sean-m-benson]

Problem

Currently, the Helm chart provides various customization options in the values.yaml file for the deployment resource, such as communicating with the MySQL DB via a CloudSQL proxy sidecar container.

However, similar customization options are not available for the migration job resource, even though both the deployment and job would likely need to communicate with the database using the same mechanisms.

What have you tried?

We are deploying Fleet in GCP. We import the helm chart, and deploy it with customizations suitable for our environment. Our application ultimately fails because it gets stuck on the job. The job times out, and when looking at the failed pod's log, it shows:
Failed to start: creating db connection: dial tcp 127.0.0.1:3306: connect: connection refused

When inspecting the job's pod's manifest, we can see that there is no proxy sidecar container, even though enableProxy from values.yaml is true, and the sidecar container is defined in the deployment's manifest.

Potential solutions

1. There should be an additional section for customizing the job resource in the values.yaml, and the job template should be updated to leverage these customizations.
2. The migration job template should be updated to reference the existing configuration options used for the deployment resource.

What is the expected workflow as a result of your proposal?

I should be able to use the helm chart, and have it deploy successfully only by modifying the exposed customization options via the values.yaml file.

[noahtalerman]

Hey @lukeheath, @allenhouchins and I looked at this request during the unpacking the why ritual. The why is clear: they're trying to use Fleet's best practice Helm chart deploy Fleet in GCP.

What we don't understand is whether this is a bug or a feature request. What do you think?

Also, it looks like we're maintaining two different Helm charts:

• https://github.com/fleetdm/fleet/tree/main/docs/Deploy/_kubernetes
• https://github.com/fleetdm/fleet/tree/main/charts/fleet

We want to only maintain one right?

[lukeheath]

@rfairburn is the best person to review and speak to this.

[pboushy]

Hey @lukeheath, @allenhouchins and I looked at this request during the unpacking the why ritual. The why is clear: they're trying to use Fleet's best practice Helm chart deploy Fleet in GCP.

What we don't understand is whether this is a bug or a feature request. What do you think?

Some background I found when when Sean first asked me about this in Slack:

• the cloudSQL stuff was added in the original version of the helm chart back in Add Helm chart #250 (3 years ago)
• The original helm chart appears to have included migration in the deployment.
• the migration was separated from the deployment into a job in Helm: Job based migrations on pre install and upgrade #3827 (2 years ago)
• It appears that several items were not accounted for in the conversion from the deployment doing the migration to the job doing the migration

Also, it looks like we're maintaining two different Helm charts:

• https://github.com/fleetdm/fleet/tree/main/docs/Deploy/_kubernetes
• https://github.com/fleetdm/fleet/tree/main/charts/fleet

We want to only maintain one right?

Kubernetes and Helm are a bit different and I think the reason you have 2 is due to the history of Helm.
Helm 2 used to require you to run some stuff in Kubernetes to use Helm. Some groups didn't allow that. Now Helm 3 is all handled locally without deploying a helm orchestrator in k8s.

At this point, IMO Helm is better for this and the Helm chart has a lot more functionality than the normal k8s item.

[lukeheath]

@noahtalerman Maybe a quick call with you, @allenhouchins, and @rfairburn makes sense? Feel free to include me if I can be helpful.

[rfairburn]

this makes 100% sense and I believe I can do this very quickly if we use the same proxy options as the primary fleet container.

Even if we have separate entries in values.yaml, it shouldn't be terrible to implement.

[rfairburn]

https://github.com/fleetdm/fleet/tree/main/docs/Deploy/_kubernetes

is direct yaml and is likely out-of-date at this point. For helm, the other (chart) location in the git repo is what is used to populate everything.

[noahtalerman]

@rfairburn if it's quick I say go for it. I added this to #g-customer-success board so y'all can prioritize.

https://github.com/fleetdm/fleet/tree/main/docs/Deploy/_kubernetes is direct yaml and is likely out-of-date at this point. For helm, the other (chart) location in the git repo is what is used to populate everything.

Also, sounds like we want to get rid of this YAML if it's out of date and we have a different chart.

[rfairburn]

#24412 is an initial draft that adds what I see as the main customization that would apply to migrations (the cloudsql sidecar). Per the PR notes, all of the sa/rbac stuff was moved to be created at init time so it is in place to be leveraged by the migration job.

This still needs to be tested in gke for validation, but looks to pass dry-run applies locally.

[fleet-release]

Migration job's voice heard,
Helm chart sings in harmony,
Fleet's connection secured.