helm: allow further customization through additional flags and secrets

[trevex]

In out cluster we use transport encryption for all components of vitess. Currently we use the upstream helm chart and apply several patches, because we are not able to configure transport encryption with just the helm chart.

The implemented changes:

• vttablet's extraMyCnf only include files with *.cnf extension to be able to provide other configuration files as well, e.g. percona vault keychain configuration
• Add extraFlags and secrets to vitess components:
  • vtctld and vtctlclient (init job and orchestrator)
  • vtgate
  • vttablet
• More documentation for readme on encryptions and TLS

Example values:

vttablet:
  extraMyCnf: vitess-extra-config
  extraFlags:
    grpc_cert: /vt/usersecrets/vttablet-tls/vttablet.pem
    grpc_key: /vt/usersecrets/vttablet-tls/vttablet-key.pem
    tablet_grpc_ca: /vt/usersecrets/vttablet-tls/vitess-ca.pem
    tablet_grpc_server_name: vttablet
  secrets:
  - vttablet-tls # mounted as /vt/usersecrets/vttablet-tls

vtctld:
  extraFlags:
    grpc_cert: /vt/usersecrets/vtctld-tls/vtctld.pem
    grpc_key: /vt/usersecrets/vtctld-tls/vtctld-key.pem
    tablet_grpc_ca: /vt/usersecrets/vtctld-tls/vitess-ca.pem
    tablet_grpc_server_name: vttablet
    tablet_manager_grpc_ca: /vt/usersecrets/vtctld-tls/vitess-ca.pem
    tablet_manager_grpc_server_name: vttablet
  secrets:
  - vtctld-tls

vtctlclient:
  extraFlags:
    vtctld_grpc_ca: /vt/usersecrets/vitess-ca/vitess-ca.pem
    vtctld_grpc_server_name: vtctld
  secrets:
  - vitess-ca

The vtctlclient section is new to the values file and is used by all components that interact using the vtctlclient (e.g. the init jobs and orchestrator). At the current time I would like to get some early feedback, whether this approach works for vitess, while I continue with testing in our local development cluster.

EDIT:
All secrets are always mounted at /vt/usersecrets/$secretname/

[derekperkins]

I think this is looking great, thanks for contributing back upstream!

Do you have any use cases for adding flags to mysqlctld?

[derekperkins]

Also, I think it'd be helpful to add some of the examples you have above to the values.yaml, plus adding a section to the readme about how to enable grpc.

[trevex]

Do you have any use cases for adding flags to mysqlctld?

Surprisingly, not yet. Traffic encryption is not necessary because it only listens on the local socket and everything else can be configured using *.cnf files.

Also, I think it'd be helpful to add some of the examples you have above to the values.yaml, plus adding a section to the readme about how to enable grpc.

I added three sections to the README including gRPC TLS configuration, slave replication traffic encryption and at rest encryption using the vault plugin.

[derekperkins]

This is looking really great, especially the documentation. Another PR was merged yesterday that introduced a few new places that use vtctlclient that will need to have your flags applied. https://github.com/vitessio/vitess/blob/master/helm/vitess/templates/_vttablet.tpl#L394-L439

Once those are taken care of, I think it LGTM.

@hmcgonig / @dkhenry Any thoughts?

[trevex]

This is looking really great, especially the documentation. Another PR was merged yesterday that introduced a few new places that use vtctlclient that will need to have your flags applied. https://github.com/vitessio/vitess/blob/master/helm/vitess/templates/_vttablet.tpl#L394-L439

Merged from current master and added VTCTL_EXTRA_FLAGS to preStop hook as well.
2c06a9f

[derekperkins]

@trevex I merged your ceph support PR first which is causing a merge conflict here. Can you resolve that? When that's done, I'll go ahead and merge this PR (just got merge rights).

[trevex]

@derekperkins All merge conflicts resolved.