add secret and configmap generator plugins

[sethpollack]

This is a POC of my plugin suggestion from #692

Plugins are loaded from ~/.kustomize/plugins so there is no risk of executing malicious code from a remote base. Also plugins are built out of tree and allows people to create their own integrations.

Also to solve the distribution problem we can provide some base plugins and export them to ~/.kustomize/plugins with a kustomize plugin save -d ~/.kustomize/plugins command.

Some example plugins are provided in the plugins/ directory. They can be built with go build -buildmode=plugin

They can be used as follows:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

secretGenerator:
- name: env-example
  plugins:
  - name: Env
    args:
    - OTHER_PREFIX_EXAMPLE_ENV
    - MY_PREFIX_EXAMPLE_ENV
    filterPrefix: MY_PREFIX_
    stripPrefix: true

[jcassee]

This is great, I see you even included a sops dotenv plugin. I would like to try it out, but I am unsure how the sops integration works exactly. Can you give a concrete example?

[sethpollack]

@jcassee cd into the sops plugin dir and build the plugin with go build -buildmode=plugin -o ~/.kustomize/plugins/sopsdotenv.so

Then do something like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

secretGenerator:
- name: sops-example
  plugins:
  - name: SopsDotEnv
    args:
    - my_sops_encrypted_file.env

[sethpollack]

@jcassee I tested out the sops plugin and added some code so that you can use relative file paths.

[monopole]

@sethpollack this is interesting.

Please take the kvPair refactor out of this PR and into its own pr, so we can get it in regardless and so it doesn't distract from the security changes. Would happily review :)

This feature is meant to replace a perceived security risk that forced a fairly large rollback of a kubectl PR. So although it's nice to have the POC, we need discussion attached to a KEP PR and review. Nothing impacting security can go into kustomize or kubectl w/o a KEP in this repo.

[sethpollack]

@monopole Ok rebased on the kvPair refactor.

Also opened a KEP PR kubernetes/enhancements#811

[jcassee]

@sethpollack Just tried it out. Works great! I like the idea, and I'm interested in what kind of feedback your KEP receives.

[sethpollack]

@jcassee thanks for the feedback!

[sethpollack]

@monopole I updated the implementation, can you take a look and let me know what you think?

[ivan4th]

this causes the linter to fail during Travis CI job:

k8sdeps/kv/plugin/testonly.go:41:25: `(*testonly).load` - `name` is unused (unparam)

[sethpollack]

@monopole I fixed your comments, can you take a look?

[monopole]

please rebase and we'll merge this.
Still needs a flag and some other mods, but we won't release till those are added.

[sethpollack]

Ok

[sethpollack]

@monopole done!

[monopole]

/lgtm

[monopole]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: monopole, sethpollack

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [monopole]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jcassee]

Thanks @sethpollack and @monopole for seeing this through!

[jcassee]

@sethpollack At some time this PR contained examples to replace the old envCommand: "sops -d somefile.env" and commands: [somefile: "sops -d somefile"]. I think those should probably not be in the kustomize code tree. Do you have them lying around somewhere? Does it make sense to publish them in a repository?

[sethpollack]

I can try and find them. @monopole what do you think about adding a repo for kustomize plugins?

[sethpollack]

@jcassee Something like this should work

func (p *sopsDotEnv) Get(root string, args []string) (map[string]string, error) {
       allKvs := make(map[string]string)

       for _, arg := range args {
               data, err := decrypt.File(fmt.Sprintf("%s/%s", root, arg), "dotenv")
               if err != nil {
                       return nil, err
               }

               kvs, err := kv.KeyValuesFromLines(data)
               if err != nil {
                       return nil, err
               }

  	        for _, pair := range kvs {
	                all[pair.Key] = pair.Value
		}
       }
       return allKvs, nil
}

var KVSource sopsDotEnv```

[jcassee]

@sethpollack Thanks, that worked! Now to create a plugin that emulates:

secretGenerator:
  - name: secret
    commands:
      somefile: "sops -d somefile.sops"

I'll post here when I get something working, for posterity.

[jcassee]

This is a plugin for replacing the commands form. Example:

secretGenerator:
  - name: example
    kvSources:
      - name: sopsfiles
        pluginType: go
        args:
          - somefile.txt=somefile.sops.txt

Code:

package main

import (
	"fmt"
	"go.mozilla.org/sops/cmd/sops/common"
	"go.mozilla.org/sops/decrypt"
	"strings"
)

type sopsFiles struct{}

var KVSource sopsFiles

func (p *sopsFiles) Get(root string, args []string) (map[string]string, error) {
	allKvs := make(map[string]string)

	for _, arg := range args {
		split := strings.SplitN(arg, "=", 2)
		if len(split) < 2 {
			return nil, fmt.Errorf("should be NAME=FILENAME: %s", arg)
		}
		format := formatForPath(split[1])
		data, err := decrypt.File(fmt.Sprintf("%s/%s", root, split[1]), format)
		if err != nil {
			return nil, err
		}

		allKvs[split[0]] = string(data)
	}

	return allKvs, nil
}

func formatForPath(path string) string {
	if common.IsYAMLFile(path) {
		return "yaml"
	} else if common.IsJSONFile(path) {
		return "json"
	} else if common.IsEnvFile(path) {
		return "dotenv"
	}
	return "binary"
}

[jcassee]

I published both plugins at https://github.com/goabout/goabout-kustomize-plugins. I'd be happy to have them included in some central plugin repository.

[fentas]

@sethpollack big thanks for this 👍
Are there any docs on this. Right now I use @jcassee code as a template (thanks) and go through the source code.

What I am wondering right now is how I am able to get information towards the plugin. For example am I able to do this in the end?

secretGenerator:
  - name: example
    kvSources:
      - name: gernerate-passwd
        pluginType: go
        chars: "asdsaf@!#!"
        length: 48
        args:
          - password1
          - password2

The args part is clear but is this the only way to pass data? Or can I access here chars and length?

[sethpollack]

@fentas No you would need to pass those as extra args.

[mousavian]

Is there any reason type of args in the Get method signature is []string instead of map[string]string ?
IMO using map can be really beneficial. For instance, plugins can add more parameters in their future versions, or even optional args without breaking backward compatibility or forcing users to follow the order.