Kustomize module for generation and var substitution

[souleb]

This module declare methods used in both kustomize-controller

and the flux cli

Signed-off-by: Soule BA bah.soule@gmail.com

[souleb]

@stefanprodan as discussed I have made sure to restore a clean repository.

[souleb]

I have tested and benchmarked the generator code on the kustomize-controller. The tests are positive.

Here the benchmark results

# existing generator code
BenchmarkKustomizationGenerator-8           1000          19454609 ns/op        11205376 B/op      90431 allocs/op
BenchmarkKustomizationGenerator-8           1000          19825319 ns/op        11217971 B/op      90432 allocs/op
BenchmarkKustomizationGenerator-8           1000          20249546 ns/op        11207548 B/op      90431 allocs/op

name                      time/op
KustomizationGenerator-8  19.8ms ± 2%

name                      alloc/op
KustomizationGenerator-8  11.2MB ± 0%

name                      allocs/op
KustomizationGenerator-8   90.4k ± 0%

## new code using Unstructured objects
BenchmarkKustomizationGenerator-8           1000          19573560 ns/op        11237382 B/op      90454 allocs/op
BenchmarkKustomizationGenerator-8           1000          19637016 ns/op        11245801 B/op      90454 allocs/op
BenchmarkKustomizationGenerator-8           1000          19789540 ns/op        11245425 B/op      90454 allocs/op

name                      time/op
KustomizationGenerator-8  19.7ms ± 1%

name                      alloc/op
KustomizationGenerator-8  11.2MB ± 0%

name                      allocs/op
KustomizationGenerator-8   90.5k ± 0%

and the benchmark function

func BenchmarkKustomizationGenerator(b *testing.B) {
	g := NewWithT(b)

	f := func() {
		os.Remove("./testdata/generator/resources/kustomization.yaml")
	}

	b.Cleanup(f)

	// Create a kustomization file with varsub
	yamlKus, err := os.ReadFile("./testdata/generator/kustomization.yaml")
	g.Expect(err).NotTo(HaveOccurred())

	clientObjects, err := readYamlObjects(strings.NewReader(string(yamlKus)))
	g.Expect(err).NotTo(HaveOccurred())

	//Get a generator
	gen := NewGenerator(clientObjects[0])
	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		_, err = gen.WriteFile(resourcePath)
	}
	
        g.Expect(err).NotTo(HaveOccurred())
}

There does not seems to be any performance reduction.

Update:

the copyFile() call to save the kustomization.yaml take 9.95% of the total time (seen with pprof). This function call is not needed in kustomize-controller.

## new code using Unstructured objects without copyFile call
name                      time/op
KustomizationGenerator-8  19.4ms ± 0%

name                      alloc/op
KustomizationGenerator-8  11.2MB ± 0%

name                      allocs/op
KustomizationGenerator-8   90.4k ± 0%

[stefanprodan]

@souleb I would've bet on the number of allocations to be way higher, that you for the benchmark. I feel better now about this change.

[souleb]

I will modify the code to use a functional option to save the kustomization.yaml, so we can avoid doing it in the controller.

[stefanprodan]

@souleb please rename this package to kustomize to match the one in apis and also please remove the Kustomize prefix from the two types.

[stefanprodan]

LGTM

Thanks @souleb please rebase with main.

[stefanprodan]

@souleb this is now released as github.com/fluxcd/pkg/kustomize@v0.0.1, you can use it in Flux CLI.