Support an integrated test workflow

[BenTheElder]

IE: replace the prototype in hack/ci/e2e-k8s.sh with a built in test command.

This will make it even easier to replicate kind's Kubernetes PR test results, and allow us to better test and support this.

kind should remain trivially scriptable for supported use csaes, but for testing Kubernetes specifically we should provide a robust tool that does this right out of the box, seeing as this is the founding purpose of the project.

More details (shared with [email protected] and [email protected]) https://docs.google.com/document/d/16WdI8VTDmNt-TlFGzW1tJaLD7NM1r6ppn0k4GXQJNHU

/assign

[BenTheElder]

added git version #988, #989
/lifecycle active

[aojea]

/cc

[aojea]

@BenTheElder is this for this milestone?

[BenTheElder]

I keep the milestone up to date. This is not a blocker for the milestone It would be nice to have and might land before then if I'm blocked on other things and finish submitting it.

[aojea]

I'd love to have a kind e2e.test --name kind --focus \[sig-network\].*Conformance" command
maybe we can add the e2e.test to the node images when we are building the node image, what do you think?

[BenTheElder]

did some more design and asked some key power users of this more about their thoughts directly ... bumping to the next milestone though because it's not quite as pressing as unblocking PV work

[BenTheElder]

status update: writing down some fresh code for v0.8.0 MVP, will PR

[BenTheElder]

reviving this, rebasing on some other test changes

[BenTheElder]

i really wanted this, but there's too many important user fixes that need to ship. v0.9.0 we will start using this and collecting e2e coverage

[BenTheElder]

Updated Proposal

NOTE: Some of this is already covered by the document above, however I'm rehashing a current proposal here.

1. We should not attempt to support testing anything other than Kubernetes core, to keep the project maintainable and focus on things clearly in-scope for the project.

Users can implement and indeed already have implemened custom test workflows using the other kind commands (see for example the hack/ci/e2e-k8s.sh that we have today) or using the kind Go pkg APIs.

This will serve both to streamline Kubernetes core e2e maintenance and as an example implementation for other workflows, which can be implemented out-of-tree. Even other subprojects (e.g. cluster-api, GatewayAPI, etc.) can follow the patterns here but should probably not be implemented in the kind repo for maintainability reasons.

Workflows not targeting kubernetes @ HEAD should be using stable kind anyhow, yet may still need to regularly tweak their test implementations. Workflows for Kubernetes core need to use the latest kind sources because sometimes we have to patch how to run Kubernetes between releases of both projects and therefore would not gain stability by being out of tree anyhow. Additionally they will be maintained by the same core maintainers of the KIND project, as they're why this project even exists.

2. We should not support any arguments / flags. Period.

Past experience from tools like kubetest / kubetest2 tells us that having flexible test tools that attempt to support every workflow quickly become confusing and unusable with loads of complex and potentially conflicting options.

We should instead have dedicated named workflows for each CI suite we support.

I.E. pull-kubernetes-e2e-kind CI job becomes kind test e2e and pull-kubernetes-e2e-kind-ipv6 becomes kind test e2e-ipv6, and so on.

This will be easy for new contributors to follow, and does not prevent us from internally sharing code between CI suites.

When we have a new core e2e CI job, that is something targeting blocking PR/release testing, we can add a new named command. These commands can have a lower stability than other kind commands, as we can explicitly note that they're for developing the Kubernetes project and not intended for other usage. They need only be stable from the POV of Kubernetes CI.

To alter the behavior of these workflows, we can test changes in the KIND repo where we have duplicate CI jobs by simply altering the sources. We do not need to expose any options, though we will respect CI env ($ARTIFACTS).

Again, other workflows can be implemented out-of-tree using either the kind Go APIs or scripts using the other commands (kind create cluster, etc.) as mentioned in 1). Additionally kubetest2 already exists as a reusable tool extensible for other tests. The goal of kind test is to provide a streamlined example used to test core Kubernetes workflows.

3. We should log subcommands used, so users can incrementally repeat them, instead of exposing options to skip/run phases.

I.E. we can log the equivalent kind create cluster, docker exec kind-control-plane /kind/e2e.test ... commands used both for transparency and so users that do not want to run the full build + up + test + down can understand and repeat them without attempting to add options for this in the kind test commands.

[BenTheElder]

cc @aojea, I'd like to rebuild a cleaner version of #1269 based on the proposal above, and solve our problems maintaining the growing e2e-k8s.sh by breaking out the core workflows into these commands, early in the Kubernetes 1.27 cycle to avoid disruptions.

I think it will be tempting to start to add options but be very difficult to maintain and unwieldy for users, we should just push some workflows out to their own scripts (which can start from the current e2e-k8s.sh) or add new commands for additional essential test workflows, I.E. workflows that we'd run in both PRs to this repo and Kubernetes/Kubernetes.

[aojea]

I like this idea, having workflows commands, we should have also a canonical example so people can reuse to implement their own workflows ...

When we have a new core e2e CI job, that is something targeting blocking PR/release testing, we can add a new named command. These commands can have a lower stability than other kind commands, as we can explicitly note that they're for developing the Kubernetes project and not intended for other usage. They need only be stable from the POV of Kubernetes CI.

something like?

/pkg/cmd/workflows/stable/e2e
/pkg/cmd/workflows/stable/e2e_ipv6
/pkg/cmd/workflows/devel/multinetwork