Release kubebuilder and k8s bundles separately

[mengqiy]

kubebuilder code doesn't have dependencies on k8s, since it's mostly scaffolding.
What have dependencies on k8s are controller-runtime and controller-tools. And both are not managed by kubebuilder.
Thus, IMO it's more reasonable to release kubebuilder and k8s bundle (i.e. kubeapiserver, kubectl and etcd) separately. We can have some mechanic to let the user to choose what k8s bundle to use with kubebuilder.

• merge ✨ stop using k8s.io/apimachinery to convert kind to resource #685
• move test to testData (part of ⚠️ controller-tools v0.2.0 scaffolding update #682), so go doesn't evaluate it to calculate deps.
• develop mechanic to let the user easily choose a k8s bundle version and install it
• change our release tools to support release kubebuilder and k8s bundle separately.

[mengqiy]

cc: @droot @DirectXMan12

[timothysc]

/assign @DirectXMan12
/cc @pwittrock @detiber @ncdc

So we (Cluster API) have a transitive dependency problem due to kube-builder deps. Kube builder forces us to take a specific version of kubernetes dependencies that we ideally would like to have abstracted. This causes a bunch of issues when trying to vendor in other Types whose versions may be newer.

This is a pretty big problem for us and would love for it to be fixed.

[DirectXMan12]

@timothysc this issue is mostly about testenv bundles, not about dependencies.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[DirectXMan12]

/lifecycle frozen

/help

[k8s-ci-robot]

@DirectXMan12:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/lifecycle frozen

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[camilamacedo86]

It was requested here a well: #1446. So +1 👍 for this request.

[DirectXMan12]

The remaining things to do here are to rev the release process and maybe add a little helper to the kb binary to download different deps. We already release a standalone copy of the deps, so we just need to to tweak the main process.

[DirectXMan12]

/good-first-issue

[k8s-ci-robot]

@DirectXMan12:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[DirectXMan12]

For what I'd look for in the download deps: list from GCS bucket's XML response, convert to version when possible, falling back to straight names in case we change the format, download to location specified in the KUBEBUILDER_ASSETS env variable, potentially with a separate folder per version.

[DirectXMan12]

don't do this automatically -- have a separate command. Do warn on kb init if we can't find them. Also, I'd probably say we should still ship a combined bundle too, just to make things easy, but I could be convinced otherwise.

[camilamacedo86]

Hi @DirectXMan12,

Shows that KB has bin shipping the bins of apiserver, kubectl and etcd in the build because they are required for the envtest.

So, I understand that for we are able to remove the bins from the kb build we need:

• Remove from the build
• Add a new target in the KB project Makefile and script to get them such as:

.PHONY: testbin
testbin:
	./test-setup.sh

`script to setup the bins (valid for mac and linux)`

set -eu

# */
# To use envtest is required etcd, kube-apiserver and kubetcl binaries in the testbin directory.
# This sript will perform this setup for linux or mac os x envs.
# */

K8S_VER=v1.18.2
ETCD_VER=v3.4.3
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m | sed 's/x86_64/amd64/')
ETCD_EXT="tar.gz"

rm -rf testbin
mkdir -p testbin

# install etcd bin
# the extension for linux env is not equals for mac os x
if [ $OS == "darwin" ]; then
   ETCD_EXT="zip"
fi
[[ -x testbin/etcd ]] || curl -L https://storage.googleapis.com/etcd/${ETCD_VER}/etcd-${ETCD_VER}-${OS}-${ARCH}.${ETCD_EXT} | tar zx -C testbin --strip-components=1 etcd-${ETCD_VER}-${OS}-${ARCH}/etcd

# install kube-apiserver and kubetcl bin
if [ $OS == "darwin" ]
then
  # kubernetes do not provide the kubernetes-server for darwin, so to have the kube-apiserver is rquired to build it locally
  # if the project is already cloned locally do nothing
  if [ ! -d $GOPATH/src/k8s.io/kubernetes ]; then
    git clone https://github.com/kubernetes/kubernetes $GOPATH/src/k8s.io/kubernetes --depth=1 -b v1.18.2
  fi

  # if the kube-apiserve is alredy built just copy
  if [ ! -f $GOPATH/src/k8s.io/kubernetes/_output/local/bin/darwin/amd64/kube-apiserver ]; then
    DIR=$(pwd)
    cd $GOPATH/src/k8s.io/kubernetes
    # Build for linux first otherwise it won't work for darwin - :(
    export KUBE_BUILD_PLATFORMS="linux/amd64"
    make WHAT=cmd/kube-apiserver
    export KUBE_BUILD_PLATFORMS="darwin/amd64"
    make WHAT=cmd/kube-apiserver
    cd ${DIR}
  fi
  cp $GOPATH/src/k8s.io/kubernetes/_output/local/bin/darwin/amd64/kube-apiserver testbin/

  # now let's get the kubectl
  curl -LO https://storage.googleapis.com/kubernetes-release/release/${K8S_VER}/bin/darwin/amd64/kubectl
  chmod +x kubectl
  mv kubectl testbin/
else
  [[ -x testbin/kube-apiserver && -x testbin/kubectl ]] || curl -L https://dl.k8s.io/${K8S_VER}/kubernetes-server-${OS}-${ARCH}.tar.gz | tar zx -C testbin --strip-components=3 kubernetes/server/bin/kube-apiserver kubernetes/server/bin/kubectl
fi

• So, it means that when we set up KB we will have a directory with the bins
• And then, we need to update the envtests setup to use it such as done by @joelanford in another project. Example:

var _ = BeforeSuite(func(done Done) {
	Expect(os.Setenv("TEST_ASSET_KUBE_APISERVER", "../../testbin/kube-apiserver")).To(Succeed())
	Expect(os.Setenv("TEST_ASSET_ETCD", "../../testbin/etcd")).To(Succeed())
	Expect(os.Setenv("TEST_ASSET_KUBECTL", "../../testbin/kubectl")).To(Succeed())

	logf.SetLogger(zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true)))
	testenv = &envtest.Environment{}

	var err error
	cfg, err = testenv.Start()
	Expect(err).NotTo(HaveOccurred())

	close(done)
}, 60)

var _ = AfterSuite(func() {
	Expect(testenv.Stop()).To(Succeed())

	Expect(os.Unsetenv("TEST_ASSET_KUBE_APISERVER")).To(Succeed())
	Expect(os.Unsetenv("TEST_ASSET_ETCD")).To(Succeed())
	Expect(os.Unsetenv("TEST_ASSET_KUBECTL")).To(Succeed())

})

Also, in order to help KB users are able to develop test we could/should at least doc it or provide the same feature in the default scaffolded project (Makefile target + script)

Note that it shows the reason for I am unable to run a few tests in the controller-runtime on Mac OS X as well. See the issue: kubernetes-sigs/controller-runtime#907

Please, let me know if I am missing something here.

c/c @joelanford

[camilamacedo86]

Hi @DirectXMan12,

The pr #1600 changes the setup for V3. However, I understand that for we are able to remove the k8s bins from the kubebuilder binary(releases) we would need to do the same for v2 which is a breaking change. Am I right? So, the change in the scripts used to do the KB release can be done just when no longer support V2 and it is removed.

c/c @joelanford @estroz

[vincepri]

To give links to the references pointed out in today's community meeting, we have a custom fetch_ext_bins.sh script that runs on make test and sets everything up properly https://github.com/kubernetes-sigs/cluster-api/blob/95fe9e2c2c48cb7c765e40fe97861f22765441ff/scripts/fetch_ext_bins.sh

The Makefile makes sure to set everything up https://github.com/kubernetes-sigs/cluster-api/blob/95fe9e2c2c48cb7c765e40fe97861f22765441ff/Makefile#L116-L118

[estroz]

Now that make test runs setup-envtest.sh, go/v3-alpha is slated to be the default plugin for v3.0.0, and kubebuilder init supports two plugins that scaffold projects with different k8s versions, I'd say this issue is resolved except for updating the release process.

The installation docs need to be updated to say something like To download envtest binaries for go/v2, do 'curl controller-runtime/v0.6.3/hack/setup-envtest.sh && source setup-envtest.sh ...' otherwise you don't have to download anything, 'make test' will do it for you.