Drop k8s.io/kubernetes dependency

[ash2k]

Please consider dropping dependency on k8s.io/kubernetes module as it makes the engine really unwieldy to use.

Imported:

• gitops-engine/pkg/diff/diff.go

  Line 27 in 1723191

  "k8s.io/kubernetes/pkg/apis/core"

  fixed in refactor: reduce k8s.io/kubernetes usage #57.

• gitops-engine/pkg/health/health.go

  Line 17 in df17961

  podutil "k8s.io/kubernetes/pkg/api/v1/pod"

  copy it?

• gitops-engine/pkg/utils/kube/convert.go

  Line 7 in 8430dc0

  "k8s.io/kubernetes/pkg/api/legacyscheme"

  a custom scheme can be constructed to avoid the dependency

• gitops-engine/pkg/utils/kube/ctl.go

  Line 29 in 4bd4f29

  "k8s.io/kubernetes/pkg/kubectl/cmd/auth"

  copy it?

• gitops-engine/pkg/utils/kube/import_known_versions.go

  Line 4 in 8430dc0

  _ "k8s.io/kubernetes/pkg/apis/apps/install"

  a custom scheme can be constructed to avoid the dependency

[ash2k]

One option would be to copy the bits that are not available in other packages. I'm willing to do the work if maintainers are ok with that. Please let me know.

[ash2k]

Hm, I have just discovered that I cannot use the engine because of this dependency. I'm getting the following error when building my project with bazel:

ERROR: /private/var/tmp/_bazel_mikhail/6a0c1de97b81a8c5de5b6b9aec394679/external/com_github_argoproj_gitops_engine/pkg/utils/kube/BUILD.bazel:3:11: in go_library rule @com_github_argoproj_gitops_engine//pkg/utils/kube:go_default_library: target '@io_k8s_kubernetes//pkg/kubectl/cmd/auth:go_default_library' is not visible from target '@com_github_argoproj_gitops_engine//pkg/utils/kube:go_default_library'. Check the visibility declaration of the former target if you think the dependency is legitimate

This is because https://github.com/kubernetes/kubernetes/blob/07d88617e55b64c243956c7b2032430beb03b159/pkg/kubectl/cmd/auth/BUILD#L16 prohibits to import this package. This is not a problem if Go modules are used, but the package cannot be used via bazel+gazelle (my case). Clearly package authors do not want anyone to depend on it.

[ash2k]

I've tried copying the code but ran into these issues:

• k8s.io/kubernetes/pkg/kubectl/cmd/auth is a lot of logic, not just trivial helpers.
• looks like creating a custom scheme is not an option because conversion functions are used but they are not exported from k/k.

So I guess I have to look for a workaround...

p.s. I think conversion functions might not be safe to use on the client side. An example: gitops-engine imports kubernetes version X and conversion functions for this version. The cluster may be of a different version Y and an object of the same group/version/kind might have a new field (I think adding fields within the same version is ok as long as default value matches absent field semantics). This field is unknown to conversion functions from version X and hence will be dropped. Because of that it is not safe to use conversion functions from any other version but Y.

[ash2k]

kubernetes/enhancements#1020

[mdvorak]

Same requirement will be for https://github.com/argoproj/argo-cd, but this needs to be fixed first (as gitops-engine is a dependency of argo-cd)

[ghostsquad]

Please consider dropping dependency on k8s.io/kubernetes module as it makes the engine really unwieldy to use.

I'm just curious what makes the dependency on kubernetes unwieldy exactly?

[mdvorak]

Please consider dropping dependency on k8s.io/kubernetes module as it makes the engine really unwieldy to use.

I'm just curious what makes the dependency on kubernetes unwieldy exactly?

go module system - replace directive, used in argocd go.mod, is not used by anything that uses argo as own dependency (they are not transitive). Which means, that anyone depending on argo needs whole replace block for each k8s dependency, and maintain it.

Plus, using k8s.io/kubernetes is explicitly deprecated, as it intentionally depends on non-existent v0.0.0 versions of all other libraries. See also kubernetes/kubernetes#81878

[ghostsquad]

ok, ya, I just ran into this:

go get: github.com/argoproj/[email protected] requires
	k8s.io/[email protected] requires
	k8s.io/[email protected]: reading k8s.io/api/go.mod at revision v0.0.0: unknown revision v0.0.0

[ghostsquad]

this one is hairy:

gitops-engine/pkg/utils/kube/ctl.go

Line 29 in 4bd4f29

"k8s.io/kubernetes/pkg/kubectl/cmd/auth"

lots to copy

as this pulls in

"k8s.io/kubernetes/pkg/registry/rbac/reconciliation"

which pulls in:

"k8s.io/kubernetes/pkg/apis/core/helper"

[ash2k]

Some good progress kubernetes/kubernetes#96145

[ash2k]

Some more good news: https://docs.google.com/document/d/121FWis7sIzdG83H9T8G2qvSB6inxJoUub5KsVwFDa60/edit#heading=h.mipaetrdowr and https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/2144-clientgo-apply#apply-functions

So after #205 merges our only blocker for dropping k/k dependency is the defaulting that we are using from scheme - see #206. Once client-go receives Server Side Apply support, my understanding is that it'll be possible to use this new API to issue a dry-run apply command to receive the new object. Then it should be possible to perform a diff between current in-cluster object and the returned one to see if something would change if we apply it for real. Obviously, I haven't tried that so I may be missing something here.

[alexmt]

Thank you for update and reminder about #205 and #206 ! I've started looking at PRs, then got distracted and totally forgot. Will try to review and test both next week.

[sbose78]

We should pick this up soon :)

[celrenheit]

Hey!
I may be missing something but isn't the package https://pkg.go.dev/k8s.io/client-go/kubernetes/scheme able to replace the current usage of k8s.io/kubernetes.

[ash2k]

@celrenheit This is the client-side scheme, it does not contain conversion/defaulting functions the server-side one contains.

[celrenheit]

Oh I see, nvm the silly question

[abursavich]

This is the client-side scheme, it does not contain conversion/defaulting functions the server-side one contains.

How do you handle custom resources for which conversions/defaulting functions aren't available?

[twmb]

The last update nearly two years ago was that this would be picked up soon -- is there a way to move the project forward?

I spent a day on removing the dependency a year ago but took an approach that ultimately did not work and would not solve the issue.

[ghostsquad]

The problem with letting this continue to linger is that this really isn't a very reusable library in it's current state. It's quite painful to have to manage the mod file manually.

[abursavich]

Bump.

Importing the Argo CD API (e.g. https://pkg.go.dev/github.com/argoproj/argo-cd/[email protected]/pkg/apis/application/v1alpha1) brings in gitops-engine which brings in all of kubernetes and is a maintenance nightmare.

[jonathan-lo]

Are there any updates on this?