Update kubernetes watcher to use official client-go libraries

[vjsamuel]

This PR moves away from using the Eric Chang client-go for Kubernetes interaction in favor of the official kube client. As a result, we now use SharedInformers. This makes sure that at any given time there is only 1 connection to the API server per resource that is requested. Moving to the client-go also makes sure that we no longer need to check ResourceVersion while watching. It is implicitly done under the hood.

This change now allows the watcher to be testable and as a follow up we could add test cases.

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[exekias]

ok to test

[exekias]

it sems you don't use inCluster anymore, how would this be used now? We would need to update the comment and some docs

[vjsamuel]

the official kube client deduces if it runs in cluster or not and hence to get a kube client we dont really need it.

if kubeconfigPath == "" && masterUrl == "" {
		klog.Warningf("Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.")
		kubeconfig, err := restclient.InClusterConfig()
		if err == nil {
			return kubeconfig, nil
		}
		klog.Warning("error creating inClusterConfig, falling back to default config: ", err)
	}

if kubeconfig path is provided then it just tries to create an out of cluster configuration.

That said. let me fix the discover node to just follow the same thing.

[exekias]

what is this line for? a comment could help here

[vjsamuel]

this just makes sure that it gets the namespace/name combination only for resources where state is not Unknown

[exekias]

Thank you for opening this @vjsamuel, this is REALLY appreciated! 🎉

I left a few comments and questions, I'm wondering about the changes in user configurations and backwards compatibility

[vjsamuel]

not sure if the failures are related. I have done manual testing on all flows and things look file. Lets have action items to now have fake client based test cases on the autodiscover/processor paths so that it is easier to find failures. there were a few issues that i fixed in the last commit one of which is to increase the default resync period. with the new client, it is no longer necessary to sync every minute. the resync period forces all items on the cache to be replayed. this is undesirable at a 1 minute interval. i have also gone ahead and removed the in_cluster config parameter as it is deduced automatically. the PR is now in mergeable state now.

[exekias]

Thank you so much for this contribution @vjsamuel! it's a real enabler for future features