[feat] ability to customize kube client qps

[DerekTBrown]

What does this PR do?

• This PR extends the gatewayapi plugin to permit increasing Kube Client QPS and Burst values.
• This PR also adds documentation of this feature.

[kostis-codefresh]

Hello and many thanks for this.

I am curious. Is this solving an issue that you have in production usage right now? Or your suggestion is a best practice that you recommend?

I am wondering though whether this change must happen centrally in the main Argo Rollouts controller so that all plugins get this capability out of the box.

[DerekTBrown]

I am curious. Is this solving an issue that you have in production usage right now? Or your suggestion is a best practice that you recommend?

TL;DR; This is solving an issue we are seeing in our real environments.

We are in the initial phases of adopting the Gateway API. We started observing issues with Argo Rollouts for services where we added HTTPRoute configuration. pprof on the main Argo Rollouts process showed us that the Rollout worker goroutines were stalled on go-plugin RPC calls. I then monkey-patched pprof into the Gateway API plugin, and observed that the plugin was stalling on the k8s client ratelimiter. Raising the QPS and burst values helps resolve that issue.

[DerekTBrown]

I am wondering though whether this change must happen centrally in the main Argo Rollouts controller so that all plugins get this capability out of the box.

The Argo Rollouts controller is a separate process/binary from any of the plugins, so it isn't currently possible to make this change within the controller in a way that would propagate to the plugins.

If the overall project were trying to achieve something like this within the plugins interface, it would need to create a standard interface of command-line flags or config options that are passed into plugins. To my knowledge, this doesn't exist. Even if it did exist, the implementation of this flag in particular would be problematic, because the QPS ratelimits would be effectively double whatever I provide as a command-line option (since the controller and the plugin are separate binaries, the ratelimiters are separate).

As an aside, I started a discussion about the limitations of the plugin architecture here: argoproj/argo-rollouts#4042 (comment).

[kostis-codefresh]

While this PR is great for the plugin, if you have the time I think it would be also useful if you also create a PR against the main rollouts controller and add code that passes down the same parameters to the plugin IF they have already been defined in the controller level
https://github.com/argoproj/argo-rollouts/blob/00b57266b9c93d29d21d897e5a7a17fc82ab2e6a/cmd/rollouts-controller/main.go#L294

Also we would need to update the docs and make it clear to the users that the QPS setting is only for the plugin. If somebody is using multiple plugins or has a different setting in the controller, right now each kubeconfig object will have it own parameters. (the doubling problem you mention). You are right that we cannot fix this, but we can document it .