sig-api-machinery: Add scale targets to CRDs to GA KEP

[jpbetz]

This proposes that our scale targets for CRD to GA be based on API call latency SLIs/SLOs and kubernetes thresholds.

For details on scalability data gathered, see: https://docs.google.com/document/d/1tEstPQvzGvaRnN-WwGUWx1H9xHPRCy_fFcGlgTkB3f8/edit#

[jpbetz]

@roycaihw @liggitt @sttts We're still gathering some data, but early feedback welcome!

[liggitt]

cc @wojtek-t @kubernetes/sig-scalability-misc

[liggitt]

the feedback from @wojtek-t was helpful. I think the approach should be:

• align with native SLI/SLO
• make it clear the overhead of the conversion webhook is either explicitly excluded or capped at Xms per call (and make sure we can measure/compensate for that in the scale tests that ensure we hit our targets)
• make the variables for our tests and claimed scale for a particular custom resource type:
  • the resource size
  • max number of custom resource instances per namespace
  • max number of custom resource instances cluster-wide

[liggitt]

I'd still like to see real-world feedback inform those max instance size and count-per-namespace and count-per-cluster numbers.

In the absence of that, setting up initial scale targets along the lines of O(64KB), O(1000) per namespace, O(10000) per cluster seems reasonable in a hand-wavy way, given other count targets in https://github.com/kubernetes/community/blob/master/sig-scalability/configs-and-limits/thresholds.md

[jpbetz]

I've rewritten this section to complement https://github.com/kubernetes/community/blob/master/sig-scalability/slos/api_call_latency.md and https://github.com/kubernetes/community/blob/master/sig-scalability/configs-and-limits/thresholds.md. The actual thresholds and max resource size are not finalized, but I've penciled in "O(10KB), O(2000) per namespace, O(10000) per cluster" and we can substitute in better value once we've nailed them down.

I agree read world feedback would be valuable, maybe there is a group we could survey? I'll ask around.

[smarterclayton]

Some known CRD scale factors:

1. Clusters with 100-200 CRDs (fixed base), many of which have 1-5 CR instances.
2. Average size of CRs 2KB, max 150KB (very large CRs which contain structured info that might represent interconnected components and chunks of text)
3. Some high churn CRDs like ProwJobs on a cluster with 10k-30k instances in a single namespace, frequent lists, infrequent gets and updates.
4. Large numbers of watches on many different CRDs for common / shared config. Example, cluster configuration represented as 30-40 CRD, of which there are 20+ controllers each having single item CR watches on some of those.

[liggitt]

1. Clusters with 100-200 CRDs (fixed base), many of which have 1-5 CR instances.

That's a good dimension (number of distinct custom resource types) we forgot to represent. The way the custom resource server is structured, each custom resource handler is independent, so this is no problem for the server to support, but is good to explicitly create/exercise O(100s) of CRDs in parallel in a test. The main impact I see for that would actually be on clients that pull down discovery docs for all group/version/resources, but that's not really specific to CRDs.

[smarterclayton]

yeah any O(N) CRD things should be caught quick. Or where there is a large constant factor per CRD (like openapi).

[smarterclayton]

For this cluster (reasonably representative of high CRD use):

$ kubectl get crd | wc -l
62
$ kubectl get crd -o json | jq -c '.' | wc -c
493222

Most of those CRDs have openapi specs with field values for explain and validation. I think it could get higher given that some may not have all the docs they need defined yet

[wojtek-t]

At the high-level, it looks fine. I would like to think more about the details, but in the worst case, I believe we could change them later too (I'm a bit overloaded now, and I don't want to block this).

[liggitt]

/lgtm

[liggitt]

/assign @lavalamp @deads2k

[lavalamp]

Why does the cluster scope have a longer SLO?

[jpbetz]

This is to be consistent with how the SLOs for native types are defined (https://github.com/kubernetes/community/blob/master/sig-scalability/slos/api_call_latency.md#api-call-latency-slisslos-details). @wojtek-t, do you know the background?

[wojtek-t]

It's connected with the number of objects - listing objects is basically some constant time + something proportional to number of objects processed. And within single namespace (scope=namespace) we offficially allow much smaller number of objects.

[lavalamp]

Is the table trying to say "return a single object in 50ms, up to 1500 10k objects in 1s, 10000 10k objects in 6 seconds"?

If so, that mathematically doesn't make much sense, the latter implies that a single object must be processed in .6ms?

[jpbetz]

I've inlined the object count details in the webhook SLOs to make it easier to understand. The 50ms for a single object is higher to account for a request serving constant factor. I've added a note.

[lavalamp]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jpbetz, lavalamp, liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• keps/sig-api-machinery/OWNERS [lavalamp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[wojtek-t]

LGTM

[wojtek-t]

It's connected with the number of objects - listing objects is basically some constant time + something proportional to number of objects processed. And within single namespace (scope=namespace) we offficially allow much smaller number of objects.