core-services/prow/02_config: Rational Azure limits

[wking]

@coverprice recently bumped the Azure limits to:

• Central US:

  • 2000 standardDSv3Family
  • 400 LowPriorityCores (Total Regional Spot vCPUs) rhbz#1888380
  • 100 Public IPs

• Our three other regions:

  • 400 standardDSv3Family
  • 200 LowPriorityCores (Total Regional Spot vCPUs)
  • 100 Public IPs

Surpassing the DSv3 limits leads to errors like:

Code=OperationNotAllowed
Message=Operation could not be completed as it results in exceeding approved standardDSv3Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: centralus, Current Limit: 1000, Current Usage: 1000, Additional Required: 4, (Minimum) New Limit Required: 1004. Submit a request for Quota increase at...

Surpassing LowPriorityCores limits leads to errors like:

Code=OperationNotAllowed
Message=Operation could not be completed as it results in exceeding approved LowPriorityCores quota. Additional details - Deployment Model: Resource Manager, Location: eastus2, Current Limit: 10, Current Usage: 8, Additional Required: 4, (Minimum) New Limit Required: 12. Submit a request for Quota increase at ...

Surpassing public IP limits leads to errors like:

Code=PublicIPCountLimitReached
Message=Cannot create more than 50 public IP addresses for this subscription in this region.

Current docs recommend 40 vCPU per 3-compute cluster with 3 public IP addresses, which makes for the following limits:

• Central US:

  • 2000 standardDSv3Family / 40 = 50 clusters
  • 400 LowPriorityCores (Total Regional Spot vCPUs) / 18 vCPUs per spot test (per @JoelSpeed) = 22 clusters
  • 100 Public IPs / 3 per cluster = 33 clusters

• Our three other regions:

  • 400 standardDSv3Family / 40 = 10 clusters
  • 200 LowPriorityCores (Total Regional Spot vCPUs) / 18 vCPUs per spot test = 11 clusters
  • 100 Public IPs / 3 per cluster = 33 clusters

• Our default limits:

  • 1000 VNets / 1 per cluster = 1000 clusters
  • 65536 network interfaces / 6+ per cluster = 10+k clusters
  • 5000 network security groups / 2+ per cluster = 2+k clusters
  • 1000 network load balancers / 3+ per cluster = 300+ clusters
  • ?? private IP addresses / 7 per cluster = ?? clusters

I dunno what our current private IP quota is. I guess we'll see when we bump into it. Anyhow, limit is 33 clusters for central US (most of our tests do not involve spot instances) and 10 in the other regions. Next thing to bump would be standardDSv3Family in the other regions, followed by public IPs.

[petr-muller]

LGTM but has conflicts

[wking]

Conflicts are with #12842. We can't move forward here until we figure out why the transition to static names didn't work.

[wking]

Rebased around #14285 with 86f548040d -> 1b664cc. That also adds some handwaving around StandardSkuPublicIPCountLimitReached and StaticPublicIPCountLimitReached to the commit message, although I'm still not clear on how OCP cluster consumption of those resources differs from PublicIPCountLimitReached.

[petr-muller]

🤞

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: petr-muller, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• core-services/prow/02_config/OWNERS [petr-muller,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@wking: Updated the following 2 configmaps:

• resources configmap in namespace ci at cluster app.ci using the following files:
  • key boskos.yaml using file core-services/prow/02_config/_boskos.yaml
• resources configmap in namespace ci at cluster api.ci using the following files:
  • key boskos.yaml using file core-services/prow/02_config/_boskos.yaml

Details

In response to this:

@coverprice recently bumped the Azure limits to:

• Central US:

  • 2000 standardDSv3Family
  • 400 LowPriorityCores (Total Regional Spot vCPUs) rhbz#1888380
  • 100 Public IPs

• Our three other regions:

  • 400 standardDSv3Family
  • 200 LowPriorityCores (Total Regional Spot vCPUs)
  • 100 Public IPs

Surpassing the DSv3 limits leads to errors like:

Code=OperationNotAllowed
Message=Operation could not be completed as it results in exceeding approved standardDSv3Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: centralus, Current Limit: 1000, Current Usage: 1000, Additional Required: 4, (Minimum) New Limit Required: 1004. Submit a request for Quota increase at...

Surpassing LowPriorityCores limits leads to errors like:

Code=OperationNotAllowed
Message=Operation could not be completed as it results in exceeding approved LowPriorityCores quota. Additional details - Deployment Model: Resource Manager, Location: eastus2, Current Limit: 10, Current Usage: 8, Additional Required: 4, (Minimum) New Limit Required: 12. Submit a request for Quota increase at ...

Surpassing public IP limits leads to errors like:

Code=PublicIPCountLimitReached
Message=Cannot create more than 50 public IP addresses for this subscription in this region.

Current docs recommend 40 vCPU per 3-compute cluster with 3 public IP addresses, which makes for the following limits:

• Central US:

  • 2000 standardDSv3Family / 40 = 50 clusters
  • 400 LowPriorityCores (Total Regional Spot vCPUs) / 18 vCPUs per spot test (per @JoelSpeed) = 22 clusters
  • 100 Public IPs / 3 per cluster = 33 clusters

• Our three other regions:

  • 400 standardDSv3Family / 40 = 10 clusters
  • 200 LowPriorityCores (Total Regional Spot vCPUs) / 18 vCPUs per spot test = 11 clusters
  • 100 Public IPs / 3 per cluster = 33 clusters

• Our default limits:

  • 1000 VNets / 1 per cluster = 1000 clusters
  • 65536 network interfaces / 6+ per cluster = 10+k clusters
  • 5000 network security groups / 2+ per cluster = 2+k clusters
  • 1000 network load balancers / 3+ per cluster = 300+ clusters
  • ?? private IP addresses / 7 per cluster = ?? clusters

I dunno what our current private IP quota is. I guess we'll see when we bump into it. Anyhow, limit is 33 clusters for central US (most of our tests do not involve spot instances) and 10 in the other regions. Next thing to bump would be standardDSv3Family in the other regions, followed by public IPs.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.