Skip to content

Providing a way for users to see limits/usage #1314

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sferich888 wants to merge 1 commit into from
Closed

Providing a way for users to see limits/usage #1314

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 102 additions & 7 deletions docs/user/aws/limits.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,29 +5,80 @@ You can find a comprehensive list of the default AWS service limits published he
[AWS Service Limits][service-limits]

Below, we'll identify OpenShift cluster needs and how those impact some of those limits.
We will also use [aws cli][aws-cli] to try and explan
where limits are set (viewable) and how to show your current usage of resources.

Many commands below are based on [Trusted Advisor][trusted-advisor] and often this can be used to print both the limit and
the usage for a service. However to check those limits/usage you need to first identify a check_id.
These check_id's can be pulled from:

```console
$ aws support describe-trusted-advisor-checks --language en --query "checks[].{id: @.id, name: @.name, category: @.category}" --output text
```

Should you need to increase a limit (with any of the following) you need to open a ticket with AWS support, according to
[AWS Service Limits][service-limits].

### Example: Using N. Virginia (us-east-1) to increase a limit for a VPC Elastic IP

To use N. Virginia (us-east-1) for a new cluster, please submit a limit increase for VPC Elastic IPs similar to the
following in the support dashboard (to create more than one cluster, a higher limit will be necessary):

![Increase Elastic IP limit in AWS](images/support_increase_elastic_ip.png)

## S3

There is a default limit of 100 S3 buckets per account. The installation creates a bucket temporarily. Also, the
registry component creates a permanent bucket. This will limit the number of clusters per account to 99 initially. To
support additional clusters, you must open a support case with AWS.

You can not view the limits for S3 buckets, however you can view how many buckets you are using (what the name is),
by running the following:

```console
$ aws s3api --query 'join(`\n`, @.Buckets[].Name)' --output text list-buckets
```

## VPC

Each cluster creates its own VPC. The default limit of VPCs per region is 5 and will allow 5 clusters. To have more
than 5 clusters, you will need to increase this limit.

Each cluster also creates a VPC Gateway Endpoint for a stable connection to S3. The default limit of VPC Gateway
Endpoints per region is 20 and will allow 20 clusters. To have more than 20 clusters, you will need to increase this
To see both the limit and the usage for VPC's you can use:

```console
$ aws support describe-trusted-advisor-check-result --check-id jL7PP0l7J9 --query 'result.flaggedResources[].{service_limit: @.metadata[2], region:@.region, status: @.status, limit: @.metadata[3], usage: @.metadata[4]}' --output text
```

Each cluster also creates a VPC Gateway Endpoint for a stable connection to S3. The default limit of VPC Gateway
Endpoints per region is 20 and will allow 20 clusters. To have more than 20 clusters, you will need to increase this
limit.

You can not view the limits for VPC Endpoints, however you can view how many endpoings you are using (what the name and
creation date and type are), by running the following:

- Be sure to change the region (from us-east-1) if you are using a different region.

```console
$ aws --region us-east-1 ec2 describe-vpc-endpoints --query "VpcEndpoints[].{id:@.VpcEndpointId, type: @.VpcEndpointType, connected_VPC: @.VpcId, connected_service: @.ServiceName, creation_timestamp: @.CreationTimestamp, state: @.State}" --output text
```

## Elastic Network Interfaces (ENI)

The default installation creates 21 + the number of availability zones of ENIs (e.g. 21 + 3 = 24 ENIs for a three-zone cluster).
The default limit per region is 350. Additional ENIs are created for additional machines and elastic load balancers
created by cluster usage and deployed workloads. A service limit increase here may be required to satisfy the needs of
additional clusters and deployed workloads.

You can not view the limits for Elastic Network Interfaces, however you can view how many interfaces you are using
(what the name and type are), by running the following:

- Be sure to change the region (from us-east-1) if you are using a different region.

```console
$ aws ec2 --region us-east-1 describe-network-interfaces --query "NetworkInterfaces[].{associated_vpc: @.VpcId, associated_subnet: @.SubnetId, zone: @.AvailabilityZone, status: @.Status, id: @.NetworkInterfaceId}" --output text
```

## Elastic IP (EIP)

By default, the installer distributes control-plane and compute machines across [all availability zones within a region][availability-zones] to provision the cluster in a highly available configuration.
Expand All @@ -40,23 +91,37 @@ In each private subnet, a separate [NAT Gateway][nat-gateways] is created and re
The default limit of 5 is sufficient for a single cluster, unless you have configured your cluster to use more than five zones.
For multiple clusters, a higher limit will likely be required (and will certainly be required to support more than five clusters, even if they are each single-zone clusters).

### Example: Using North Virginia (us-east-1)
To see both the limit and the usage for EIP's you can use:

North Virginia (us-east-1) has six availablity zones, so a higher limit is required unless you configure your cluster to use fewer zones.
To support the default, all-zone installation, please submit a limit increase for VPC Elastic IPs similar to the following in the support dashboard (to create more than one cluster, a higher limit will be necessary):

![Increase Elastic IP limit in AWS](images/support_increase_elastic_ip.png)
```console
$ aws support describe-trusted-advisor-check-result --check-id lN7RR0l7J9 --query 'result.flaggedResources[].{service_limit: @.metadata[2], region:@.region, status: @.status, limit: @.metadata[3], usage: @.metadata[4]}' --output text
```

## NAT Gateway

The default limit for NAT Gateways is 5 per availability zone. This is sufficient for up to 5 clusters in a dedicated
account. If you intend to create more than 5 clusters, you will need to request an increase to this limit.

You can not view the limits for VPC Endpoints, however you can view how many endpoings you are using (what the name and
creation date and type are), by running the following:

- Be sure to change the region (from us-east-1) if you are using a different region.

```console
$ aws --region us-east-1 ec2 describe-nat-gateways --query "NatGateways[].{id: @.NatGatewayId, state: @.State, associated_subnet_id: @.SubnetId, associated_vpc_id: @.VpcId}" --output text
```

## VPC Gateway

The default limit of VPC Gateways (for S3 access) is 20. Each cluster will create a single S3 gateway endpoint within
the new VPC. If you intend to create more than 20 clusters, you will need to request an increase to this limit.

To see both the limit and the usage for EIP's you can use:

```console
$ aws support describe-trusted-advisor-check-result --check-id kM7QQ0l7J9 --query 'result.flaggedResources[].{service_limit: @.metadata[2], region:@.region, status: @.status, limit: @.metadata[3], usage: @.metadata[4]}' --output text
```

## Security Groups

Each cluster creates distinct security groups. The default limit of 2,500 for new accounts allows for many clusters
Expand All @@ -67,6 +132,13 @@ to be created. The security groups which exist after the default install are:
1. Worker
1. Router/Ingress

You can not view the limits for Security Groups, however you can view how many Security Groups you have defined using,
by running the following:

```console
$ aws ec2 describe-security-groups --query "SecurityGroups[].{id: @.GroupId, name: @.GroupName}" --output text
```

## Instance Limits

By default, a cluster will create:
Expand All @@ -80,15 +152,38 @@ If you intend to start with a higher number of workers, enable autoscaling and l
or a different instance type, please ensure you have the necessary remaining instance count within the instance type's
limit to satisfy the need. If not, please ask AWS to increase the limit via a support case.

You can then confirm the limits and usage (for on Demand Instances) by running the following:

```console
$ aws support describe-trusted-advisor-check-result --check-id 0Xc6LMYG8P --query 'result.flaggedResources[].{service_limit: @.metadata[2], region:@.region, status: @.status, limit: @.metadata[3], usage: @.metadata[4]}' --output text
```

## Elastic Load Balancing (ELB/NLB)

By default, each cluster will create 2 network load balancers for the master API server (1 internal, 1 external) and a
single classic elastic load balancer for the router. Additional Kubernetes LoadBalancer Service objects will create
additional [load balancers][load-balancing].

You can then confirm the limits and usage (for Classic Load Balancers) by running the following:

```console
$ aws support describe-trusted-advisor-check-result --check-id iK7OO0l7J9 --query 'result.flaggedResources[].{service_limit: @.metadata[2], region:@.region, status: @.status, limit: @.metadata[3], usage: @.metadata[4]}' --output text
```

And for Network Load Balancers:

- Be sure to change the region (from us-east-1) if you are using a different region.

```sh
$ aws --region us-east-1 elbv2 describe-account-limits --query "Limits[? @.Name == 'network-load-balancers'].Max" --output text
$ aws --region us-east-1 elbv2 describe-load-balancers --query 'join(`\n`, @.LoadBalancers[].LoadBalancerArn)' --output text | wc -l
```

[availability-zones]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html
[az-map]: https://aws.amazon.com/about-aws/global-infrastructure/
[elastic-ip]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html
[load-balancing]: https://aws.amazon.com/elasticloadbalancing/
[nat-gateways]: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html
[service-limits]: https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html
[aws-cli]: https://aws.amazon.com/cli/
[trusted-advisor]: https://aws.amazon.com/premiumsupport/technology/trusted-advisor/