VPC with private subnet, using Nat Instance not connected to the internet.

[icecold21]

I created a VPC Stack using AWS CDK, which has 3 public subnets, 3 private subnets, and 3 isolated subnets. And use natInstance as an internet provider. But EC2 instances in private subnet were not able to connect to the internet.

Reproduction Steps

This AMI: ami-01514bb1776d5c018 is ap-southeast-1 ami for NatInstance.

import { Construct, Stack, StackProps } from "@aws-cdk/core";
import { Vpc, SubnetType, GenericLinuxImage, NatProvider, InstanceType } from "@aws-cdk/aws-ec2";

interface EnvProps {
}

export class VpcStackTestNat extends Stack {
  public readonly vpc: Vpc;

  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);
    this.vpc = new Vpc(this, 'VpcStackTestNat', {
      maxAzs: 3,
      cidr: '10.0.0.0/16',
      enableDnsHostnames: true,
      enableDnsSupport: true,
      natGatewayProvider: NatProvider.instance({
        instanceType: new InstanceType('t3.nano'),
        machineImage: new GenericLinuxImage({
          'ap-southeast-1': 'ami-01514bb1776d5c018'
        }),
        keyName: 'nat-instance-key',
      }),
      natGateways: 1,
      subnetConfiguration: [
        {
          subnetType: SubnetType.PUBLIC,
          name: 'Ingress',
          cidrMask: 22,
        },
        {
          cidrMask: 22,
          name: 'Application',
          subnetType: SubnetType.PRIVATE,
        },
        {
          cidrMask: 22,
          name: 'Database',
          subnetType: SubnetType.ISOLATED,
        }
      ]
    });
  }
}

Error Log

Ping not success
[ec2-user@ip-10-0-16-205 ~]$ ping google.com
PING google.com (74.125.68.138) 56(84) bytes of data.
----- empty -----

Environment

• CLI Version :1.31.0 (build 8f3ac79)
• Framework Version: 1.31.0
• OS : Mac OS
• Language : TypeScript

Other

---

This is 🐛 Bug Report

[rix0rrr]

Thanks for the report. Can you help us debug a little more?

• Did you figure out what was wrong with the setup? Is it routing tables?
• How did you SSH to your private instance?
• Do you have a completer reproducing sample?

Cheers

[icecold21]

Hi,

• I haven't found out what issue with the setup, currently scheduling time with AWS support to see what's the issue.

• Other than NAT instance, I created 1 ec2 in private subnet, and 1 ec2 in public subnet, then I ssh to the private subnet through the public one.

• Here is the code i use to create EKS stack, which has no access to internet, but I confirm that plain EC instance created via AWS console also have no access to internet.

import { Construct, Stack, StackProps, Duration } from "@aws-cdk/core";
import { Vpc, SubnetType, InstanceType, Peer, Port } from "@aws-cdk/aws-ec2";
import { Cluster } from "@aws-cdk/aws-eks";
import { Role, AccountRootPrincipal } from "@aws-cdk/aws-iam";


export class MainEksStack extends Stack {
  public readonly cluster: Cluster;

  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    const vpc = Vpc.fromLookup(scope, 'mainVpc', {
      vpcId: "vpc-id",
      vpcName: "vpc-name",
    }) ;

    this.cluster = new Cluster(this, "MainEksCluster", {
      vpc: vpc,
      defaultCapacity: 0,
      kubectlEnabled: true,
      clusterName: "MainEksCluster"
    });

    const mastersRole = new Role(this, 'MainEksAdminRole', { assumedBy: new AccountRootPrincipal() });
    this.cluster.awsAuth.addMastersRole(mastersRole);

    const asg = this.cluster.addCapacity('MainEksPrivateNodes', {
      instanceType: new InstanceType('m5.xlarge'),
      desiredCapacity: 2,
      minCapacity: 2,
      maxCapacity: 2,
      vpcSubnets: { subnetType: SubnetType.PRIVATE },
      keyName: 'private-eks-nodes',
      rollingUpdateConfiguration: {
      }
    })

    asg.scaleOnCpuUtilization('up', {
      targetUtilizationPercent: 80
    });
  }
}

[rix0rrr]

The problem has to be in the setup of your VPC.

When you do this:

    const vpc = Vpc.fromLookup(scope, 'mainVpc', {
      vpcId: "vpc-id",
      vpcName: "vpc-name",
    }) ;

Is that the same VPC as the one created in your initial post?

    this.vpc = new Vpc(this, 'VpcStackTestNat', {
      maxAzs: 3,
      cidr: '10.0.0.0/16',
      enableDnsHostnames: true,
      enableDnsSupport: true,
      natGatewayProvider: NatProvider.instance({
        // ...etc...

?

[icecold21]

yes, it is the same VPC. still waiting for a meeting with AWS Support. I will get back once I know the problem.

[icecold21]

It turns out the default security group of the NatInstance only allows TCP Connection.
While I did not notice that ping is actually an ICMP traffic. do you think the default security group for NatInstance should allow ICMP connection? because I believe a lot of people are actually testing internet connection by ping.