"awsvpc trunking" cause task launch to fail (maybe app mesh related)

[SleeperSmith]

Summary

I have a task definition that has an envoy proxy attached as sidecar. When I turn on awsvpc trunking via CLI, the tasks no longer run after I rotate in new instances.

Description

I am setting up awsvpc trunking. While setting up AWS app mesh it often fails to launch the tasks complaining "RESOURCE:ENI".

However, no tasks start at all after I have turned on the awsvpc trunking with:

aws ecs put-account-setting-default --name awsvpcTrunking --value enabled --region ap-southeast-2

Turning it off again allows the tasks to be launched successfully.

Expected Behavior

That the task would launch successfully on ECS.

Observed Behavior

Task does not launch. No reason given in:
ECS Service Events
ECS Task Detail
No Docker logs available. The task did not even actually start running.

Environment Details

Ubuntu 16.04
ECS agent set up using docker run, as per the documentation on this github.
ECS agent version 1.29.0
ap-southeast-2

Supporting Log Snippets

vpc-branch-eni.log

2019-06-21T07:08:46Z [INFO] Plugin vpc-branch-eni version  executing CNI command.
2019-06-21T07:08:46Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.0 Name: Type:vpc-branch-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[] Domain: Search:[] Options:[]}} TrunkName: TrunkMACAddress:0a:53:03:6d:4c:1c BranchVlanID:1 BranchMACAddress:0a:70:da:f2:35:da BranchIPAddress:10.0.90.125/24 BranchGatewayIPAddress:10.0.90.1 InterfaceType:vlan UserName: BlockIMDS:false}.
2019-06-21T07:08:46Z [INFO] Searching for netns /host/proc/17079/ns/net.
2019-06-21T07:08:46Z [INFO] Creating branch link ens6.1.
2019-06-21T07:08:46Z [INFO] Creating vlan link for branch [ens6.1]: &{LinkAttrs:{Index:0 MTU:0 TxQLen:-1 Name:ens6.1 HardwareAddr:0a:70:da:f2:35:da Flags:0 RawFlags:0 ParentIndex:4 MasterIndex:0 Namespace:<nil> Alias: Statistics:<nil> Promisc:0 Xdp:<nil> EncapType: Protinfo:<nil> OperState:unknown NetNsID:0 NumTxQueues:0 NumRxQueues:0} VlanId:1}
2019-06-21T07:08:46Z [INFO] Moving branch link {linkName:ens6.1 macAddress:0a:70:da:f2:35:da} to netns /host/proc/17079/ns/net.
2019-06-21T07:08:46Z [INFO] Renaming branch link {linkName:ens6.1 macAddress:0a:70:da:f2:35:da} to eth0.
2019-06-21T07:08:46Z [INFO] Setting branch link state up.
2019-06-21T07:08:46Z [ERROR] Failed to set branch link {linkName:eth0 macAddress:0a:70:da:f2:35:da} state: network is down.
2019-06-21T07:08:46Z [ERROR] Failed to setup the link: network is down.
2019-06-21T07:08:46Z [ERROR] CNI command failed: network is down
2019-06-21T07:08:46Z [INFO] Plugin vpc-branch-eni version  executing CNI command.
2019-06-21T07:08:46Z [INFO] Executing DEL with netconfig: &{NetConf:{CNIVersion:0.3.0 Name: Type:vpc-branch-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[] Domain: Search:[] Options:[]}} TrunkName: TrunkMACAddress:0a:53:03:6d:4c:1c BranchVlanID:1 BranchMACAddress:0a:70:da:f2:35:da BranchIPAddress:10.0.90.125/24 BranchGatewayIPAddress:10.0.90.1 InterfaceType:vlan UserName: BlockIMDS:false}.
2019-06-21T07:08:46Z [INFO] Deleting branch link: eth0.
2019-06-21T07:08:46Z [INFO] Plugin aws-appmesh version  executing CNI command.
2019-06-21T07:08:46Z [INFO] Executing DEL with netconfig: &{NetConf:{CNIVersion:0.3.0 Name: Type:aws-appmesh Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[] Domain: Search:[] Options:[]}} IgnoredUID:1337 IgnoredGID: ProxyIngressPort:15000 ProxyEgressPort:15001 AppPorts:80 EgressIgnoredPorts: EgressIgnoredIPv4s:169.254.170.2,169.254.169.254 EgressIgnoredIPv6s: EnableIPv6:false}.
2019-06-21T07:08:46Z [ERROR] Delete the rule in PREROUTING chain failed: running [/sbin/iptables -t nat -D PREROUTING -p tcp -m addrtype ! --src-type LOCAL -j APPMESH_INGRESS --wait]: exit status 2: iptables v1.6.1: Couldn't load target `APPMESH_INGRESS':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.

2019-06-21T07:08:46Z [ERROR] Failed to delete ip rules: running [/sbin/iptables -t nat -D PREROUTING -p tcp -m addrtype ! --src-type LOCAL -j APPMESH_INGRESS --wait]: exit status 2: iptables v1.6.1: Couldn't load target `APPMESH_INGRESS':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.
.
2019-06-21T07:08:46Z [ERROR] CNI command failed: running [/sbin/iptables -t nat -D PREROUTING -p tcp -m addrtype ! --src-type LOCAL -j APPMESH_INGRESS --wait]: exit status 2: iptables v1.6.1: Couldn't load target `APPMESH_INGRESS':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.

aws-appmesh.log

2019-06-21T06:43:59Z [INFO] Plugin aws-appmesh version  executing CNI command.
2019-06-21T06:43:59Z [INFO] Executing DEL with netconfig: &{NetConf:{CNIVersion:0.3.0 Name: Type:aws-appmesh Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[] Domain: Search:[] Options:[]}} IgnoredUID:1337 IgnoredGID: ProxyIngressPort:15000 ProxyEgressPort:15001 AppPorts:80 EgressIgnoredPorts: EgressIgnoredIPv4s:169.254.170.2,169.254.169.254 EgressIgnoredIPv6s: EnableIPv6:false}.
2019-06-21T06:43:59Z [ERROR] Delete the rule in PREROUTING chain failed: running [/sbin/iptables -t nat -D PREROUTING -p tcp -m addrtype ! --src-type LOCAL -j APPMESH_INGRESS --wait]: exit status 2: iptables v1.6.1: Couldn't load target `APPMESH_INGRESS':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.

2019-06-21T06:43:59Z [ERROR] Failed to delete ip rules: running [/sbin/iptables -t nat -D PREROUTING -p tcp -m addrtype ! --src-type LOCAL -j APPMESH_INGRESS --wait]: exit status 2: iptables v1.6.1: Couldn't load target `APPMESH_INGRESS':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.
.
2019-06-21T06:43:59Z [ERROR] CNI command failed: running [/sbin/iptables -t nat -D PREROUTING -p tcp -m addrtype ! --src-type LOCAL -j APPMESH_INGRESS --wait]: exit status 2: iptables v1.6.1: Couldn't load target `APPMESH_INGRESS':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.

Full log capture:
https://cf-templates-1963v75346lrz-us-east-2.s3.us-east-2.amazonaws.com/collect-i-09e138fad7313da6a.tgz
Please note the full log capture is not from the same session as the log pasted here.

[fenxiong]

Hi,
The awsvpcTrunking feature currently only works on ECS optimized AMI, or other Amazon Linux AMI that has ec2-net-utils installed. Apologies for not mentioning this in the documentation. This is because ECS attaches a secondary network interface (known as the trunk network interface) and we rely on ec2-net-utils to bring it up. On the Ubuntu AMI I'm guessing there's no such script available by default, so the interface is not automatically brought up, and awsvpc task will fail to launch on the instance. We will look into whether we can bring up the interface automatically on our side.

[SleeperSmith]

@fenxiong thanks for the reply. There were no issue with attaching and using ENIs in awsvpc networking mode, if I don't turn on trunking.

This only happens specifically when trunking is turned on.

awsvpc networking tasks was discussed here already:
#1083

?

[fenxiong]

There were no issue with attaching and using ENIs in awsvpc networking mode, if I don't turn on trunking.

Yes, the non-trunking version of awsvpc task can work on Ubuntu, just the trunking version can't right now (at least by default), because it currently relies on ec2-net-utils or other similar scripts that detect and bring up newly attached ENI.

We've made a change in #2093 that removes awsvpcTrunking's dependency on such script. The change will be included in our next release. By then you should be able to use the feature on Ubuntu.

[SleeperSmith]

Oh right.

Thank you so much for those changes. Really appreciate it. Really looking forward to the next release.

Regards,

[sharanyad]

This is fixed as part of https://github.com/aws/amazon-ecs-agent/releases/tag/v1.29.1
Closing the issue now.