IPAM: Adds AWS ENI IPv6 Prefix Delegation Support by danehans · Pull Request #32352 · cilium/cilium

danehans · 2024-05-03T21:34:17Z

Previously, ENI prefix delegation support was limited to IPv4.
This PR updates existing structs, methods, etc. to support IPv6
prefix delegation. The overall implementation follows the existing
approach taken for IPv4 prefix delegation.

Although AWS assigns a /80 prefix to an ENI, 64 addresses are the
maximum number of allocatable addresses. This restriction can be user
configurable in the future if needed.

Fixes: #19251
Fixes: #18405

Adds "aws-enable-ipv6-prefix-delegation" configuration flag to enable IPv6 prefix delegation support for ENI IPAM mode in AWS.

Includes commit 3b20d9c from #31145. When #31145 is merged, I will rebase to remove this commit so ignore this commit from your review.

tommyp1ckles · 2024-05-08T22:49:31Z

@danehans The main commit here is quite long, would it be possible to break this down into a couple smaller commits to make the changes easier to review/understand?

daemon/cmd/daemon_main.go

pkg/ipam/node_manager.go

tommyp1ckles · 2024-05-08T22:20:04Z

pkg/azure/ipam/node.go

+func (n *Node) GetMaximumAllocatableIP(family ipam.Family) int {
+	if family == ipam.IPv6 {
+		// not implemented
+		return 0
+	}


It doesn't seem like Azure distinguishes between ip4/ip6 for this limit, so we can probably just leave this as is and make a note for the follow metrics work.

If the family parameter is removed, the Azure Node type will not not implement the IPAM NodeOperations interface.

pkg/alibabacloud/eni/node.go

danehans · 2024-05-10T16:11:44Z

@danehans The main commit here is quite long, would it be possible to break this down into a couple smaller commits to make the changes easier to review/understand?

@tommyp1ckles thanks for reviewing the PR. I originally broke the PR into ~4 commits but the Check if build works for every commit CI job was failing. Let me know if you have any suggestions on the best way to break the PR up into multiple commits that will pass this job.

asauber

Let me know if you have any suggestions on the best way to break the PR up into multiple commits that will pass this job.

As an example of something that could be broken down into its own commit, you have some new logic in ipam.go which determine if a new masquerade option should be set. I'm not entirely sure how self contained this concept is, but it looks like it could be broken down into its own commit, which adds that new logic and produces a version of Cilium that can build and pass tests with the new functionality.

A PR that adds over 3200 lines of new code I would expect to contain at least 6 - 12 commits with detailed descriptions, or the PR will be quite difficult to review.

asauber · 2024-05-10T21:26:45Z

Some example PRs from recent large-size change sets:

https://github.com/cilium/cilium/pull/32336/commits
https://github.com/cilium/cilium/pull/32125/commits

In these case I believe the reviewers also held some conversations to explain the changes to the reviewers.

danehans · 2024-05-13T19:06:21Z

@asauber thanks for the review and guidance. Let me look into refactoring the PR into multiple commits.

danehans · 2024-05-13T19:45:55Z

To verify the e2e functionality of this PR, perform the following steps:

Create an EC2 cluster:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: cilium
  region: us-west-2
kubernetesNetworkConfig:
  ipFamily: IPv6
addons:
  - name: vpc-cni
  - name: coredns
  - name: kube-proxy
iam:
  withOIDC: true
managedNodeGroups:
- name: ng-1
  desiredCapacity: 2
  instanceType: t3a.small
  ssh:
    allow: true
  # privateNetworking: true
  # taint nodes so that application pods are
  # not scheduled/executed until Cilium is deployed.
  # Alternatively, see the note below.
  taints:
   - key: "node.cilium.io/agent-not-ready"
     value: "true"
     effect: "NoExecute"

Note: withOIDC: true is required for IPv6 based on the eksctl docs and I disabled privateNetworking to verify the kernel version of instances using SSH.

Patch VPC CNI:

kubectl -n kube-system patch daemonset aws-node --type='strategic' -p='{"spec":{"template":{"spec":{"nodeSelector":{"io.cilium/aws-node-enabled":"true"}}}}}'

Before installing Cilium, a few dependencies must be met. First, create an IAM policy doc required for the operator to function:

$ cat eni-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ec2:DescribeNetworkInterfaces",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "ec2:ModifyNetworkInterfaceAttribute",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "ec2:AssignPrivateIpAddresses",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "ec2:AttachNetworkInterface",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "ec2:DescribeTags",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "ec2:CreateTags",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "ec2:CreateNetworkInterface",
      "Resource": "*"
    }
  ]
}

Create an IAM policy using the above policy document:

aws iam create-policy --policy-name CiliumENIPolicy --policy-document file://eni-policy.json

Attach the IAM policy to the node instance role created by eksctl. First, extract the node role from the node group:

aws eks describe-nodegroup --cluster-name cilium --nodegroup-name ng-1 | jq -r '.nodegroup.nodeRole'

Attach the IAM policy to the node group:

aws iam attach-role-policy --role-name eksctl-cilium-no-NodeInstanceRole-XmmjKuDUfvxd --policy-arn arn:aws:iam::552234177002:policy/CiliumENIPolicy

Next, the Cilium ipv6NativeRoutingCIDR configuration flag must be set to IPv6 network of the AWS VPC. Get the VPC ID and save it as an environment variable:

export VPC_ID=$(aws ec2 describe-vpcs --filters "Name=tag:alpha.eksctl.io/cluster-name,Values=cilium" --query "Vpcs[].VpcId" --output text)

Get the IPv6 CIDR for the VPC and save it as an environment variable:

export VPC_CIDR=$(aws ec2 describe-vpcs --vpc-ids $VPC_ID --query "Vpcs[].Ipv6CidrBlockAssociationSet[].Ipv6CidrBlock" --output text)

Build and push images:

agent build/tag/push:

ARCH=amd64 DOCKER_DEV_ACCOUNT=docker.io/danehans DOCKER_IMAGE_TAG=ipv6-pd-latest make dev-docker-image

aws-operator build/tag/push:

ARCH=amd64 DOCKER_DEV_ACCOUNT=docker.io/danehans DOCKER_IMAGE_TAG=ipv6-pd-latest make docker-operator-aws-image

Due to the following Cilium daemon error, set the VPC IPv6 CIDR range for each node:

`level=fatal msg="Cannot autocomplete node addresses" error="IPv6 allocation CIDR is not configured. Please specify --ipv6-range" subsys=daemon`

Annotate each node with network.cilium.io/ipv6-pod-cidr=$VPC_CIDR. For example:

kubectl annotate node $MY_K8S_NODE_NAME network.cilium.io/ipv6-pod-cidr=$VPC_CIDR

Install Cilium using Helm:

helm install cilium ./cilium \
  --namespace kube-system \
  --set annotateK8sNode=true \
  --set debug.enabled=true \
  --set eni.enabled=true \
  --set ipam.mode=eni \
  --set ipv4.enabled=false \
  --set ipv6.enabled=true \
  --set eni.awsEnableIPv6PrefixDelegation=true \
  --set egressMasqueradeInterfaces=eth0 \
  --set routingMode=native \
  --set ipv6NativeRoutingCIDR=$VPC_CIDR \
  --set image.override="docker.io/danehans/cilium-dev:ipv6-pd-latest" \
  --set operator.image.override="docker.io/danehans/operator-aws:ipv6-pd-latest" \
  --set operator.replicas=1

If you built your own agent and operator images, replace the image.override and operator.image.override values.

Wait for Cilium to be ready:

cilium status --wait

I used my CiliumCon IPv6 demo to verify functionality. Note that the kernel version for instance t3a.small does not support BIG TCP.

danehans · 2024-05-15T21:43:06Z

/test

youngnick

LGTM from sig-k8s.

adrianmace · 2024-05-21T06:36:49Z

@tommyp1ckles Once this has been approved and merged in, can you advise how frequently you release new minor versions? We're trying to gauge how far off this feature is from being GA. TIA

squeed · 2024-05-27T13:02:02Z

daemon/cmd/daemon_main.go

 		log.Warn("IPSec encrypted overlay is enabled but IPSec is not. Ignoring option.")
 	}

+	// AWS does not support dualstack: https://docs.aws.amazon.com/eks/latest/userguide/cni-ipv6.html


nit: should this "AWS does not support" or "EKS does not support"?

(I definitely don't understand the whole configuration space here, so feel free to tell me I'm wrong).

squeed · 2024-05-27T13:06:49Z

pkg/alibabacloud/eni/node.go

-// GetMinimumAllocatableIPv4 returns the minimum amount of IPv4 addresses that
-// must be allocated to the instance.
-func (n *Node) GetMinimumAllocatableIPv4() int {
+// GetMinimumAllocatableIPv4 returns the minimum amount of addresses that


nit: docblock is wrong here.

squeed · 2024-05-27T13:52:01Z

pkg/ipam/crd.go

-// in the allocation pool. It also returns the number of IPs required and
-// available.
-func (n *nodeStore) hasMinimumIPsInPool(localNodeStore *node.LocalNodeStore) (minimumReached bool, required, numAvailable int) {
+func (n *nodeStore) autoDetectIPv6NativeRoutingCIDR(localNodeStore *node.LocalNodeStore) bool {


This seems copy-pasty from autoDetectIPv4NativeRoutingCIDR, can those be combined?

gandro

Thanks a lot for tackling this! I know this has been a heavily requested feature.

We (@bimmlerd and I) started reviewing the code (noticing some nits), but stopped half way, as we think overall there is a discussion to be had if this is the right approach to implement IPv6 support on top of PD.

I'd rather have us re-think how the prefix delegation control plane works, see comment below for more.

gandro · 2024-05-28T12:02:45Z

daemon/cmd/daemon_main.go

+
+	// IPv6 is not supported for the Azure IPAM plugin.
+	if option.Config.IPAM == ipamOption.IPAMAzure && option.Config.IPv6Enabled() {
+		log.Fatalf("%s ENI IPAM plugin does not support IPv6", ipamOption.IPAMAzure)


Suggested change

log.Fatalf("%s ENI IPAM plugin does not support IPv6", ipamOption.IPAMAzure)

log.Fatalf("%s IPAM mode does not support IPv6", ipamOption.IPAMAzure)

Yes, although Alibaba Cloud does call their network interfaces "ENI"s, Azure does not call them "ENI"s, so best to leave that out here.

gandro · 2024-05-28T12:02:53Z

daemon/cmd/daemon_main.go

+
+	// IPv6 is not supported for the Alibaba IPAM plugin.
+	if option.Config.IPAM == ipamOption.IPAMAlibabaCloud && option.Config.IPv6Enabled() {
+		log.Fatalf("%s ENI IPAM plugin does not support IPv6", ipamOption.IPAMAlibabaCloud)


Suggested change

log.Fatalf("%s ENI IPAM plugin does not support IPv6", ipamOption.IPAMAlibabaCloud)

log.Fatalf("%s ENI IPAM mode does not support IPv6", ipamOption.IPAMAlibabaCloud)

gandro · 2024-05-28T12:10:56Z

daemon/cmd/ipam.go

 	// by the caller.
 	// This will also cause disruption of networking until all endpoints
 	// have been regenerated.
+	primaryFam := ipam.IPv4


This looks off: If we are restoring a router IP from before a cilium-agent restart, allocateDatapathIPs is called with fromFS being a valid IPv6. Thus the family of the allocated IP would be IPv6. We ought to check the family type of fromK8s/fromFS here

gandro · 2024-05-28T12:15:41Z

daemon/cmd/ipam.go

+			if d.healthEndpointRouting, err = parseRoutingInfo(result); err != nil {
+				log.WithError(err).Warn("Unable to allocate health information for ENI")
+			}
+		}


Given the amount of duplication here, have you considered refactoring out the body of allocateHealthIPs into a family-agnostic function similar to allocateDatapathIPs?

gandro · 2024-05-28T12:27:52Z

pkg/aws/eni/node.go

+		if family == ipam.IPv6 {
+			numPrefixes := ip.PrefixCeil(a.IPv6.AvailableForAllocation, option.ENIPDBlockSizeIPv6)
+			if err := n.manager.api.AssignENIIPv6Prefixes(ctx, a.InterfaceID, int32(numPrefixes)); err != nil {
+				return err


In contrast to the isSubnetAtPrefixCapacity logic below (which returns early if there is no error), this will not return early if there is no error. This means that if AssignENIIPv6Prefixes succeeded, we'll still call n.manager.api.AssignPrivateIpAddresses at the end of the function, which I think we don't want.

gandro · 2024-05-28T12:45:44Z

pkg/ipam/option/option.go

+// ENIPDBlockSizeIPv6 is the number of IPs available on an ENI IPv6 prefix. AWS assigns a /80 prefixto an IPv6 ENI
+// which contains 2^(128-80) IP addresses, so the number of IPv6 addresses is scaled down to this value.
+// See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-prefix-eni.html#ec2-prefix-basics for more details
+const ENIPDBlockSizeIPv6 = 64


The downside of limiting each PD to only 64 IPs means that we're having to allocate multiple PDs per node (assuming ~100-300 pods per node), even though a single PD contains more than enough IPs we'll ever need.

While I understand that building on top of the existing PD infrastructure is the easiest way to add IPv6 support, I fear that we're painting ourselves in a corner here. I'd rather have us re-think how the PD control plane is supposed to work. I think it would simplify things a lot if it worked more like cluster-pool or multi-pool IPAM, where the operator assigns a full CIDR to a node, and then node itself then carves out IPs from that CIDR. This also reduces the amount of CiliumNode updates, as we no longer have to update the resource for every IP allocation/release or enumerate every single pod IP in the CiliumNode CRD.

I understand that this is a larger refactor than the approach presented in this PR, but doing these refactors before we add a feature like IPv6 gives us the chance to fix such shortcomings in the controlplane, as any changes after the feature has been merged needs to be backwards compatible.

Agreed. Offering more of the IPv6 space to each ENI (at by least an order of magnitude) allows for a new range of use cases. There are applications of which I'm aware that use an IPv6 address per connection.

AWS Node allocates /80 prefix from the private subnet exclusively for pods. Should the cilium IPAM/ENI not just do the same?

asauber

This is clearly a feature that we need at some point. This is a great first pass at the implementation. There are ways that it could be modified to more fully take advantage of the IPv6 address space, so let's discuss that.

Reviewed for CLI and Config changes, and did one pass looking for logic problems.

I will leave a more detailed IPAM review to others.

asauber · 2024-05-30T14:46:47Z

daemon/cmd/daemon_main.go


+	// AWS does not support dualstack: https://docs.aws.amazon.com/eks/latest/userguide/cni-ipv6.html
+	if option.Config.IPAM == ipamOption.IPAMENI && option.Config.IsDualStack() {
+		log.Fatalf("Both %s and %s options cannot be set since AWS does not support dualstack",


If we decide to change the comment above, this error message should also say "EKS" rather than "AWS".

asauber · 2024-05-30T14:49:35Z

daemon/cmd/daemon_main.go

+
+	// IPv6 is not supported for the Azure IPAM plugin.
+	if option.Config.IPAM == ipamOption.IPAMAzure && option.Config.IPv6Enabled() {
+		log.Fatalf("%s ENI IPAM plugin does not support IPv6", ipamOption.IPAMAzure)


Yes, although Alibaba Cloud does call their network interfaces "ENI"s, Azure does not call them "ENI"s, so best to leave that out here.

asauber · 2024-05-30T15:36:52Z

daemon/cmd/ipam.go

+		if option.Config.EnableIPv6Masquerade &&
+			option.Config.IPAM == ipamOption.IPAMENI &&
+			result != nil &&
+			len(result.CIDRs) > 0 {
+			result.CIDRs = coalesceCIDRs(result.CIDRs)
+		}


Is this case not already handled by the block on line 284?

asauber · 2024-05-30T15:46:47Z

pkg/aws/ec2/ec2.go

 	InvalidParameterValueStr = "InvalidParameterValue"
+
+	// subnetAvailableIPv6Addrs is the available number of addresses for an EC2 IPv6 subnet.
+	subnetAvailableIPv6Addrs = 10000


Where does the number 10000 come from? The /80 (per ENI) mentioned in your PR description would support billions of addresses per subnet.

Since the maximum number of addresses in an IPv4 subnet is 2^24 (10.0.0.0/8), I would expect IPv6 to support at least that many addresses as a maximum, and preferably more.

Per your tests, six m5.large instances could exceed 10,000 addresses.

asauber · 2024-05-30T15:51:55Z

pkg/aws/ec2/ec2.go

 	if iface.MacAddress != nil {
-		eni.MAC = aws.ToString(iface.MacAddress)
+		eni.MAC = *iface.MacAddress
 	}

 	if iface.NetworkInterfaceId != nil {
-		eni.ID = aws.ToString(iface.NetworkInterfaceId)
+		eni.ID = *iface.NetworkInterfaceId
 	}

 	if iface.Description != nil {
-		eni.Description = aws.ToString(iface.Description)
+		eni.Description = *iface.Description
 	}


Is there a reason here to remove the use of aws.ToString? It coalesces nil values to empty strings.

asauber · 2024-05-30T15:54:59Z

pkg/aws/ec2/ec2.go

+	if iface.PrivateIpAddress != nil {
+		eni.IP = *iface.PrivateIpAddress
+		for _, ip := range iface.PrivateIpAddresses {
+			if !usePrimary && ip.Primary != nil && aws.ToBool(ip.Primary) {


nit: aws.ToBool does the nil check

asauber · 2024-05-30T16:13:37Z

pkg/aws/eni/node.go

-			return 0, unableToCreateENI, fmt.Errorf("%s: %w", errUnableToCreateENI, err)
+			scopedLog.WithField(logfields.Node, n.k8sObj.Name).Warning("Subnet might be out of IPv4 prefixes, Cilium will not allocate prefixes on this node anymore")
+			eniID, eni, err = n.manager.api.CreateNetworkInterface(ctx, int32(toAllocate), subnet.ID, desc, securityGroupIDs, false, family)
+			if err != nil {


Moving this error handling inside the isSubnetAtPrefixCapacity(err) block here removes the unableToCreateENI error handling for other cases where err != nil on line 598.

It should remain outside this block.

asauber · 2024-05-30T16:15:23Z

pkg/aws/eni/node.go

-// GetMinimumAllocatableIPv4 returns the minimum amount of IPv4 addresses that
-// must be allocated to the instance.
-func (n *Node) GetMinimumAllocatableIPv4() int {
+// GetMinimumAllocatableIPv4 returns the minimum amount of IPa that must be allocated


asauber · 2024-05-30T16:18:36Z

pkg/ipam/option/option.go

+// ENIPDBlockSizeIPv6 is the number of IPs available on an ENI IPv6 prefix. AWS assigns a /80 prefixto an IPv6 ENI
+// which contains 2^(128-80) IP addresses, so the number of IPv6 addresses is scaled down to this value.
+// See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-prefix-eni.html#ec2-prefix-basics for more details
+const ENIPDBlockSizeIPv6 = 64


Agreed. Offering more of the IPv6 space to each ENI (at by least an order of magnitude) allows for a new range of use cases. There are applications of which I'm aware that use an IPv6 address per connection.

asauber · 2024-05-30T16:24:38Z

pkg/ipam/types/types.go

+	// IPv6MaxAboveWatermark is the maximum number of IPv6 addresses to allocate
+	// beyond the addresses needed to reach the IPv6PreAllocate watermark.
+	// Going above the watermark can help reduce the number of API calls to
+	// allocate IPs, e.g. when a new ENI is allocated, as many secondary
+	// IPv6 adresses as possible are allocated. Limiting the amount can help reduce
+	// waste of IPs.
+	//
+	// +kubebuilder:validation:Minimum=0
+	IPv6MaxAboveWatermark int `json:"ipv6-max-above-watermark,omitempty"`


This concept shouldn't be necessary with the IPv6 address space.

github-actions · 2024-07-08T01:50:41Z

This pull request has been automatically marked as stale because it
has not had recent activity. It will be closed if no further activity
occurs. Thank you for your contributions.

joestringer · 2024-07-11T23:36:19Z

Heads up that #31145 will need to be also added back into this effort if someone revives this PR.

sylr · 2024-07-17T20:25:15Z

@joestringer I talked to @danehans and he is not going to be able to move forward with this PR. Do you think a maintainer could take over his work ?

github-actions · 2024-08-17T01:49:40Z

This pull request has been automatically marked as stale because it
has not had recent activity. It will be closed if no further activity
occurs. Thank you for your contributions.

github-actions · 2024-08-31T01:54:45Z

This pull request has not seen any activity since it was marked stale.
Closing.

wfclark5 · 2024-09-07T19:51:26Z

Hello! Is the work ever going to merged in the future?

sylr · 2024-09-25T10:22:08Z

@joestringer I talked to @danehans and he is not going to be able to move forward with this PR. Do you think a maintainer could take over his work ?

@joestringer is there anything you or one of cilium's maintainer could do ?

joestringer · 2024-09-25T17:58:55Z

Cilium relies on the contributions from volunteers in the community to implement functionality that they find important. I would encourage contributors to chat with one another, collaborate on what they see as the solutions, and make proposals to move this forward step by step. Personally I don't have the bandwidth to tackle this at the moment.

francisko0 · 2024-10-03T08:40:28Z

Hi, I'm going to be working on this in a few months, I could do some PR on @danehans repo

Defman · 2024-12-09T14:20:09Z

What work does the @danehans need to be acceptable? The above review looks to be mostly minor requests?

gandro · 2024-12-09T14:27:45Z

I believe #32352 (comment) is the biggest concern which I think will need some serious engineering work to get resolved.

danehans requested review from a team as code owners May 3, 2024 21:34

danehans requested a review from youngnick May 3, 2024 21:34

maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label May 3, 2024

danehans requested review from asauber, christarazi, squeed and tommyp1ckles May 3, 2024 21:34

danehans force-pushed the issue_19251_aio branch 5 times, most recently from 9c8e75c to 984e02a Compare May 8, 2024 00:02

tommyp1ckles reviewed May 8, 2024

View reviewed changes

danehans force-pushed the issue_19251_aio branch from 984e02a to f353505 Compare May 10, 2024 16:44

asauber reviewed May 10, 2024

View reviewed changes

asauber self-requested a review May 10, 2024 21:24

maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label May 14, 2024

youngnick approved these changes May 16, 2024

View reviewed changes

squeed reviewed May 27, 2024

View reviewed changes

squeed requested a review from gandro May 27, 2024 13:04

squeed reviewed May 27, 2024

View reviewed changes

gandro requested changes May 28, 2024

View reviewed changes

asauber requested changes May 30, 2024

View reviewed changes

christarazi marked this pull request as draft June 7, 2024 18:12

christarazi mentioned this pull request Jun 26, 2024

Revert "IPAM: Adds AWS IPv6 Prefix Delegation Config Option" #33394

Merged

github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Jul 8, 2024

github-actions bot removed the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Jul 12, 2024

github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Aug 17, 2024

github-actions bot closed this Aug 31, 2024

This comment was marked as off-topic.

Sign in to view

This was referenced Feb 27, 2026

🌱 CNCF mission generation 2026-02-27 kubestellar/console-kb#6

Closed

🌱 CNCF mission generation 2026-02-27 kubestellar/console-kb#11

Merged

	log.Fatalf("%s ENI IPAM plugin does not support IPv6", ipamOption.IPAMAzure)
	log.Fatalf("%s IPAM mode does not support IPv6", ipamOption.IPAMAzure)

	log.Fatalf("%s ENI IPAM plugin does not support IPv6", ipamOption.IPAMAlibabaCloud)
	log.Fatalf("%s ENI IPAM mode does not support IPv6", ipamOption.IPAMAlibabaCloud)

Conversation

danehans commented May 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tommyp1ckles commented May 8, 2024

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

danehans commented May 10, 2024

Uh oh!

asauber left a comment

Choose a reason for hiding this comment

Uh oh!

asauber commented May 10, 2024

Uh oh!

danehans commented May 13, 2024

Uh oh!

danehans commented May 13, 2024

Uh oh!

danehans commented May 15, 2024

Uh oh!

youngnick left a comment

Choose a reason for hiding this comment

Uh oh!

adrianmace commented May 21, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gandro left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Defman Jan 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asauber left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

danehans commented May 3, 2024 •

edited

Loading

Defman Jan 1, 2025 •

edited

Loading