CIlium ENI mode issues in 1.14

[T-Kukawka]

On cilium release 1.14 in ENI mode pods on worker nodes don't have internal or external communication the first time it boots, once we restart the agent it fixes the pods on those nodes.

At first we thought this was only happening on new nodepool creation, but this also happens when scaling up any node.

We followed another clue without success, when executing "cilium status --verbose" there is a list of Devices and it only shows eth0 on first boot and eth0, eth1, eth2... after restart and we thought that for some reason it was not detecting the new ENIs, unfortunately I this also happens in release 1.13 but nodes don't have any issue in there.

The last thing I want to compare with 1.13 is the route tables that are generated inside the pods and compare between clusters.

Another thing I would like to try, but it might be a bit more difficult is to upgrade to release 1.15 and see if the issue is still happening there on ENI mode.

We haven't found any issue upstream about this issue.

[paurosello]

cilium/cilium#29363 here is the upstream issue, I have added some comments.

[AndiDog]

From CAPA testing issue:

v1.14.5 seems to work. I created a CAPA WC and pods on worker nodes had connectivity to the internet. Also on newly-added worker nodes.

[T-Kukawka]

this is sorted with 1.14.5