Flannel and kube-proxy race for postrouting chain

[bprashanth]

More recent versions of flannel when started with the --ip-masq flag, force a jump to the FLANNEL chain where there's an 'ACCEPT all traffic from node subnet' rule. i.e something like:

$ iptables -t nat -L flannel
$ iptables -t nat -A POSTROUTING -s $NODE_CIDR -j FLANNEL
$ iptables -t nat -A FLANNEL -d $NODE_CIDR -j ACCEPT

$ sudo service kube-proxy start

From: https://github.com/coreos/flannel/blob/master/network/ipmasq.go#L32

Since ACCEPT is a built in target like DROP, it'll stop processing any of the kube service rules. This manifests as a bug that looks like a misconfigured hairpin mode, i.e, if a pod gets loadbalanced to itself when accessing the service dns name, packets get dropped because of a martian source.

@kubernetes/goog-cluster how do we co-ordinate? is it safe to always have kube-proxy prepend?

[ArtfulCoder]

Users are hitting this issue. (referenced issue #22717)

[ArtfulCoder]

Other option could be along these lines: #22717 (comment)

[mwhooker]

we're hitting this, too. Does anyone have a workaround? Tried iptables -t nat -I FLANNEL -s 10.2.0.0/16 -d 10.2.0.0/16 -j MASQUERADE without any luck.

I think we could get by setting --proxy-mode=userspace, but according to the invocation that's qualitatively worse than iptables

[thockin]

Can flannel maybe ONLY install this rule if the policy for POSTROUTING is DROP ? @steveej what do you think? We need to coordinate a bit...

[martynd]

@mwhooker
At a guess, did you try that command based on this comment?

If you are still experiencing the issue, try running . /var/run/flannel/subnet.env && iptables -t nat -I FLANNEL -s $FLANNEL_NETWORK -d $FLANNEL_NETWORK -j MASQUERADE to automatically add the rule using the flannels runtime config.

If that isnt where your env file is stored, you can just use the ip/subnet on the flannel.1 interface as that should match. Just switch out the 10.2.0.0/16's in the original command. Alternatively just edit /var/run/flannel/subnet.env ion the previous command accordingly.

RE the userspace proxy, it would work as a short term bandaid, but it wont scale as well, so id limit its use just to testing.

[mwhooker]

@martynd yes, that's the comment. I looked at /var/run/flannel/subnet.env and found FLANNEL_NETWORK=10.2.0.0/16, so I have to assume the result will be the same as when I tried it earlier.

I can try again but it will have to wait until later this week.

thanks for the thought, but still looking for alternate ideas.

[martynd]

I've come across a similar issue to this one.
Give this a go, it solved the other issue
. /var/run/flannel/subnet.env && iptables -t nat -A POSTROUTING -s $FLANNEL_SUBNET -o docker0 -j ACCEPT

[thockin]

@steveej ping?

[lxpollitt]

cc @tomdee

[steveej]

@thockin

ping?

I'll stick my head together with @tomdee to figure this out.
Some information upfront will be good, this seems also related to installation/configuration (documentation) of kubernetes. What is the official way of setting up flannel with kubernetes? Does kubernetes rely on flannel's --ip-masq behavior in any way?

[bprashanth]

What is the official way of setting up flannel with kubernetes?

There is no "official way", but there is a way that should just work. Run kube-up with NETWORK_PROVIDER=flannel (https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/config-default.sh#L134).

Does kubernetes rely on flannel's --ip-masq behavior in any way?

Someone needs to add that masq rule (flannel-io/flannel#318)

[tomdee]

I don't think it's right that flannel does an ACCEPT. A much better option would be a RETURN. As @thockin points out above that could be an issue if the default for the POSTROUTING table was DROP but as far as I understand it that would be an extremely strange thing to do.

I'm going to put up a PR for flannel to change ACCEPT to RETURN and that should resolve this bug.

[martynd]

Perhaps it could use ACCEPT/RETURN based on the default policy? Probably a better fit that way.

[changleicn]

any updates for this issue？

[tomdee]

@changleicn It looks like I fixed this back in May in flannel, so maybe this isn't an issue any more?

[Hades32]

Happening for me in a fresh "kubeadm init" installation.

My workaround is finding the "RETURN" rule in "POSTROUTING" that "ignores" intra-pod traffic ans simply delete it. Works fine until k8s restarts...

Could someone care to explain why this rule is needed in the first place? Is it just for optimization?

# symptom is like this:

[root@pine64 ~]# docker exec -it 3c93 bash
root@bash-2239139833-xavbo:/# dig grafana
;; reply from unexpected source: 10.244.0.47#53, expected 10.96.0.10#53
# find NAT rule
Chain POSTROUTING (policy ACCEPT 4 packets, 255 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1    22421 1487K KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
2      803 52102 RETURN     all  --  *      *       10.244.0.0/16        10.244.0.0/16
# delete it
iptables -t nat -D POSTROUTING 2

Versions: (system is arm64)

ubeadm/kubernetes-xenial,now 1.5.0-alpha.2-421-a6bea3d79b8bba-00 arm64 [installed]
kubectl/kubernetes-xenial,now 1.4.4-00 arm64 [installed]
kubelet/kubernetes-xenial,now 1.4.4-01 arm64 [installed]
kubernetes-cni/kubernetes-xenial,now 0.3.0.1-07a8a2-00 arm64 [installed]
[root@pine64 ~]# docker ps | grep flann
2b7a51958274        quay.io/coreos/flannel-git:v0.6.1-28-g5dde68d-arm64

[thockin]

@tomdee I don't have a flannel install up right now to poke at, but maybe we can hash out a protocol for this between kube and flannel? Is flannel still installing rules into POSTROUTING unilaterally?

[tomdee]

@thockin sounds good. I'd need to go and check but yes, I think flannel is still installing POSTROUTING rules unilaterally.

[thockin]

Should we plan a slack meeting or hangout or something maybe next week or week after?

…

On Thu, Dec 1, 2016 at 1:10 PM, Tom Denham ***@***.***> wrote: @thockin <https://github.com/thockin> sounds good. I'd need to go and check but yes, I think flannel is still installing POSTROUTING rules unilaterally. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#20391 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVBWmCQZwCeElO5cMceWL5lxAabCuks5rDzeogaJpZM4HQH9o> .

[abemusic]

Has there been any progress on this? I think we may also be hitting this, but not sure. All pod to pod interactions across nodes seem to be fine when using their service, but host to service works until the service gives back a pod running on the host. If that makes sense :)

[phillydogg28]

Bump this again. I am not able to use the service endpoints from inside a pod of the service.

PodA is part of ServiceA.
PodB is part of ServiceB.
In PodA, I can curl ServiceB. In PodB I can curl ServiceA.
In PodA, I *cannot* curl ServiceA. In PodB I *cannot* curl ServiceB.

Should I be able to access ServiceA from inside PodA?

[jl-dos]

Same issue. Any update?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[codebreach]

Hey @phillydogg28, @abemusic and @jl-dos

We are facing the same issue and were wondering if any of you had found a workaround or a resolution?

Happening on GKE with no plugins on both 1.9.4-gke.1 and 1.7.12-gke.1. Node/Master OS is ContainerOS

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[guoshimin]

/remove-lifecycle rotten

[rikatz]

A non related but related thing here is that after changing to a pure Calico stack (only Calico, without flannel) solved my race condition issue in a big environment.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[poidag-zz]

A non related but related thing here is that after changing to a pure Calico stack (only Calico, without flannel) solved my race condition issue in a big environment.

Hi @rikatz Do you have any guidelines on what you did for migration from Flannel to Calico?

[discostur]

Just installed a k8s v1.24.7 cluster with flannel v0.20.1 and the issue still seems to exist:

;; reply from unexpected source: 10.244.0.45#53, expected 10.96.0.10#53
;; reply from unexpected source: 10.244.0.45#53, expected 10.96.0.10#53
;; reply from unexpected source: 10.244.0.45#53, expected 10.96.0.10#53

DNS only works if the coreDNS pod runs on the same node like the client. Pod to pod communication via services is broken ...

[aojea]

but that is a flannel problem, isn't it?

it should be reported on its repo

[discostur]

@aojea you are right it is specific to flannel not to k8s - please forget about my post.

[aojea]

no worries, sometimes is the other way around