Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[backport] fix: [NPM] [Linux] improve iptables version detection and cleanup #3110

Merged
merged 2 commits into from
Nov 7, 2024

Conversation

huntergregory
Copy link
Contributor

Backport #3090 and add a fix per cd4cfcf

cd4cfcf causes NPM to crash if it fails to detect which iptables version kube-proxy is using (whether it fails since the kube chains don't exist or for any other reason).

Without cd4cfcf, #3090 would introduce an issue where NPM could use nft when it should use legacy (if iptables -nL failed for whatever reason, or if kube-proxy somehow hadn't installed its chains yet).

)

* fix: improve iptables version detection

Signed-off-by: Hunter Gregory <[email protected]>

* fix: redo everything and add tests

Signed-off-by: Hunter Gregory <[email protected]>

* fix: address comments

Signed-off-by: Hunter Gregory <[email protected]>

* fix: avoid segfault by only listing one chain

Signed-off-by: Hunter Gregory <[email protected]>

* style: log the kernel version

Signed-off-by: Hunter Gregory <[email protected]>

* style: fix lints

Signed-off-by: Hunter Gregory <[email protected]>

* fix: don't use stale chains. add comments. minor style change

Signed-off-by: Hunter Gregory <[email protected]>

* fix: listing kube chain. get stderr too. also add missing ut

Signed-off-by: Hunter Gregory <[email protected]>

* fix: log messages

Signed-off-by: Hunter Gregory <[email protected]>

* fix: stop checking kernel version. default nft, never crash

Signed-off-by: Hunter Gregory <[email protected]>

* style: fix lint

Signed-off-by: Hunter Gregory <[email protected]>

* style: try fixing gci/gofumpt lint

Signed-off-by: Hunter Gregory <[email protected]>

* test: fix unit tests referencing iptables legacy

Signed-off-by: Hunter Gregory <[email protected]>

* style: fix lint in iptm_test.go

Signed-off-by: Hunter Gregory <[email protected]>

---------

Signed-off-by: Hunter Gregory <[email protected]>
@huntergregory huntergregory added npm Related to NPM. linux labels Nov 7, 2024
@huntergregory huntergregory requested a review from a team as a code owner November 7, 2024 02:42
@huntergregory huntergregory requested a review from matmerr November 7, 2024 02:42
@huntergregory huntergregory changed the title Huntergregory/backport npm legacy [backport] fix: [NPM] [Linux] improve iptables version detection and cleanup Nov 7, 2024
@huntergregory
Copy link
Contributor Author

/azp run Azure Container Networking PR

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@vakalapa vakalapa enabled auto-merge November 7, 2024 02:47
@huntergregory huntergregory merged commit 957f147 into release/v1.5 Nov 7, 2024
15 of 18 checks passed
@huntergregory huntergregory deleted the huntergregory/backport-npm-legacy branch November 7, 2024 02:49
@huntergregory
Copy link
Contributor Author

Manual test of crash logic:

I1107 02:53:59.943767       1 chain-management_linux.go:253] first attempt detecting iptables version. looking for hint/canary chain in iptables-nft
I1107 02:53:59.943774       1 chain-management_linux.go:523] executing iptables command [iptables-nft] with args [-w 60 -L FAKE-KUBE-IPTABLES-HINT -t mangle -n]
I1107 02:53:59.946813       1 chain-management_linux.go:523] executing iptables command [iptables-nft] with args [-w 60 -L FAKE-KUBE-KUBELET-CANARY -t mangle -n]
2024/11/07 02:53:59 [1] error: There was an error running command: [iptables-nft -w 60 -L FAKE-KUBE-IPTABLES-HINT -t mangle -n] Stderr: [exit status 1, # Warning: iptables-legacy tables present, use iptables-legacy to see them
iptables: No chain/target/match by that name.]
2024/11/07 02:53:59 [1] error: There was an error running command: [iptables-nft -w 60 -L FAKE-KUBE-KUBELET-CANARY -t mangle -n] Stderr: [exit status 1, # Warning: iptables-legacy tables present, use iptables-legacy to see them
iptables: No chain/target/match by that name.]
I1107 02:53:59.948626       1 chain-management_linux.go:259] second attempt detecting iptables version. looking for hint/canary chain in iptables-legacy
I1107 02:53:59.948632       1 chain-management_linux.go:523] executing iptables command [iptables] with args [-w 60 -L FAKE-KUBE-IPTABLES-HINT -t mangle -n]
2024/11/07 02:53:59 [1] error: There was an error running command: [iptables -w 60 -L FAKE-KUBE-IPTABLES-HINT -t mangle -n] Stderr: [exit status 1, iptables: No chain/target/match by that name.]
I1107 02:53:59.952128       1 chain-management_linux.go:523] executing iptables command [iptables] with args [-w 60 -L FAKE-KUBE-KUBELET-CANARY -t mangle -n]
E1107 02:53:59.955719       1 dataplane.go:118] Failed to reset dataplane: Operation [BootupDataplane] failed with error code [999], full cmd [], full error failed to reset policy dataplane: Operation [BootupPolicyManager] failed with error code [999], full cmd [], full error failed to bootup policy manager: failed to detect iptables version: unable to locate which iptables version kube proxy is using
Error: failed to create dataplane with error Operation [BootupDataplane] failed with error code [999], full cmd [], full error failed to reset policy dataplane: Operation [BootupPolicyManager] failed with error code [999], full cmd [], full error failed to bootup policy manager: failed to detect iptables version: unable to locate which iptables version kube proxy is using
Usage:
  azure-npm start [flags]

Flags:
  -h, --help                help for start
      --kubeconfig string   path to kubeconfig

Error: failed to create dataplane with error Operation [BootupDataplane] failed with error code [999], full cmd [], full error failed to reset policy dataplane: Operation [BootupPolicyManager] failed with error code [999], full cmd [], full error failed to bootup policy manager: failed to detect iptables version: unable to locate which iptables version kube proxy is using
2024/11/07 02:53:59 [1] error: There was an error running command: [iptables -w 60 -L FAKE-KUBE-KUBELET-CANARY -t mangle -n] Stderr: [exit status 1, iptables: No chain/target/match by that name.]
2024/11/07 02:53:59 [1] error: failed to bootup policy manager: failed to detect iptables version: unable to locate which iptables version kube proxy is using
2024/11/07 02:53:59 [1] error: failed to create dataplane with error Operation [BootupDataplane] failed with error code [999], full cmd [], full error failed to reset policy dataplane: Operation [BootupPolicyManager] failed with error code [999], full cmd [], full error failed to bootup policy manager: failed to detect iptables version: unable to locate which iptables version kube proxy is using

@huntergregory
Copy link
Contributor Author

Other detection/cleanup logic works for this NPM image built on release/v1.5 branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
linux npm Related to NPM.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants