Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: [NPM] [Linux] improve iptables version detection and cleanup #3090

Merged
merged 15 commits into from
Nov 7, 2024

Conversation

huntergregory
Copy link
Contributor

@huntergregory huntergregory commented Oct 29, 2024

Reason for Change:

Requirements:

New bootup logic

Policy Manager bootup will:

  1. Detect whether NPM should use nft or legacy iptables.
  2. Clean up NPM state from the other iptables version. Crash if unsuccessful.
  3. Regular bootup: reset NPM chains in the current iptables and prepare chains for NetPols.

Changes

detecting iptables version

Fix #3094.

avoid segmentation fault at high scale

NPM has logic to detect whether kube proxy is using nft or legacy iptables. NPM must use the same version as kube proxy. That logic is based on this article.

We need to check whether kube chains are in mangle table. We used to get the whole table's output with iptables-nft-save. When there exist ~20K rules in iptables-nft, running this command can result in a segmentation fault error.

A fix is to only list the kube chains we mean to check for via iptables-nft -nL <chain>.

fix non-happy path

We used to use legacy tables for any failure to run the command detecting iptables version.

Now we update the logic based on this article to check for either kube chain in NFT. If either is there, use nft. If it's not there or there is any failure, check for either kube chain in legacy. If found, use legacy. Otherwise, default to nft.

clean up other iptables version

clean up nft tables if we use legacy

Previously, we only cleaned up legacy if using nft. Support the opposite.

prevent possibility of failing to clean up other iptables version

Fixes #3088

NPM should crash if it fails to guarantee that it has cleaned up the other iptables enough so that packets won't be accepted/dropped.

The criteria for crashing are 1) failed to delete any jump to AZURE-NPM chain and 2) failed to flush AZURE-NPM chain.

first, try flushing all chains

Used to flush each chain one by one. Now try restoring all flushes. Flush one by one if restore fails.

Testing

All the below tests were performed where NPM v1.5.37 was getting segmentation fault. I added 30K iptables rules in nft, causing the segmentation fault for iptables-nft-save.

detect nft based on hint chain. Then clean up legacy:
I1106 21:12:17.945668       1 chain-management_linux.go:249] first attempt detecting iptables version. looking for hint/canary chain in iptables-nft
I1106 21:12:17.945675       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -L KUBE-IPTABLES-HINT -t mangle -n]
I1106 21:12:18.191043       1 const.go:271] setting iptables to nft
I1106 21:12:18.191076       1 chain-management_linux.go:298] detected nft iptables. cleaning up legacy iptables
I1106 21:12:18.191082       1 const.go:278] setting iptables to legacy
I1106 21:12:18.191089       1 chain-management_linux.go:520] executing iptables command [iptables] with args [-w 60 -D FORWARD -j AZURE-NPM]
I1106 21:12:18.195561       1 chain-management_linux.go:532] didn't delete deprecated jump rule from FORWARD chain to AZURE-NPM chain likely because NPM v1 was not used prior. not able to run iptables command [iptables -w 60 -D FORWARD -j AZURE-NPM]. exit code: 1, output: iptables: No chain/target/match by that name.
I1106 21:12:18.195585       1 chain-management_linux.go:520] executing iptables command [iptables] with args [-w 60 -D FORWARD -j AZURE-NPM -m conntrack --ctstate NEW]
I1106 21:12:18.199208       1 chain-management_linux.go:332] [cleanup] deleted jump rule from FORWARD chain to AZURE-NPM chain
I1106 21:12:18.207708       1 chain-management_linux.go:350] [cleanup] 11 chains to clean up
I1106 21:12:18.207939       1 restore.go:188] running this restore command: [iptables-restore -w 60 -T filter --noflush]
I1106 21:12:18.211648       1 chain-management_linux.go:520] executing iptables command [iptables] with args [-w 60 -X AZURE-NPM]
I1106 21:12:18.216786       1 chain-management_linux.go:520] executing iptables command [iptables] with args [-w 60 -X AZURE-NPM-INGRESS-4043283756]
I1106 21:12:18.221363       1 chain-management_linux.go:520] executing iptables command [iptables] with args [-w 60 -X AZURE-NPM-EGRESS-2775630891]
I1106 21:12:18.226225       1 chain-management_linux.go:520] executing iptables command [iptables] with args [-w 60 -X AZURE-NPM-INGRESS]
I1106 21:12:18.229053       1 chain-management_linux.go:520] executing iptables command [iptables] with args [-w 60 -X AZURE-NPM-EGRESS]
I1106 21:12:18.232292       1 chain-management_linux.go:520] executing iptables command [iptables] with args [-w 60 -X AZURE-NPM-EGRESS-3618314628]
I1106 21:12:18.235060       1 chain-management_linux.go:520] executing iptables command [iptables] with args [-w 60 -X AZURE-NPM-EGRESS-4043283756]
I1106 21:12:18.237865       1 chain-management_linux.go:520] executing iptables command [iptables] with args [-w 60 -X AZURE-NPM-EGRESS-474303581]
I1106 21:12:18.240564       1 chain-management_linux.go:520] executing iptables command [iptables] with args [-w 60 -X AZURE-NPM-INGRESS-474303581]
I1106 21:12:18.243626       1 chain-management_linux.go:520] executing iptables command [iptables] with args [-w 60 -X AZURE-NPM-INGRESS-ALLOW-MARK]
I1106 21:12:18.246427       1 chain-management_linux.go:520] executing iptables command [iptables] with args [-w 60 -X AZURE-NPM-ACCEPT]
I1106 21:12:18.302664       1 chain-management_linux.go:307] cleaned up legacy iptables
I1106 21:12:18.302722       1 const.go:271] setting iptables to nft
I1106 21:12:18.302731       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -D FORWARD -j AZURE-NPM]
I1106 21:12:18.305868       1 chain-management_linux.go:532] in nft tables, didn't delete deprecated jump rule from FORWARD chain to AZURE-NPM chain likely because AZURE-NPM chain doesn't exist. not able to run iptables command [iptables-nft -w 60 -D FORWARD -j AZURE-NPM]. exit code: 2, output: iptables v1.8.4 (nf_tables): Chain 'AZURE-NPM' does not exist
Try `iptables -h' or 'iptables --help' for more information.
I1106 21:12:18.408790       1 chain-management_linux.go:225] found 0 current chains in the default iptables
I1106 21:12:18.408892       1 restore.go:188] running this restore command: [iptables-nft-restore -w 60 -T filter --noflush]
I1106 21:12:18.426571       1 chain-management_linux.go:652] Inserting jump from FORWARD chain to AZURE-NPM chain
I1106 21:12:18.426614       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -I FORWARD 8 -j AZURE-NPM -m conntrack --ctstate NEW]
I1106 21:12:18.503676       1 ipsetmanager_linux.go:147] running this command while resetting ipsets: [ [ipset list --name | grep -q -v azure-npm-]]
I1106 21:12:18.508405       1 ipsetmanager_linux.go:158] running this command while resetting ipsets: [bash -c 'ipset flush && ipset destroy']
detect legacy based on kube canary chain. Then clean up nft:
I1106 21:42:43.354193       1 chain-management_linux.go:249] first attempt detecting iptables version. looking for hint/canary chain in iptables-nft
I1106 21:42:43.354199       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -L FAKE-KUBE-IPTABLES-HINT -t mangle -n]
2024/11/06 21:42:43 [1] error: There was an error running command: [iptables-nft -w 60 -L FAKE-KUBE-IPTABLES-HINT -t mangle -n] Stderr: [exit status 1, # Warning: iptables-legacy tables present, use iptables-legacy to see them
iptables: No chain/target/match by that name.]
I1106 21:42:43.357790       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -L FAKE-KUBE-KUBELET-CANARY -t mangle -n]
2024/11/06 21:42:43 [1] error: There was an error running command: [iptables-nft -w 60 -L FAKE-KUBE-KUBELET-CANARY -t mangle -n] Stderr: [exit status 1, # Warning: iptables-legacy tables present, use iptables-legacy to see them
iptables: No chain/target/match by that name.]
I1106 21:42:43.360393       1 chain-management_linux.go:255] second attempt detecting iptables version. looking for hint/canary chain in iptables-legacy
I1106 21:42:43.360399       1 chain-management_linux.go:520] executing iptables command [iptables] with args [-w 60 -L FAKE-KUBE-IPTABLES-HINT -t mangle -n]
2024/11/06 21:42:43 [1] error: There was an error running command: [iptables -w 60 -L FAKE-KUBE-IPTABLES-HINT -t mangle -n] Stderr: [exit status 1, iptables: No chain/target/match by that name.]
I1106 21:42:43.363502       1 chain-management_linux.go:520] executing iptables command [iptables] with args [-w 60 -L FAKE-KUBE-KUBELET-CANARY -t mangle -n]
I1106 21:42:43.366904       1 const.go:278] setting iptables to legacy
I1106 21:42:43.366930       1 chain-management_linux.go:301] detected legacy iptables. cleaning up nft iptables
I1106 21:42:43.366935       1 const.go:271] setting iptables to nft
I1106 21:42:43.366941       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -D FORWARD -j AZURE-NPM]
I1106 21:42:43.371405       1 chain-management_linux.go:532] probably tried to delete a jump rule that didn't exist in nft tables. not able to run iptables command [iptables-nft -w 60 -D FORWARD -j AZURE-NPM]. exit code: 1, output: iptables: Bad rule (does a matching rule exist in that chain?).
I1106 21:42:43.371421       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -D FORWARD -j AZURE-NPM -m conntrack --ctstate NEW]
I1106 21:42:43.381578       1 chain-management_linux.go:332] [cleanup] deleted jump rule from FORWARD chain to AZURE-NPM chain
I1106 21:42:43.507593       1 chain-management_linux.go:350] [cleanup] 11 chains to clean up
I1106 21:42:43.507855       1 restore.go:188] running this restore command: [iptables-nft-restore -w 60 -T filter --noflush]
I1106 21:42:43.525717       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -X AZURE-NPM]
I1106 21:42:43.545780       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -X AZURE-NPM-EGRESS-3618314628]
I1106 21:42:43.569607       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -X AZURE-NPM-INGRESS-474303581]
I1106 21:42:43.601877       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -X AZURE-NPM-EGRESS-4043283756]
I1106 21:42:43.617641       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -X AZURE-NPM-INGRESS-ALLOW-MARK]
I1106 21:42:43.633793       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -X AZURE-NPM-INGRESS]
I1106 21:42:43.657670       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -X AZURE-NPM-EGRESS]
I1106 21:42:43.674917       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -X AZURE-NPM-ACCEPT]
I1106 21:42:43.697652       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -X AZURE-NPM-EGRESS-474303581]
I1106 21:42:43.709662       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -X AZURE-NPM-EGRESS-2775630891]
I1106 21:42:43.725514       1 chain-management_linux.go:520] executing iptables command [iptables-nft] with args [-w 60 -X AZURE-NPM-INGRESS-4043283756]
I1106 21:42:43.745650       1 chain-management_linux.go:310] cleaned up nft tables
I1106 21:42:43.745672       1 const.go:278] setting iptables to legacy
I1106 21:42:43.745680       1 chain-management_linux.go:520] executing iptables command [iptables] with args [-w 60 -D FORWARD -j AZURE-NPM]
I1106 21:42:43.747959       1 chain-management_linux.go:532] didn't delete deprecated jump rule from FORWARD chain to AZURE-NPM chain likely because AZURE-NPM chain doesn't exist. not able to run iptables command [iptables -w 60 -D FORWARD -j AZURE-NPM]. exit code: 2, output: iptables v1.8.4 (legacy): Couldn't load target `AZURE-NPM':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.
I1106 21:42:43.754704       1 chain-management_linux.go:225] found 0 current chains in the default iptables
I1106 21:42:43.754789       1 restore.go:188] running this restore command: [iptables-restore -w 60 -T filter --noflush]
I1106 21:42:43.771972       1 chain-management_linux.go:652] Inserting jump from FORWARD chain to AZURE-NPM chain
I1106 21:42:43.771997       1 chain-management_linux.go:520] executing iptables command [iptables] with args [-w 60 -I FORWARD -j AZURE-NPM -m conntrack --ctstate NEW]
I1106 21:42:43.774533       1 ipsetmanager_linux.go:147] running this command while resetting ipsets: [ [ipset list --name | grep -q -v azure-npm-]]
I1106 21:42:43.805982       1 ipsetmanager_linux.go:158] running this command while resetting ipsets: [bash -c 'ipset flush && ipset destroy']
crash on failure to clean up other iptables
I1106 03:53:38.124460       1 chain-management_linux.go:386] [cleanup] 2 chains to clean up
I1106 03:53:38.124554       1 chain-management_linux.go:406] sleeping 20 seconds to test cleanup
I1106 03:53:58.125349       1 chain-management_linux.go:410] [cleanup] failed to flush all chains with error: FAKE
I1106 03:53:58.125408       1 chain-management_linux.go:560] executing iptables command [iptables-nft] with args [-w 60 -F AZURE-NPM]
2024/11/06 03:53:58 [1] [cleanup] failed to flush all chains with error: FAKE
I1106 03:53:58.130418       1 chain-management_linux.go:346] cleaned up nft tables
E1106 03:53:58.130476       1 dataplane.go:118] Failed to reset dataplane: Operation [BootupDataplane] failed with error code [999], full cmd [], full error failed to reset policy dataplane: Operation [BootupPolicyManager] failed with error code [999], full cmd [], full error failed to bootup policy manager: failed to cleanup other iptables chains: [cleanup] must crash and retry. failed to delete jump rule and flush chain %s with error: failed to run iptables command [iptables-nft -w 60 -F AZURE-NPM] Stderr: [iptables: No chain/target/match by that name.]. err: [exit status 1]
Error: failed to create dataplane with error Operation [BootupDataplane] failed with error code [999], full cmd [], full error failed to reset policy dataplane: Operation [BootupPolicyManager] failed with error code [999], full cmd [], full error failed to bootup policy manager: failed to cleanup other iptables chains: [cleanup] must crash and retry. failed to delete jump rule and flush chain %s with error: failed to run iptables command [iptables-nft -w 60 -F AZURE-NPM] Stderr: [iptables: No chain/target/match by that name.]. err: [exit status 1]
2024/11/06 03:53:58 [1] error: There was an error running command: [iptables-nft -w 60 -F AZURE-NPM] Stderr: [exit status 1, iptables: No chain/target/match by that name.]
2024/11/06 03:53:58 [1] error: failed to bootup policy manager: failed to cleanup other iptables chains: [cleanup] must crash and retry. failed to delete jump rule and flush chain %!s(MISSING) with error: failed to run iptables command [iptables-nft -w 60 -F AZURE-NPM] Stderr: [iptables: No chain/target/match by that name.]. err: [exit status 1]
2024/11/06 03:53:58 [1] error: failed to create dataplane with error Operation [BootupDataplane] failed with error code [999], full cmd [], full error failed to reset policy dataplane: Operation [BootupPolicyManager] failed with error code [999], full cmd [], full error failed to bootup policy manager: failed to cleanup other iptables chains: [cleanup] must crash and retry. failed to delete jump rule and flush chain %!s(MISSING) with error: failed to run iptables command [iptables-nft -w 60 -F AZURE-NPM] Stderr: [iptables: No chain/target/match by that name.]. err: [exit status 1]

@huntergregory huntergregory added npm Related to NPM. linux labels Oct 29, 2024
@huntergregory huntergregory requested review from a team as code owners October 29, 2024 00:58
go.mod Outdated Show resolved Hide resolved
npm/pkg/dataplane/dataplane_linux.go Outdated Show resolved Hide resolved
npm/pkg/dataplane/dataplane_linux.go Outdated Show resolved Hide resolved
@huntergregory huntergregory force-pushed the huntergregory/npm-legacy branch 2 times, most recently from d3d96ce to 716f6f1 Compare October 31, 2024 01:18
@huntergregory huntergregory changed the title fix: [NPM] [Linux] improve iptables version detection fix: [NPM] [Linux] improve iptables version detection and cleanup Oct 31, 2024
npm/util/sysinfo_linux.go Outdated Show resolved Hide resolved
npm/util/sysinfo_linux.go Outdated Show resolved Hide resolved
Signed-off-by: Hunter Gregory <[email protected]>
@huntergregory
Copy link
Contributor Author

/azp list

@huntergregory
Copy link
Contributor Author

/azp run NPM Conformance Tests

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@huntergregory
Copy link
Contributor Author

/azp run NPM Scale Test

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@huntergregory
Copy link
Contributor Author

/azp run Azure Container Networking PR

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@huntergregory
Copy link
Contributor Author

85dc587 takes the fix found in #3104

@huntergregory huntergregory force-pushed the huntergregory/npm-legacy branch 2 times, most recently from 27a8174 to acc84e3 Compare November 6, 2024 01:17
Signed-off-by: Hunter Gregory <[email protected]>
@huntergregory
Copy link
Contributor Author

/azp run Azure Container Networking PR

@huntergregory
Copy link
Contributor Author

/azp run NPM Conformance Tests

@huntergregory
Copy link
Contributor Author

/azp run NPM Scale Test

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

2 similar comments
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@rbtr rbtr requested a review from timraymond November 6, 2024 20:41
Signed-off-by: Hunter Gregory <[email protected]>
@huntergregory
Copy link
Contributor Author

/azp run Azure Container Networking PR

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@huntergregory
Copy link
Contributor Author

/azp run NPM Conformance Tests

@huntergregory
Copy link
Contributor Author

/azp run NPM Scale Test

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

1 similar comment
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@huntergregory
Copy link
Contributor Author

Just repeated manual testing for this latest version. Confirmed that this version works and that v1.5.37 still seg faults

@huntergregory huntergregory added this pull request to the merge queue Nov 6, 2024
@rbtr
Copy link
Contributor

rbtr commented Nov 6, 2024

should this be against the release/v1.5 branch instead of master? or planning to backport?

@rbtr rbtr added release/latest Change affects latest release train needs-backport Change needs to be backported to previous release trains labels Nov 6, 2024
@huntergregory
Copy link
Contributor Author

Planning to backport to v1.5

Merged via the queue into master with commit 73a22fc Nov 7, 2024
33 checks passed
@huntergregory huntergregory deleted the huntergregory/npm-legacy branch November 7, 2024 00:48
huntergregory added a commit that referenced this pull request Nov 7, 2024
)

* fix: improve iptables version detection

Signed-off-by: Hunter Gregory <[email protected]>

* fix: redo everything and add tests

Signed-off-by: Hunter Gregory <[email protected]>

* fix: address comments

Signed-off-by: Hunter Gregory <[email protected]>

* fix: avoid segfault by only listing one chain

Signed-off-by: Hunter Gregory <[email protected]>

* style: log the kernel version

Signed-off-by: Hunter Gregory <[email protected]>

* style: fix lints

Signed-off-by: Hunter Gregory <[email protected]>

* fix: don't use stale chains. add comments. minor style change

Signed-off-by: Hunter Gregory <[email protected]>

* fix: listing kube chain. get stderr too. also add missing ut

Signed-off-by: Hunter Gregory <[email protected]>

* fix: log messages

Signed-off-by: Hunter Gregory <[email protected]>

* fix: stop checking kernel version. default nft, never crash

Signed-off-by: Hunter Gregory <[email protected]>

* style: fix lint

Signed-off-by: Hunter Gregory <[email protected]>

* style: try fixing gci/gofumpt lint

Signed-off-by: Hunter Gregory <[email protected]>

* test: fix unit tests referencing iptables legacy

Signed-off-by: Hunter Gregory <[email protected]>

* style: fix lint in iptm_test.go

Signed-off-by: Hunter Gregory <[email protected]>

---------

Signed-off-by: Hunter Gregory <[email protected]>
huntergregory added a commit that referenced this pull request Nov 7, 2024
…cleanup (#3110)

* fix: [NPM] [Linux] improve iptables version detection and cleanup (#3090)

* fix: improve iptables version detection

Signed-off-by: Hunter Gregory <[email protected]>

* fix: redo everything and add tests

Signed-off-by: Hunter Gregory <[email protected]>

* fix: address comments

Signed-off-by: Hunter Gregory <[email protected]>

* fix: avoid segfault by only listing one chain

Signed-off-by: Hunter Gregory <[email protected]>

* style: log the kernel version

Signed-off-by: Hunter Gregory <[email protected]>

* style: fix lints

Signed-off-by: Hunter Gregory <[email protected]>

* fix: don't use stale chains. add comments. minor style change

Signed-off-by: Hunter Gregory <[email protected]>

* fix: listing kube chain. get stderr too. also add missing ut

Signed-off-by: Hunter Gregory <[email protected]>

* fix: log messages

Signed-off-by: Hunter Gregory <[email protected]>

* fix: stop checking kernel version. default nft, never crash

Signed-off-by: Hunter Gregory <[email protected]>

* style: fix lint

Signed-off-by: Hunter Gregory <[email protected]>

* style: try fixing gci/gofumpt lint

Signed-off-by: Hunter Gregory <[email protected]>

* test: fix unit tests referencing iptables legacy

Signed-off-by: Hunter Gregory <[email protected]>

* style: fix lint in iptm_test.go

Signed-off-by: Hunter Gregory <[email protected]>

---------

Signed-off-by: Hunter Gregory <[email protected]>

* fix: crash NPM if unable to locate kube chain

Signed-off-by: Hunter Gregory <[email protected]>

---------

Signed-off-by: Hunter Gregory <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
linux needs-backport Change needs to be backported to previous release trains npm Related to NPM. release/latest Change affects latest release train
Projects
None yet
5 participants