[v3.30] Fix CNI delete timer to start after acquiring IPAM lock

[sridhartigera]

Cherry-pick history

• Pick onto release-v3.30: Fix CNI delete timer to start after acquiring IPAM lock #11824

Description

Type: Bug fix

Why this should be merged:

This PR fixes a critical bug in the CNI plugin's DELETE operation where the 90-second context timeout was incorrectly started before acquiring the IPAM lock, rather than after. This inconsistency
with ADD operations (AssignIP and AutoAssign) caused DELETE operations to timeout when waiting for the lock during high pod churn.

Problem:

• During concurrent pod deletions, CNI processes queue for the host-wide IPAM lock
• The DELETE operation started its 90-second timer before acquiring the lock
• Lock wait time counted against the timeout budget
• By the time a DELETE acquired the lock, the context was often already expired or near expiration
• This resulted in "client rate limiter Wait returned an error: context deadline exceeded" errors

Solution:
In cni-plugin/pkg/ipamplugin/ipam_plugin.go.

• Lock is now acquired first, then the 90-second timer starts
• This matches the pattern already used in ADD operations
• DELETE operations now get the full 90 seconds for API calls, regardless of lock wait time

Components affected:

• CNI plugin IPAM delete operations (cmdDel function)
• Specifically the ReleaseByHandle flow

Impact:

• Low risk: Only reorders existing code, no logic changes
• High benefit: Eliminates cascading DELETE failures during high pod churn
• No API changes, backward compatible

Related issues/PRs

8822f57

Todos

• Tests - Existing tests should pass; new tests for lock/timer ordering would be beneficial
• Documentation - Code comment added explaining the timer ordering rationale
• Release note - Included below

Release Note

Fix CNI delete timeout to start after IPAM lock acquisition, preventing "context deadline exceeded" failures during high pod churn.

[Copilot]

Pull request overview

This PR cherry-picks a critical bug fix from the main branch to the v3.30 release branch. The fix corrects the timing of the 90-second context timeout in CNI DELETE operations to start after acquiring the IPAM lock, rather than before. This inconsistency with ADD operations was causing DELETE operations to timeout during high pod churn scenarios when multiple CNI processes queued for the host-wide IPAM lock.

Changes:

• Moved context timeout initialization in cmdDel to occur after IPAM lock acquisition, matching the pattern used in ADD operations
• This ensures DELETE operations get the full 90 seconds for API calls, regardless of lock wait time

[Copilot]

The timeout initialization should include an explanatory comment similar to the one in the ADD operations (lines 183-184 and 284-285). This comment explains why the timeout is started after acquiring the lock. Consider adding: "Only start the timeout after we get the lock. When there's a thundering herd of pod deletions, acquiring the lock can take a while."

Suggested change

	ctx := context.Background()
	ctx := context.Background()
	// Only start the timeout after we get the lock. When there's a thundering herd of pod deletions, acquiring the lock can take a while.