Conformance: Improve test reliability by mazdakn · Pull Request #353 · kubernetes-sigs/network-policy-api

mazdakn · 2026-01-19T23:25:16Z

ClusterNetworkPolicy conformance tests are flaky due to two issues:

Re-use of contexts in some tests.
Lack of a retry in connectivity checker.

This PR improve the quality of conformance tests, and removes sources of flakiness. The proposed changes include:

Replacing many redundant steps like getting or patching pod and cnp resources with helper functions, which helps with test readability.
Using a new context for each operation. Some tests re-used context which led to flaky behavior.
Adding a retry mechanism to PokeServer utility function, as the desired states are expected to happen eventually. This seems to be a known issue and mentioned here: Enhance pokeServer in helper utilities #108

In the Calico implementation of ClusterNetworkPolicy, by applying these changes conformance tests passes consistently.

linux-foundation-easycla · 2026-01-19T23:25:24Z

The committers listed above are authorized under a signed CLA.

✅ login: mazdakn / name: Mazdak Nasab (0246a85, 8137394, c55b4d0)

k8s-ci-robot · 2026-01-19T23:25:25Z

Welcome @mazdakn!

It looks like this is your first PR to kubernetes-sigs/network-policy-api 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/network-policy-api has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2026-01-19T23:25:26Z

Hi @mazdakn. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

netlify · 2026-01-19T23:25:50Z

✅ Deploy Preview for kubernetes-sigs-network-policy-api ready!

Name	Link
🔨 Latest commit	`0246a85`
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-network-policy-api/deploys/698cd8b3b5b38d0008c98b10
😎 Deploy Preview	https://deploy-preview-353--kubernetes-sigs-network-policy-api.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

fasaxc · 2026-01-20T11:08:51Z

/ok-to-test

danwinship · 2026-01-22T22:32:46Z

@mazdakn this is a very large PR and would be much easier to review if it was broken up into multiple smaller commits for the different pieces...

mazdakn · 2026-01-22T22:56:00Z

@danwinship this PR looks like a large PR but honestly the main changes are in the 2 files under conformance/utils directory. Basically:

PokeServer function is updated to retry checking connectivity.
3 utility functions are added to:
- get a pod resource.
- get and patch a clusternetworkpolicy resource.

The tests are only changed to:

Use the new utility functions.
Reflect changes made to the PokeServer function.

I'll try to break this PR into smaller commits, but I'm afraid the result would not be much different. Most likely it would be 2 commits each with roughly half of total LOC changes. This is due to the fact that changes to utility functions needs to be applied to all test files.

npinaeva

Some nits, otherwise nice change!

npinaeva · 2026-02-11T12:31:58Z

+		Namespace: namespace,
+		Name:      name,
+	}, pod)
+	require.NoErrorf(t, err, "unable to fetch %s/%s", namespace, name)


should we say "unable to fetch pod ..."?

Right. Fixed it.

npinaeva · 2026-02-11T12:43:13Z

 }

-// PokeServer is a utility function that checks if the connection from the provided clientPod in clientNamespace towards the targetHost:targetPort
+func PokeServer(t *testing.T, client k8sclient.Interface, kubeConfig *rest.Config, clientNamespace, clientPod, protocol, targetHost string, targetPort int32, timeoutConfig config.TimeoutConfig, shouldConnect bool) {


I think this could use a comment. We should explain that this function waits for the expected connection result, and then checks it once more to make sure the original result wasn't an accident. I think retrying once is a good balance for now between making sure the new condition is stable and keeping the total test time reasonable, but we may want to increase it in the future

Yes, make sense. Add some comments.

I think retrying once is a good balance for now between making sure the new condition is stable and keeping the total test time reasonable, but we may want to increase it in the future

Exactly. I was tempted to assert connectivity more than once, but I thought it can be done later if needed.

mazdakn · 2026-02-11T19:09:14Z

@npinaeva thanks for the review. Addressed your comments. PTAL.

npinaeva · 2026-02-12T15:29:48Z

/lgtm
/approve
Nice improvement, thanks!

k8s-ci-robot · 2026-02-12T15:29:58Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mazdakn, npinaeva

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [npinaeva]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot requested review from aojea and tssurya January 19, 2026 23:25

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jan 19, 2026

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jan 19, 2026

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jan 19, 2026

Conformance: improve test reliablity

8137394

mazdakn force-pushed the improve-conformance-tests branch from 4524c64 to 8137394 Compare January 20, 2026 00:03

mazdakn changed the title ~~Conformance: improve test reliablity~~ Conformance: Improve test reliablity Jan 20, 2026

mazdakn changed the title ~~Conformance: Improve test reliablity~~ Conformance: Improve test reliability Jan 20, 2026

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 20, 2026

npinaeva self-assigned this Feb 11, 2026

npinaeva reviewed Feb 11, 2026

View reviewed changes

Add comments for PokeServer func

c55b4d0

mazdakn requested a review from npinaeva February 11, 2026 19:10

Update comment to satisfy linter

0246a85

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 12, 2026

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 12, 2026

k8s-ci-robot merged commit 412bf65 into kubernetes-sigs:main Feb 12, 2026
8 checks passed

mazdakn mentioned this pull request Feb 12, 2026

CNP: pick conformance improvements and enable it projectcalico/calico#11833

Merged

3 tasks

npinaeva mentioned this pull request Feb 13, 2026

[release-0.1] Conformance: improve test reliablity #359

Merged

This was referenced Feb 25, 2026

[conformance] increase test timeout due to the added retries #362

Merged

Update to the API changes for ClusterNetworkPolicy kubernetes-sigs/kube-network-policies#303

Merged

Conversation

mazdakn commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linux-foundation-easycla Bot commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Jan 19, 2026

Uh oh!

k8s-ci-robot commented Jan 19, 2026

Uh oh!

netlify Bot commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for kubernetes-sigs-network-policy-api ready!

Uh oh!

fasaxc commented Jan 20, 2026

Uh oh!

danwinship commented Jan 22, 2026

Uh oh!

mazdakn commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

npinaeva left a comment

Choose a reason for hiding this comment

Uh oh!

npinaeva Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

mazdakn Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

npinaeva Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

mazdakn Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mazdakn commented Feb 11, 2026

Uh oh!

npinaeva commented Feb 12, 2026

Uh oh!

k8s-ci-robot commented Feb 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mazdakn commented Jan 19, 2026 •

edited

Loading

linux-foundation-easycla Bot commented Jan 19, 2026 •

edited

Loading

netlify Bot commented Jan 19, 2026 •

edited

Loading

mazdakn commented Jan 22, 2026 •

edited

Loading

mazdakn Feb 11, 2026 •

edited

Loading