E2E Test Case TestPodTrafficShaping Failed on Photon 3

[ZhangYW18]

Matrix compatibility test job runs weekly, and the E2E test running under Photon 3 fails on TestPodTrafficShaping test case in recent two weeks:

--- FAIL: TestPodTrafficShaping (197.39s)
    fixtures.go:107: Creating 'antrea-test' K8s Namespace
    fixtures.go:70: Applying Antrea YAML
    fixtures.go:74: Waiting for all Antrea DaemonSet Pods
    fixtures.go:78: Checking CoreDNS deployment
    --- FAIL: TestPodTrafficShaping/limited_by_egress_bandwidth (91.03s)
        bandwidth_test.go:145: Error when waiting for the perftest client Pod: timed out waiting for the condition
        fixtures.go:281: Deleting Pod 'client-a-0'
    --- FAIL: TestPodTrafficShaping/limited_by_ingress_bandwidth (90.02s)
        bandwidth_test.go:145: Error when waiting for the perftest client Pod: timed out waiting for the condition
        fixtures.go:281: Deleting Pod 'client-a-1'
    fixtures.go:167: Exporting test logs to '/var/lib/jenkins/workspace/antrea-weekly-matrix-compatibility-test/IS_MATRIX_TEST/True/K8S_VERSION/v1.18.2/TEST_OS/photon-3/labels/antrea-test-node/antrea-test-logs/TestPodTrafficShaping/beforeTeardown.Nov01-00-40-22'
    fixtures.go:265: Error when exporting kubelet logs: error when running journalctl on Node 'matrix-photon-3-k8s-v1-18-2-build-num-3-mlp85', is it available?
    fixtures.go:274: Deleting 'antrea-test' K8s Namespace

[antoninbas]

Token Bucket Filter (TBF) does not seem to be available in Photon OS: https://github.com/vmware/photon/blob/master/SPECS/linux/config-esx#L1310

However it is required by the bandwidth plugin: https://github.com/containernetworking/plugins/blob/ccd872bd7a834e0948758cc039a90721d817a8ed/plugins/meta/bandwidth/ifb_creator.go#L116

At the very least we should:

• document this issue there: https://github.com/vmware-tanzu/antrea/blob/master/docs/os-issues.md
• disable the test for Photon

[antoninbas]

Maybe confix-esx is actually not the right place to look

@ZhangYW18 could you do the following on one of the Photon Nodes:

• check the kernel config parameters in /boot/config-$(uname -r), in particular CONFIG_NET_SCHED and all the "Queueing/Scheduling" config parameters (CONFIG_NET_SCH_*)
• can the sch_tbf kernel module be loaded with sudo modprobe sch_tbf?
• what is the exact Photon version / kernel version? (output of uname -a)

[antoninbas]

@tnqn pointed out to me that the issue has already been reported to the Photon OS team. It seems that it is actually the ifb kernel module which is missing: https://github.com/vmware/photon/blob/dca7e9c3821f5151d5f44c822ace06aa868a86d2/SPECS/linux/config_x86_64#L2346

So on the Antrea side, we indeed just need to include this information in the OS documentation and disable this test for Photon.

[ZhangYW18]

OK, you can check the TEST_OS env variable to disable this test on Photon. It's valued photon-3 when running matrix tests on Photon 3.0.

• Phonton version and kernal version:

VMware Photon OS 3.0
PHOTON_BUILD_NUMBER=11dd065
Linux 4.19.115-2.ph3

[inforly]

Token Bucket Filter (TBF) does not seem to be available in Photon OS: https://github.com/vmware/photon/blob/master/SPECS/linux/config-esx#L1310

However it is required by the bandwidth plugin: https://github.com/containernetworking/plugins/blob/ccd872bd7a834e0948758cc039a90721d817a8ed/plugins/meta/bandwidth/ifb_creator.go#L116

At the very least we should:

• document this issue there: https://github.com/vmware-tanzu/antrea/blob/master/docs/os-issues.md
• disable the test for Photon

@antoninbas do you know if the Photon OS have a plan to add the missing module? Or do we have some workaround? Could users add it manually?

[antoninbas]

@inforly I believe the Photon team intends to enable IFB in the kernel, but it seems that it hasn't been done yet. Maybe you could reach out to them directly on Github?

I am not Photon expert, but I believe you can modify the config file and build your own kernel.

[inforly]

@inforly I believe the Photon team intends to enable IFB in the kernel, but it seems that it hasn't been done yet. Maybe you could reach out to them directly on Github?

I am not Photon expert, but I believe you can modify the config file and build your own kernel.

Got it, thank you very much！