for copp test case class scope fixture not working correctly.

[abdosi]

When running copp test case noticed that class based fixture are getting invoked for all protocol type for test_policier.

For eg: copp_testbed gets called for test_class and then again for test_class for 2nd parameter (somehow not for 1st one)

    def test_class1(self, copp_testbed):
        pass

    @pytest.mark.parametrize("protocol", ["ARP0", "ARP1"])
    def test_class(self, copp_testbed, protocol):
        pass

08/02/2022 03:15:49 test_copp.copp_testbed                   L0142 INFO   | Class fixture
08/02/2022 03:15:49 __init__._fixture_generator_decorator    L0075 INFO   | -------------------- fixture copp_testbed setup ends --------------------
08/02/2022 03:15:49 __init__.loganalyzer                     L0048 INFO   | Log analyzer is disabled
08/02/2022 03:15:49 __init__._fixture_func_decorator         L0059 INFO   | -------------------- fixture ignore_expected_loganalyzer_exceptions setup starts --------------------
08/02/2022 03:15:49 __init__._fixture_func_decorator         L0066 INFO   | -------------------- fixture ignore_expected_loganalyzer_exceptions setup ends --------------------
08/02/2022 03:15:49 __init__._log_sep_line                   L0160 INFO   | ==================== copp/test_copp.py::TestCOPP::test_class1[str2-8800-lc1-1] call ====================
08/02/2022 03:15:49 __init__._log_sep_line                   L0160 INFO   | ==================== copp/test_copp.py::TestCOPP::test_class1[str2-8800-lc1-1] teardown ====================
08/02/2022 03:15:49 __init__._log_sep_line                   L0160 INFO   | ==================== copp/test_copp.py::TestCOPP::test_class[str2-8800-lc1-1-ARP0] setup  ====================
08/02/2022 03:15:49 __init__.set_default                     L0049 INFO   | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic
08/02/2022 03:15:49 __init__.check_test_completeness         L0138 INFO   | Test has no defined levels. Continue without test completeness checks
08/02/2022 03:15:49 __init__.loganalyzer                     L0048 INFO   | Log analyzer is disabled
08/02/2022 03:15:49 __init__._fixture_func_decorator         L0059 INFO   | -------------------- fixture ignore_expected_loganalyzer_exceptions setup starts --------------------
08/02/2022 03:15:49 __init__._fixture_func_decorator         L0066 INFO   | -------------------- fixture ignore_expected_loganalyzer_exceptions setup ends --------------------
08/02/2022 03:15:49 __init__._log_sep_line                   L0160 INFO   | ==================== copp/test_copp.py::TestCOPP::test_class[str2-8800-lc1-1-ARP0] call ====================
08/02/2022 03:15:49 __init__._log_sep_line                   L0160 INFO   | ==================== copp/test_copp.py::TestCOPP::test_class[str2-8800-lc1-1-ARP0] teardown ====================
08/02/2022 03:15:49 __init__._log_sep_line                   L0160 INFO   | ==================== copp/test_copp.py::TestCOPP::test_class[str2-8800-lc1-1-ARP1] setup  ====================
08/02/2022 03:15:49 __init__.set_default                     L0049 INFO   | Completeness level not set during test execution. Setting to default level: CompletenessLevel.basic
08/02/2022 03:15:49 __init__.check_test_completeness         L0138 INFO   | Test has no defined levels. Continue without test completeness checks
08/02/2022 03:15:49 __init__._fixture_generator_decorator    L0083 INFO   | -------------------- fixture copp_testbed teardown starts --------------------
08/02/2022 03:15:49 __init__._fixture_generator_decorator    L0092 INFO   | -------------------- fixture copp_testbed teardown ends --------------------
08/02/2022 03:15:49 __init__._fixture_generator_decorator    L0071 INFO   | -------------------- fixture copp_testbed setup starts --------------------
08/02/2022 03:15:49 test_copp.copp_testbed                   L0142 INFO   | Class fixture
08/02/2022 03:15:49 __init__._fixture_generator_decorator    L0075 INFO   | -------------------- fixture copp_testbed setup ends --------------------
08/02/2022 03:15:49 __init__.loganalyzer                     L0048 INFO   | Log analyzer is disabled
08/02/2022 03:15:49 __init__._fixture_func_decorator         L0059 INFO   | -------------------- fixture ignore_expected_loganalyzer_exceptions setup starts --------------------
08/02/2022 03:15:49 __init__._fixture_func_decorator         L0066 INFO   | -------------------- fixture ignore_expected_loganalyzer_exceptions setup ends --------------------
08/02/2022 03:15:49 __init__._log_sep_line                   L0160 INFO   | ==================== copp/test_copp.py::TestCOPP::test_class[str2-8800-lc1-1-ARP1] call ====================
08/02/2022 03:15:49 __init__._log_sep_line                   L0160 INFO   | ==================== copp/test_copp.py::TestCOPP::test_class[str2-8800-lc1-1-ARP1] teardown ====================
08/02/2022 03:15:49 __init__._fixture_generator_decorator    L0083 INFO   | -------------------- fixture copp_testbed teardown starts --------------------
08/02/2022 03:15:49 __init__._fixture_generator_decorator    L0092 INFO   | -------------------- fixture copp_testbed teardown ends --------------------
08/02/2022 03:15:49 __init__._fixture_generator_decorator    L0083 INFO   | -------------------- fixture copy_ptftests_directory teardown starts --------------------

[bingwang-ms]

The issue can be reproed.
It happened when a fixture is parametrized, and the test function is parametrized as well.
For example, the fixture in test_copp.py is parametrized by enum_rand_one_per_hwsku_frontend_hostname
https://github.com/Azure/sonic-mgmt/blob/180d8911dfada2ad79724e8db2c40e7e3c16f02c/tests/copp/test_copp.py#L197-L208

At the same time, the testing function test_policer is parametrized as well
https://github.com/Azure/sonic-mgmt/blob/180d8911dfada2ad79724e8db2c40e7e3c16f02c/tests/copp/test_copp.py#L73-L90

In this scenario, the class level fixtute dut_type is called for each testing function, i.e. test_policer[ARP], test_policer[IP2ME], test_policer[SNMP] and etc.

Currently, I have no idea if it's a bug of pytest or the use of enum_rand_one_per_hwsku_frontend_hostname breaks the infra of pytest. Before the issue is fixed, we have to workaround the issue by aviod the above usage.

[abdosi]

looks like we are hitting similar to this: pytest-dev/pytest#634

this looks more relevant: pytest-dev/pytest#3161

[vcheketx]

@abdosi @bingwang-ms I opened PR to address this issue some time ago - 4961. Could you please take a look at it?