[chassis][202405]: snmp service enters failed state during test_pfcwd_basic_single_lossless_prio_service_restart test #20637
Labels
Triaged
this issue has been triaged
Comments
|
Hi @abdosi , @stepanblyschak As per comments in #2790, snmp restart is expected. Is that condition still true? Please suggest. |
|
Can we handle this in the test to reset the snmp service for short term. For long term fix, we need to investigate if the snmp services needs to restart on swss restart |
8 tasks
yejianquan
pushed a commit
to sonic-net/sonic-mgmt
that referenced
this issue
Nov 20, 2024
Description of PR Summary: The pfcwd_basic service-restart cases keep failing due to: sonic-net/sonic-buildimage#20637 The ask is not to restart swss multiple times without doing a config reload in between. So in this PR: we are doing config-reload for every iteration of the test The swss restart is done only once in one DUT. The asic is randomly picked, and the swss of that ASIC is restarted instead of doing the restart for all asics. Also added checks to make sure the services, interfaces and bgp are up before proceding with the ixia traffic. Approach What is the motivation for this PR? The issue: sonic-net/sonic-buildimage#20637 How did you do it? Pls see the description. How did you verify/test it? Ran it on my TB. =========================================================================================================================== PASSES =========================================================================================================================== ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss] _____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-False-swss] ____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-True-swss] _____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-False-swss] ____________________________________________________________________________________ ----------------------------------------------------------------------------- generated xml file: /run_logs/ixia/restart-service/2024-11-14-00-05-11/tr_2024-11-14-00-05-11.xml ------------------------------------------------------------------------------ INFO:root:Can not get Allure report URL. Please check logs ------------------------------------------------------------------------------------------------------------------- live log sessionfinish ------------------------------------------------------------------------------------------------------------------- 01:31:34 __init__.pytest_terminal_summary L0067 INFO | Can not get Allure report URL. Please check logs ================================================================================================================== short test summary info =================================================================================================================== PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-False-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-False-swss] ========================================================================================================= 4 passed, 7 warnings in 5180.68s (1:26:20) ========================================================================================================= sonic@ixia-sonic-mgmt-whitebox:/data/tests$ -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =========================================================================================================================== PASSES =========================================================================================================================== ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss] _____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-False-swss] ____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-True-swss] _____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-False-swss] ____________________________________________________________________________________ ---------------------------------------------------------------------------- generated xml file: /run_logs/ixia/restart-service-2/2024-11-14-02-47-47/tr_2024-11-14-02-47-47.xml ----------------------------------------------------------------------------- INFO:root:Can not get Allure report URL. Please check logs ------------------------------------------------------------------------------------------------------------------- live log sessionfinish ------------------------------------------------------------------------------------------------------------------- 04:14:03 __init__.pytest_terminal_summary L0067 INFO | Can not get Allure report URL. Please check logs ================================================================================================================== short test summary info =================================================================================================================== PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-False-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-False-swss] ========================================================================================================= 4 passed, 7 warnings in 5173.22s (1:26:13) ========================================================================================================= sonic@ixia-sonic-mgmt-whitebox:/data/tests$ =========================================================================================================================== PASSES =========================================================================================================================== ____________________________________________________________________________________ test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-True-swss] ____________________________________________________________________________________ ___________________________________________________________________________________ test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-False-swss] ____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-True-swss] ____________________________________________________________________________________ ___________________________________________________________________________________ test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-False-swss] ____________________________________________________________________________________ ---------------------------------------------------------------------------- generated xml file: /run_logs/ixia/restart-service-2/2024-11-14-06-39-15/tr_2024-11-14-06-39-15.xml ----------------------------------------------------------------------------- INFO:root:Can not get Allure report URL. Please check logs ------------------------------------------------------------------------------------------------------------------- live log sessionfinish ------------------------------------------------------------------------------------------------------------------- 08:10:42 __init__.pytest_terminal_summary L0067 INFO | Can not get Allure report URL. Please check logs ================================================================================================================== short test summary info =================================================================================================================== PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-False-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-False-swss] ========================================================================================================= 4 passed, 7 warnings in 5484.86s (1:31:24) ========================================================================================================= sonic@ixia-sonic-mgmt-whitebox:/data/tests$ co-authorized by: [email protected]
mssonicbld
pushed a commit
to mssonicbld/sonic-mgmt
that referenced
this issue
Nov 20, 2024
Description of PR Summary: The pfcwd_basic service-restart cases keep failing due to: sonic-net/sonic-buildimage#20637 The ask is not to restart swss multiple times without doing a config reload in between. So in this PR: we are doing config-reload for every iteration of the test The swss restart is done only once in one DUT. The asic is randomly picked, and the swss of that ASIC is restarted instead of doing the restart for all asics. Also added checks to make sure the services, interfaces and bgp are up before proceding with the ixia traffic. Approach What is the motivation for this PR? The issue: sonic-net/sonic-buildimage#20637 How did you do it? Pls see the description. How did you verify/test it? Ran it on my TB. =========================================================================================================================== PASSES =========================================================================================================================== ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss] _____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-False-swss] ____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-True-swss] _____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-False-swss] ____________________________________________________________________________________ ----------------------------------------------------------------------------- generated xml file: /run_logs/ixia/restart-service/2024-11-14-00-05-11/tr_2024-11-14-00-05-11.xml ------------------------------------------------------------------------------ INFO:root:Can not get Allure report URL. Please check logs ------------------------------------------------------------------------------------------------------------------- live log sessionfinish ------------------------------------------------------------------------------------------------------------------- 01:31:34 __init__.pytest_terminal_summary L0067 INFO | Can not get Allure report URL. Please check logs ================================================================================================================== short test summary info =================================================================================================================== PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-False-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-False-swss] ========================================================================================================= 4 passed, 7 warnings in 5180.68s (1:26:20) ========================================================================================================= sonic@ixia-sonic-mgmt-whitebox:/data/tests$ -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =========================================================================================================================== PASSES =========================================================================================================================== ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss] _____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-False-swss] ____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-True-swss] _____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-False-swss] ____________________________________________________________________________________ ---------------------------------------------------------------------------- generated xml file: /run_logs/ixia/restart-service-2/2024-11-14-02-47-47/tr_2024-11-14-02-47-47.xml ----------------------------------------------------------------------------- INFO:root:Can not get Allure report URL. Please check logs ------------------------------------------------------------------------------------------------------------------- live log sessionfinish ------------------------------------------------------------------------------------------------------------------- 04:14:03 __init__.pytest_terminal_summary L0067 INFO | Can not get Allure report URL. Please check logs ================================================================================================================== short test summary info =================================================================================================================== PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-False-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-False-swss] ========================================================================================================= 4 passed, 7 warnings in 5173.22s (1:26:13) ========================================================================================================= sonic@ixia-sonic-mgmt-whitebox:/data/tests$ =========================================================================================================================== PASSES =========================================================================================================================== ____________________________________________________________________________________ test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-True-swss] ____________________________________________________________________________________ ___________________________________________________________________________________ test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-False-swss] ____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-True-swss] ____________________________________________________________________________________ ___________________________________________________________________________________ test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-False-swss] ____________________________________________________________________________________ ---------------------------------------------------------------------------- generated xml file: /run_logs/ixia/restart-service-2/2024-11-14-06-39-15/tr_2024-11-14-06-39-15.xml ----------------------------------------------------------------------------- INFO:root:Can not get Allure report URL. Please check logs ------------------------------------------------------------------------------------------------------------------- live log sessionfinish ------------------------------------------------------------------------------------------------------------------- 08:10:42 __init__.pytest_terminal_summary L0067 INFO | Can not get Allure report URL. Please check logs ================================================================================================================== short test summary info =================================================================================================================== PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-False-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-False-swss] ========================================================================================================= 4 passed, 7 warnings in 5484.86s (1:31:24) ========================================================================================================= sonic@ixia-sonic-mgmt-whitebox:/data/tests$ co-authorized by: [email protected]
8 tasks
mssonicbld
pushed a commit
to sonic-net/sonic-mgmt
that referenced
this issue
Nov 21, 2024
Description of PR Summary: The pfcwd_basic service-restart cases keep failing due to: sonic-net/sonic-buildimage#20637 The ask is not to restart swss multiple times without doing a config reload in between. So in this PR: we are doing config-reload for every iteration of the test The swss restart is done only once in one DUT. The asic is randomly picked, and the swss of that ASIC is restarted instead of doing the restart for all asics. Also added checks to make sure the services, interfaces and bgp are up before proceding with the ixia traffic. Approach What is the motivation for this PR? The issue: sonic-net/sonic-buildimage#20637 How did you do it? Pls see the description. How did you verify/test it? Ran it on my TB. =========================================================================================================================== PASSES =========================================================================================================================== ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss] _____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-False-swss] ____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-True-swss] _____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-False-swss] ____________________________________________________________________________________ ----------------------------------------------------------------------------- generated xml file: /run_logs/ixia/restart-service/2024-11-14-00-05-11/tr_2024-11-14-00-05-11.xml ------------------------------------------------------------------------------ INFO:root:Can not get Allure report URL. Please check logs ------------------------------------------------------------------------------------------------------------------- live log sessionfinish ------------------------------------------------------------------------------------------------------------------- 01:31:34 __init__.pytest_terminal_summary L0067 INFO | Can not get Allure report URL. Please check logs ================================================================================================================== short test summary info =================================================================================================================== PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-False-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-False-swss] ========================================================================================================= 4 passed, 7 warnings in 5180.68s (1:26:20) ========================================================================================================= sonic@ixia-sonic-mgmt-whitebox:/data/tests$ -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =========================================================================================================================== PASSES =========================================================================================================================== ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss] _____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-False-swss] ____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-True-swss] _____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-False-swss] ____________________________________________________________________________________ ---------------------------------------------------------------------------- generated xml file: /run_logs/ixia/restart-service-2/2024-11-14-02-47-47/tr_2024-11-14-02-47-47.xml ----------------------------------------------------------------------------- INFO:root:Can not get Allure report URL. Please check logs ------------------------------------------------------------------------------------------------------------------- live log sessionfinish ------------------------------------------------------------------------------------------------------------------- 04:14:03 __init__.pytest_terminal_summary L0067 INFO | Can not get Allure report URL. Please check logs ================================================================================================================== short test summary info =================================================================================================================== PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info0-False-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_multi_lossless_prio_restart_service[multidut_port_info1-False-swss] ========================================================================================================= 4 passed, 7 warnings in 5173.22s (1:26:13) ========================================================================================================= sonic@ixia-sonic-mgmt-whitebox:/data/tests$ =========================================================================================================================== PASSES =========================================================================================================================== ____________________________________________________________________________________ test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-True-swss] ____________________________________________________________________________________ ___________________________________________________________________________________ test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-False-swss] ____________________________________________________________________________________ ____________________________________________________________________________________ test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-True-swss] ____________________________________________________________________________________ ___________________________________________________________________________________ test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-False-swss] ____________________________________________________________________________________ ---------------------------------------------------------------------------- generated xml file: /run_logs/ixia/restart-service-2/2024-11-14-06-39-15/tr_2024-11-14-06-39-15.xml ----------------------------------------------------------------------------- INFO:root:Can not get Allure report URL. Please check logs ------------------------------------------------------------------------------------------------------------------- live log sessionfinish ------------------------------------------------------------------------------------------------------------------- 08:10:42 __init__.pytest_terminal_summary L0067 INFO | Can not get Allure report URL. Please check logs ================================================================================================================== short test summary info =================================================================================================================== PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info0-False-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-True-swss] PASSED snappi_tests/multidut/pfcwd/test_multidut_pfcwd_basic_with_snappi.py::test_pfcwd_basic_single_lossless_prio_service_restart[multidut_port_info1-False-swss] ========================================================================================================= 4 passed, 7 warnings in 5484.86s (1:31:24) ========================================================================================================= sonic@ixia-sonic-mgmt-whitebox:/data/tests$ co-authorized by: [email protected]
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
The changes in sonic-net/sonic-mgmt#13483 fixes an issue with swss restart test case for multiasic.
After this change we ar eobserving failure on multiasic DUTs as multiple swss services are restarted.
With every swss service restart, snmp service also gets restarted and this causes snmp service to enter the failed state.
The test case does not bring snmp service out of failed state and restart it.
This causes subsequent test cases to fail as snmp doe snot run.
Steps to reproduce the issue:
Describe the results you received:
Describe the results you expected:
Output of
show version:Output of
show techsupport:Additional information you deem important (e.g. issue happens only occasionally):
The text was updated successfully, but these errors were encountered: