Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes to salt tests #493

Merged
merged 5 commits into from
Nov 15, 2022
Merged

Fixes to salt tests #493

merged 5 commits into from
Nov 15, 2022

Conversation

marmarek
Copy link
Member

Workaround for a warning message (fixed upstream already, but waiting to be backported)

Fixes QubesOS/qubes-issues#7834

Copy link
Contributor

@DemiMarie DemiMarie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typos in comments

state_output_json = json.loads(state_output[len(expected_output):])
json_data = state_output[len(expected_output):]
# workaround for https://github.com/saltstack/salt/issues/60476
# (fixed upstream, but haven't flowd into Fedora yet)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# (fixed upstream, but haven't flowd into Fedora yet)
# (fixed upstream, but hasen't flowed into Fedora yet)

@@ -399,10 +411,17 @@ def test_001_multi_state_highstate(self):
expected_output = vmname + ':\n'
self.assertTrue(state_output.startswith(expected_output),
'Full output: ' + state_output)
json_data = state_output[len(expected_output):]
# workaround for https://github.com/saltstack/salt/issues/60476
# (fixed upstream, but haven't flowd into Fedora yet)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# (fixed upstream, but haven't flowd into Fedora yet)
# (fixed upstream, but hasen't flowed into Fedora yet)

@@ -469,6 +488,12 @@ def test_002_grans_id(self):
tpl_output = tpl_output[len(tplname + ':\n'):]

for name, output in ((tplname, tpl_output), (vmname, appvm_output)):
# workaround for https://github.com/saltstack/salt/issues/60476
# (fixed upstream, but haven't flowd into Fedora yet)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# (fixed upstream, but haven't flowd into Fedora yet)
# (fixed upstream, but hasen't flowed into Fedora yet)

@qubesos-bot
Copy link

qubesos-bot commented Nov 13, 2022

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.1&build=2022111223-4.1&flavor=pull-requests

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.1&build=2022101806-4.1&flavor=update

  • system_tests_basic_vm_qrexec_gui@hw1

    • TC_00_AppVM_debian-11: test_223_audio_play_hvm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 60 secon...

    • TC_00_AppVM_fedora-36: test_223_audio_play_hvm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 60 secon...

  • system_tests_pvgrub_salt_storage

    • TC_41_HVMGrub_debian-11: test_010_template_based_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 90 secon...
  • system_tests_guivm_gui_interactive

    • update_templates: unnamed test (unknown)
    • update_templates: Failed (test died)
      # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...

Failed tests

18 failures
  • system_tests_basic_vm_qrexec_gui@hw1

  • system_tests_suspend@hw1

    • [unstable] suspend: unnamed test (unknown)
    • [unstable] suspend: Failed (test died)
      # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
  • system_tests_pvgrub_salt_storage

    • TC_41_HVMGrub_debian-11: test_010_template_based_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 90 secon...
  • system_tests_splitgpg

  • system_tests_guivm_gui_interactive

    • update_templates: unnamed test (unknown)
    • update_templates: Failed (test died)
      # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
  • system_tests_network_ipv6

  • system_tests_dispvm

    • [unstable] TC_20_DispVM_fedora-36: test_100_open_in_dispvm (failure)
      self.assertEqual(test_txt_content.s... AssertionError: b'' != b'test1'

    • [unstable] TC_20_DispVM_whonix-ws-16: test_100_open_in_dispvm (failure)
      AssertionError: './open-file test.txt' failed with ./open-file test...

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/48243#dependencies

8 fixed

Unstable tests

  • system_tests_suspend@hw1

    suspend/ (1/5 times with errors)
    suspend/Failed (1/5 times with errors)
    • job 53470 # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
    suspend/Failed (1/5 times with errors)
    • job 48287 # Test died: command 'qvm-run -ap sys-whonix 'LC_ALL=C whonixcheck ...
    suspend/wait_serial (1/5 times with errors)
    • job 53470 # wait_serial expected: "xl info; echo 8Ye1l-\$?-"...
    suspend/wait_serial (1/5 times with errors)
    • job 53470 # wait_serial expected: qr/8Ye1l-\d+-/...
    suspend/wait_serial (1/5 times with errors)
    • job 48287 # wait_serial expected: qr/wG5VX-\d+-/...
  • system_tests_network

    VmNetworking_debian-11/test_020_simple_proxyvm_nm (1/5 times with errors)
    • job 53042 AssertionError: 1 != 0 : nm-applet window not found
  • system_tests_splitgpg

    TC_10_Thunderbird_debian-11/test_000_send_receive_default (3/5 times with errors)
    • job 47171 dogtail.tree.SearchError: descendent of [frame | Write: Test messag...
    • job 48266 dogtail.tree.SearchError: descendent of [frame | Write: Test messag...
    • job 53046 dogtail.tree.SearchError: descendent of [frame | Write: Test messag...
    TC_10_Thunderbird_whonix-ws-16/test_000_send_receive_default (2/5 times with errors)
    • job 47171 dogtail.tree.SearchError: descendent of [frame | Write: Test messag...
    • job 53046 dogtail.tree.SearchError: descendent of [frame | Write: Test messag...
    TC_10_Thunderbird_debian-11/test_010_send_receive_inline_signed_only (3/5 times with errors)
    • job 47171 dogtail.tree.SearchError: descendent of [frame | Write: Test messag...
    • job 48266 dogtail.tree.SearchError: descendent of [frame | Write: Test messag...
    • job 53046 dogtail.tree.SearchError: descendent of [frame | Write: Test messag...
    TC_10_Thunderbird_whonix-ws-16/test_010_send_receive_inline_signed_only (2/5 times with errors)
    • job 47171 dogtail.tree.SearchError: descendent of [frame | Write: Test messag...
    • job 53046 dogtail.tree.SearchError: descendent of [frame | Write: Test messag...
    TC_10_Thunderbird_debian-11/test_020_send_receive_inline_with_attachment (3/5 times with errors)
    • job 47171 dogtail.tree.SearchError: descendent of [frame | Write: Test messag...
    • job 48266 dogtail.tree.SearchError: descendent of [frame | Write: Test messag...
    • job 53046 dogtail.tree.SearchError: descendent of [frame | Write: Test messag...
    TC_10_Thunderbird_whonix-ws-16/test_020_send_receive_inline_with_attachment (3/5 times with errors)
    • job 46885 dogtail.tree.SearchError: descendent of [application | Thunderbird]...
    • job 47171 dogtail.tree.SearchError: descendent of [frame | Write: Test messag...
    • job 53046 dogtail.tree.SearchError: descendent of [frame | Write: Test messag...
  • system_tests_extra

    TC_00_InputProxy_fedora-36/test_050_mouse_late_attach (1/5 times with errors)
    • job 53037 AssertionError: unexpectedly None : Device 'test-inst-input: Test i...
  • system_tests_network_ipv6

    VmIPv6Networking_debian-11/test_520_ipv6_simple_proxyvm_nm (2/5 times with errors)
    • job 47168 AssertionError: 1 != 0 : nm-applet window not found
    • job 47809 AssertionError: 1 != 0 : nm-applet window not found
  • system_tests_network_updates

    TC_10_QvmTemplate_debian-11/test_000_template_list (1/5 times with errors)
    • job 48264 AssertionError: libvirt event impl drain timeout
    TC_11_QvmTemplateMgmtVM_whonix-gw-16/test_000_template_list (2/5 times with errors)
    • job 47169 subprocess.CalledProcessError: Command 'systemcheck --cli' returned...
    • job 47817 subprocess.CalledProcessError: Command 'systemcheck --cli' returned...
    TC_10_QvmTemplate_debian-11/test_010_template_install (1/5 times with errors)
    • job 47817 AssertionError: libvirt event impl drain timeout
    TC_10_QvmTemplate_fedora-36/test_010_template_install (1/5 times with errors)
    • job 53044 AssertionError: libvirt event impl drain timeout
    TC_10_QvmTemplate_whonix-gw-16/test_010_template_install (1/5 times with errors)
    • job 48264 AssertionError: libvirt event impl drain timeout
    TC_11_QvmTemplateMgmtVM_debian-11/test_010_template_install (1/5 times with errors)
    • job 47817 AssertionError: libvirt event impl drain timeout
    TC_11_QvmTemplateMgmtVM_whonix-gw-16/test_010_template_install (3/5 times with errors)
    • job 46861 subprocess.CalledProcessError: Command 'systemcheck --cli' returned...
    • job 47169 AssertionError: libvirt event impl drain timeout
    • job 47817 subprocess.CalledProcessError: Command 'systemcheck --cli' returned...
  • system_tests_dispvm

    TC_20_DispVM_fedora-36/test_100_open_in_dispvm (4/5 times with errors)
    • job 47793 self.assertEqual(test_txt_content.s... AssertionError: b'' != b'test1'
    • job 48246 self.assertEqual(test_txt_content.s... AssertionError: b'' != b'test1'
    • job 52384 self.assertEqual(test_txt_content.s... AssertionError: b'' != b'test1'
    • job 53026 self.assertEqual(test_txt_content.s... AssertionError: b'' != b'test1'
    TC_20_DispVM_whonix-ws-16/test_100_open_in_dispvm (5/5 times with errors)
    • job 46883 AssertionError: libvirt event impl drain timeout
    • job 47793 AssertionError: libvirt event impl drain timeout
    • job 48246 AssertionError: './open-file test.txt' failed with ./open-file test...
    • job 52384 AssertionError: './open-file test.txt' failed with ./open-file test...
    • job 53026 AssertionError: libvirt event impl drain timeout
  • system_tests_suspend

    suspend/ (1/5 times with errors)
    suspend/Failed (1/5 times with errors)
    • job 53470 # Test died: no candidate needle with tag(s) 'xscreensaver-prompt' ...
    suspend/Failed (1/5 times with errors)
    • job 48287 # Test died: command 'qvm-run -ap sys-whonix 'LC_ALL=C whonixcheck ...
    suspend/wait_serial (1/5 times with errors)
    • job 53470 # wait_serial expected: "xl info; echo 8Ye1l-\$?-"...
    suspend/wait_serial (1/5 times with errors)
    • job 53470 # wait_serial expected: qr/8Ye1l-\d+-/...
    suspend/wait_serial (1/5 times with errors)
    • job 48287 # wait_serial expected: qr/wG5VX-\d+-/...

Workaround for saltstack/salt#60476
It prints warning to stdout(?!). The issue is fixed upstream already,
but the fix isn't packaged yet. At this time, it affects Fedora 37 only.

Fixes QubesOS/qubes-issues#7834
Unset also 'management_dispvm' property as part of breaking dependency
loops.
Default test env is now a copy of the host env.
Instead of requiring manually decorating test method, do that
automatically when QUBES_TEST_WAIT_ON_FAIL=1 env variable is set.
@codecov
Copy link

codecov bot commented Nov 15, 2022

Codecov Report

Merging #493 (914b8b7) into master (c177aac) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #493   +/-   ##
=======================================
  Coverage   65.83%   65.83%           
=======================================
  Files          53       53           
  Lines       10020    10020           
=======================================
  Hits         6597     6597           
  Misses       3423     3423           
Flag Coverage Δ
unittests 65.83% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@marmarek marmarek merged commit 914b8b7 into QubesOS:master Nov 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Build 2022102511-4.1-fedora-37] openQA test fails in TC_10_VMSalt_fedora-37
3 participants