Skip to content

The qa-ctl threads are not stopping correctly when pressing ctrl+c #1999

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
3 of 4 tasks
fernandolojano opened this issue Oct 7, 2021 · 5 comments
Closed
3 of 4 tasks

The qa-ctl threads are not stopping correctly when pressing ctrl+c #1999

fernandolojano opened this issue Oct 7, 2021 · 5 comments
Assignees
@fernandolojano
Labels
tool/qactl

Comments

@fernandolojano
Copy link
Contributor

fernandolojano commented Oct 7, 2021 •
edited
Loading

It is needed to check that the qa-ctl process is stopped correctly if some of the tool steps fail. As for now, It looks like the current stop implementation is not working properly.
The tasks that need to be done for this issue are:

  • Research about Python Threads
  • Research about possible causes of the current implementation not working
  • Modify the qa-ctl implementation for solving the not-stopping process issue when pressing ctrl+c on Linux.
  • Modify the qa-ctl implementation for solving the not-stopping process issue when pressing ctrl+c on Windows.
@fernandolojano fernandolojano self-assigned this Oct 7, 2021
@fernandolojano
Copy link
Contributor Author

After making some research about the second issue, I found that python threads not stopping after pressing ctrl+c (for stopping a process) is pretty common. However, it does not have a clear solution, there are multiple approaches instead. I'm currently trying to kill a python thread with the second approach suggested in this page, that is, by using Python Events.

@jmv74211 jmv74211 changed the title QA-CTL: Fix qa-ctl threads not stopping when pressing ctrl+c qa-ctl threads are not stopping correctly when pressing ctrl+c Oct 11, 2021
@jmv74211 jmv74211 mentioned this issue Oct 11, 2021
Closed
22 tasks
@jmv74211 jmv74211 changed the title qa-ctl threads are not stopping correctly when pressing ctrl+c The qa-ctl threads are not stopping correctly when pressing ctrl+c Oct 11, 2021
fernandolojano pushed a commit that referenced this issue Oct 14, 2021
fix: fixed threads from qa-ctl not stopping properly #1999
3301a27
@fernandolojano
Copy link
Contributor Author

The working branch for this issue is: 1999-qa-ctl-stopping-threads

@fernandolojano
Copy link
Contributor Author

fernandolojano commented Oct 14, 2021 •
edited
Loading

The issue is now partially solved as qa-ctl is able to stop when the user desires on Linux systems. That is thanks to the changes made in the commit 3301a27.

However, this implementation is not working in Windows as the parallelization is implemented with an alternative method different from the implemented parallelization in Linux. As for now, the process stops correctly on Windows only in the Deployment section, in the other 2 sections, it does not stop immediately, it waits until the qa_ctl_docker_run method is fully completed. It still remains to solve this issue for qa-ctl on Windows.

stopping deployment process on Windows 
qa-ctl -r test_general_settings_enabled --qa-branch 1999-qa-ctl-stopping-threads -dd
2021-10-14 11:42:14,557 - INFO - Pulling remote repository changes in C:\Users\Reko\AppData\Local\Temp\wazuh_qa_ctl\wazuh-qa local repository
2021-10-14 11:42:15,273 - INFO - Validating input parameters
2021-10-14 11:42:16,581 - INFO - Input parameters validation has passed successfully
2021-10-14 11:42:16,582 - DEBUG - Generating configuration file
2021-10-14 11:42:17,286 - DEBUG - Configuration file has been created sucessfully in C:\Users\Reko\AppData\Local\Temp\wazuh_qa_ctl\config_1634204536.582916.yaml
2021-10-14 11:42:17,287 - DEBUG - Reading configuration_data
2021-10-14 11:42:17,295 - DEBUG - The configuration data has been read successfully
2021-10-14 11:42:17,295 - DEBUG - Validating configuration schema
2021-10-14 11:42:17,299 - DEBUG - Schema validation has passed successfully
2021-10-14 11:42:17,300 - INFO - Starting 1 instances deployment
Bringing machine 'manager-test-general-settings-enabled-1634204537.283939' up with 'virtualbox' provider...
==> manager-test-general-settings-enabled-1634204537.283939: Importing base box 'qactl/ubuntu_20_04'...
==> manager-test-general-settings-enabled-1634204537.283939: Matching MAC address for NAT networking...
==> manager-test-general-settings-enabled-1634204537.283939: Setting the name of the VM: manager-test-general-settings-enabled-1634204537.283939
==> manager-test-general-settings-enabled-1634204537.283939: Clearing any previously set network interfaces...
==> manager-test-general-settings-enabled-1634204537.283939: Preparing network interfaces based on configuration...
  manager-test-general-settings-enabled-1634204537.283939: Adapter 1: nat
  manager-test-general-settings-enabled-1634204537.283939: Adapter 2: hostonly
==> manager-test-general-settings-enabled-1634204537.283939: Forwarding ports...
  manager-test-general-settings-enabled-1634204537.283939: 22 (guest) => 2222 (host) (adapter 1)
==> manager-test-general-settings-enabled-1634204537.283939: Running 'pre-boot' VM customizations...
==> manager-test-general-settings-enabled-1634204537.283939: Booting VM...
==> manager-test-general-settings-enabled-1634204537.283939: Waiting for machine to boot. This may take a few minutes...
  manager-test-general-settings-enabled-1634204537.283939: SSH address: 127.0.0.1:2222
  manager-test-general-settings-enabled-1634204537.283939: SSH username: vagrant
  manager-test-general-settings-enabled-1634204537.283939: SSH auth method: private key
Process MultiprocessingThread-1:
Traceback (most recent call last):
File "C:\Users\Reko\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py", line 315, in _bootstrap
  self.run()
File "C:\Users\Reko\Downloads\testing-qaclt\testing-qa\lib\site-packages\wazuh_testing-4.3.0-py3.9.egg\wazuh_testing\tools\multiprocessing_thread.py", line 32, in run
  self._run()
File "C:\Users\Reko\Downloads\testing-qaclt\testing-qa\lib\site-packages\wazuh_testing-4.3.0-py3.9.egg\wazuh_testing\tools\multiprocessing_thread.py", line 24, in _run
  self._return = self.function(**self.parameters)
File "C:\Users\Reko\Downloads\testing-qaclt\testing-qa\lib\site-packages\wazuh_testing-4.3.0-py3.9.egg\wazuh_testing\qa_ctl\deployment\vagrant_wrapper.py", line 62, in run
  self.vagrant.up()
File "C:\Users\Reko\Downloads\testing-qaclt\testing-qa\lib\site-packages\vagrant\__init__.py", line 337, in up
  self._call_vagrant_command(args)
File "C:\Users\Reko\Downloads\testing-qaclt\testing-qa\lib\site-packages\vagrant\__init__.py", line 962, in _call_vagrant_command
  subprocess.check_call(command, cwd=self.root, stdout=out_fh,
File "C:\Users\Reko\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 368, in check_call
  retcode = call(*popenargs, **kwargs)
File "C:\Users\Reko\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 351, in call
  return p.wait(timeout=timeout)
File "C:\Users\Reko\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 1189, in wait
  return self._wait(timeout=timeout)
File "C:\Users\Reko\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 1470, in _wait
  result = _winapi.WaitForSingleObject(self._handle,
KeyboardInterrupt
==> manager-test-general-settings-enabled-1634204537.283939: Waiting for cleanup before exiting...
2021-10-14 11:42:52,253 - INFO - Stopping 1 instances
Vagrant exited after cleanup due to external interrupt.
==> manager-test-general-settings-enabled-1634204537.283939: Attempting graceful shutdown of VM...
  manager-test-general-settings-enabled-1634204537.283939: Guest communication could not be established! This is usually because
  manager-test-general-settings-enabled-1634204537.283939: SSH is not running, the authentication information was changed,
  manager-test-general-settings-enabled-1634204537.283939: or some other networking issue. Vagrant will force halt, if
  manager-test-general-settings-enabled-1634204537.283939: capable.
==> manager-test-general-settings-enabled-1634204537.283939: Forcing shutdown of VM...
2021-10-14 11:43:12,428 - INFO - The instances have been stopped sucessfully

@fernandolojano
Copy link
Contributor Author

fernandolojano commented Oct 15, 2021 •
edited
Loading

After making extensive research about the subprocess library and docker, I've come to the conclusion that qa-ctl is not stopping in the provision and testing stages because docker does not receive any signal for being stopped when pressing ctrl + c. That issue got solved by adding the arguments -t -i to the docker run command as explained in this issue comment. The stages now stop when the combination ctrl + c gets pressed, but the current problem is that Docker does not send any signal back of being stop (SIGINT) after pressing the combination, so qa-ctl keeps with the execution jumping to the next stage instead of jumping to the finally condition, where all the files created, are deleted.

stopping qa-ctl on Windows 
qa-ctl -r test_general_settings_enabled --qa-branch 1999-qa-ctl-stopping-threads -dd
2021-10-15 13:45:47,905 - INFO - Pulling remote repository changes in C:\Users\Reko\AppData\Local\Temp\wazuh_qa_ctl\wazuh-qa local repository
2021-10-15 13:45:49,231 - INFO - Validating input parameters
2021-10-15 13:45:51,194 - INFO - Input parameters validation has passed successfully
2021-10-15 13:45:51,195 - DEBUG - Generating configuration file
2021-10-15 13:45:51,947 - DEBUG - Configuration file has been created sucessfully in C:\Users\Reko\AppData\Local\Temp\wazuh_qa_ctl\config_1634298351.195665.yaml
2021-10-15 13:45:51,947 - DEBUG - Reading configuration_data
2021-10-15 13:45:51,957 - DEBUG - The configuration data has been read successfully
2021-10-15 13:45:51,957 - DEBUG - Validating configuration schema
2021-10-15 13:45:51,965 - DEBUG - Schema validation has passed successfully
2021-10-15 13:45:51,967 - INFO - Starting 1 instances deployment
Bringing machine 'manager-test-general-settings-enabled-1634298351.943103' up with 'virtualbox' provider...
==> manager-test-general-settings-enabled-1634298351.943103: Importing base box 'qactl/ubuntu_20_04'...
==> manager-test-general-settings-enabled-1634298351.943103: Matching MAC address for NAT networking...
==> manager-test-general-settings-enabled-1634298351.943103: Setting the name of the VM: manager-test-general-settings-enabled-1634298351.943103
==> manager-test-general-settings-enabled-1634298351.943103: Fixed port collision for 22 => 2222. Now on port 2200.
==> manager-test-general-settings-enabled-1634298351.943103: Clearing any previously set network interfaces...
==> manager-test-general-settings-enabled-1634298351.943103: Preparing network interfaces based on configuration...
  manager-test-general-settings-enabled-1634298351.943103: Adapter 1: nat
  manager-test-general-settings-enabled-1634298351.943103: Adapter 2: hostonly
==> manager-test-general-settings-enabled-1634298351.943103: Forwarding ports...
  manager-test-general-settings-enabled-1634298351.943103: 22 (guest) => 2200 (host) (adapter 1)
==> manager-test-general-settings-enabled-1634298351.943103: Running 'pre-boot' VM customizations...
==> manager-test-general-settings-enabled-1634298351.943103: Booting VM...
==> manager-test-general-settings-enabled-1634298351.943103: Waiting for machine to boot. This may take a few minutes...
  manager-test-general-settings-enabled-1634298351.943103: SSH address: 127.0.0.1:2200
  manager-test-general-settings-enabled-1634298351.943103: SSH username: vagrant
  manager-test-general-settings-enabled-1634298351.943103: SSH auth method: private key
  manager-test-general-settings-enabled-1634298351.943103: Warning: Connection reset. Retrying...
  manager-test-general-settings-enabled-1634298351.943103: Warning: Connection aborted. Retrying...
==> manager-test-general-settings-enabled-1634298351.943103: Machine booted and ready!
==> manager-test-general-settings-enabled-1634298351.943103: Checking for guest additions in VM...
==> manager-test-general-settings-enabled-1634298351.943103: Setting hostname...
==> manager-test-general-settings-enabled-1634298351.943103: Configuring and enabling network interfaces...
2021-10-15 13:46:53,418 - INFO - The instances deployment has finished sucessfully
2021-10-15 13:46:53,421 - INFO - Building docker image for provisioning the instances
2021-10-15 13:46:58,019 - INFO - Running the Linux container for provisioning the instances
2021-10-15 11:47:07,209 - DEBUG - Reading configuration_data
2021-10-15 11:47:07,216 - DEBUG - The configuration data has been read successfully
2021-10-15 11:47:07,216 - DEBUG - Validating configuration schema
2021-10-15 11:47:07,222 - DEBUG - Schema validation has passed successfully
2021-10-15 11:47:07,223 - INFO - Checking hosts SSH connection

PLAY [all] *********************************************************************

TASK [Waiting for SSH hosts connection are reachable] **************************
^C
2021-10-15 13:47:09,821 - INFO - Building docker image for launching the tests
2021-10-15 13:47:12,199 - INFO - Running the Linux container for launching the tests
2021-10-15 11:47:19,241 - DEBUG - Reading configuration_data
2021-10-15 11:47:19,269 - DEBUG - The configuration data has been read successfully
2021-10-15 11:47:19,269 - DEBUG - Validating configuration schema
2021-10-15 11:47:19,274 - DEBUG - Schema validation has passed successfully
2021-10-15 11:47:19,283 - INFO - Launching 1 tests
2021-10-15 11:47:19,284 - INFO - Waiting for tests to finish

PLAY [10.150.50.5] *************************************************************

TASK [lineinfile] **************************************************************
^C
2021-10-15 13:47:22,217 - INFO - The results of /tmp/wazuh_qa_ctl/wazuh-qa/tests/integration/test_vulnerability_detector/test_general_settings/test_general_settings_enabled.py tests have been saved in C:\Users\Reko\AppData\Local\Temp/wazuh_qa_ctl/test_general_settings_enabled_1634298351.94408/
2021-10-15 13:47:22,218 - INFO - Destroying 1 instances
==> manager-test-general-settings-enabled-1634298351.943103: Forcing shutdown of VM...
==> manager-test-general-settings-enabled-1634298351.943103: Destroying VM and associated drives...
2021-10-15 13:47:28,018 - INFO - The instances have been destroyed sucessfully

@fernandolojano
Copy link
Contributor Author

After talking with @jmv74211, we conclude that the best approach for stopping the threads is to catch the SIGINT signal and send a docker stop command to stop the container directly. As for now this part of the implementation is done, however, it remains to get a way to obtain the id of the docker that has been created, and check if the new approach works correctly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fernandolojano fernandolojano

Labels
tool/qactl
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jmv74211 @fernandolojano