CRM_Queue_Runner - Add basic/skeletal support for runAllInteractively()

[totten]

Overview

The helper class CRM_Queue_Runner provides methods for working through the tasks in a queue. This adds another one (runAllInteractively()) and a setting (enableBackgroundQueue). This arrangement allows the sysadmin to decide how to run tasks. It is flagged EXPERIMENTAL.

The method runAllInteractively() is similar to the existing runAllViaWeb(), but it is not necessarily executed via web-browser - it can be supplemented (depending on system config) by background/multiprocessing tasks, and it has somewhat tighter requirements/constraints. (These constraints are validated upfront.)

Before

There are two mechanisms/helpers:

• runAll() synchronously executes all tasks in a queue (until the queue is empty).
• runAllViaWeb() redirects the user another web-page and runs the queue. This other page displays the progress of the queue, and it actively executes each task, 1-by-1.

After

Adds a third mechanism/helper:

• runAllInteractively() redirects the user to another web-page and runs the queue. However, it do so in a few different ways. It depends on the setting enableBackgroundQueue:
  • By default (enableBackgroundQueue=0), it will do exactly the same as runAllViaWeb().
  • With extra configuration (enableBackgroundQueue=1), it will send the queue for execution by a background task-runner. The web browser will redirect to a monitoring page.

The new setting enableBackgroundQueue is flagged "EXPERIMENTAL". The default is fully functional/equivalent/compatible with current behavior.

Comments

General:

• The addition of runAllInteractively() should have no impact on existing callers that use runAll() or runAllViaWeb().
• This is basic/skeletal implementation. The biggest omission is in the page civicrm/queue/monitor. It needs a widget to actually... monitor the progress.
• runAllInteractively() and civicrm/queue/monitor do not support the option onEnd. There are a couple reasons:
  • Based on grepping universe, this option is often used for manipulating session-status-messages or to issue HTTP redirects. This is not portable to background queues.
  • onEnd is/was a configurable callback. The most obvious place to store this is (civicrm_user_job.metadata), and that is API-CRUDable - which means its not safe to store a callback there.
  • However, to compensate, we have the preceding PR Queue - When UserJob has no more tasks, update status & fire hook #23711 which added hook_civicrm_queueStatus.

To test this, I used demoqueue and coworker, ie

cv en demoqueue
cv open civicrm/demoqueue?reset=1

Then I had console runner coworker. It received an executed the tasks

[bknix-max:~/bknix/build/dmaster/web/sites/all/modules/civicrm] ~/src/coworker/bin/coworker run -v

[2022-06-09T16:46:08.465362-07:00] PipeConnection_1.INFO: Start: cv ev "Civi::pipe();"
[2022-06-09T16:46:08.671740-07:00] CtlConn.NOTICE: Connected
[2022-06-09T16:46:08.672392-07:00] WorkPool.INFO: Start
[2022-06-09T16:46:08.673342-07:00] CiviQueueWatcher.INFO: Starting...
[2022-06-09T16:46:09.714231-07:00] CiviQueueWatcher.INFO: Claimed queue item(s): array[{"id":"26","queue":"demoqueue_1654818331"}]
[2022-06-09T16:46:09.794671-07:00] PipeConnection_2.INFO: Start: cv ev "Civi::pipe();"
[2022-06-09T16:46:10.960117-07:00] CiviQueueWatcher.INFO: Claimed queue item(s): array[{"id":"27","queue":"demoqueue_1654818331"}]
[2022-06-09T16:46:12.035409-07:00] CiviQueueWatcher.INFO: Claimed queue item(s): array[{"id":"28","queue":"demoqueue_1654818331"}]
[2022-06-09T16:46:13.064324-07:00] CiviQueueWatcher.INFO: Claimed queue item(s): array[{"id":"29","queue":"demoqueue_1654818331"}]
[2022-06-09T16:46:14.093760-07:00] CiviQueueWatcher.INFO: Claimed queue item(s): array[{"id":"32","queue":"demoqueue_1654818331"}]
[2022-06-09T16:46:15.114301-07:00] CiviQueueWatcher.INFO: Claimed queue item(s): array[{"id":"33","queue":"demoqueue_1654818331"}]
[2022-06-09T16:46:16.131495-07:00] CiviQueueWatcher.INFO: Claimed queue item(s): array[{"id":"34","queue":"demoqueue_1654818331"}]
[2022-06-09T16:46:17.160107-07:00] CiviQueueWatcher.INFO: Claimed queue item(s): array[{"id":"35","queue":"demoqueue_1654818331"}]
[2022-06-09T16:46:18.190567-07:00] CiviQueueWatcher.INFO: Claimed queue item(s): array[{"id":"36","queue":"demoqueue_1654818331"}]
[2022-06-09T16:46:19.223099-07:00] CiviQueueWatcher.INFO: Claimed queue item(s): array[{"id":"30","queue":"demoqueue_1654818331"}]
[2022-06-09T16:46:20.242231-07:00] CiviQueueWatcher.INFO: Claimed queue item(s): array[{"id":"31","queue":"demoqueue_1654818331"}]
[2022-06-09T16:46:20.247858-07:00] PipeConnection_3.INFO: Start: cv ev "Civi::pipe();"

[civibot]

(Standard links)

• If this is your first pull-request for CiviCRM, please browse CONTRIBUTING.md for information about the development and testing processes.
• If you are reviewing this pull-request, you may wish to consult the test sites and the Review Standards (long template, short template).

[eileenmcnaughton]

test this please

[eileenmcnaughton]

unrelated fail