Perform more realistic simulations with large blasts

[totten]

@lcdservices reported on Stackexchange an issue when running a large blast. The cause/scope of that issue of that is unknown, but it would be good to provide some baseline by doing tests with real(ish) SMTP deliveries.

[totten]

I did trial-run today with these steps:

1. Rebuild with latest dmaster. Installed mosaico, flexmailer.

2. Start mailcatcher (fake SMTP service).

3. Created ~15k contacts by running the following script ~15 times. (The actual number was 15730 unique email addresses. It's not a round number because of a DB failure while inserting the contacts.)

$offset = getenv('OFFSET');
if ($offset === '' || !is_numeric($offset)) {
  exit('Please set OFFSET');
}

$count = getenv('COUNT');
if (!$count) $count = 1000;

for ($i = 0; $i < $count; $i++) {
  $j = $offset + $i;
  echo "[$j]\n";
  civicrm_api3('contact', 'create', [
    'contact_type' => 'Individual',
    'first_name' => "First$j",
    'last_name' => "Last$j",
    'email' => "person-$j@example.org",
  ]);
}

4. Put these contacts in a group, "Fifteen K".

5. Create and schedule a new Mosaico mailing to "Fifteen K".

6. Execute cv api -U admin job.process_mailing. This crashed after a few moments due to memory exhaustion.

PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 23373 bytes) in /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/ext/mosaico/CRM/Mosaico/UrlFilter.php on line 54
PHP Stack trace:
PHP   1. {main}() /Users/totten/bknix/civicrm-buildkit/bin/cv:0
PHP   2. require() /Users/totten/bknix/civicrm-buildkit/bin/cv:10
PHP   3. Civi\Cv\Application::main() phar:///Users/totten/bknix/civicrm-buildkit/bin/cv/bin/cv:26
PHP   4. Symfony\Component\Console\Application->run() phar:///Users/totten/bknix/civicrm-buildkit/bin/cv/src/Application.php:17
PHP   5. Symfony\Component\Console\Application->doRun() phar:///Users/totten/bknix/civicrm-buildkit/bin/cv/vendor/symfony/console/Application.php:124
PHP   6. Symfony\Component\Console\Application->doRunCommand() phar:///Users/totten/bknix/civicrm-buildkit/bin/cv/vendor/symfony/console/Application.php:193
PHP   7. Symfony\Component\Console\Command\Command->run() phar:///Users/totten/bknix/civicrm-buildkit/bin/cv/vendor/symfony/console/Application.php:850
PHP   8. Civi\Cv\Command\ApiCommand->execute() phar:///Users/totten/bknix/civicrm-buildkit/bin/cv/vendor/symfony/console/Command/Command.php:257
PHP   9. civicrm_api() phar:///Users/totten/bknix/civicrm-buildkit/bin/cv/src/Command/ApiCommand.php:53
PHP  10. Civi\API\Kernel->runSafe() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/api/api.php:23
PHP  11. Civi\API\Kernel->runRequest() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/Civi/API/Kernel.php:100
PHP  12. Civi\API\Provider\MagicFunctionProvider->invoke() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/Civi/API/Kernel.php:169
PHP  13. civicrm_api3_job_process_mailing() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/Civi/API/Provider/MagicFunctionProvider.php:89
PHP  14. CRM_Mailing_BAO_Mailing::processQueue() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/api/v3/Job.php:324
PHP  15. CRM_Mailing_BAO_MailingJob::runJobs() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/CRM/Mailing/BAO/Mailing.php:2829
PHP  16. Civi\Core\Resolver->call() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/CRM/Mailing/BAO/MailingJob.php:196
PHP  17. call_user_func_array:{/Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/Civi/Core/Resolver.php:119}() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/Civi/Core/Resolver.php:119
PHP  18. Civi\FlexMailer\FlexMailer::createAndRun() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/Civi/Core/Resolver.php:119
PHP  19. Civi\FlexMailer\FlexMailer->run() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/ext/flexmailer/src/FlexMailer.php:149
PHP  20. Civi\FlexMailer\FlexMailer->fireWalkBatches() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/ext/flexmailer/src/FlexMailer.php:187
PHP  21. Civi\Core\CiviEventDispatcher->dispatch() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/ext/flexmailer/src/FlexMailer.php:222
PHP  22. Symfony\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/Civi/Core/CiviEventDispatcher.php:47
PHP  23. Symfony\Component\EventDispatcher\EventDispatcher->dispatch() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/vendor/symfony/event-dispatcher/Symfony/Component/EventDispatcher/ContainerAwareEventDispatcher.php:167
PHP  24. Symfony\Component\EventDispatcher\EventDispatcher->doDispatch() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/vendor/symfony/event-dispatcher/Symfony/Component/EventDispatcher/EventDispatcher.php:53
PHP  25. call_user_func:{/Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/vendor/symfony/event-dispatcher/Symfony/Component/EventDispatcher/EventDispatcher.php:164}() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/vendor/symfony/event-dispatcher/Symfony/Component/EventDispatcher/EventDispatcher.php:164
PHP  26. Civi\FlexMailer\Listener\DefaultBatcher->onWalk() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/vendor/symfony/event-dispatcher/Symfony/Component/EventDispatcher/EventDispatcher.php:164
PHP  27. Civi\FlexMailer\Event\WalkBatchesEvent->visit() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/ext/flexmailer/src/Listener/DefaultBatcher.php:74
PHP  28. call_user_func:{/Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/ext/flexmailer/src/Event/WalkBatchesEvent.php:71}() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/ext/flexmailer/src/Event/WalkBatchesEvent.php:71
PHP  29. Civi\FlexMailer\FlexMailer->Civi\FlexMailer\{closure}() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/ext/flexmailer/src/Event/WalkBatchesEvent.php:71
PHP  30. Civi\FlexMailer\FlexMailer->fireComposeBatch() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/ext/flexmailer/src/FlexMailer.php:184
PHP  31. Civi\Core\CiviEventDispatcher->dispatch() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/ext/flexmailer/src/FlexMailer.php:238
PHP  32. Symfony\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/Civi/Core/CiviEventDispatcher.php:47
PHP  33. Symfony\Component\EventDispatcher\EventDispatcher->dispatch() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/vendor/symfony/event-dispatcher/Symfony/Component/EventDispatcher/ContainerAwareEventDispatcher.php:167
PHP  34. Symfony\Component\EventDispatcher\EventDispatcher->doDispatch() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/vendor/symfony/event-dispatcher/Symfony/Component/EventDispatcher/EventDispatcher.php:53
PHP  35. call_user_func:{/Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/vendor/symfony/event-dispatcher/Symfony/Component/EventDispatcher/EventDispatcher.php:164}() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/vendor/symfony/event-dispatcher/Symfony/Component/EventDispatcher/EventDispatcher.php:164
PHP  36. CRM_Mosaico_UrlFilter->onCompose() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/vendor/symfony/event-dispatcher/Symfony/Component/EventDispatcher/EventDispatcher.php:164
PHP  37. Civi\FlexMailer\Listener\SimpleFilter::byColumn() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/ext/mosaico/CRM/Mosaico/UrlFilter.php:24
PHP  38. call_user_func_array:{/Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/ext/flexmailer/src/Listener/SimpleFilter.php:96}() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/ext/flexmailer/src/Listener/SimpleFilter.php:96
PHP  39. CRM_Mosaico_UrlFilter->filterHtml() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/ext/flexmailer/src/Listener/SimpleFilter.php:96
PHP  40. preg_replace_callback() /Users/totten/bknix/build/dmaster/sites/all/modules/civicrm/ext/mosaico/CRM/Mosaico/UrlFilter.php:54

7. Retry delivery, but without any memory limit.

php -d memory_limit=-1 `which cv` api -U admin job.process_mailing

This took about ten minutes ultimately reported:

{
    "is_error": 0,
    "version": 3,
    "count": 1,
    "id": "processed",
    "values": {
        "processed": 15730
    }
}

8. Check the mailcatcher UI. It agreed that 15730 messages were delivered. Spot-check to see that messages look right.

9. Check report in CiviMail to see that the opens registered.

So, overall, I'd that was a successful run. Some thoughts:

• @lcdservices, it might be interesting if you reported all the CiviMail batching parameters from your system. I could take another go at it with those settings.
• mailcatcher's web UI seems to struggle a bit when handling 15k messages. I might want to try with fakesmtp
• It might worth trying again at 25k.

[totten]

This seems related: veda-consulting-company/uk.co.vedaconsulting.mosaico#167 (comment)