Full Exports: improve error handling during Bigquery load phase

[heldchen]

during the bigquery data load the full export returns a meaningless error when the data could not be loaded:

Start of Boxalino Data Integration (DI) ...

Fatal error: Uncaught Boxalino\DataIntegrationDoc\Service\ErrorHandler\FailDocLoadException: Doc Load failed. Exception: Client error: `POST https://boxalino-di-full-xxx-ew.a.run.app/load/bq` resulted in a `400 Bad Request` response:
"Fail to load gs:\/\/prod_rtux-data-integration_xxx\/doc_order_F_20220109192640.json to BQ with exception: (truncated...)
 in /var/www/xxx/src/www/vendor/boxalino/data-integration-doc-php/src/Service/Flow/LoadBqTrait.php:40
Stack trace:
#0 /var/www/xxx/src/www/vendor/boxalino/data-integration-magento2/Service/Document/DiIntegrateTrait.php(62): Boxalino\DataIntegrationDoc\Service\Integration\Doc\DocOrder->loadBq()
#1 /var/www/xxx/src/www/vendor/boxalino/data-integration-magento2/Service/Document/DiIntegrateTrait.php(97): Boxalino\DataIntegration\Service\Document\Order\DocHandler->integrate()
#2 /var/www/xxx/src/www/vendor/boxalino/data-integration-magento2/Service/Document/DiIntegrateTrait.php(52): Boxalino\DataIntegration\Service\Document\O in /var/www/xxx/src/www/vendor/boxalino/data-integration-doc-php/src/Service/Flow/LoadBqTrait.php on line 40

• please expose the underlying data error/exception
• please add an (optional, opt-in) mode that ignores bad records (this is already possible using manual data loads in the bq gui)

[dannegwork]

please add an (optional, opt-in) mode that ignores bad records (this is already possible using manual data loads in the bq gui)

This is something to be reviewed at a later stage.
We recognize the value for "test" loads, howether there are more valuable integration options which are not allowed to fail. Most integrators never review the status of their data export (be it done via plugins or self-developed), allowing bad records to be skipped would wrongly lead to the idea that everything is fine in their integration.

[heldchen]

sure. but as discussed with sylvain we won't implement a data filter on our side, and adding it to the exporter itself might slow things down. so exposing the ignore-errornous-rows BQ load option as an opt-in export mode would solve this for now. and this is not in regards of test-loads, but the production exports.

[dannegwork]

we won't implement a data filter on our side
This is not needed :) There will be a data format sanitization (it was a WIP update in the plugin as is)