[migrations] Throw error if reindex task fails

[tylersmalley]

We are not correctly handing the re-index task failing. This is a problem in a scenerio where there are missing shards, which mean that some or all of the documents will be lost.

This PR is only to resolve the issue of possible data loss. We will work on subsequent PR's to address re-try logic where possible.

Testing:

Start ES with a Kibana index pre 6.5.0, which requires a migration and has a missing segment file. This will result in a re-index failure of .kibana > .kibana_1. Alternatively, you can start ES using my data directory here.

Start Kibana, which will trigger the re-index, which should fail.

[elasticmachine]

Pinging @elastic/kibana-operations

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[LeeDr]

Testing this on 6.6.0 BC3, I used the Elasticsearch data from the original description (on debian 9 with Elasticsearch and Kibana running as a service);

Here's the reindex fails and Kibana shuts down;

Jan 11 22:16:05 packer-virtualbox-iso-1518710300 kibana[5429]: {"type":"log","@timestamp":"2019-01-11T22:16:05Z","tags":["info","migrations"],"pid":5429,"message":"Creating index .kibana_2."}
Jan 11 22:16:06 packer-virtualbox-iso-1518710300 kibana[5429]: {"type":"log","@timestamp":"2019-01-11T22:16:06Z","tags":["info","migrations"],"pid":5429,"message":"Reindexing .kibana to .kibana_1"}
Jan 11 22:16:07 packer-virtualbox-iso-1518710300 kibana[5429]: {"type":"log","@timestamp":"2019-01-11T22:16:07Z","tags":["debug","root"],"pid":5429,"message":"shutting root down"}
Jan 11 22:16:07 packer-virtualbox-iso-1518710300 kibana[5429]: {"type":"log","@timestamp":"2019-01-11T22:16:07Z","tags":["fatal","root"],"pid":5429,"message":"Error: Re-index failed [search_phase_execution_exception] all shards failed :: {\"type\":\"search_phase_execution_exception\",\"reaso                                            n\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[]}\n    at callCluster.then.result (/usr/share/kibana/src/server/saved_objects/migrations/core/elastic_index.js:283:23)\n    at tryCatcher (/usr/share/kibana/node_modules/bluebird/js/main/util.js:26:23)\n    at                                             Promise._settlePromiseFromHandler (/usr/share/kibana/node_modules/bluebird/js/main/promise.js:503:31)\n    at Promise._settlePromiseAt (/usr/share/kibana/node_modules/bluebird/js/main/promise.js:577:18)\n    at Promise._settlePromises (/usr/share/kibana/node_modules/bluebird/js/main/promise                                            .js:693:14)\n    at Async._drainQueue (/usr/share/kibana/node_modules/bluebird/js/main/async.js:123:16)\n    at Async._drainQueues (/usr/share/kibana/node_modules/bluebird/js/main/async.js:133:10)\n    at Immediate.Async.drainQueues [as _onImmediate] (/usr/share/kibana/node_modules/bluebird/                                            js/main/async.js:15:14)\n    at runCallback (timers.js:705:18)\n    at tryOnImmediate (timers.js:676:5)\n    at processImmediate (timers.js:658:5)"}
Jan 11 22:16:07 packer-virtualbox-iso-1518710300 kibana[5429]: {"type":"log","@timestamp":"2019-01-11T22:16:07Z","tags":["debug","server"],"pid":5429,"message":"stopping server"}
Jan 11 22:16:07 packer-virtualbox-iso-1518710300 kibana[5429]: {"type":"log","@timestamp":"2019-01-11T22:16:07Z","tags":["debug","legacy-service"],"pid":5429,"message":"stopping legacy service"}
Jan 11 22:16:07 packer-virtualbox-iso-1518710300 kibana[5429]: {"type":"log","@timestamp":"2019-01-11T22:16:07Z","tags":["debug","plugins-service"],"pid":5429,"message":"Stopping plugins service"}
Jan 11 22:16:07 packer-virtualbox-iso-1518710300 kibana[5429]: {"type":"log","@timestamp":"2019-01-11T22:16:07Z","tags":["debug","http","server"],"pid":5429,"message":"stopping http server"}
Jan 11 22:16:07 packer-virtualbox-iso-1518710300 kibana[5429]: {"type":"log","@timestamp":"2019-01-11T22:16:07Z","tags":["debug","legacy-proxy"],"pid":5429,"message":"Event is being forwarded: close"}
Jan 11 22:16:07 packer-virtualbox-iso-1518710300 kibana[5429]:  FATAL  Error: Re-index failed [search_phase_execution_exception] all shards failed :: {"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]}
Jan 11 22:16:07 packer-virtualbox-iso-1518710300 systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Jan 11 22:16:07 packer-virtualbox-iso-1518710300 systemd[1]: kibana.service: Unit entered failed state.
Jan 11 22:16:07 packer-virtualbox-iso-1518710300 systemd[1]: kibana.service: Failed with result 'exit-code'.
Jan 11 22:16:07 packer-virtualbox-iso-1518710300 systemd[1]: kibana.service: Service hold-off time over, scheduling restart.
Jan 11 22:16:07 packer-virtualbox-iso-1518710300 systemd[1]: Stopped Kibana.
Jan 11 22:16:07 packer-virtualbox-iso-1518710300 systemd[1]: Started Kibana.

But the service tries to start it again ^.

Then I get;

Jan 11 22:16:12 packer-virtualbox-iso-1518710300 kibana[5451]: {"type":"log","@timestamp":"2019-01-11T22:16:12Z","tags":["info","migrations"],"pid":5451,"message":"Creating index .kibana_2."}
Jan 11 22:16:12 packer-virtualbox-iso-1518710300 kibana[5451]: {"type":"log","@timestamp":"2019-01-11T22:16:12Z","tags":["warning","migrations"],"pid":5451,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_2 and restarting Kibana."}

From this point on, Kibana page only says "Kibana server is not ready yet ".

It's up to the user to stop Kibana, fix the shards (hopefully from a snapshot), and restart it.