[WIP] Migrations/major version

[chrisdavies]

This is one in a series of several PRs that will go towards resolving #15100.

This builds on the original PR, so it's probably best to wait to review this until that one is approved.

This introduces the following changes:

• Adds core logic for migrating indices between major versions of Kibana
• Adds helper functions to:
  • Compute what types / docs would be dropped if a migration were run
  • Compute what types are currently supported by active plugins
• Does not provide a mechanism to run major migrations, as such a mechanism is TBD

Testing this PR

• To run automated tests, see the readme
• To manually test,
  • You can test against your own custom indices, as the migraiton logic is generic, but these instructions are for the kibana index
    • If you want to test against your own indices, just change the var index = '.kibana;` in the REPL example below
  • Unless you want to test against an empty index,
    • Launch Kibana and make some things so that the Kibana index has data (e.g. use Nate's nice sample data feature or something)
  • Create a plugin, or modify an existing one (such as the Kibana plugin)
  • Add a migration to the plugin definition (see readme.md for an example)
  • Launch Kibana in repl mode: yarn start --repl
  • Wait for it to boot
  • Run migrations using the sample code provided below
• Things to look for as you test:
  • Try making breaking mapping changes
  • Try renaming a type
  • Try dropping support for a type
  • Does it behave as you expect?
  • If you get the system to error during migrations, does it give a good explanation of what's wrong?
  • Is it clear how to fix / recover from errors?

We haven't determined how major migrations will be launched (via UI, via cli, automatically on start (probably not))? So, this PR is only the core of major index migrations. To actually run migrations, you have to tinker with some code, which is easily done via the REPL. The following code provides a way to migrate the index, and a way to undo the migration (undoMigrate()) if the migration fails. If the undo function is handy enough, we should probably incororate it into the codebase.

Note: it's helpful to use var in the REPL, as it allows you to more easily revover from mistakes.

// Some helpful values that you'll use when tinkering w/ migrations
var callCluster = kbnServer.server.plugins.elasticsearch.getCluster('admin').callWithInternalUser;
var kibanaVersion = kbnServer.version;
var index = '.kibana';
var log = (...args) => kbnServer.server.log(...args);
var { getMigrationPlugins, migrateIndex } = require('./src/server/saved_objects/migrations');
var plugins = getMigrationPlugins(kbnServer)
var opts = { callCluster, index, kibanaVersion, log, plugins, dropUnsupportedTypes: false };

// A helpful funtion for reverting the index back to its previous state
async function undoMigrate() {
  await callCluster('indices.deleteAlias', { name: '.kibana', index: '.kibana-7' });
  var mappings = await callCluster('indices.getMapping', { index: '.kibana-7' });
  await callCluster('indices.create', { index: '.kibana', body: { mappings: Object.values(mappings)[0].mappings } });
  await callCluster('reindex', { body: { dest: { index: '.kibana', }, source: { index: '.kibana-7', }, }, refresh: true, waitForCompletion: true, });
  await callCluster('indices.delete', { index: '.kibana-7' });
  await callCluster('indices.delete', { index: '.kibana-6' });
}

// Migrate the index
migrateIndex(opts);

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[chrisdavies]

Jenkins test this

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request