Skip to content

[esArchiver] v6 kibana index support#12854

Closed
spalger wants to merge 15 commits intoelastic:masterfrom
spalger:implement/es-archiver/v6-index-support
Closed

[esArchiver] v6 kibana index support#12854
spalger wants to merge 15 commits intoelastic:masterfrom
spalger:implement/es-archiver/v6-index-support

Conversation

@spalger
Copy link
Contributor

@spalger spalger commented Jul 14, 2017

Requires #12794

When we remove support for v5 kibana indices we will need to update our es archives to use v6 format. To do that we add a convertToV6Stream to the esArchiver that converts doc and index records to v6 format on the fly. This enabled:

  • loading v5 data files with the load command, which write the data into elasticsearch in v6 format which powers...
  • the new reindex command which will read data from elasticsearch in v5 format and put it back into elasticsearch in v6 format
  • and allows the rebuild-all command to read the archives from disk in v5 format and write them back to disk in v6 format

As always, use node scripts/es_archiver --help for CLI syntax. For the impatient, the following will read the .kibana index (or any indices specified after reindex) to a tmp directory on disk, then delete the indices, and recreate them by first converting the docs into v6 format

node scripts/es_archiver reindex .kibana 

The existing archives were also updated to use v6 format using the rebuild-all command.

tylersmalley and others added 11 commits July 11, 2017 15:56
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
When creating an id, we will always prefix the type to either an id which was provided or one we generate. This alows us to simplify all actions for managing documents.

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
Remove entire compatibility layer and updates id generation
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@spalger spalger requested a review from epixa July 14, 2017 06:56
@spalger spalger changed the title [esArchiver] v6 index support [esArchiver] v6 kibana index support Jul 14, 2017
@spalger spalger force-pushed the implement/es-archiver/v6-index-support branch 3 times, most recently from f732b17 to 3e2383d Compare July 14, 2017 07:05
@spalger spalger force-pushed the implement/es-archiver/v6-index-support branch from afdcd50 to 95b3063 Compare July 14, 2017 07:24
@spalger spalger force-pushed the implement/es-archiver/v6-index-support branch from 95b3063 to 3ab51d6 Compare July 14, 2017 07:42
@spalger
Copy link
Contributor Author

spalger commented Jul 14, 2017

Merged esArchive updates into #12794, since reindex command would only be useful for a hot minute we aren't going to merge it into the project. If devs want to migrate existing kibana indices to v6 format they can use the upgrade UI in 5.6. Otherwise starting from a fresh kibana index will do the trick

@spalger spalger closed this Jul 14, 2017
@spalger spalger deleted the implement/es-archiver/v6-index-support branch July 14, 2017 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants