[Index management] Prepare support Index template V2 format by sebelga · Pull Request #61588 · elastic/kibana

sebelga · 2020-03-27T09:44:14Z

This PR prepares the index_management plugin to support the upcoming Index Template V2 (elastic/elasticsearch#53101).

Context

In v7.8 Elasticsearch is going to release a new API to read/create/update/delete index templates. Those index templates will have a different format (V2) that the current index template (V1).

The main difference being:

// V1 format

{
  "templateName": {
    "aliases: {},
    "mappings": {}.
    "settings": {},
    ...
  }
}

// V2 format
{
  "templateName: {
    "template": { // wrapped under _template_ parameter
      "aliases: {},
      "mappings": {}.
      "settings": {},
    },
    // new parameter. An array of template components (upcoming new API for 7.8),
    "composed_of": []
    // new parameter. An integer to set the priority of the index template to apply in case 2 index patterns matches an index name
    "priority: 1,
    ... // remaining parameters are the same
  }
}

There will be 2 different APIs to manage index templates:

PUT _template for the V1 format (supported until the end of 8.x cycle)
PUT _index_template for the V2 format (supported from 7.8)

This means that 2 index templates can have the same name with a different version.

Changes in this PR

This PR is a preparation work to support both formats in the UI. The client code has been refactored to work only with V2 index template format (it expects V2 format to be thrown at it) as it will be the future of index template. A _kbnMeta object is added from the server to let the client know the format version of the template.

Knowing the format version is important for the client as, when we will fetch or delete an index template, we need to tell our API server route the format version so we can proxy the call to the correct Elasticsearch API.

We will also, in a future PR, enhance the table view to clearly indicate to the user the format of the index template. And probably add a deprecation warning for v1 template and a button to convert V1 index templates to the V2 format.

This PR still creates V1 index templates but our goal for 7.8 is to create V2 index templates by default.

How to review

Create an index template and verify that the Request preview (in Review step) is the V1 format

Edit the index template (from the pencil icon in the table, the context menu in the table and the context menu in the details flyout), make sure it saves correctly
Clone an index template (from the context menu in the table and the context menu in the details flyout), make sure it saves correctly

Fixes #57288

… one

elasticmachine · 2020-03-30T14:58:24Z

Pinging @elastic/es-ui (Team:Elasticsearch UI)

sebelga · 2020-03-31T17:08:24Z

@elasticmachine merge upstream

alisonelizabeth

Great start with this @sebelga! Code LGTM. Tested locally and everything worked as expected. I did notice a test that was previously skipped failed on the latest run; there might be some more investigation needed there.

alisonelizabeth · 2020-03-31T22:07:40Z

x-pack/plugins/index_management/common/constants/index_templates.ts

+
+/**
+ * Up until the end of the 8.x release cycle we need to support both
+ * V1 and V2 index template formats. This constant keeps track of wether


Suggested change

* V1 and V2 index template formats. This constant keeps track of wether

* V1 and V2 index template formats. This constant keeps track of whether

alisonelizabeth · 2020-03-31T22:11:31Z

x-pack/plugins/index_management/common/types/mappings.ts

+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+// TODO: Move mappings type from Mappings editor here


Do you plan to address this as part of this PR?

I didn't plan on touching the mappings editor in this PR. As I plan on adding test coverage to the mappings editor I will do it then.

alisonelizabeth · 2020-03-31T22:15:02Z

.../plugins/index_management/public/application/components/template_form/steps/use_json_step.ts

    const value = isValid && content.trim() !== '' ? JSON.parse(content) : {};
-    const data = { [prop]: value };
-    return Promise.resolve({ isValid, data });
+    // If no key has been added to the JSON object, we strip it out so an empty objecct is not sent in the request


Suggested change

// If no key has been added to the JSON object, we strip it out so an empty objecct is not sent in the request

// If no key has been added to the JSON object, we strip it out so an empty object is not sent in the request

alisonelizabeth · 2020-03-31T22:31:00Z

x-pack/plugins/index_management/public/application/lib/index_templates.ts

+import { Location } from 'history';
+
+export const getFormatVersionFromQueryparams = (location: Location): 1 | 2 | undefined => {
+  const { v } = parse(location.search.substring(1));


nit: this might be more clear as const { v: version } = ...

alisonelizabeth · 2020-03-31T22:56:19Z

x-pack/plugins/index_management/__jest__/client_integration/template_edit.test.tsx


-    // TODO: Flakey test
-    describe.skip('form payload', () => {
+    describe('form payload', () => {


Looks like this failed on the latest run. Might be more work that needs to be done. Related issue: #57288.

sebelga · 2020-04-01T11:51:33Z

Thanks for the review @alisonelizabeth! Indeed the flakiness was still there. I added a new helper in the Testbed utils: waitFor('myTestSubject') which should help us for this type of flakiness related to hooks cycle update. (no need to add an estimated guess in await nextTick(43); // wait 43 millisecond anymore 😄)

I will run the CI on this PR a few times to make sure.

sebelga · 2020-04-01T21:09:41Z

@elasticmachine merge upstream

cjcenizal · 2020-04-02T00:48:41Z

@elasticmachine merge upstream

sebelga · 2020-04-02T11:27:11Z

@elasticmachine merge upstream

sebelga · 2020-04-02T15:54:00Z

@elasticmachine merge upstream

kibanamachine · 2020-04-02T17:39:54Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 65451f5

History

💚 Build #38138 succeeded 6e95fd7
💛 Build #38095 was flaky 6176123
💚 Build #38081 succeeded d93ac4e
💚 Build #38053 succeeded 6176123
💚 Build #37995 succeeded 6176123

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* master: Switch to embeddable factory interface with optional override (elastic#61165) fix text error in diagrams (elastic#62101) [Index management] Prepare support Index template V2 format (elastic#61588) Updates dashboard images (elastic#62011) [Maps] remove MapBounds type (elastic#62332) [Uptime] Convert anomaly Jobs name to lowercase to comply with… (elastic#62293) Make d3 place nicely with object values (elastic#62004) EMT-287: update schema with elastic agent id (elastic#62252) [Maps] fix replaceLayerList to handle case where map is not intialized (elastic#62202) Remove support for deprecated xpack.telemetry configurations (elastic#51142) [Uptime] Remove static constant for index name completely (elastic#62256) [APM] E2E: install dependencies for vanilla workspaces (elastic#62178) [backport] Bump to 5.1.3 (elastic#62286) Show server name in Remote Cluster detail panel (elastic#62250) Rename some alert types (elastic#61693) changing duration type to ms, s, m (elastic#62265) [ML] Clear Kibana index pattern cache on creation or form reset. (elastic#62184) Move `src/legacy/server/index_patterns` to data plugin (server) (Remove step) (elastic#61618) [NP] Remove IndexedArray usage in Schemas (elastic#61410)

cjcenizal · 2020-04-21T20:09:39Z

@sebelga Can you confirm that I understand the changes introduced by this PR correctly?

The GET /templates route currently fetches only v1 templates. In the future, we intend to update it to fetch both v1 and v2 index templates.
This route has been updated to normalize all fetched templates to the v2 shape before returning them to the client. As part of this normalization step, we add the _kbnMeta property to index templates to differentiate between those fetched from the v1 API and those fetched from the v2 API.
The client has been updated to accommodate the new v2 shape, and no longer supports the v1 shape.
The PUT /templates/{name} and POST /templates/{name} endpoints for creating and updating index templates have been updated to check the version stored in the _kbnMeta property, and either convert to v1 shape or throw an error. In the future, we intend to update it to save both v1 and v2 index templates using their respective APIs.

sebelga · 2020-04-22T07:19:30Z

@cjcenizal this is correct. Apart from the PUT and POST, the DELETE also will need to be updated to support both APIs.

sebelga force-pushed the index-template-v2/table-view branch from d805d58 to ac7efa0 Compare March 27, 2020 13:13

sebelga added 20 commits March 30, 2020 16:07

Add helper method to get template format version (1 or 2)

ec1bd97

Rename interface to make explicit deserialized template vs serialized…

1a91c00

… one

Refactor template list deserialization

c98c1a5

Move utils to common folder

57fb494

Differentiate serialisation to V1 or V2 template format

ac46764

Improve typings and make V2 format the default for client

0054cb0

Fix api integration tests

570e61f

Refactor client code to work with V2 template format (in and out)

4763cef

[useRequest] Fix TS type for data returned

20d24b7

Add constant to keep track of the default index template format

1415fb7

Fix TemplateList and TS issues

a96abc7

Fix component integration tests

b936e31

Set default template name to empty string

c2dc9f0

Add support for v2 format serialization in step review

199763c

Add deserializer for V2 template format (server side)

8b0caf2

Only allow template V1 to be created or edited

7fb48f8

Require version to be provided when fetching template

2a3fc01

Fix template V1 serialization

e3e3ccb

Update API delete template route to receive template format version

d220224

Fix API integration tests

ca0f0ec

sebelga force-pushed the index-template-v2/table-view branch from c2f7104 to ca0f0ec Compare March 30, 2020 14:07

Fix component integration tests

74dfe83

sebelga requested a review from alisonelizabeth March 30, 2020 14:53

sebelga marked this pull request as ready for review March 30, 2020 14:53

sebelga added Feature:Index Management Index and index templates UI Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more t// v7.8 v8.0.0 labels Mar 30, 2020

Merge branch 'master' into index-template-v2/table-view

6afa4d8

alisonelizabeth approved these changes Mar 31, 2020

View reviewed changes

sebelga added 4 commits April 1, 2020 13:42

[TestBed] Add "waitFor" helper for flaxy tests

41efbc3

Make CR suggestions

b0f193c

[Form lib] Fix FormDataProvider initial state

ede49bf

Fix flaxy component integration test

0f12470

sebelga added 2 commits April 1, 2020 18:04

Fix component integration tests

d2ffe6c

Fix flaky test for <LoadMappingsProvider />

4b2c9bb

Merge branch 'master' into index-template-v2/table-view

6176123

Merge branch 'master' into index-template-v2/table-view

d93ac4e

shahzad31 requested a review from a team as a code owner April 2, 2020 06:32

Refactor TestBed.waitFor() and wait up to 30 sec

6e95fd7

Merge branch 'master' into index-template-v2/table-view

65451f5

sebelga merged commit 74d8413 into elastic:master Apr 2, 2020

sebelga deleted the index-template-v2/table-view branch April 2, 2020 18:14

sebelga mentioned this pull request Apr 3, 2020

[7.x] [Index management] Prepare support Index template V2 format (#61588) #62400

Merged

rayafratkina added v7.8.0 and removed v7.8 labels Apr 6, 2020

kibanamachine mentioned this pull request Feb 7, 2022

[IM] Add missing values to the index details flyout #124731

Merged

1 task

	* V1 and V2 index template formats. This constant keeps track of wether
	* V1 and V2 index template formats. This constant keeps track of whether

	// If no key has been added to the JSON object, we strip it out so an empty objecct is not sent in the request
	// If no key has been added to the JSON object, we strip it out so an empty object is not sent in the request

Conversation

sebelga commented Mar 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Changes in this PR

How to review

Uh oh!

elasticmachine commented Mar 30, 2020

Uh oh!

sebelga commented Mar 31, 2020

Uh oh!

alisonelizabeth left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alisonelizabeth Mar 31, 2020

Choose a reason for hiding this comment

Uh oh!

alisonelizabeth Mar 31, 2020

Choose a reason for hiding this comment

Uh oh!

sebelga Apr 1, 2020

Choose a reason for hiding this comment

Uh oh!

alisonelizabeth Mar 31, 2020

Choose a reason for hiding this comment

Uh oh!

alisonelizabeth Mar 31, 2020

Choose a reason for hiding this comment

Uh oh!

alisonelizabeth Mar 31, 2020

Choose a reason for hiding this comment

Uh oh!

sebelga commented Apr 1, 2020

Uh oh!

sebelga commented Apr 1, 2020

Uh oh!

cjcenizal commented Apr 2, 2020

Uh oh!

sebelga commented Apr 2, 2020

Uh oh!

sebelga commented Apr 2, 2020

Uh oh!

kibanamachine commented Apr 2, 2020

💚 Build Succeeded

History

Uh oh!

cjcenizal commented Apr 21, 2020

Uh oh!

sebelga commented Apr 22, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sebelga commented Mar 27, 2020 •

edited

Loading

alisonelizabeth left a comment •

edited

Loading