Use single ES document type

[tylersmalley]

Technical Summary:

• Removed compatibility with multi-type indices
• ES Plugin status will be red when multiple, or zero, types are detected
  • devs will need to delete their kibana index or use the migration assistant UI to upgrade the index while still using 5.x
• the ensureTypesExist part of the health check is now patchKibanaIndex, it no longer creates types and applies all patches in a single request
• SavedObjectClient
  • Fixed #find() support for filtering on specific fields
    • improved test coverage by breaking createFindQuery() into pieces and testing each individually
  • Removed normalize_es_doc as the abstractions was subtly different for each method and seemed better implemented inline
  • fields and searchFields must be arrays after the API.
    • respective query params will always be cast to an array by using Joi.array().single()
• server/mappings module added
  • includes what used to be ui/ui_mappings
  • exposes an instance of IndexMappings on kbnServer that should not be used by code outside of the uiExports
    • all mapping properties (except for the defaults) should be defined with uiExports type mappings in a plugin
  • defines server.getKibanaIndexMappingsDsl(), which was previously exposed via kibana.uiExports.mappings.getCombined()
    • code that used to pass around the mappings object and the server now just pass around the server
  • added helpers for reading mapping DSL
  • kbnServer.mappings which is the instance of IndexMappings maintained by the kibana server and extended by the UiExports class using kbnServer.mappings.addRootProperties()
• when in the context of mappings management/es index types, SavedObject "types" are now disambiguated from es types by calling them the "root properties" of the "root type" of the "kibana index mappings dsl"
• ui
  • SavedObjectLoader now sends * (not undefined*) when search string is undefined
  • Converted several uses of indexPattern.id to indexPattern.title, where appropriate
  • Added a mapBreadcrumbs() argument to routes that will allow them to set the display value for the index pattern id in the breadcrumb as the title of the index pattern
  • Removed a stray async function that snuck into angular code
  • When an index pattern doesn't have a title, the id is assumed to be the title (BWC)
• functional tests
  • updated kibanaServer.uiSettings to use SavedObjectClient
  • added kibanaIndex service that fetches a copy of the kibanaIndex mapping and exposes the name of the kibana index to tests, should they need it
  • retry is now in the common services, along with kibanaServer and kibanaIndex

Release Note:

Starting in Elasticsearch 6.0, you are no longer able to create new indices with multiple index types. To accomplish the same effect as index types, a new type field was added to identify the document and the value mapped under the index to allow for strict mapping.

Example

"mappings": {
  "dashboard": {
    "properties": { ... }
  }
}

is now

"mappings": {
  "doc": {
    "properties": {
      "type": { "type": "keyword" },
      "dashboard": {
        "type": "object",
        "properties": { ... }
      }
    }
  }
}

[spalger]

This shouldn't be necessary right? We can just add this to the doc type?

[tylersmalley]

Jenkins, test it.

[tylersmalley]

When a document is created in 5.x with multi-types, we create the document with the id provided.
When we re-index, we are pre-pending the type to the id due to uniqueness only being guaranteed per-type.
Currently, we are utilizing Elastisearch to provide the unique ID's and if we attempt to create a document with a specific ID, then we continue to prepend it.

I believe we should change using uuid.v1() or similar and auto-generate the ids ourselves. This would allow us to, beginning in 6.0, to do lookups/updates/gets by ID.

The change here fixes the issue and I am working to implement and test what I mentioned above.

[tylersmalley]

Here is a PR I would like to pull into this branch: tylersmalley#2

@spalger @epixa, I added you to my fork so you're able to merge.

[spalger]

LGTM, but we probably want to extend the health check to go red when the index is using v5-format before merging

[stacey-gammon]

My little testing with dashboards and dashboard only mode, looks good to me, no issues! (Minus the index id thing showing up in the prefilled list of indexes, but looks like that might have been mentioned already above?)

[tylersmalley]

@epixa - @spalger has a few more tests he is going to push up, then thoughts on merging to unblock on running against ES master? We will create issues for anything that comes up.

[tylersmalley]

@spalger's changes LGTM

[epixa]

@tylersmalley I think that's a good idea.

[epixa]

We need to backport the necessary pieces as well. The longer they sit unbackported, the more likely we'll miss them and cause some really unfortunate regressions with the data, which are the hardest regressions to fix.

[spalger]

LGTM

[jbudz]

jenkins, test it. It failed on the view edit more test, #12014

[spalger]

Actually, failure seems legit,

edit: @tylersmalley Looks like we made the wrong call, filtering the fields caused title^3 to be treated like an invalid field for dashboard, which is why that search failed to find anything^

[epixa]

Green!

[tylersmalley]

MERGED! cc: @elastic/kibana-operations

Working on a backport for the 5.x specific changes contained in this PR.

[tylersmalley]

Partial backport to 5.x: cf5a044

[jimgoodwin]

@tylersmalley can we get a Release Note: summary of the changes