Saved objects compatibility layer

[jbudz]

This adds a compatibility layer to the saved objects client allowing it to support and return the same responses for crud operations on kibana indices under a single type(6,) and kibana indices using multiple types v(4,5)

To test, create v6 mappings:

curl -XPUT "http://localhost:9200/_template/kibana6" -H 'Content-Type: application/json' -d'
{
  "index_patterns": [".kibana-6*"],
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "doc": {
      "properties": {
        "id": {
          "type": "keyword"
        },
        "type": {
          "type": "keyword"
        },
        "config": {
          "properties": {
            "buildNum": {
              "type": "keyword"
            }
          }
        },
        "index-pattern": {
          "properties": {
            "fieldFormatMap": {
              "type": "text"
            },
            "fields": {
              "type": "text"
            },
            "intervalName": {
              "type": "keyword"
            },
            "notExpandable": {
              "type": "boolean"
            },
            "sourceFilters": {
              "type": "text"
            },
            "timeFieldName": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            }
          }
        },
        "visualization": {
          "properties": {
            "description": {
              "type": "text"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "savedSearchId": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            },
            "uiStateJSON": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            },
            "visState": {
              "type": "text"
            }
          }
        },
        "search": {
          "properties": {
            "columns": {
              "type": "keyword"
            },
            "description": {
              "type": "text"
            },
            "hits": {
              "type": "integer"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "sort": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            }
          }
        },
        "dashboard": {
          "properties": {
            "description": {
              "type": "text"
            },
            "hits": {
              "type": "integer"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "optionsJSON": {
              "type": "text"
            },
            "panelsJSON": {
              "type": "text"
            },
            "refreshInterval": {
              "properties": {
                "display": {
                  "type": "keyword"
                },
                "pause": {
                  "type": "boolean"
                },
                "section": {
                  "type": "integer"
                },
                "value": {
                  "type": "integer"
                }
              }
            },
            "timeFrom": {
              "type": "keyword"
            },
            "timeRestore": {
              "type": "boolean"
            },
            "timeTo": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            },
            "uiStateJSON": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            }
          }
        },
        "url": {
          "properties": {
            "accessCount": {
              "type": "long"
            },
            "accessDate": {
              "type": "date"
            },
            "createDate": {
              "type": "date"
            },
            "url": {
              "type": "text",
                "fields": {
                "keyword": {
                  "type": "keyword",
                    "ignore_above": 2048
                }
              }
            }
          }
        },
        "server": {
          "properties": {
            "uuid": {
              "type": "keyword"
            }
          }
        },
        "timelion-sheet": {
          "properties": {
            "description": {
              "type": "text"
            },
            "hits": {
              "type": "integer"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "timelion_chart_height": {
              "type": "integer"
            },
            "timelion_columns": {
              "type": "integer"
            },
            "timelion_interval": {
              "type": "keyword"
            },
            "timelion_other_interval": {
              "type": "keyword"
            },
            "timelion_rows": {
              "type": "integer"
            },
            "timelion_sheet": {
              "type": "text"
            },
            "title": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            }
          }
        }    
      }
    } 
  }
}'

Reindex:

curl -XPOST "http://localhost:9200/_reindex" -H 'Content-Type: application/json' -d'
{
  "source": {
    "index": ".kibana"
  },
  "dest": {
    "index": ".kibana-6"
  },
  "script": {
    "inline": "    ctx._source = [ ctx._type : ctx._source ];\n    ctx._source.type = ctx._type;\n    ctx._source.id = ctx._id;\n    ctx._id = null;\n    ctx._type = \"doc\";"
  }
}'

Set kibana.index to .kibana-6.

The server will error until everything is switched over to use the api, but endpoints should be available. Test cases detailed in the saved objects pr should behave the same.

[jbudz]

WIP until I have tests covering falling back to v5 indices when v6 fails. Queries should be good to review.

[tylersmalley]

This might be an appropriate time to make a function to handle the returned object from these methods.

[tylersmalley]

Make sure to cover bulkGet. It's currently used for the dashboard import/export.

GET /api/kibana/dashboards/export
POST /api/kibana/dashboards/import

More details here: #10858

[tylersmalley]

Looks like you need to also export getDocType

[tylersmalley]

@jbudz Might want to update the template to set strict mappings

[tylersmalley]

Getting an error prior to the re-index on bulkCreate when used through the dashboard importer.

"error": {
  "message": "type[doc] missing"
}

To reproduce:

if you don't have a Metricbeat-filesystem dashboard, swap out the query param

curl -o dashboard-export.json "http://localhost:5601/api/kibana/dashboards/export?dashboard=Metricbeat-filesystem"
curl -X POST -H "Content-Type: application/json" -d @dashboard-export.json "http://localhost:5601/api/kibana/dashboards/import" | python -m json.tool

[tylersmalley]

We can add an option to define the format to use and default to v6. If it fails, we can re-call bulkCreate with v5 if it's v6.

We can also create a module for each formatting helpers.

[tylersmalley]

Will re-open with changes.