Saved objects compatibility layer

[tylersmalley]

Made updates to #12334 and re-opened while @jbudz is out.

This adds a compatibility layer to the client of the saved object allowing it to support and return the same responses for crud operations on kibana indices under a single type (v6) and kibana indices using multiple types v{5,6}

To test v6 mappings:

curl -XPUT "http://localhost:9200/.kibana-6" -H 'Content-Type: application/json' -d'
{
  "settings" : {
      "number_of_shards": 1,
      "index.mapper.dynamic": false
  },
  "mappings": {
    "doc": {
      "dynamic": false,
      "properties": {
        "type": {
          "type": "keyword"
        },
        "config": {
          "properties": {
            "buildNum": {
              "type": "keyword"
            }
          }
        },
        "index-pattern": {
          "properties": {
            "fieldFormatMap": {
              "type": "text"
            },
            "fields": {
              "type": "text"
            },
            "intervalName": {
              "type": "keyword"
            },
            "notExpandable": {
              "type": "boolean"
            },
            "sourceFilters": {
              "type": "text"
            },
            "timeFieldName": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            }
          }
        },
        "visualization": {
          "properties": {
            "description": {
              "type": "text"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "savedSearchId": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            },
            "uiStateJSON": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            },
            "visState": {
              "type": "text"
            }
          }
        },
        "search": {
          "properties": {
            "columns": {
              "type": "keyword"
            },
            "description": {
              "type": "text"
            },
            "hits": {
              "type": "integer"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "sort": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            }
          }
        },
        "dashboard": {
          "properties": {
            "description": {
              "type": "text"
            },
            "hits": {
              "type": "integer"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "optionsJSON": {
              "type": "text"
            },
            "panelsJSON": {
              "type": "text"
            },
            "refreshInterval": {
              "properties": {
                "display": {
                  "type": "keyword"
                },
                "pause": {
                  "type": "boolean"
                },
                "section": {
                  "type": "integer"
                },
                "value": {
                  "type": "integer"
                }
              }
            },
            "timeFrom": {
              "type": "keyword"
            },
            "timeRestore": {
              "type": "boolean"
            },
            "timeTo": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            },
            "uiStateJSON": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            }
          }
        },
        "url": {
          "properties": {
            "accessCount": {
              "type": "long"
            },
            "accessDate": {
              "type": "date"
            },
            "createDate": {
              "type": "date"
            },
            "url": {
              "type": "text",
                "fields": {
                "keyword": {
                  "type": "keyword",
                    "ignore_above": 2048
                }
              }
            }
          }
        },
        "server": {
          "properties": {
            "uuid": {
              "type": "keyword"
            }
          }
        },
        "timelion-sheet": {
          "properties": {
            "description": {
              "type": "text"
            },
            "hits": {
              "type": "integer"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "timelion_chart_height": {
              "type": "integer"
            },
            "timelion_columns": {
              "type": "integer"
            },
            "timelion_interval": {
              "type": "keyword"
            },
            "timelion_other_interval": {
              "type": "keyword"
            },
            "timelion_rows": {
              "type": "integer"
            },
            "timelion_sheet": {
              "type": "text"
            },
            "title": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            }
          }
        }    
      }
    }
  }
}'

Reindex:

curl -XPOST "http://localhost:9200/_reindex" -H 'Content-Type: application/json' -d'
{
  "source": {
    "index": ".kibana"
  },
  "dest": {
    "index": ".kibana-6"
  },
  "script": {
    "inline": "ctx._source = [ ctx._type : ctx._source ];\n ctx._source.type = ctx._type;\n ctx._id = ctx._type + \":\" + ctx._id;\n ctx._type = \"doc\";"
  }
}'

Set kibana.index to .kibana-6.

The server will error until everything is switched over to use the api, but endpoints should be available. Test cases detailed in the saved objects pr should behave the same.

[epixa]

stray console.log

[epixa]

stray console.log

[epixa]

Nitpick: this can use the short property syntax

[epixa]

Rather than mutating acc, how about using spread:

return [...acc, {}, createIdQuery({ type, id })];

That suggestion out of the way, why push an empty object here (ie the first arg)?

[tylersmalley]

It's the msearch header. We can leave it empty as we're specifying the index on the request instead of per query.

[epixa]

This can be done with a single map rather than closing over and mutating a results accumulator inside a forEach.

[epixa]

Is this trailing comma to appease the linter?

[tylersmalley]

Thought it was needed to ignore the additional assignment, but appears it's not necessary. Will remove.

[epixa]

short property syntax

[epixa]

There's no need to use the Promise constructor here. Since _withKibanaIndex returns a promise itself, you can just chain off of that and return the result.

[tylersmalley]

Here was my manual testing procedure - you will need to update id's before copy/pasting if you use:

Starting with .kibana index

Create a single object

curl -X POST -H "Content-Type: application/json" -H "kbn-xsrf: 6.0.0-alpha3" -d '{ "attributes": { "title": "Test 1 - from v5" } }' http://localhost:5601/api/saved_objects/config/

response: {"id":"AVz7PI7Bo2AvNui3Rihk","type":"config","version":1,"attributes":{"title":"Test 1 - from v5"}}

Update the object

curl -X PUT -H "Content-Type: application/json" -H "kbn-xsrf: 6.0.0-alpha3" -d '{ "attributes": { "title": "Test 1, updated - from v5" } }' http://localhost:5601/api/saved_objects/config/AVz7PI7Bo2AvNui3Rihk

response: {"id":"AVz7PI7Bo2AvNui3Rihk","type":"config","version":2,"attributes":{"title":"Test 1, updated - from v5"}}

Get the object

curl -H "Content-Type: application/json" -H "kbn-xsrf: 6.0.0-alpha3" http://localhost:5601/api/saved_objects/config/AVz7PI7Bo2AvNui3Rihk

response: {"id":"AVz7PI7Bo2AvNui3Rihk","type":"config","version":2,"attributes":{"title":"Test 1, updated - from v5"}}

Import dashboard

curl -X POST -H "Content-Type: application/json" -H "kbn-xsrf: 6.0.0-alpha3" -d '{ "version": "6.0.0-alpha3", "objects": [{ "id": "test", "type": "dashboard", "version": 1, "attributes": { "title": "My Test Dashboard" }}]}' http://localhost:5601/api/kibana/dashboards/import

response:
{"objects":[{"id":"test","type":"dashboard","version":3,"attributes":{"title":"My Test Dashboard"}}]}

Export dashboard

curl "http://localhost:5601/api/kibana/dashboards/export?dashboard=test"

response:
{ "version": "6.0.0-alpha3", "objects": [ { "id": "test", "type": "dashboard", "version": 3, "attributes": { "title": "My Test Dashboard" } } ]

Create template

use template creation script in issue description

Reindex

use reindex script in issue description

Use .kibana-v6

Config should now reflect kibana.index: .kibana-v6

Create a single object

curl -X POST -H "Content-Type: application/json" -H "kbn-xsrf: 6.0.0-alpha3" -d '{ "attributes": { "title": "Test 2 - from v6" } }' http://localhost:5601/api/saved_objects/config

response: {"id":"AVz7Tzwto2AvNui3RjTd","type":"config","version":1,"attributes":{"title":"Test 2 - from v6"}}

Update the object

curl -X PUT -H "Content-Type: application/json" -H "kbn-xsrf: 6.0.0-alpha3" -d '{ "attributes": { "title": "Test 2, updated - from v6" } }' http://localhost:5601/api/saved_objects/config/AVz7Tzwto2AvNui3RjTd

response: {"id":"AVz7Tzwto2AvNui3RjTd","type":"config","version":2,"attributes":{"title":"Test 2, updated - from v6"}}

Get the object

curl -H "Content-Type: application/json" -H "kbn-xsrf: 6.0.0-alpha3" http://localhost:5601/api/saved_objects/config/AVz7Tzwto2AvNui3RjTd

response: {"id":"AVz7Tzwto2AvNui3RjTd","type":"config","version":2,"attributes":{"title":"Test 2, updated - from v6"}}

Get an object from v5

curl -H "Content-Type: application/json" -H "kbn-xsrf: 6.0.0-alpha3" http://localhost:5601/api/saved_objects/config/AVz7PI7Bo2AvNui3Rihk

response: {"id":"AVz7TGDso2AvNui3RjMA","type":"config","version":2,"attributes":{"title":"Test 1, updated - from v5"}}

Bulk get both objects

curl -X POST -H "Content-Type: application/json" -H "kbn-xsrf: 6.0.0-alpha3" -d '[{"type": "config", "id": "AVz7PI7Bo2AvNui3Rihk"}, {"type": "config", "id": "AVz7Tzwto2AvNui3RjTd"}]' http://localhost:5601/api/saved_objects/bulk_get

response: {"id":"AVz7PI7Bo2AvNui3Rihk","type":"config","version":2,"attributes":{"title":"Test 1, updated - from v5"}}

Delete the object

curl -X DELETE -H "Content-Type: application/json" -H "kbn-xsrf: 6.0.0-alpha3" http://localhost:5601/api/saved_objects/config/AVz7PI7Bo2AvNui3Rihk

[kobelb]

One small nitpick; but hitting the APIs directly everything seems to work correctly. Like you mentioned, I'm unable to actually start Kibana without #12669 being done as well.

[kobelb]

Can we move the V6_TYPE constant from saved_objects_client.js and use it here instead of hard-coding 'doc'?

[thomasneirynck]

I went through the test-scenario, and no immediate red flags. Seems to work well!

Below a couple of comments, unsure if they are all related to this PR.

---

After reindexing, object are assigned a new ID property. Is that a property of the re-indexing of ES? This seems to create some inconsistencies.

For example, wrt. dashboard. It was create with the "id": "test", and after re-indexing it gets assigned a new id, e.g.
{"id":"AV0UtNI1urO7MVHxeLol","type":"dashboard","version":1,"attributes":{"title":"My Test Dashboard"}}.

The REST-API then can return that same document using either of the IDs.

• curl http://localhost:5601/api/saved_objects/dashboard/test
• curl http://localhost:5601/api/saved_objects/dashboard/AV0UtNI1urO7MVHxeLol

will return the same document. Is there a reason we cannot reuse the same ID from v5?

---

A couple times, the bulk_get failed for me. Not sure how to reproduce consistently though

> ~/repos/kibana$ curl -X POST -H "Contenton" -H "kbn-xsrf: 6.0.0-alpha3" -d '[{"type": "config", "id": "AV0UqXp3urO7MVHxeLob"}, {"type": "config", "id": "AV0UuO54urO7MVHxeLon"}]' http://localhost:5601/api/saved_objects/bulk_get
                             curl: (7) Failed to connect to localhost port 5601: Connection refused
> ~/repos/kibana$ curl -X POST -H "Content-Type: application/json" -H "kbn-xsrf: 6.0.0-alpha3" -d '[{"type": "config", "id": "AV0UqXp3urO7MVHxeLob"}, {"type": "config", "id": "AV0UuO54urO7MVHxeLon"}]' http://localhost:5601/api/saved_objects/bulk_get
                             {"saved_objects":[{"id":"AV0UtNI1urO7MVHxeLok","type":"config","version":1,"attributes":{"title":"Test 1, updated - from v5"}},{"id":"AV0UuO54urO7MVHxeLon","type":"config","version":2,"attributes":{"title":"Test 2, updated - from v6"}}]}

---

Any reason why the export-method API turns on content-disposition, but the REST-API does not? May help if responses are aligned. (is easier to debug if removed as well).

HTTP/1.1 200 OK
kbn-name: kibana
kbn-version: 6.0.0-alpha3
vary: origin
content-type: application/json; charset=utf-8
cache-control: no-cache
content-length: 103
accept-ranges: bytes
Date: Wed, 05 Jul 2017 22:21:37 GMT
Connection: keep-alive

{"id":"AV0UtNI1urO7MVHxeLol","type":"dashboard","version":1,"attributes":{"title":"My Test Dashboard"}}```
vs
```~/repos/kibana$ ~url http://localhost:5601/api/kibana/dashboards/export?dashboard=test -i
HTTP/1.1 200 OK
content-disposition: attachment; filename="kibana-dashboards.2017-07-05-22-22-21.json"
content-type: application/json; charset=utf-8
content-length: 213
kbn-name: kibana
kbn-version: 6.0.0-alpha3
vary: origin
cache-control: no-cache
accept-ranges: bytes
Date: Wed, 05 Jul 2017 22:22:21 GMT
Connection: keep-alive

{
  "version": "6.0.0-alpha3",
  "objects": [
    {
      "id": "AV0UtNI1urO7MVHxeLol",
      "type": "dashboard",
      "version": 1,
      "attributes": {
        "title": "My Test Dashboard"
      }
    }
  ]
}```

[thomasneirynck]

I guess this explains my earlier id-comment.

Isn't there a possible conflict here?

E.g., suppose I create a new dashboard after re-indexing with the same human readable id I had from an older v5 dashboard.

curl -X POST -H "Content-Type: application/json" -H "kbn-xsrf: 6.0.0-alpha3" -d '{ "version": "6.0.0-alpha3", "objects": [{ "id": "test", "type": "dashboard", "version": 1, "attributes": { "title": "My Test Dashboard From V6" }}]}' http://localhost:5601/api/kibana/dashboards/import
{"objects":[{"id":"test","type":"doc","version":1,"attributes":{"title":"My Test Dashboard From V6"}}]}

This creates a new dashboard:

curl http://localhost:5601/api/saved_objects/dashboard
{"saved_objects":[{"id":"AV0UtNI1urO7MVHxeLol","type":"dashboard","version":1,"attributes":{"title":"My Test Dashboard"}},{"id":"test","type":"dashboard","version":1,"attributes":{"title":"My Test Dashboard From V6"}}],"total":2,"per_page":20,"page":1}

But when I now query by id, I get the old v5 dashboard like earlier.

curl http://localhost:5601/api/saved_objects/dashboard/test
{"id":"AV0UtNI1urO7MVHxeLol","type":"dashboard","version":1,"attributes":{"title":"My Test Dashboard"}}

[tylersmalley]

When attempting to import that dashboard I get an error that the document already exists:

curl -X POST -H "Content-Type: application/json" -H "kbn-xsrf: 6.0.0-alpha3" -d '{ "version": "6.0.0-alpha3", "objects": [{ "id": "test", "type": "dashboard", "attributes": { "title": "My Test Dashboard From V6" }}]}' http://localhost:5601/api/kibana/dashboards/import
{"objects":[{"id":"test","type":"dashboard","attributes":{"title":"My Test Dashboard From V6"},"error":{"message":"[dashboard][test]: version conflict, document already exists (current version [3])"}}]}

Ping me tomorrow and let's go over this live.

[tylersmalley]

@thomasneirynck and I worked through and were able to distil the problem.

A user on 5.6 has the following visualization object in ES:

{
  _id: 'foo',
  _type: 'visualization'
}

They export the saved object, then run the Kibana index migration.

This will result in their ES object being transformed into the following:

{
  _id: 'abc123',
  _type: 'doc',
  _source: {
    type: 'visualization',
    legacyId: 'foo'
  }
}

If the user then re-imports the export previously made in 5.6 they would be forcing the document to be created with the id specified in order to avoid breaking associations.

{
  _id: 'foo',
  _type: 'doc',
  _source: {
    type: 'visualization'
  }
}

The current thought is we need to first query for the document before inserting. If a document is returned, and we're forcing the overwrite we will simply update the document. For bulkCreate we would first need to perform a bulkGet on the documents. This doesn't seem like a great solution, as it's not atomic and feels like a hack we might not be able to get away from.

[tylersmalley]

@epixa and I discussed. There are a few changes we're going to make to address this.

First, instead of moving the existing _id to legacyId and auto-generating _id, we're going to set to it something we can expect: ${_type}:${_id}.

Secondly, we're going to begin including a saved_object_version in the exports, beginning with version 2. This will allow us to assume all existing exports are version 1 and should use the concatenated type and ID as the id.

[tylersmalley]

@thomasneirynck, thanks for the review.

After reindexing, object are assigned a new ID property. Is that a property of the re-indexing of ES? This seems to create some inconsistencies.

Previously id's were only unique to a single document type. Since the removal of types, we must move all of our documents under a single type which causes id conflicts. To address this, we are re-indexing and copying the existing _id to a legacyId property. This will still allow us to fetch the records using this id for legacy purposes, but we will use the new ID to update/delete the record. This should allow previously saved/shared searches, visualizations, and dashboards to effectively work with their existing id. This legacyId is only visible on the Elasticsearch document itself and is not surfaced through the API to prevent us from miss-using it. It's my current opinion this should only exist for legacy lookups.

Any reason why the export-method API turns on content-disposition, but the REST-API does not?

@simianhacker, can you provide insights into the reason specifying the content-disposition for the dashboard export API?

[tylersmalley]

@epixa @kobelb, mind taking another look over this one?

[kobelb]

@tylersmalley functionally this appears to be working properly. However, the SavedObjectsClient tests seem to only test the v5 functionality for create/update methods and not the fallbacks.

[tylersmalley]

@kobelb, good catch - will add those.

[tylersmalley]

@kobelb - tests have been added surrounding the v6 fallback

[tylersmalley]

@kobelb, let me know if this looks good to you and I will go ahead an merge.

[kobelb]

Can we use the V6_TYPE from the 'saved_objects_client' here?

[kobelb]

What's preventing this from removing the ${type}: from weird unconverted v5 ids? Could we do the doc._type === V6_TYPE check that's below here as well?

[kobelb]

LGTM

[tylersmalley]

5.x: 7932e62

[spalger]

💃 💃