Corrupt Saved Object documents can cause a migration every time Kibana starts up

[rudolf]

When starting up, Kibana scans the .kibana alias/index for documents that require a migration. For each document being migrated, we do a basic validation that checks if the type and namespace in the id matches the type and namespace fields. If these don't match, we don't apply the migration and just write the document as-is. Because the document's migrationVersion wasn't bumped, the next time Kibana starts up it will again see outdated documents and start a new migration.

kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.ts

Line 41 in ea7c78d

if (serializer.isRawSavedObject(raw)) {

Because the Saved Objects API ensures that written documents are always valid, this will only affect users which write saved object documents directly into Elasticsearch instead of using the Kibana API's.

As a first step we can add logging to highlight this case to users.

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[rudolf]

This bug can also lead to data loss if multiple Kibana nodes are using the same Elasticsearch index. If one of the nodes are restarted while the other nodes accept writes, the restarted node will create a new .kibana_n+1 index and start migrating documents over to it. If old nodes update documents in .kibana_n which were already migrated by the restarted node, or create new documents, these writes will be lost once the restarted node moves the .kibana alias from .kibana_n to .kibana_n+1.

[rudolf]

Once we complete #75780 Kibana upgrade migrations will fail if there are corrupt saved object documents encountered. Users will have to delete the corrupt document to proceed. This prevents hard to debug migration problems where we're unable to transform the corrupt document but still try persist it leading which might lead to a mappings exception.