[#192624] multiple_es_nodes

[afharo]

Summary

Related #192624.

ES complained because the archive was created with 8.0.0, but ES 9.0.0 requires the datastore to be upgraded to 8.16.0.
Then, it complained because the index from the archives was created on 7.13.0, and required reindexing.

The following steps have been followed to update the archives:

1. Start both nodes of ES 8.16 with the affected data-archive on separate terminals:
   1. Node 01: yarn es snapshot --version=8.16.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip --base-path .es/node01
   2. Node 02: yarn es snapshot --version=8.16.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip --base-path .es/node02
2. After ES is ready (without starting Kibana), reindex the index .kibana_7.13.0_001
   1. Retrieve the settings from the original index via curl -L 'http://localhost:9200/.kibana_7.13.0_001' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d ''
   2. Create the target index with those settings:

   curl -L -X PUT 'http://localhost:9200/.kibana_7.13.0_002' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{
   "mappings": {
       "properties": {
           "bar": {
               "properties": {
                   "status": {
                       "type": "text",
                       "fields": {
                           "keyword": {
                               "type": "keyword",
                               "ignore_above": 256
                           }
                       }
                   }
               }
           },
           "coreMigrationVersion": {
               "type": "keyword"
           },
           "foo": {
               "properties": {
                   "status": {
                       "type": "text",
                       "fields": {
                           "keyword": {
                               "type": "keyword",
                               "ignore_above": 256
                           }
                       }
                   }
               }
           },
           "migrationVersion": {
               "dynamic": "true",
               "properties": {
                   "bar": {
                       "type": "text",
                       "fields": {
                           "keyword": {
                               "type": "keyword",
                               "ignore_above": 256
                           }
                       }
                   },
                   "foo": {
                       "type": "text",
                       "fields": {
                           "keyword": {
                               "type": "keyword",
                               "ignore_above": 256
                           }
                       }
                   }
               }
           },
           "namespace": {
               "type": "keyword"
           },
           "namespaces": {
               "type": "keyword"
           },
           "originId": {
               "type": "keyword"
           },
           "references": {
               "type": "nested",
               "properties": {
                   "id": {
                       "type": "keyword"
                   },
                   "name": {
                       "type": "keyword"
                   },
                   "type": {
                       "type": "keyword"
                   }
               }
           },
           "type": {
               "type": "keyword"
           },
           "updated_at": {
               "type": "date"
           }
       }
   },
   "settings": {
       "index": {
           "hidden": "true",
           "number_of_shards": "1",
           "number_of_replicas": "0"
       }
   }
   }'

   3. Reindex the content: curl -L 'http://localhost:9200/_reindex' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "source": { "index": ".kibana_7.13.0_001" }, "dest": { "index": ".kibana_7.13.0_002" } }'
   4. Remove the old index and recreate the aliases

   curl -L 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{
     "actions": [
       { "add": { "index": ".kibana_7.13.0_002", "alias": ".kibana_7.13.0_001" } },
       { "remove_index": {"index": ".kibana_7.13.0_001" } },
       { "add": { "index": ".kibana_7.13.0_002", "alias": ".kibana_7.13.0" } },
       { "add": { "index": ".kibana_7.13.0_002", "alias": ".kibana" } }
     ]
   }'

3. Stop both ES nodes.
4. Compress both archives

   cd .es/node01/8.16.0
   rm -rf data/nodes # we need to remove this dir or it fails to start again
   zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip data  -x "*/\.*"
   cd ../../../
   cd .es/node02/8.16.0
   rm -rf data/nodes # we need to remove this dir or it fails to start again
   zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip data  -x "*/\.*"
   cd ../../../

5. Run the tests to confirm that the issue is fixed: yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[afharo]

zen is no longer supported

[jbudz]

comment can be removed

[afharo]

Addressed in a225053 (#193899)

[jbudz]

Do we know why the file size increased? mostly curious

[gsoldevila]

#193741 (review)

[afharo]

I'll use ditto, sorry!

[afharo]

Addressed in 6a699b5 (#193899)

[afharo]

Reverted in 97b4937 (#193899). We noticed that the documents were gone 😨

[afharo]

I re-ran it locally and it included them now 🤷
However, somehow, the .zip generated via ditto were even larger than the ones generated with zip, so I stuck to the existing files 😇

[gsoldevila]

LGTM, thanks!

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 97b4937

Failed CI Steps

• Jest Tests #11

Test Failures

• [job] [logs] Jest Tests #11 / Configuration button renders the tooltip correctly when hovering the button

Metrics [docs]

✅ unchanged

History

• 💚 Build #236735 succeeded 1c09fe2

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @afharo

[jbudz]

packages/kbn-test/src/es/test_es_cluster.ts

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation