Skip to content

Commit 3065bbd

Browse files
gchapsgchaps
authored
[DOCS] Updates the Upgrade docs (#125288) (#125327)
* [DOCS] Adds redirect for Standard Upgrade doc * [DOCS] Updates Upgrade docs * [DOCS] Incorporates review comments * [DOCS] Adds info on saved objects * [DOCS] Fixes broken link * [DOCS] Fixes placement of section on unknown saved objects * [DOCS] Removes duplicate topic (cherry picked from commit a13dafc)
1 parent 0a67100 commit 3065bbd

File tree

9 files changed

+261
-247
lines changed

9 files changed

+261
-247
lines changed

‎docs/developer/architecture/core/index.asciidoc‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -36,9 +36,3 @@ The services that core provides are:
3636
* <<logging-service, Logging service>>
3737
* <<saved-objects-service, Saved Objects service>>
3838
* <<ui-settings-service, UI settings service>>
39-
40-
41-
42-
43-
44-

‎docs/setup/upgrade/logging-configuration-changes.asciidoc‎ renamed to ‎docs/developer/architecture/core/logging-configuration-migration.asciidoc‎

File renamed without changes.

‎docs/developer/architecture/index.asciidoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,8 @@ include::core/http-service.asciidoc[leveloffset=+1]
4040

4141
include::core/logging-service.asciidoc[leveloffset=+1]
4242

43+
include::core/logging-configuration-migration.asciidoc[leveloffset=+1]
44+
4345
include::core/saved-objects-service.asciidoc[leveloffset=+1]
4446

4547
include::core/uisettings-service.asciidoc[leveloffset=+1]

‎docs/redirects.asciidoc‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -386,16 +386,16 @@ This content has moved. Refer to <<managing-data-views>>.
386386

387387
This content has moved. Refer to <<kibana-role-management>>.
388388

389-
[role="exclude",id="logging-configuration-changes"]
390-
== Logging configuration changes
391-
392-
This content has moved. Refer to <<logging-config-changes>>.
393-
394389
[role="exclude",id="upgrade-migrations"]
395390
== Upgrade migrations
396391

397392
This content has moved. Refer to <<saved-object-migrations>>.
398393

394+
[role="exclude",id="upgrade-standard"]
395+
== Standard Upgrade
396+
397+
This content has moved. Refer to {stack-ref}/upgrading-kibana.html[Upgrade Kibana].
398+
399399
[role="exclude",id="upgrade-assistant"]
400400
== Upgrade Assistant
401401

‎docs/setup/upgrade.asciidoc‎

Lines changed: 49 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,72 @@
11
[[upgrade]]
22
== Upgrade {kib}
33

4-
To upgrade from 7.16.0 or earlier to {version},
4+
To upgrade from 7.16.0 or earlier to {version},
55
**you must first upgrade to {prev-major-last}**, which enables you to use the *Upgrade Assistant* to
66
{stack-ref}/upgrading-elastic-stack.html#prepare-to-upgrade[prepare for the upgrade].
7-
Before you upgrade, you must resolve all critical issues identified by the *Upgrade Assistant*.
7+
Before you upgrade, you must resolve all critical issues identified by the *Upgrade Assistant*.
88

9-
Rolling upgrades are unsupported in {kib}. To upgrade,
9+
Rolling upgrades are unsupported in {kib}. To upgrade,
1010
you must shut down all {kib} instances, install the new software, and restart {kib}.
1111
Upgrading while older {kib} instances are running can cause data loss or upgrade failures.
1212

1313
[WARNING]
1414
====
15-
When required, {kib} automatically migrates <<saved-object-migrations, saved objects>>.
15+
When required, {kib} automatically migrates <<saved-object-migrations, saved objects>>.
1616
In case of an upgrade failure, you can roll back to an
1717
earlier version of {kib}. To roll back, you **must** have a
1818
{ref}/snapshot-restore.html[backup snapshot] that includes the `kibana` feature
1919
state. By default, snapshots include the `kibana` feature state.
2020
====
2121

22-
For more information about upgrading,
22+
For more information about upgrading,
2323
refer to {stack-ref}/upgrading-elastic-stack.html[Upgrading to Elastic {version}.]
2424

25-
IMPORTANT: You can upgrade to pre-release versions for testing,
25+
IMPORTANT: You can upgrade to pre-release versions for testing,
2626
but upgrading from a pre-release to the General Available version is unsupported.
2727
You should use pre-release versions only for testing in a temporary environment.
2828

29-
include::upgrade/upgrade-migrations.asciidoc[leveloffset=-1]
29+
[float]
30+
=== Upgrading multiple {kib} instances
31+
When upgrading several {kib} instances connected to the same {es} cluster,
32+
ensure that all outdated instances are shut down before starting the upgrade.
3033

31-
include::upgrade/logging-configuration-changes.asciidoc[]
34+
Rolling upgrades are unsupported in {kib}. However, when outdated instances are shut down, you can start all upgraded instances in parallel,
35+
which allows all instances to participate in the upgrade migration in parallel.
36+
37+
For large deployments with more than 10 {kib} instances, and more than 10,000 saved objects,
38+
you can reduce the upgrade downtime by bringing up a single {kib} instance and waiting for it to
39+
complete the upgrade migration before bringing up the remaining instances.
40+
41+
[float]
42+
[[preventing-migration-failures]]
43+
=== Preparing for migration
44+
45+
There are extra steps you can follow to ensure you are ready for migration.
46+
47+
[float]
48+
==== Ensure your {es} cluster is healthy
49+
Problems with your {es} cluster can prevent {kib} upgrades from succeeding. Ensure that your cluster has:
50+
51+
* Enough free disk space, at least twice the amount of storage taken up by the `.kibana` and `.kibana_task_manager` indices
52+
* Sufficient heap size
53+
* A "green" cluster status
54+
55+
[float]
56+
==== Ensure that all {kib} instances are the same
57+
When you perform an upgrade migration of different {kib} versions, the migration can fail.
58+
Ensure that all {kib} instances are running the same version, configuration, and plugins.
59+
60+
[float]
61+
==== Back up your data
62+
Be sure to have a {ref}/snapshot-restore.html[snapshot] of all your data before attempting a migration.
63+
If something goes wrong during migration, you can restore from the snapshot and try again.
64+
65+
Review the <<resolve-migrations-failures,common causes of {kib} upgrade failures>> and how to prevent them.
66+
67+
68+
include::upgrade/saved-objects-migration.asciidoc[]
69+
70+
include::upgrade/resolving-migration-failures.asciidoc[]
71+
72+
include::upgrade/rollback-migration.asciidoc[]

‎docs/setup/upgrade/resolving-migration-failures.asciidoc‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
[[resolve-migrations-failures]]
2+
=== Resolve migration failures
3+
4+
Migrating {kib} primarily involves migrating saved object documents to be compatible
5+
with the new version.
6+
7+
[float]
8+
==== Resolve saved object migration failures
9+
10+
If {kib} unexpectedly terminates while migrating a saved object index, {kib} automatically attempts to
11+
perform the migration again when the process restarts. Do not delete any saved objects indices to
12+
to fix a failed migration. Unlike previous versions, {kib} 7.12.0 and
13+
later does not require deleting indices to release a failed migration lock.
14+
15+
If upgrade migrations fail repeatedly, refer to
16+
<<preventing-migration-failures, preparing for migration>>.
17+
When you address the root cause for the migration failure,
18+
{kib} automatically retries the migration.
19+
If you're unable to resolve a failed migration, contact Support.
20+
21+
22+
[float]
23+
[[upgrade-migrations-old-indices]]
24+
==== Handle old `.kibana_N` indices
25+
26+
After the migrations complete, multiple {kib} indices are created in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0` etc).
27+
{kib} only uses the index that the `.kibana` and `.kibana_task_manager` aliases point to.
28+
The other {kib} indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version.
29+
30+
[float]
31+
==== Handle known issues with {fleet} beta
32+
If you see a`timeout_exception` or `receive_timeout_transport_exception` error,
33+
it might be from a known known issue in 7.12.0 if you tried the {fleet} beta.
34+
Upgrade migrations fail because of a large number of documents in the `.kibana` index,
35+
which causes {kib} to log errors such as:
36+
37+
[source,sh]
38+
--------------------------------------------
39+
Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms]
40+
41+
Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54]
42+
--------------------------------------------
43+
44+
For instructions on how to mitigate the known issue, refer to https://github.com/elastic/kibana/issues/95321[the GitHub issue].
45+
46+
47+
[float]
48+
==== Handle corrupt saved objects
49+
To find and remedy problems caused by corrupt documents, we highly recommend testing your {kib} upgrade in a development cluster,
50+
especially when there are custom integrations that create saved objects in your environment.
51+
52+
Saved objects that are corrupted through manual editing or integrations cause migration
53+
failures with a log message, such as `Unable to migrate the corrupt Saved Object document ...`.
54+
For a successful upgrade migration, you must fix or delete corrupt documents.
55+
56+
For example, you receive the following error message:
57+
58+
[source,sh]
59+
--------------------------------------------
60+
Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index.
61+
--------------------------------------------
62+
63+
To delete the documents that cause migrations to fail, take the following steps:
64+
65+
. Remove the write block which the migration system has placed on the previous index:
66+
+
67+
[source,sh]
68+
--------------------------------------------
69+
PUT .kibana_7.12.1_001/_settings
70+
{
71+
"index": {
72+
"blocks.write": false
73+
}
74+
}
75+
--------------------------------------------
76+
77+
. Delete the corrupt document:
78+
+
79+
[source,sh]
80+
--------------------------------------------
81+
DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275
82+
--------------------------------------------
83+
84+
. Restart {kib}.
85+
+
86+
The dashboard with the `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` ID that belongs to the `marketing_space` space **is no longer available**.
87+
88+
[float]
89+
[[unknown-saved-object-types]]
90+
==== Handle documents for unknown saved objects
91+
Migrations will fail if saved objects belong to an unknown
92+
saved object type. Unknown saved objects are typically caused by
93+
to the {es} index, or by disabling a plugin that had previously
94+
created a saved object.
95+
96+
We recommend using the {kibana-ref-all}/7.17/upgrade-assistant.html[Upgrade Assistant]
97+
to discover and remedy any unknown saved object types. {kib} version 7.17.0 deployments containing unknown saved
98+
object types will also log the following warning message:
99+
100+
[source,sh]
101+
--------------------------------------------
102+
CHECK_UNKNOWN_DOCUMENTS Upgrades will fail for 8.0+ because documents were found for unknown saved object types. To ensure that upgrades will succeed in the future, either re-enable plugins or delete these documents from the ".kibana_7.17.0_001" index after the current upgrade completes.
103+
--------------------------------------------
104+
105+
If you fail to remedy this, your upgrade to 8.0+ will fail with a message like:
106+
107+
[source,sh]
108+
--------------------------------------------
109+
Unable to complete saved object migrations for the [.kibana] index: Migration failed because documents were found for unknown saved object types. To proceed with the migration, please delete these documents from the ".kibana_7.17.0_001" index.
110+
--------------------------------------------
111+
112+
[float]
113+
==== Handle incompatible settings or mappings
114+
Matching index templates that specify `settings.refresh_interval` or
115+
`mappings` are known to interfere with {kib} upgrades.
116+
This can happen when index templates are defined manually.
117+
118+
To make sure the index templates won't apply to new `.kibana*` indices, narrow down the {data-sources} of any user-defined index templates.
119+
120+
[float]
121+
==== Handle incompatible `xpack.tasks.index` configuration setting
122+
In {kib} 7.5.0 and earlier, when the task manager index is set to `.tasks`
123+
with the configuration setting `xpack.tasks.index: ".tasks"`,
124+
upgrade migrations fail. In {kib} 7.5.1 and later, the incompatible configuration
125+
setting prevents upgrade migrations from starting.

‎docs/setup/upgrade/rollback-migration.asciidoc‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
[[upgrade-migrations-rolling-back]]
2+
=== Roll back to a previous version of {kib}
3+
4+
If you've followed <<preventing-migration-failures, preparing for migration>>
5+
and <<resolve-migrations-failures, resolving migration failures>>, and
6+
{kib} is still unable to successfully upgrade, rollback {kib} until
7+
you're able to identify and fix the root cause.
8+
9+
WARNING: Before you roll back {kib}, ensure that the version you want to roll back to is compatible with
10+
your {es} cluster. If the version you want to roll back to is not compatible, you must also rollback {es}.
11+
Any changes made after an upgrade are lost when you roll back to a previous version.
12+
13+
To roll back after a failed upgrade migration, you must also rollback the saved object indices to be compatible with the previous {kib} version.
14+
15+
[float]
16+
==== Roll back by restoring a backup snapshot
17+
18+
. Before proceeding, {ref}/snapshots-take-snapshot.html[take a snapshot] that contains the `kibana` feature state.
19+
By default, snapshots include the `kibana` feature state.
20+
. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances.
21+
. To delete all saved object indices, use `DELETE /.kibana*`.
22+
. {ref}/snapshots-restore-snapshot.html[Restore] the `kibana` feature state from the snapshot.
23+
. Start all {kib} instances on the older version you want to rollback to.
24+
25+
[float]
26+
==== (Not recommended) Roll back without a backup snapshot
27+
28+
. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances.
29+
. {ref}/snapshots-take-snapshot.html[Take a snapshot] that includes the `kibana` feature state. By default, snapshots include the `kibana` feature state.
30+
. Delete the version-specific indices created by the failed upgrade migration.
31+
+
32+
For example, to rollback from a failed upgrade
33+
to v7.12.0, use `DELETE /.kibana_7.12.0_*,.kibana_task_manager_7.12.0_*`.
34+
. Inspect the output of `GET /_cat/aliases`.
35+
+
36+
If the `.kibana` or `.kibana_task_manager` aliases are missing, you must create them manually.
37+
Find the latest index from the output of `GET /_cat/indices` and create the missing alias to point to the latest index.
38+
For example, if the `.kibana` alias is missing, and the latest index is `.kibana_3`, create a new alias using `POST /.kibana_3/_aliases/.kibana`.
39+
. To remove the write block from the roll back indices, use
40+
`PUT /.kibana,.kibana_task_manager/_settings {"index.blocks.write": false}`
41+
. Start {kib} on the older version you want to rollback to.

‎docs/setup/upgrade/saved-objects-migration.asciidoc‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
[[saved-object-migrations]]
2+
=== Migrate saved objects
3+
4+
Each time you upgrade {kib}, an upgrade migration is performed to ensure that all <<managing-saved-objects,saved objects>> are compatible with the new version.
5+
6+
NOTE: To help you prepare for the upgrade to 8.0.0, 7.17.0 includes an https://www.elastic.co/guide/en/kibana/7.17/upgrade-assistant.html[*Upgrade Assistant*].
7+
To access the assistant, go to *Stack Management > Upgrade Assistant*.
8+
9+
WARNING: {kib} 7.12.0 and later uses a new migration process and index naming scheme. Before you upgrade, read the documentation for your version of {kib}.
10+
11+
WARNING: The following instructions assumes {kib} is using the default index names. If the `kibana.index` or `xpack.tasks.index` configuration settings are different from the default, adapt the instructions accordingly.
12+
13+
[float]
14+
[[upgrade-migrations-process]]
15+
==== How saved objects migrations work
16+
17+
Saved objects are stored in two indices:
18+
19+
* `.kibana_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_7.12.0_001`.
20+
* `.kibana_task_manager_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_task_manager_7.12.0_001`.
21+
22+
The index aliases `.kibana` and `.kibana_task_manager` always point to
23+
the most up-to-date saved object indices.
24+
25+
When you start a new {kib} installation, an upgrade migration is performed before starting plugins or serving HTTP traffic.
26+
Before you upgrade, shut down old nodes to prevent losing acknowledged writes.
27+
To reduce the likelihood of old nodes losing acknowledged writes, {kib} 7.12.0 and later
28+
adds a write block to the outdated index. Table 1 lists the saved objects indices used by previous {kib} versions.
29+
30+
.Saved object indices and aliases per {kib} version
31+
[options="header"]
32+
|=======================
33+
|Upgrading from version | Outdated index (alias)
34+
| 6.5.0 through 7.3.x | `.kibana_N` (`.kibana` alias)
35+
| 7.4.0 through 7.11.x
36+
| `.kibana_N` (`.kibana` alias)
37+
38+
`.kibana_task_manager_N` (`.kibana_task_manager` alias)
39+
|=======================

0 commit comments

Comments
 (0)