-
Notifications
You must be signed in to change notification settings - Fork 242
Rework migrate data docs (Take 2) #5244
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 14 commits
Commits
Show all changes
57 commits
Select commit
Hold shift + click to select a range
a582571
Rework migrate data docs
8ec1725
fixup
dee3acf
Move in snapshot-restore steps
a1f09c5
fix links
2396ab1
link fix
3a1f2be
link fix
a4376c6
Expand reindex from remote steps
e0b9f57
fixup
336b4e9
renmate migrate.md for easier diff / reviewing
e7fb4cf
fix links
8c874cf
fix labels
4473f0e
add redirect
8b9d2c9
add variables
e52e358
cleanup
8c3b51f
Add reindex method for Serverless -> ECE
c7568e3
Add dual ingestion method
5b66faa
typo
ab3a4a5
fix preview label for serverless reindex
bab16ae
Add Edu's suggestions
0a9ce5a
fix tagging
d271b91
fix label again
57bb75a
Merge branch 'main' into 4353/rework-ech2ess-mig-v2
kilfoyle 85bae34
review comment fixes
a60750c
Merge branch 'main' into 4353/rework-ech2ess-mig-v2
kilfoyle 51bbbda
edits
e3b48c7
Address comments, part uno
f9cd891
fix links
818d2cd
Merge branch 'main' into 4353/rework-ech2ess-mig-v2
kilfoyle e2b8956
ci fixes
5929fca
Reorganize snapshot restore page
7c80328
Move methods to table; fix link
4961fdd
typo
f8618ee
link fix
9dca097
Apply suggestion from @shainaraskas
kilfoyle 7c4a90a
Apply suggestion from @shainaraskas
kilfoyle 27b85c4
Apply suggestion from @shainaraskas
kilfoyle 8296de0
Apply suggestion from @shainaraskas
kilfoyle fa8076b
Apply suggestion from @shainaraskas
kilfoyle 1051dee
Apply suggestion from @shainaraskas
kilfoyle 378c44a
Apply suggestion from @shainaraskas
kilfoyle 2b795a1
Update manage-data/migrate.md
kilfoyle 8cc4cd5
Update manage-data/migrate.md
kilfoyle 65e886a
Update manage-data/migrate.md
kilfoyle fe6bd2d
Update manage-data/migrate.md
kilfoyle 4732c1b
Update manage-data/migrate.md
kilfoyle 8071cf9
Update manage-data/migrate.md
kilfoyle 2c97cb1
Update manage-data/migrate.md
kilfoyle 3a9ae2e
Remove one prereq
425e55d
Merge branch 'main' into 4353/rework-ech2ess-mig-v2
kilfoyle 1d640e8
Merge branch 'main' into 4353/rework-ech2ess-mig-v2
kilfoyle a36f40d
Merge branch 'main' into 4353/rework-ech2ess-mig-v2
kilfoyle adb6b2f
Add variable for ES API (from 6147 merge conflict)
7150cd8
Merge branch 'main' into 4353/rework-ech2ess-mig-v2
kilfoyle 33d402d
Merge branch 'main' into 4353/rework-ech2ess-mig-v2
yetanothertw 12f327c
Merge branch 'main' into 4353/rework-ech2ess-mig-v2
georgewallace 0e883a5
Merge branch 'main' into 4353/rework-ech2ess-mig-v2
shainaraskas 6a10ce9
Merge branch 'main' into 4353/rework-ech2ess-mig-v2
georgewallace File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| --- | ||
| mapped_pages: | ||
| - https://www.elastic.co/guide/en/cloud/current/ec-migrating-data.html | ||
| - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-migrating-data.html | ||
| - https://www.elastic.co/guide/en/cloud-heroku/current/ech-migrate-data2.html | ||
| applies_to: | ||
| stack: ga | ||
| serverless: ga | ||
| products: | ||
| - id: elasticsearch | ||
| - id: cloud-hosted | ||
| - id: cloud-enterprise | ||
| - id: cloud-kubernetes | ||
| --- | ||
|
|
||
| # Migrate your {{es}} data [migrate-your-elasticsearch-data] | ||
|
|
||
| Transitioning between Elastic deployment types involves migrating your {{es}} data. This page helps you plan your migration by describing the main categories of data you may need to move (ingested user data, {{es}} system data, {{kib}} saved objects, and feature-specific data), the migration methods available for each, and where to find step-by-step guides for your scenario. | ||
|
|
||
| ## Data types [migration-data-types] | ||
|
|
||
| Your migration options depend on the type of data that you need to migrate, which can be categorized into four groups: | ||
|
|
||
| - **Ingested user data**: All of the data that you've added into {{es}}, structured or unstructured, for your own applications. | ||
|
|
||
| - **{{es}} system data**: Configuration and state information stored in {{es}} [system indices](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#system-indices). This data is used by {{es}} for its internal operations. | ||
|
kilfoyle marked this conversation as resolved.
Outdated
|
||
|
|
||
| - **{{kib}} saved objects**: Dashboards, visualizations, maps, data views, Canvas workpads, and any other objects that you've saved in {{kib}}. | ||
|
|
||
| - **Feature and component data**: Data stored in {{es}} that is specific to a given Elastic feature or component. This includes, for example, configuration data for {{fleet}} and {{integrations}}, {{watcher}} data, alerting and security detection rules, security data such as role mappings, API keys, and service tokens, and others. | ||
|
|
||
| ## Migration options [migration-options] | ||
|
|
||
| Depending on the type of data that you need to move, various migration options are available: | ||
|
|
||
| - **Reindex from source**: For your own data, reindexing into your new, destination deployment from the data's original source is typically the most straightforward approach, since it's available without any need to consider differing {{es}} versions or deployment types. | ||
|
kilfoyle marked this conversation as resolved.
Outdated
kilfoyle marked this conversation as resolved.
Outdated
kilfoyle marked this conversation as resolved.
Outdated
|
||
|
|
||
| If you still have access to the original data source, outside of your former {{es}} cluster, you can load the data from there. You have the option to use any ingestion method that you want—{{ls}}, {{agent}}, {{beats}}, the {{es}} clients, or whatever works best for you. | ||
|
|
||
| If the original source isn’t available or has other issues that make it non-viable, you can choose from one of the other migration options described here. | ||
|
|
||
| - **Snapshot and restore**: Use a snapshot to create a backup of your running {{es}} cluster, and then migrate by restoring your data into a new cluster. | ||
| - **Reindex API**: Copy documents from a source index to a destination index. You can reindex across clusters and deployment types and transform the data en route. | ||
| - **{{ls}}**: With {{ls}} you can collect, process, and forward data from a variety of sources to a variety of destinations. It serves as a highly configurable option available for migrating data across any deployment types. | ||
|
kilfoyle marked this conversation as resolved.
Outdated
|
||
| - **Saved objects API**: Use this API to migrate objects that you've saved in {{kib}}. | ||
| - **{{kib}} saved object management**: You can also use the {{kib}} UI to migrate your saved objects. | ||
|
|
||
| The following table describes the migration options available for each data type and where to find guidance. | ||
|
|
||
| | Data type | Migration options | | ||
| | ------ | ------ | | ||
| | Ingested user data | The reindex API, snapshot and restore, and {{ls}} migration options are available for your user data, with some restrictions based on the source and target deployment type. Refer to [User data migration guides](#data-migration-guides) on this page to learn more. | | ||
| | {{es}} system data | System indices must be migrated using the snapshot and restore [feature states](/deploy-manage/tools/snapshot-and-restore.md#feature-state) component. Refer to [Migrate system indices](/manage-data/migrate/migrate-internal-indices.md) for detailed migration steps. | | ||
|
kilfoyle marked this conversation as resolved.
Outdated
|
||
| | {{kib}} saved objects | {{kib}} saved objects can be migrated using the snapshot and restore [feature states](/deploy-manage/tools/snapshot-and-restore.md#feature-state) component or the {{kib}} import and export tools. The tools include the import and export endpoints of the [Saved objects API](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-saved-objects) and the [import and export](/explore-analyze/find-and-organize/saved-objects.md#saved-objects-import-and-export) options in the {{kib}} UI.<br><br>Snapshot and restore is generally the preferred migration method due to both speed and ease of use. In case you need to migrate {{fleet}} configuration data through snapshot and restore, this requires also restoring the {{kib}} feature state. | | ||
|
kilfoyle marked this conversation as resolved.
Outdated
|
||
| | Elastic feature and component data | Configuration data for products such as {{fleet}}, {{integrations}}, and {{watcher}} is typically migrated using the snapshot and restore feature. Refer to [Snaphot and restore](/deploy-manage/tools/snapshot-and-restore.md) and to the documentation for each specific product for additional detail. | | ||
|
kilfoyle marked this conversation as resolved.
Outdated
|
||
|
|
||
| <!-- | ||
| ## Data migration guides [data-migration-guides] | ||
|
|
||
| Choose one of our guides for detailed steps to migrate your {{es}} data. | ||
|
|
||
| Migrate your user data to {{serverless-full}}: | ||
|
|
||
| - [Migrate with the reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md) | ||
| - [Migrate with {{ls}}](/manage-data/migrate/migrate-with-logstash.md) | ||
|
|
||
| Migrate your user data to {{ech}} or {{ece}}: | ||
|
|
||
| - [Migrate to {{ech}} or ECE](/manage-data/migrate/migrate-to-ech-or-ece.md) | ||
| - [Reindex using a private CA](/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md) {applies_to}`ece: unavailable` | ||
|
|
||
| Migrate data using snapshot and restore, including system data: | ||
|
|
||
| - [Minimal-downtime migration using snapshots](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md) | ||
| - [Migrate system indices](/manage-data/migrate/migrate-internal-indices.md) | ||
| --> | ||
|
|
||
|
|
||
| ## User data migration guides [data-migration-guides] | ||
|
|
||
| To migrate your {{es}} ingested user data, choose one of the available migration options depending on your source and target deployment types. | ||
|
kilfoyle marked this conversation as resolved.
Outdated
|
||
|
|
||
| **Migrate data to {{ech}}**: | ||
|
kilfoyle marked this conversation as resolved.
Outdated
|
||
|
|
||
| | From | To | Supported Methods | | ||
| | --- | --- | --- | | ||
| | ECH | ECH | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | ECE | ECH | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | ECK | ECH | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | {{serverless-short}} | ECH | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | Self-managed | ECH | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md)*, [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| * See also [Reindex from a Self-managed cluster using a private CA](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4914/manage-data/migrate/migrate-from-a-Self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex) | ||
|
|
||
| **Migrate data to {{ece}}**: | ||
|
|
||
| | From | To | Supported Methods | | ||
| | --- | --- | --- | | ||
| | ECH | ECE | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | ECE | ECE | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | ECK | ECE | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | {{serverless-short}} | ECE | Logstash | | ||
|
kilfoyle marked this conversation as resolved.
Outdated
|
||
| | Self-managed | ECE | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
|
|
||
| **Migrate data to {{eck}}**: | ||
|
|
||
| | From | To | Supported Methods | | ||
| | --- | --- | --- | | ||
| | ECH | ECK | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | ECE | ECK | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | ECK | ECK | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | {{serverless-short}} | ECK | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | Self-managed | ECK | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
|
|
||
| **Migrate data to {{serverless-full}}**: | ||
|
|
||
| | From | To | Supported Methods | | ||
| | --- | --- | --- | | ||
| | ECH | {{serverless-short}} | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md)*, [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | ECE | {{serverless-short}} | [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | ECK | {{serverless-short}} | [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | {{serverless-short}} | {{serverless-short}} | [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | Self-managed | {{serverless-short}} | Not currently supported | | ||
|
|
||
| * {applies_to}`stack: preview 9.3+` | ||
|
|
||
| **Migrate data to Elastic self-managed**: | ||
|
|
||
| | From | To | Supported Methods | | ||
| | --- | --- | --- | | ||
| | ECH | Self-managed | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | ECE | Self-managed | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | ECK | Self-managed | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | {{serverless-short}} | Self-managed | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
| | Self-managed | Self-managed | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | | ||
|
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.