diff --git a/deploy-manage/deploy/cloud-enterprise/create-deployment.md b/deploy-manage/deploy/cloud-enterprise/create-deployment.md index 554dbd2ea6..b76ef4807f 100644 --- a/deploy-manage/deploy/cloud-enterprise/create-deployment.md +++ b/deploy-manage/deploy/cloud-enterprise/create-deployment.md @@ -47,7 +47,7 @@ To create a deployment in ECE: 8. Once the deployment is ready, select **Continue** to open the deployment’s main page. -After a deployment is spun up, you can scale the size and add other features; however, the instance configuration and computing ratios cannot be changed. If you need to change an existing deployment to another template, we recommend [migrating your data](../../../manage-data/migrate.md). +After a deployment is spun up, you can scale the size and add other features; however, the instance configuration and computing ratios cannot be changed. If you need to change an existing deployment to another template, we recommend [migrating your data](/manage-data/migrate-data.md). ## Next steps diff --git a/deploy-manage/deploy/elastic-cloud/azure-native-isv-service.md b/deploy-manage/deploy/elastic-cloud/azure-native-isv-service.md index a0e5bcc813..5056c5ff39 100644 --- a/deploy-manage/deploy/elastic-cloud/azure-native-isv-service.md +++ b/deploy-manage/deploy/elastic-cloud/azure-native-isv-service.md @@ -260,7 +260,7 @@ $$$azure-integration-migrate$$$How do I migrate my data from the classic Azure m 3. Navigate to Azure and log in as the user that manages the {{es}} resources. 4. Before proceeding, ensure the new account is configured according to the [prerequisites](#azure-integration-get-started). 5. Create a [new {{es}} resource](#azure-integration-get-started) for each existing deployment that needs migration from the classic Azure account. - 6. In the new {{es}} resource, follow the steps in [Restore from a snapshot](../../../manage-data/migrate.md#ec-restore-snapshots) to register the custom snapshot repository from Step 1. + 6. In the new {{es}} resource, follow the steps in [Restore from a snapshot](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md) to register the custom snapshot repository from Step 1. 7. In the same set of steps, restore the snapshot data from the snapshot repository that you registered. 8. Confirm the data has moved successfully into your new {{es}} resource on Azure. 9. To remove the old Azure subscription and the old deployments, go to the [Azure SaaS page](https://portal.azure.com/#blade/HubsExtension/BrowseResourceBlade/resourceType/Microsoft.SaaS%2Fresources) and unsubscribe from the {{ecloud}} ({{es}}) marketplace subscription. This action triggers the existing deployments termination. diff --git a/deploy-manage/deploy/elastic-cloud/heroku.md b/deploy-manage/deploy/elastic-cloud/heroku.md index ffd4cbda16..fb4b6c5ed9 100644 --- a/deploy-manage/deploy/elastic-cloud/heroku.md +++ b/deploy-manage/deploy/elastic-cloud/heroku.md @@ -73,7 +73,7 @@ There are several ways to ingest data into the deployment: * Use the sample data available from the {{kib}} home page without loading your own data. There are multiple data sets available and you can add them with one click. * Ingest your own data. [Learn more](/manage-data/ingest.md). -* Have existing {{es}} data? Consider your [migration options](../../../manage-data/migrate.md). +* Have existing {{es}} data? Consider your [migration options](/manage-data/migrate-data.md). ### Increase security [ech-increase-security] diff --git a/manage-data/index.md b/manage-data/index.md index 7abb29a8b5..7de26f87bb 100644 --- a/manage-data/index.md +++ b/manage-data/index.md @@ -44,4 +44,4 @@ Strategies for managing data depend on the type of data and how you're using it. If you move to new infrastructure, there are several options for moving existing data between {{es}} clusters. -**Learn more in [Migrate your {{es}} data](/manage-data/migrate.md).** \ No newline at end of file +**Learn more in [Migrate your {{es}} data](/manage-data/migrate-data.md).** \ No newline at end of file diff --git a/manage-data/migrate-data.md b/manage-data/migrate-data.md new file mode 100644 index 0000000000..3bba6f9343 --- /dev/null +++ b/manage-data/migrate-data.md @@ -0,0 +1,133 @@ +--- +mapped_pages: + - https://www.elastic.co/guide/en/cloud/current/ec-migrating-data.html + - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-migrating-data.html + - https://www.elastic.co/guide/en/cloud-heroku/current/ech-migrate-data2.html +applies_to: + stack: ga + serverless: ga +products: + - id: elasticsearch + - id: cloud-hosted + - id: cloud-enterprise + - id: cloud-kubernetes +--- + +# Migrate your {{es}} data [migrate-your-elasticsearch-data] + +Transitioning between Elastic deployment types involves migrating your {{es}} data. This page helps you plan your migration by describing the main categories of data you may need to move (ingested user data, {{es}} system data, {{kib}} saved objects, and feature-specific data), the migration methods available for each, and where to find step-by-step guides for your scenario. + +## Data types [migration-data-types] + +Your migration options depend on the type of data that you need to migrate, which can be categorized into four groups: + +- **Ingested user data**: All of the data that you've added into {{es}}, structured or unstructured, for your own applications. + +- **{{es}} system data**: Configuration and state information stored in {{es}} [system indices](elasticsearch://reference/elasticsearch/rest-apis/api-conventions.md#system-indices). This data is used by {{es}} for its internal operations. + +- **{{kib}} saved objects**: Dashboards, visualizations, maps, data views, Canvas workpads, and any other objects that you've saved in {{kib}}. + +- **Feature and component data**: Data stored in {{es}} that is specific to a given Elastic feature or component. This includes, for example, configuration data for {{fleet}} and {{integrations}}, {{watcher}} data, alerting and security detection rules, security data such as role mappings, API keys, and service tokens, and others. + +## Migration options [migration-options] + +Depending on the type of data that you need to move, various migration options are available: + + - **Reindex from source**: For your own data, reindexing into your new, destination deployment from the data's original source is typically the most straightforward approach, since it's available without any need to consider differing {{es}} versions or deployment types. + + If you still have access to the original data source, outside of your former {{es}} cluster, you can load the data from there. You have the option to use any ingestion method that you want—{{ls}}, {{agent}}, {{beats}}, the {{es}} clients, or whatever works best for you. + + If the original source isn’t available or has other issues that make it non-viable, you can choose from one of the other migration options described here. + - **Dual ingest**: For data with a limited lifecycle (logs and metrics, for example), another approach is to ingest into both the original and new environment at the same time, for a set duration. You can ingest into both environments for long enough for the data retention period to elapse. Then, after confirming that everything is working well in the new environment, the original environment can be shut down. + - **Snapshot and restore**: Use a snapshot to create a backup of your running {{es}} cluster, and then migrate by restoring your data into a new cluster. + - **Reindex API**: Copy documents from a source index to a destination index. You can reindex across clusters and deployment types and transform the data en route. + - **{{ls}}**: With {{ls}} you can collect, process, and forward data from a variety of sources to a variety of destinations. It serves as a highly configurable option available for migrating data across any deployment types. + - **Saved objects API**: Use this API to migrate objects that you've saved in {{kib}}. + - **{{kib}} saved object management**: You can also use the {{kib}} UI to migrate your saved objects. + +The following table describes the migration options available for each data type and where to find guidance. + +| Data type | Migration options | +| ------ | ------ | +| Ingested user data | The reindex API, snapshot and restore, and {{ls}} migration options are available for your user data, with some restrictions based on the source and target deployment type. Refer to [User data migration guides](#data-migration-guides) on this page to learn more. | +| {{es}} system data | System indices must be migrated using the snapshot and restore [feature states](/deploy-manage/tools/snapshot-and-restore.md#feature-state) component. Refer to [Migrate system indices](/manage-data/migrate/migrate-internal-indices.md) for detailed migration steps. | +| {{kib}} saved objects | {{kib}} saved objects can be migrated using the snapshot and restore [feature states](/deploy-manage/tools/snapshot-and-restore.md#feature-state) component or the {{kib}} import and export tools. The tools include the import and export endpoints of the [Saved objects API](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-saved-objects) and the [import and export](/explore-analyze/find-and-organize/saved-objects.md#saved-objects-import-and-export) options in the {{kib}} UI.

Snapshot and restore is generally the preferred migration method due to both speed and ease of use. In case you need to migrate {{fleet}} configuration data through snapshot and restore, this requires also restoring the {{kib}} feature state. | +| Elastic feature and component data | Configuration data for products such as {{fleet}}, {{integrations}}, and {{watcher}} is typically migrated using the snapshot and restore feature. Refer to [Snaphot and restore](/deploy-manage/tools/snapshot-and-restore.md) and to the documentation for each specific product for additional detail. | + + + + +## User data migration guides [data-migration-guides] + +To migrate your {{es}} ingested user data, choose one of the available migration options depending on your source and target deployment types. + +**Migrate data to {{ech}}**: + +| From | To | Supported Methods | +| --- | --- | --- | +| ECH | ECH | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| ECE | ECH | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| ECK | ECH | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| {{serverless-short}} | ECH | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| Self-managed | ECH | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md)*, [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +* See also [Reindex from a Self-managed cluster using a private CA](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4914/manage-data/migrate/migrate-from-a-Self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex) + +**Migrate data to {{ece}}**: + +| From | To | Supported Methods | +| --- | --- | --- | +| ECH | ECE | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| ECE | ECE | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| ECK | ECE | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| {{serverless-short}} | ECE | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| Self-managed | ECE | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | + +**Migrate data to {{eck}}**: + +| From | To | Supported Methods | +| --- | --- | --- | +| ECH | ECK | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| ECE | ECK | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| ECK | ECK | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| {{serverless-short}} | ECK | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| Self-managed | ECK | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | + +**Migrate data to {{serverless-full}}**: + +| From | To | Supported Methods | +| --- | --- | --- | +| ECH | {{serverless-short}} | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md) {applies_to}`serverless: preview 9.3+`, [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| ECE | {{serverless-short}} | [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| ECK | {{serverless-short}} | [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| {{serverless-short}} | {{serverless-short}} | [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| Self-managed | {{serverless-short}} | [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | + +**Migrate data to Elastic self-managed**: + +| From | To | Supported Methods | +| --- | --- | --- | +| ECH | Self-managed | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| ECE | Self-managed | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| ECK | Self-managed | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| {{serverless-short}} | Self-managed | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | +| Self-managed | Self-managed | [Reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md), [Snapshot and restore](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md), [{{ls}}](/manage-data/migrate/migrate-with-logstash.md) | + diff --git a/manage-data/migrate.md b/manage-data/migrate.md deleted file mode 100644 index 6b8918a5f1..0000000000 --- a/manage-data/migrate.md +++ /dev/null @@ -1,222 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud/current/ec-migrating-data.html - - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-migrating-data.html - - https://www.elastic.co/guide/en/cloud-heroku/current/ech-migrate-data2.html -applies_to: - deployment: - ess: ga - ece: ga -products: - - id: cloud-hosted - - id: cloud-enterprise ---- - -# Migrate your {{es}} data - -You might have switched to {{ech}} (ECH) or {{ece}} (ECE) for any number of reasons, and you’re likely wondering how to get your existing {{es}} data into your new infrastructure. Along with easily creating as many new deployments with {{es}} clusters that you need, you have several options for moving your data over. Choose the option that works best for you: - -* Index your data from the original source, which is the simplest method and provides the greatest flexibility for the {{es}} version and ingestion method. -* Reindex from a remote cluster, which rebuilds the index from scratch. -* Restore from a snapshot, which copies the existing indices. - -::::{note} -Although this guide focuses on migrating data from a self-managed cluster to an {{ech}} or {{ece}} deployment, the steps can also be adapted for other scenarios, such as when the source cluster is managed by {{eck}}, or when migrating from {{ece}} to {{ech}}. - -If both clusters belong to the same {{ech}} or {{ece}} environment, refer to [](/deploy-manage/tools/snapshot-and-restore/ece-restore-across-clusters.md). -:::: - -## Before you begin [ec_migrate_before_you_begin] - -Depending on which option you choose, you might have limitations or need to do some preparation beforehand. - -Indexing from the source -: The new cluster must be the same size as your old one, or larger, to accommodate the data. - -Reindex from a remote cluster -: The new cluster must be the same size as your old one, or larger, to accommodate the data. Depending on your security settings for your old cluster, you might need to temporarily allow TCP traffic on port 9243 for this procedure. - - For {{ech}}, if your old cluster is self-managed and uses [TLS certificates](/deploy-manage/security/set-up-basic-security-plus-https.md) signed by a private (non–publicly trusted) certificate authority, follow [this guide](migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md) to establish trust and configure remote reindex to ECH. - -Restore from a snapshot -: The new cluster must be the same size as your old one, or larger, to accommodate the data. The new cluster must also be an {{es}} version that is compatible with the old cluster (check [Elasticsearch snapshot version compatibility](/deploy-manage/tools/snapshot-and-restore.md#snapshot-restore-version-compatibility) for details). If you have not already done so, you need to [set up snapshots for your old cluster](/deploy-manage/tools/snapshot-and-restore/self-managed.md) using a repository that the new cluster can access. - -:::{admonition} Migrating system {{es}} indices -In {{es}} 8.0 and later versions, to back up and restore system indices and system data streams such as `.kibana` or `.security`, you must snapshot and restore the related feature's [feature state](/deploy-manage/tools/snapshot-and-restore.md#feature-state). - -Refer to [Migrate system indices](./migrate/migrate-internal-indices.md) to learn how to restore the internal {{es}} system indices from a snapshot. -::: - -## Index from the source [ec-index-source] - -If you still have access to the original data source, outside of your old {{es}} cluster, you can load the data from there. This might be the simplest option, allowing you to choose the {{es}} version and take advantage of the latest features. You have the option to use any ingestion method that you want—Logstash, Beats, the {{es}} clients, or whatever works best for you. - -If the original source isn’t available or has other issues that make it non-viable, there are still two more migration options, getting the data from a remote cluster or restoring from a snapshot. - -## Reindex from a remote cluster [ech-reindex-remote] - -Through the {{es}} [reindex API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex), you can connect your new {{es}} deployment remotely to your old {{es}} cluster. This pulls the data from your old cluster and indexes it into your new one. Reindexing essentially rebuilds the index from scratch and it can be more resource intensive to run than a [snapshot restore](#ec-restore-snapshots). - -::::{warning} -Reindex operations do not migrate index mappings, settings, or associated index templates from the source cluster. - -Before migrating your {{es}} data, define the necessary [mappings](/manage-data/data-store/mapping.md) and [templates](/manage-data/data-store/templates.md) on the new cluster. The easiest way to do this is to copy the relevant index templates from the old cluster to the new one before starting reindex operations. -:::: - -::::{note} -:applies_to: { ess: } - -For {{ech}}, if your old cluster uses [TLS certificates](/deploy-manage/security/set-up-basic-security-plus-https.md) signed by a private (non–publicly trusted) certificate authority, follow [this guide](migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md) to establish trust. -:::: - -Follow these steps to reindex data remotely: - -1. Log in to {{ech}} or {{ece}}. -2. Select a deployment or create one. -3. Ensure that the new deployment can access your old cluster to perform the reindex operation. Access is controlled by the {{es}} `reindex.remote.whitelist` user setting. - - Domains matching the patterns `["*.io:*", "*.com:*"]` are allowed by default, so if your remote host URL matches that pattern you do not need to explicitly define `reindex.remote.whitelist`. - - Otherwise, if your remote endpoint is not covered by the default patterns, adjust the setting to add the remote {{es}} cluster as an allowed host: - - 1. From your deployment menu, go to the **Edit** page. - 2. In the **Elasticsearch** section, select **Manage user settings and extensions**. For deployments with existing user settings, you might have to expand the **Edit elasticsearch.yml** caret for each node type instead. - 3. Add the following `reindex.remote.whitelist: [REMOTE_HOST:PORT]` user setting, where `REMOTE_HOST` is a pattern matching the URL for the remote {{es}} host that you are reindexing from, and `PORT` is the host port number. Do not include the `https://` prefix. - - If you override the parameter, it replaces the defaults: `["*.io:*", "*.com:*"]`. If you still want these patterns to be allowed, you need to specify them explicitly in the value. - - For example: - - `reindex.remote.whitelist: ["*.us-east-1.aws.found.io:9243", "*.com:*"]` - - 4. Save your changes. - -4. Using the **API Console** or within {{kib}}, either create the destination index with the appropriate settings and [mappings](/manage-data/data-store/mapping.md), or ensure that the relevant [index templates](/manage-data/data-store/templates.md) are in place. - -5. Using the **API Console** or [{{kib}} DevTools Console](/explore-analyze/query-filter/tools/console.md), reindex the data remotely from the old cluster: - - ```sh - POST _reindex - { - "source": { - "remote": { - "host": "https://:", - "username": "USER", - "password": "PASSWORD" - }, - "index": "INDEX_NAME", - "query": { - "match_all": {} - } - }, - "dest": { - "index": "INDEX_NAME" - } - } - ``` - - For additional options and details, refer to the [reindex API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex). - -6. Verify that the new index is present: - - ```sh - GET INDEX-NAME/_search?pretty - ``` - -7. If you are not planning to reindex more data from the remote, you can remove the `reindex.remote.whitelist` user setting that you added previously. - -## Restore from a snapshot [ec-restore-snapshots] - -[Restoring from a snapshot](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md) is often the fastest and most reliable way to migrate data between {{es}} clusters. It preserves mappings, settings, and optionally parts of the cluster state such as index templates, component templates, and system indices. - -You can restore system indices by including their corresponding [feature states](/deploy-manage/tools/snapshot-and-restore.md#feature-state) in the restore operation, allowing you to retain internal configurations related to security, {{kib}}, or other stack features. - -This method is especially useful when: - -* You want to fully replicate the old cluster or when remote reindexing is not feasible, for example if the old cluster is in a degraded or unreachable state. -* Your old cluster contains mostly static data, allowing you to snapshot the old cluster, restore in the new cluster, and continue operations. - -When your source cluster is actively ingesting data, such as logs, metrics, or traces, and you need a seamless migration with minimal downtime, consider using the [minimal downtime migration](migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md) guide. - -### Requirements [ec-restore-snapshots-requirements] - -* The new cluster must have access to the snapshot repository that contains the data from the old cluster. If snapshots are not already configured on the old cluster, refer to [Snapshot and restore](/deploy-manage/tools/snapshot-and-restore/self-managed.md) to enable and configure snapshots, or use a different data migration method. -* Both clusters must use [compatible versions](/deploy-manage/tools/snapshot-and-restore.md#snapshot-compatibility). - -For more information, refer to [Restore into a different cluster](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-different-cluster). - -::::{note} -For {{ece}}, Amazon S3 is the most common snapshot storage, but you can restore from any accessible external storage that contains your {{es}} snapshots. -:::: - -### Step 1: Set up the repository in the new cluster [migrate-repo-setup] - -In this step, you’ll configure a read-only snapshot repository in the new cluster that points to the storage location used by the old cluster. This allows the new cluster to access and restore snapshots created in the original environment. - -::::{tip} -If your new {{ech}} or {{ece}} deployment cannot connect to the same repository used by your self-managed cluster, for example if it's a private Network File System (NFS) share, consider one of the following alternatives: - -* [Back up your repository](/deploy-manage/tools/snapshot-and-restore/self-managed.md#snapshots-repository-backup) to a supported storage system such as AWS S3, Google Cloud Storage, or Azure Blob Storage, and then configure your new cluster to use that location for the data migration. -* Expose the repository contents over `ftp`, `http`, or `https`, and use a [read-only URL repository](/deploy-manage/tools/snapshot-and-restore/read-only-url-repository.md) type in your new deployment to access the snapshots. -:::: - -1. On your old {{es}} cluster, retrieve the snapshot repository configuration: - - ```sh - GET /_snapshot/_all - ``` - - Take note of the repository name and type (for example, `s3`, `gcs`, or `azure`), its base path, and any additional settings. Authentication credentials are often stored in the [secure settings](/deploy-manage/security/secure-settings.md) on each node. You’ll need to replicate all this configuration when registering the repository in the new ECH or ECE deployment. - - If your old cluster has multiple repositories configured, identify the repository with the snapshots containing the data that you want to migrate. - -2. Add the snapshot repository on the new cluster. - - Considerations: - - * If you're migrating [searchable snapshots](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md), the repository name must be identical in the old and new clusters. - * If the old cluster still has write access to the repository, register the repository as read-only to avoid data corruption. You can do this using the `readonly: true` option. - - To connect the existing snapshot repository to your new deployment, follow the steps for the storage provider where the repository is hosted: - - * **Amazon Web Services (AWS) Storage** - * [Store credentials in the keystore](/deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md#ec-snapshot-secrets-keystore) - * [Create the repository](/deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md#ec-create-aws-repository) - * **Google Cloud Storage (GCS)** - * [Store credentials in the keystore](/deploy-manage/tools/snapshot-and-restore/ec-gcs-snapshotting.md#ec-configure-gcs-keystore) - * [Create the repository](/deploy-manage/tools/snapshot-and-restore/ec-gcs-snapshotting.md#ec-create-gcs-repository) - * **Azure Blob Storage** - * [Store credentials in the keystore](/deploy-manage/tools/snapshot-and-restore/ec-azure-snapshotting.md#ec-configure-azure-keystore). - * [Create the repository](/deploy-manage/tools/snapshot-and-restore/ec-azure-snapshotting.md#ec-create-azure-repository). - - ::::{important} - Although these instructions focus on {{ech}}, you should follow the same steps for {{ece}} by configuring the repository directly **at the deployment level**. - - **Do not** configure the repository as an [ECE-managed repository](/deploy-manage/tools/snapshot-and-restore/cloud-enterprise.md), which is intended for automatic snapshots of deployments. In this case, you need to add a custom repository that already contains snapshots from another cluster. - :::: - - -### Step 2: Run the snapshot restore [migrate-restore] - -After you have registered and verified the repository, you are ready to restore any data from any of its snapshots to your new cluster. - -You can run a restore operation using the {{kib}} Management UI, or using the {{es}} API. Refer to [Restore a snapshot](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md) for more details, including API-based examples. - -For details about the contents of a snapshot, refer to [](/deploy-manage/tools/snapshot-and-restore.md#snapshot-contents). - -To start the restore process: - -1. Open Kibana and go to the **Snapshot and Restore** management page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). -2. Under the **Snapshots** tab, you can find the available snapshots from your newly added snapshot repository. Select any snapshot to view its details, and from there you can choose to restore it. -3. Select **Restore**. -4. Select the index or indices you wish to restore. -5. Optionally, configure additional restore options, such as **Restore aliases**, **Restore global state**, or **Restore feature state**. -6. Select **Restore snapshot** to begin the process. - -7. Verify that each restored index is available in your deployment. You can do this using {{kib}} **Index Management** UI, or by running this query: - - ```sh - GET INDEX_NAME/_search?pretty - ``` - - If you have restored many indices you can also run `GET _cat/indices?s=index` to list all indices for verification. diff --git a/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md b/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md index 43bc969b0f..1ab96d452e 100644 --- a/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md +++ b/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md @@ -1,7 +1,8 @@ --- -navigation_title: Minimal-downtime migration using snapshots +navigation_title: Migrate data using snapshots applies_to: stack: ga + serverless: unavailable products: - id: elasticsearch - id: cloud-hosted @@ -11,7 +12,7 @@ products: # Migrate {{es}} data with minimal downtime using snapshots [migrate-elasticsearch-data-with-minimal-downtime] -When moving your data and services from one {{es}} cluster to another, such as to {{ech}}, {{ece}}, new on-premises hardware, or any other {{es}} environment, you can migrate with minimal downtime using incremental snapshots. +When moving your data and services from one {{es}} cluster to another, such as to {{ech}}, {{ece}}, or new on-premises hardware, you can migrate with minimal downtime using incremental snapshots. Migrating with incremental snapshots is useful when you want to: @@ -34,18 +35,22 @@ Before you migrate, review the prerequisites and requirements. ### Prerequisites * Learn how to [set up and manage snapshot repositories](/deploy-manage/tools/snapshot-and-restore/manage-snapshot-repositories.md). * If restoring to a different cluster, review [Restore to a different cluster](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-different-cluster). -* As an alternative migration method, you can [reindex from a remote cluster](/manage-data/migrate.md#ech-reindex-remote). +* As an alternative migration method, you can [reindex from a remote cluster](/manage-data/migrate/migrate-data-using-reindex-api.md). ### Requirements * **Cluster size** – The new cluster must be the same size or larger than the old cluster. -* **Version compatibility** – Both clusters must use compatible {{es}} versions. To check if your cluster versions are compatible, check [Snapshot version compatibility](/deploy-manage/tools/snapshot-and-restore.md#snapshot-restore-version-compatibility). +* **Version compatibility** – Both clusters must use [compatible {{es}} versions](/deploy-manage/tools/snapshot-and-restore.md#snapshot-compatibility). To check if your cluster versions are compatible, check [Snapshot version compatibility](/deploy-manage/tools/snapshot-and-restore.md#snapshot-restore-version-compatibility). * **Storage requirements** - Ensure sufficient repository storage. Usage grows with snapshot frequency and data volume. * **Network overhead** – Transferring snapshots across networks, regions, or providers can be time consuming and incur costs. * **Resource usage** – Snapshot and restore operations can be resource intensive and affect cluster performance. * **Custom integrations** – Some integrations that use the {{es}} API directly, such as the [Elasticsearch Java Client library](elasticsearch-java://reference/index.md), can require additional handling during cutover. +::::{note} +For {{ece}}, Amazon S3 is the most common snapshot storage, but you can restore from any accessible external storage that contains your {{es}} snapshots. +:::: + ## Recommended migration timeline [recommended-migration-timeline] -Tp complete the migration with minimal downtime, use incremental snapshots. While the exact sequence may differ depending on your infrastructure and operational requirements, you can use the recommended migration timeline as a reliable baseline that you can adapt. Adjust the steps and times to fit your own operational needs. +To complete the migration with minimal downtime, use incremental snapshots. While the exact sequence may differ depending on your infrastructure and operational requirements, you can use the recommended migration timeline as a reliable baseline that you can adapt. Adjust the steps and times to fit your own operational needs. 1. **09:00**: Take the initial full snapshot of the old cluster. You can also take the initial full snapshot the day before. 2. **09:30**: Restore the snapshot to the new cluster. @@ -57,5 +62,88 @@ Tp complete the migration with minimal downtime, use incremental snapshots. Whil 4. Change ingestion and querying to the new cluster. 5. Open the indices in the new cluster. + +## Migrate your data using snapshot and restore [migrate-data-snapshot-restore] + +Follow these steps to migrate your {{es}} data. + +### Step 1: Set up the repository in the new cluster [migrate-repo-setup] + +In this step, you’ll configure a read-only snapshot repository in the new cluster that points to the storage location used by the old cluster. This allows the new cluster to access and restore snapshots created in the original environment. + +::::{tip} +If your new deployment cannot connect to the same repository used by your self-managed cluster, for example if it's a private Network File System (NFS) share, consider one of the following alternatives: + +* [Back up your repository](/deploy-manage/tools/snapshot-and-restore/self-managed.md#snapshots-repository-backup) to a supported storage system such as AWS S3, Google Cloud Storage, or Azure Blob Storage, and then configure your new cluster to use that location for the data migration. +* Expose the repository contents over `ftp`, `http`, or `https`, and use a [read-only URL repository](/deploy-manage/tools/snapshot-and-restore/read-only-url-repository.md) type in your new deployment to access the snapshots. +:::: + +1. On your old {{es}} cluster, retrieve the snapshot repository configuration: + + ```sh + GET /_snapshot/_all + ``` + + Take note of the repository name and type (for example, `s3`, `gcs`, or `azure`), its base path, and any additional settings. Authentication credentials are often stored in the [secure settings](/deploy-manage/security/secure-settings.md) on each node. You’ll need to replicate all this configuration when registering the repository in the new ECH or ECE deployment. + + If your old cluster has multiple repositories configured, identify the repository with the snapshots containing the data that you want to migrate. + +2. Add the snapshot repository on the new cluster. + + Considerations: + + * If you're migrating [searchable snapshots](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md), the repository name must be identical in the old and new clusters. + * If the old cluster still has write access to the repository, register the repository as read-only to avoid data corruption. You can do this using the `readonly: true` option. + + To connect the existing snapshot repository to your new deployment, follow the steps for the storage provider where the repository is hosted: + + * **Amazon Web Services (AWS) Storage** + * [Store credentials in the keystore](/deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md#ec-snapshot-secrets-keystore) + * [Create the repository](/deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md#ec-create-aws-repository) + * **Google Cloud Storage (GCS)** + * [Store credentials in the keystore](/deploy-manage/tools/snapshot-and-restore/ec-gcs-snapshotting.md#ec-configure-gcs-keystore) + * [Create the repository](/deploy-manage/tools/snapshot-and-restore/ec-gcs-snapshotting.md#ec-create-gcs-repository) + * **Azure Blob Storage** + * [Store credentials in the keystore](/deploy-manage/tools/snapshot-and-restore/ec-azure-snapshotting.md#ec-configure-azure-keystore). + * [Create the repository](/deploy-manage/tools/snapshot-and-restore/ec-azure-snapshotting.md#ec-create-azure-repository). + + ::::{important} + Although these instructions focus on {{ech}}, you should follow the same steps for {{ece}} by configuring the repository directly **at the deployment level**. + + **Do not** configure the repository as an [ECE-managed repository](/deploy-manage/tools/snapshot-and-restore/cloud-enterprise.md), which is intended for automatic snapshots of deployments. In this case, you need to add a custom repository that already contains snapshots from another cluster. + :::: + + +### Step 2: Run the snapshot restore [migrate-restore] + +After you have registered and verified the repository, you are ready to restore any data from any of its snapshots to your new cluster. + +You can run a restore operation using the {{kib}} Management UI, or using the {{es}} API. Refer to [Restore a snapshot](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md) for more details, including API-based examples. + +For details about the contents of a snapshot, refer to [](/deploy-manage/tools/snapshot-and-restore.md#snapshot-contents). + +To start the restore process: + +1. Open Kibana and go to the **Snapshot and Restore** management page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). +2. Under the **Snapshots** tab, you can find the available snapshots from your newly added snapshot repository. Select any snapshot to view its details, and from there you can choose to restore it. +3. Select **Restore**. +4. Select the index or indices you wish to restore. +5. Optionally, configure additional restore options, such as **Restore aliases**, **Restore global state**, or **Restore feature state**. +6. Select **Restore snapshot** to begin the process. + +7. Verify that each restored index is available in your deployment. You can do this using {{kib}} **Index Management** UI, or by running this query: + + ```sh + GET INDEX_NAME/_search?pretty + ``` + + If you have restored many indices you can also run `GET _cat/indices?s=index` to list all indices for verification. + + + + + + + ## Additional support [incremental-snapshots-additional-support] To get expert assistance for your {{es}} migrations, go to [Elastic Professional Services](https://www.elastic.co/consulting). diff --git a/manage-data/migrate/migrate-data-using-reindex-api.md b/manage-data/migrate/migrate-data-using-reindex-api.md new file mode 100644 index 0000000000..f2d7e220c4 --- /dev/null +++ b/manage-data/migrate/migrate-data-using-reindex-api.md @@ -0,0 +1,124 @@ +--- +navigation_title: Migrate data using the reindex API +applies_to: + serverless: preview + stack: all +products: + - id: elasticsearch + - id: cloud-hosted +--- + +# Migrate {{es}} data using the reindex API [migrate-reindex-from-remote] + +The [reindex API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) offers a convenient way for you to copy your {{es}} documents from a source index, data stream, or alias in one deployment to another. + +This guide gives the example of reindexing a full index from an {{ech}} deployment to an {{serverless-full}} project using the remote host parameters as shown in the [reindex from remote](elasticsearch://reference/elasticsearch/rest-apis/reindex-indices.md#reindex-from-remote) example. These steps can be adapted to other deployment types as well. When you copy your data to deployment types other than {{serverless-short}}, there are [additional considerations](#migrate-reindex-from-remote-others) to make note of. + +For more advanced use cases, including data modification using scripts or ingest pipelines, refer to the [Reindex indices examples](elasticsearch://reference/elasticsearch/rest-apis/reindex-indices.md#reindex-from-remote) and the [reindex API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) documentation. + +## Prerequisites [migrate-reindex-from-remote-prereqs] + +- An {{ech}} deployment with data to migrate +- An [{{serverless-full}}](/deploy-manage/deploy/elastic-cloud/serverless.md) project configured and running +- An [API key](/deploy-manage/api-keys/elastic-cloud-api-keys.md) for authentication with the {{ech}} deployment + + Basic authentication can be used in place of an API key, but an API key is recommended as a more secure option. + +- The target deployment must be able to access your original source cluster to perform the reindex operation. Access is controlled by the {{es}} `reindex.remote.whitelist` user setting. + + Domains matching the patterns `["*.io:*", "*.com:*"]` are allowed by default, so if your remote host URL matches that pattern you do not need to explicitly define `reindex.remote.whitelist`. + + Otherwise, if your remote endpoint is not covered by the default patterns, adjust the setting to add the remote {{es}} cluster as an allowed host: + + 1. From your deployment menu, go to the **Edit** page. + 2. In the **Elasticsearch** section, select **Manage user settings and extensions**. For deployments with existing user settings, you might have to expand the **Edit elasticsearch.yml** caret for each node type instead. + 3. Add the following `reindex.remote.whitelist: [REMOTE_HOST:PORT]` user setting, where `REMOTE_HOST` is a pattern matching the URL for the remote {{es}} host that you are reindexing from, and `PORT` is the host port number. Do not include the `https://` prefix. + + If you override the parameter, it replaces the defaults: `["*.io:*", "*.com:*"]`. If you still want these patterns to be allowed, you need to specify them explicitly in the value. + + For example: + + `reindex.remote.whitelist: ["*.us-east-1.aws.found.io:9243", "*.com:*"]` + + 4. Save your changes. + + + +:::{important} +Kibana assets must be migrated separately using the {{kib}} [export/import APIs](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-saved-objects) or recreated manually. Refer to [Migration options](/manage-data/migrate-data.md#migration-options) for details about migrating different types of {{es}} data. + +Index templates, data stream definitions, and data lifecycle settings must be in place _before_ you start reindexing data. However, be mindful that if you have any [ingest pipelines](/manage-data/ingest/transform-enrich/ingest-pipelines.md) configured, it's typically best to add these _after_ data migration so as to avoid re-transforming data that had already been transformed at the time that it was ingested into your source deployment (though if the data is idempotent, re-transforming is not a concern). + +Visual components, such dashboard and visualizations, can be migrated after you have migrated the data. +::: + +## Migrate documents from {{ech}} to {{serverless-short}} [migrate-reindex-from-remote-ech-serverless] + +1. In your {{ech}} deployment: + + 1. Navigate to the deployment home page and copy the {{es}} endpoint. + + 1. Go to the **Index Management** page in the navigation menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). + + 1. Use the search field to identify the indices that you want to migrate. + +1. In your {{serverless-short}} project: + + 1. Open the Developer Tools [Console](/explore-analyze/query-filter/tools/console.md). + + 1. Call the reindex API to migrate your index. If you have multiiple indices to migrate, we recommend doing a separate call for each. + + Example: Reindex from an {{ech}} deployment to a {[serverless-short]} project using an API key: + + ``` + POST _reindex + { + "source": { + "remote": { + "host": "https://:443", <1> + "api_key": "" <2> + }, + "index": "" <3> + }, + "dest": { + "index": "" <4> + } + } + ``` + 1. The URL for your {{serverless-short}} project. This is the {{es}} endpoint that you copied in Step 1. If you're migrating to, for example, an {{ech}} cluster, you can modify the remote host address accordingly. + 1. The API key for authenticating the connection to your {{ech}} deployment. + 1. The source index to copy from your {{ech}} deployment. + 1. The destination index in your {{serverless-short}} project. + + 1. Verify that the new index is present: + + ```sh + GET INDEX-NAME/_search?pretty + ``` + + 1. If you are not planning to reindex more data from the remote and you configured a `reindex.remote.whitelist` user setting, that setting can now be removed. + + +## Notes for migrating between other deployment types [migrate-reindex-from-remote-others] + +The page demonstrates copying data from an {{ech}} deployment to {{serverless-short}}. When you use the reindex API to copy data across other deployment types, consider the following: + + - If you're migrating from a self-managed cluster that uses non–publicly trusted TLS certificates, including self-signed certificates and certificates signed by a private certificate authority (CA), refer to our guide [Reindex from a self-managed cluster using a private CA](/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md). + + - The target deployment must be able to access your original source cluster to perform the reindex operation. When you migrate to {{serverless-short}}, access to all {{ech}} endpoints is allowed automatically. For migrating to other deployment types, access is controlled by the {{es}} `reindex.remote.whitelist` user setting. + + Domains matching the patterns `["*.io:*", "*.com:*"]` are allowed by default, so if your remote host URL matches that pattern you do not need to explicitly define `reindex.remote.whitelist`. + + Otherwise, if your remote endpoint is not covered by the default patterns, adjust the setting to add the remote {{es}} cluster as an allowed host: + + 1. From your deployment menu, go to the **Edit** page. + 2. In the **Elasticsearch** section, select **Manage user settings and extensions**. For deployments with existing user settings, you might have to expand the **Edit elasticsearch.yml** caret for each node type instead. + 3. Add the following `reindex.remote.whitelist: [REMOTE_HOST:PORT]` user setting, where `REMOTE_HOST` is a pattern matching the URL for the remote {{es}} host that you are reindexing from, and `PORT` is the host port number. Do not include the `https://` prefix. + + If you override the parameter, it replaces the defaults: `["*.io:*", "*.com:*"]`. If you still want these patterns to be allowed, you need to specify them explicitly in the value. + + For example: + + `reindex.remote.whitelist: ["*.us-east-1.aws.found.io:9243", "*.com:*"]` + + 4. Save your changes. \ No newline at end of file diff --git a/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md b/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md index ba3d943187..1f7ddd0616 100644 --- a/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md +++ b/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md @@ -70,7 +70,7 @@ The `Destination` cluster should be the same or newer version as the `Source` cl ## Step 5: Reindex from remote `Source` cluster [ec-remote-reindex-step5] -You can now run a remote reindex operation on the {{ech}} `Destination` cluster from the `Source` cluster, as described in the [migration guide](/manage-data/migrate.md#ech-reindex-remote): +You can now run a `reindex from remote` reindex operation on the {{ech}} `Destination` cluster from the `Source` cluster, as described in [Migrate Elasticsearch data using the reindex API](/manage-data/migrate/migrate-data-using-reindex-api.md): ```text POST _reindex diff --git a/manage-data/migrate/migrate-internal-indices.md b/manage-data/migrate/migrate-internal-indices.md index 76c45d5e84..c8c43f284b 100644 --- a/manage-data/migrate/migrate-internal-indices.md +++ b/manage-data/migrate/migrate-internal-indices.md @@ -18,7 +18,7 @@ However, using snapshot and restore for system indices does not mean you must us ## Migrate system indices using snapshot and restore -To restore system indices from a snapshot, follow the same procedure described in [](../migrate.md#ec-restore-snapshots) and select the appropriate feature states when preparing the restore operation, such as `kibana` or `security`. +To restore system indices from a snapshot, follow the same procedure described in [Migrate Elasticsearch data with minimal downtime using snapshots](/manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md) and select the appropriate feature states when preparing the restore operation, such as `kibana` or `security`. For more details about restoring feature states, or the entire cluster state, refer to [](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-feature-state). diff --git a/manage-data/migrate/migrate-with-logstash.md b/manage-data/migrate/migrate-with-logstash.md index c786b4453b..cdd31b339a 100644 --- a/manage-data/migrate/migrate-with-logstash.md +++ b/manage-data/migrate/migrate-with-logstash.md @@ -1,5 +1,5 @@ --- -navigation_title: Migrate Elastic Cloud Hosted data to Serverless with Logstash +navigation_title: Migrate data using Logstash applies_to: serverless: deployment: @@ -10,7 +10,7 @@ products: - id: cloud-hosted --- -# Migrate {{ech}} data to {{serverless-full}} with {{ls}} [migrate-with-ls] +# Migrate {{es}} data using {{ls}} [migrate-with-ls] [{{ls}}](logstash://reference/index.md) is a data collection engine that uses a large ecosystem of [plugins](logstash-docs-md://lsr/index.md) to collect, process, and forward data from a variety of sources to a variety of destinations. Here we focus on using the [Elasticsearch input](logstash-docs-md://lsr/plugins-inputs-elasticsearch.md) plugin to read from your {{ech}} deployment, and the [Elasticsearch output](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md) plugin to write to your {{{serverless-full}} project. @@ -30,7 +30,7 @@ The Elasticsearch input plugin offers [additional configuration options](#additi - API keys in {{ls}} format for authentication with both deployments :::{important} -Kibana assets much be migrated separately using the {{kib}} [export/import APIs](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-saved-objects) or recreated manually. +Kibana assets much be migrated separately using the {{kib}} [export/import APIs](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-saved-objects) or recreated manually. Refer to [Migration options](/manage-data/migrate-data.md#migration-options) for details about migrating different types of {{es}} data. Templates, data stream definitions, and ILM policies, must be in place _before_ you start data migration. Visual components, such dashboard and visualizations, can be migrated after you have migrated the data. diff --git a/manage-data/toc.yml b/manage-data/toc.yml index fe9fda40d2..461b55f6a0 100644 --- a/manage-data/toc.yml +++ b/manage-data/toc.yml @@ -172,10 +172,12 @@ toc: - file: lifecycle/rollup/rollup-aggregation-limitations.md - file: lifecycle/rollup/rollup-search-limitations.md - file: lifecycle/rollup/migrating-from-rollup-to-downsampling.md - - file: migrate.md + - file: migrate-data.md children: - - file: migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md - - file: migrate/migrate-internal-indices.md - file: migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md + - file: migrate/migrate-data-using-reindex-api.md + children: + - file: migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md - file: migrate/migrate-with-logstash.md + - file: migrate/migrate-internal-indices.md - file: use-case-use-elasticsearch-to-manage-time-series-data.md diff --git a/redirects.yml b/redirects.yml index cbb54d4e20..72e0fbd205 100644 --- a/redirects.yml +++ b/redirects.yml @@ -702,6 +702,9 @@ redirects: 'solutions/observability/observability-ai-assistant.md': 'solutions/observability/ai/observability-ai-assistant.md' 'solutions/observability/llm-performance-matrix.md': 'solutions/observability/ai/llm-performance-matrix.md' +# Related to https://github.com/elastic/docs-content/pull/5244 + 'manage-data/migrate.md': 'manage-data/migrate-data.md' + # Related to cases and alerting documentation restructuring # Main pages 'explore-analyze/alerts-cases.md': 'explore-analyze/alerting.md'