Skip to content

[9.1] [ska] relocate esArchives to platform/test/fixtures (#229750) #230161

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dmlemeshko merged 1 commit into from
Aug 1, 2025
Merged

[9.1] [ska] relocate esArchives to platform/test/fixtures (#229750) #230161

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
90 changes: 43 additions & 47 deletions .github/CODEOWNERS
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion scripts/archive_migration_functions.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -350,7 +350,7 @@ migrate() {

load_logstash() {
set -x
node scripts/es_archiver.js load x-pack/test/functional/es_archives/logstash_functional --config "$test_config"
node scripts/es_archiver.js load x-pack/platform/test/fixtures/es_archives/logstash_functional --config "$test_config"
set +x
}

Expand Down
30 changes: 20 additions & 10 deletions src/platform/packages/shared/kbn-es-archiver/README.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,21 @@
---
id: kibDevDocsOpsEsArchiver
slug: /kibana-dev-docs/ops/es-archiver
title: "ES Archiver"
title: 'ES Archiver'
description: A tool which helps developers capture and restore ES indexes
date: 2024-01-11
tags: ['kibana', 'dev', 'contributor', 'operations', 'ci', 'es-archiver', 'load', 'es-archiver-load-action', 'performance']
tags:
[
'kibana',
'dev',
'contributor',
'operations',
'ci',
'es-archiver',
'load',
'es-archiver-load-action',
'performance',
]
---

The ES Archiver is a service primarily used by the Functional Tests to load up ES indexes using the bulk API which makes the archives more resilient to ES upgrades and easier to inspect/edit locally because they are just plain text files containing newline-delimited JSON (though they are sometimes compressed).
Expand All @@ -19,26 +30,25 @@ Additional information about what functionality the CLI provides can be found by

## Performance Option Parameter

We now have a performance parameter for the es-archiver#load(), entry-point
We now have a performance parameter for the es-archiver#load(), entry-point
function.
This parameter is optional, with defaults:

- Batch size: 5000
- Concurrency (maximum number of bulk requests that can be active in parallel): 4
- Batch size: 5000
- Concurrency (maximum number of bulk requests that can be active in parallel): 4

According to our benchmarks, these default settings are giving the best results in terms of loading time for
the majority of the archives.
According to our benchmarks, these default settings are giving the best results in terms of loading time for
the majority of the archives.
However, there might be cases where different settings are needed, so they can be overridden when loading an archive.


### How to override the default performance settings in test files

To control the batch size and concurrency

#### Example
#### Example

```typescript
await esArchiver.load('x-pack/test/functional/es_archives/getting_started/shakespeare', {
await esArchiver.load('x-pack/platform/test/fixtures/es_archives/getting_started/shakespeare', {
performance: {
batchSize: 300,
concurrency: 1,
Expand Down
39 changes: 18 additions & 21 deletions ...ages/shared/kbn-es-archiver/src/fixtures/override_saved_objects_index/exception_list.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,35 +1,32 @@
[
"x-pack/test/functional/es_archives/action_task_params",
"x-pack/test/functional/es_archives/actions",
"x-pack/test/functional/es_archives/alerting/8_2_0",
"x-pack/test/functional/es_archives/alerts",
"x-pack/test/functional/es_archives/alerts_legacy/rules",
"x-pack/test/functional/es_archives/alerts_legacy/tasks",
"x-pack/test/functional/es_archives/cases/default",
"x-pack/test/functional/es_archives/cases/migrations/7.11.1",
"x-pack/test/functional/es_archives/cases/migrations/7.13.2",
"x-pack/test/functional/es_archives/cases/migrations/7.13_user_actions",
"x-pack/test/functional/es_archives/cases/migrations/7.16.0_space",
"x-pack/test/functional/es_archives/cases/migrations/8.8.0",
"x-pack/test/functional/es_archives/data/search_sessions",
"x-pack/platform/test/fixtures/es_archives/action_task_params",
"x-pack/platform/test/fixtures/es_archives/actions",
"x-pack/platform/test/fixtures/es_archives/alerting/8_2_0",
"x-pack/platform/test/fixtures/es_archives/alerts",
"x-pack/platform/test/fixtures/es_archives/cases/default",
"x-pack/platform/test/fixtures/es_archives/cases/migrations/7.11.1",
"x-pack/platform/test/fixtures/es_archives/cases/migrations/7.13.2",
"x-pack/platform/test/fixtures/es_archives/cases/migrations/7.13_user_actions",
"x-pack/platform/test/fixtures/es_archives/cases/migrations/7.16.0_space",
"x-pack/platform/test/fixtures/es_archives/cases/migrations/8.8.0",
"x-pack/platform/test/fixtures/es_archives/data/search_sessions",
"x-pack/test/functional/es_archives/endpoint/telemetry/agent_only",
"x-pack/test/functional/es_archives/endpoint/telemetry/cloned_endpoint_different_states",
"x-pack/test/functional/es_archives/endpoint/telemetry/cloned_endpoint_installed",
"x-pack/test/functional/es_archives/endpoint/telemetry/cloned_endpoint_uninstalled",
"x-pack/test/functional/es_archives/endpoint/telemetry/endpoint_malware_disabled",
"x-pack/test/functional/es_archives/endpoint/telemetry/endpoint_malware_enabled",
"x-pack/test/functional/es_archives/endpoint/telemetry/endpoint_uninstalled",
"x-pack/test/functional/es_archives/event_log_legacy_ids",
"x-pack/test/functional/es_archives/event_log_multiple_indicies",
"x-pack/test/functional/es_archives/fleet/agents",
"x-pack/test/functional/es_archives/lists",
"x-pack/test/functional/es_archives/rules_scheduled_task_id/rules",
"x-pack/test/functional/es_archives/rules_scheduled_task_id/tasks",
"x-pack/platform/test/fixtures/es_archives/event_log_legacy_ids",
"x-pack/platform/test/fixtures/es_archives/event_log_multiple_indicies",
"x-pack/platform/test/fixtures/es_archives/fleet/agents",
"x-pack/platform/test/fixtures/es_archives/rules_scheduled_task_id/rules",
"x-pack/platform/test/fixtures/es_archives/rules_scheduled_task_id/tasks",
"x-pack/test/functional/es_archives/security_solution/import_rule_connector",
"x-pack/test/functional/es_archives/security_solution/migrations",
"x-pack/test/functional/es_archives/security_solution/resolve_read_rules/7_14",
"x-pack/test/functional/es_archives/security_solution/timelines/7.15.0",
"x-pack/test/functional/es_archives/security_solution/timelines/7.15.0_space",
"x-pack/test/functional/es_archives/task_manager_removed_types",
"x-pack/test/functional/es_archives/task_manager_tasks"
"x-pack/platform/test/fixtures/es_archives/task_manager_removed_types",
"x-pack/platform/test/fixtures/es_archives/task_manager_tasks"
]
6 changes: 4 additions & 2 deletions src/platform/test/examples/discover_customization_examples/customizations.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,9 @@ export default ({ getService, getPageObjects }: FtrProviderContext) => {
describe('Customizations', () => {
before(async () => {
await kibanaServer.savedObjects.cleanStandardList();
await esArchiver.loadIfNeeded('x-pack/test/functional/es_archives/logstash_functional');
await esArchiver.loadIfNeeded(
'x-pack/platform/test/fixtures/es_archives/logstash_functional'
);
await kibanaServer.importExport.load(
'src/platform/test/functional/fixtures/kbn_archiver/discover'
);
Expand All @@ -46,7 +48,7 @@ export default ({ getService, getPageObjects }: FtrProviderContext) => {
await kibanaServer.importExport.unload(
'src/platform/test/functional/fixtures/kbn_archiver/discover'
);
await esArchiver.unload('x-pack/test/functional/es_archives/logstash_functional');
await esArchiver.unload('x-pack/platform/test/fixtures/es_archives/logstash_functional');
await kibanaServer.savedObjects.cleanStandardList();
});

Expand Down
12 changes: 8 additions & 4 deletions src/platform/test/examples/unified_field_list_examples/field_stats.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,15 @@ export default ({ getService, getPageObjects }: FtrProviderContext) => {
describe('Field stats', () => {
before(async () => {
await kibanaServer.savedObjects.cleanStandardList();
await esArchiver.loadIfNeeded('x-pack/test/functional/es_archives/logstash_functional');
await esArchiver.loadIfNeeded(
'x-pack/platform/test/fixtures/es_archives/logstash_functional'
);
await kibanaServer.importExport.load(
'x-pack/test/functional/fixtures/kbn_archiver/visualize/default'
);
await esArchiver.loadIfNeeded('x-pack/test/functional/es_archives/pre_calculated_histogram');
await esArchiver.loadIfNeeded(
'x-pack/platform/test/fixtures/es_archives/pre_calculated_histogram'
);
await PageObjects.common.navigateToApp('unifiedFieldListExamples');
await PageObjects.header.waitUntilLoadingHasFinished();
await retry.waitFor('combobox is ready', async () => {
Expand All @@ -47,8 +51,8 @@ export default ({ getService, getPageObjects }: FtrProviderContext) => {
});

after(async () => {
await esArchiver.unload('x-pack/test/functional/es_archives/logstash_functional');
await esArchiver.unload('x-pack/test/functional/es_archives/pre_calculated_histogram');
await esArchiver.unload('x-pack/platform/test/fixtures/es_archives/logstash_functional');
await esArchiver.unload('x-pack/platform/test/fixtures/es_archives/pre_calculated_histogram');
await kibanaServer.savedObjects.cleanStandardList();
await PageObjects.unifiedFieldList.cleanSidebarLocalStorage();
});
Expand Down
2 changes: 1 addition & 1 deletion src/platform/test/examples/unified_tabs_examples/manage_tabs.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ export default ({ getService, getPageObjects }: FtrProviderContext) => {
});

after(async () => {
await esArchiver.unload('x-pack/test/functional/es_archives/logstash_functional');
await esArchiver.unload('x-pack/platform/test/fixtures/es_archives/logstash_functional');
await kibanaServer.savedObjects.cleanStandardList();
});

Expand Down
2 changes: 1 addition & 1 deletion src/platform/test/functional/apps/discover/group4/_adhoc_data_views.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ export default function ({ getService, getPageObjects }: FtrProviderContext) {

after(async () => {
await kibanaServer.savedObjects.cleanStandardList();
await esArchiver.unload('x-pack/test/functional/es_archives/logstash_functional');
await esArchiver.unload('x-pack/platform/test/fixtures/es_archives/logstash_functional');
});

it('should navigate back correctly from to surrounding and single views', async () => {
Expand Down
2 changes: 1 addition & 1 deletion src/platform/test/functional/apps/discover/group4/_data_view_edit.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ export default function ({ getService, getPageObjects }: FtrProviderContext) {

after(async () => {
await kibanaServer.savedObjects.cleanStandardList();
await esArchiver.unload('x-pack/test/functional/es_archives/logstash_functional');
await esArchiver.unload('x-pack/platform/test/fixtures/es_archives/logstash_functional');
await es.transport.request({
path: '/data-view-index-000001',
method: 'DELETE',
Expand Down
2 changes: 1 addition & 1 deletion src/platform/test/functional/apps/discover/group4/_field_list_new_fields.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ export default function ({ getService, getPageObjects }: FtrProviderContext) {

after(async () => {
await kibanaServer.savedObjects.cleanStandardList();
await esArchiver.unload('x-pack/test/functional/es_archives/logstash_functional');
await esArchiver.unload('x-pack/platform/test/fixtures/es_archives/logstash_functional');
await es.transport.request({
path: '/my-index-000001',
method: 'DELETE',
Expand Down
2 changes: 1 addition & 1 deletion x-pack/performance/journeys_e2e/aiops_log_rate_analysis.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import dataView from '../kbn_archives/large_arrays_data_view.json';

export const journey = new Journey({
kbnArchives: ['x-pack/performance/kbn_archives/large_arrays_data_view'],
esArchives: ['x-pack/test/functional/es_archives/large_arrays'],
esArchives: ['x-pack/platform/test/fixtures/es_archives/large_arrays'],
})
.step('Go to AIOps Log Rate Analysis', async ({ page, kbnUrl, kibanaPage }) => {
// Navigate to Log Rate Analysis with just a prepopulated time range.
Expand Down
4 changes: 2 additions & 2 deletions x-pack/platform/plugins/private/discover_enhanced/test/scout/ui/fixtures/constants.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,10 @@ export const LOGSTASH_IN_RANGE_DATES = {
};

export const ES_ARCHIVES = {
LOGSTASH: 'x-pack/test/functional/es_archives/logstash_functional',
LOGSTASH: 'x-pack/platform/test/fixtures/es_archives/logstash_functional',
NO_TIME_FIELD:
'src/platform/test/functional/fixtures/es_archiver/index_pattern_without_timefield',
ECOMMERCE: 'x-pack/test/functional/es_archives/reporting/ecommerce',
ECOMMERCE: 'x-pack/platform/test/fixtures/es_archives/reporting/ecommerce',
};

export const KBN_ARCHIVES = {
Expand Down
6 changes: 4 additions & 2 deletions x-pack/platform/test/accessibility/apps/group1/roles.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,17 @@ export default function ({ getService, getPageObjects }: FtrProviderContext) {

describe('Kibana roles page Accessibility', () => {
before(async () => {
await esArchiver.loadIfNeeded('x-pack/test/functional/es_archives/logstash_functional');
await esArchiver.loadIfNeeded(
'x-pack/platform/test/fixtures/es_archives/logstash_functional'
);
await kibanaServer.uiSettings.update({
defaultIndex: 'logstash-*',
});
await PageObjects.security.clickElasticsearchRoles();
});

after(async () => {
await esArchiver.unload('x-pack/test/functional/es_archives/logstash_functional');
await esArchiver.unload('x-pack/platform/test/fixtures/es_archives/logstash_functional');
});

it('a11y test for Roles main page', async () => {
Expand Down
6 changes: 4 additions & 2 deletions x-pack/platform/test/accessibility/apps/group1/search_profiler.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,16 @@ export default function ({ getService, getPageObjects }: FtrProviderContext) {

describe('Search Profiler Editor Accessibility', () => {
before(async () => {
await esArchiver.loadIfNeeded('x-pack/test/functional/es_archives/logstash_functional');
await esArchiver.loadIfNeeded(
'x-pack/platform/test/fixtures/es_archives/logstash_functional'
);
await PageObjects.common.navigateToApp('searchProfiler');
await a11y.testAppSnapshot();
expect(await testSubjects.exists('searchProfilerEditor')).to.be(true);
});

after(async () => {
await esArchiver.unload('x-pack/test/functional/es_archives/logstash_functional');
await esArchiver.unload('x-pack/platform/test/fixtures/es_archives/logstash_functional');
});

it('input the JSON in the editor', async () => {
Expand Down
4 changes: 2 additions & 2 deletions x-pack/platform/test/accessibility/apps/group2/lens.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,14 @@ export default function ({ getService, getPageObjects }: FtrProviderContext) {
describe('Lens Accessibility', () => {
const lensChartName = 'MyLensChart';
before(async () => {
await esArchiver.load('x-pack/test/functional/es_archives/logstash_functional');
await esArchiver.load('x-pack/platform/test/fixtures/es_archives/logstash_functional');
await kibanaServer.importExport.load(
'x-pack/test/functional/fixtures/kbn_archiver/lens/lens_basic.json'
);
});

after(async () => {
await esArchiver.unload('x-pack/test/functional/es_archives/logstash_functional');
await esArchiver.unload('x-pack/platform/test/fixtures/es_archives/logstash_functional');
await kibanaServer.importExport.unload(
'x-pack/test/functional/fixtures/kbn_archiver/lens/lens_basic.json'
);
Expand Down
22 changes: 14 additions & 8 deletions x-pack/platform/test/accessibility/apps/group2/ml.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,11 +72,15 @@ export default function ({ getService }: FtrProviderContext) {
);

before(async () => {
await esArchiver.loadIfNeeded('x-pack/test/functional/es_archives/ml/ihp_outlier');
await esArchiver.loadIfNeeded('x-pack/test/functional/es_archives/ml/egs_regression');
await esArchiver.loadIfNeeded('x-pack/test/functional/es_archives/ml/bm_classification');
await esArchiver.loadIfNeeded('x-pack/platform/test/fixtures/es_archives/ml/ihp_outlier');
await esArchiver.loadIfNeeded(
'x-pack/test/functional/es_archives/ml/module_sample_ecommerce'
'x-pack/platform/test/fixtures/es_archives/ml/egs_regression'
);
await esArchiver.loadIfNeeded(
'x-pack/platform/test/fixtures/es_archives/ml/bm_classification'
);
await esArchiver.loadIfNeeded(
'x-pack/platform/test/fixtures/es_archives/ml/module_sample_ecommerce'
);
await ml.testResources.createDataViewIfNeeded(ihpIndexName);
await ml.testResources.createDataViewIfNeeded(egsIndexName);
Expand All @@ -97,10 +101,12 @@ export default function ({ getService }: FtrProviderContext) {
await ml.testResources.deleteDataViewByTitle(egsIndexName);
await ml.testResources.deleteDataViewByTitle(bmIndexName);
await ml.testResources.deleteDataViewByTitle(ecIndexName);
await esArchiver.unload('x-pack/test/functional/es_archives/ml/ihp_outlier');
await esArchiver.unload('x-pack/test/functional/es_archives/ml/egs_regression');
await esArchiver.unload('x-pack/test/functional/es_archives/ml/bm_classification');
await esArchiver.unload('x-pack/test/functional/es_archives/ml/module_sample_ecommerce');
await esArchiver.unload('x-pack/platform/test/fixtures/es_archives/ml/ihp_outlier');
await esArchiver.unload('x-pack/platform/test/fixtures/es_archives/ml/egs_regression');
await esArchiver.unload('x-pack/platform/test/fixtures/es_archives/ml/bm_classification');
await esArchiver.unload(
'x-pack/platform/test/fixtures/es_archives/ml/module_sample_ecommerce'
);
await ml.testResources.resetKibanaTimeZone();
});

Expand Down
16 changes: 10 additions & 6 deletions x-pack/platform/test/accessibility/apps/group2/ml_anomaly_detection.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,12 +199,12 @@ export default function ({ getService }: FtrProviderContext) {
};

before(async () => {
await esArchiver.loadIfNeeded('x-pack/test/functional/es_archives/ml/farequote');
await esArchiver.loadIfNeeded('x-pack/platform/test/fixtures/es_archives/ml/farequote');
await esArchiver.loadIfNeeded(
'x-pack/test/functional/es_archives/ml/module_sample_ecommerce'
'x-pack/platform/test/fixtures/es_archives/ml/module_sample_ecommerce'
);
await esArchiver.loadIfNeeded(
'x-pack/test/functional/es_archives/ml/categorization_small'
'x-pack/platform/test/fixtures/es_archives/ml/categorization_small'
);
await ml.testResources.createDataViewIfNeeded(fqIndexName, '@timestamp');
await ml.testResources.createDataViewIfNeeded(ecIndexName, 'order_date');
Expand Down Expand Up @@ -243,9 +243,13 @@ export default function ({ getService }: FtrProviderContext) {
await ml.testResources.deleteDataViewByTitle(fqIndexName);
await ml.testResources.deleteDataViewByTitle(ecIndexName);
await ml.testResources.deleteDataViewByTitle(categorizationIndexName);
await esArchiver.unload('x-pack/test/functional/es_archives/ml/farequote');
await esArchiver.unload('x-pack/test/functional/es_archives/ml/module_sample_ecommerce');
await esArchiver.unload('x-pack/test/functional/es_archives/ml/categorization_small');
await esArchiver.unload('x-pack/platform/test/fixtures/es_archives/ml/farequote');
await esArchiver.unload(
'x-pack/platform/test/fixtures/es_archives/ml/module_sample_ecommerce'
);
await esArchiver.unload(
'x-pack/platform/test/fixtures/es_archives/ml/categorization_small'
);
await ml.testResources.resetKibanaTimeZone();
});

Expand Down
4 changes: 2 additions & 2 deletions x-pack/platform/test/accessibility/apps/group2/transform.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ export default function ({ getService }: FtrProviderContext) {
const latestTransformDestinationIndex = `user-${latestTransformId}`;

before(async () => {
await esArchiver.loadIfNeeded('x-pack/test/functional/es_archives/ml/ecommerce');
await esArchiver.loadIfNeeded('x-pack/platform/test/fixtures/es_archives/ml/ecommerce');
await transform.testResources.createDataViewIfNeeded(ecIndexName, 'order_date');
await transform.testResources.setKibanaTimeZoneToUTC();
});
Expand All @@ -96,7 +96,7 @@ export default function ({ getService }: FtrProviderContext) {
await transform.testResources.deleteDataViewByTitle(pivotTransformDestinationIndex);
await transform.testResources.deleteDataViewByTitle(latestTransformDestinationIndex);
await transform.testResources.deleteDataViewByTitle(ecIndexName);
await esArchiver.unload('x-pack/test/functional/es_archives/ml/ecommerce');
await esArchiver.unload('x-pack/platform/test/fixtures/es_archives/ml/ecommerce');
await transform.testResources.resetKibanaTimeZone();
});

Expand Down
Loading