[Lens] Move TSDB and LOGSDB test into separate testing groups

.buildkite/ftr_oblt_serverless_configs.yml

@@ -38,6 +38,8 @@ enabled:
   - x-pack/platform/test/serverless/functional/configs/observability/config.group10.ts
   - x-pack/platform/test/serverless/functional/configs/observability/config.group11.ts
   - x-pack/platform/test/serverless/functional/configs/observability/config.group12.ts
+  - x-pack/platform/test/serverless/functional/configs/observability/config.group13.ts
+  - x-pack/platform/test/serverless/functional/configs/observability/config.group14.ts
   - x-pack/platform/test/serverless/functional/configs/observability/config.logs_essentials.group1.ts
   - x-pack/solutions/observability/test/serverless/functional/configs/config.screenshots.ts
   - x-pack/solutions/observability/test/serverless/functional/configs/config.telemetry.ts

.buildkite/ftr_platform_stateful_configs.yml

@@ -232,6 +232,10 @@ enabled:
   - x-pack/platform/test/functional/apps/lens/group5/config.ts
   - x-pack/platform/test/functional/apps/lens/group6/config.ts
   - x-pack/platform/test/functional/apps/lens/group7/config.ts
+  - x-pack/platform/test/functional/apps/lens/group8/config.ts
+  - x-pack/platform/test/functional/apps/lens/group9/config.ts
+  - x-pack/platform/test/functional/apps/lens/group10/config.ts
+  - x-pack/platform/test/functional/apps/lens/group11/config.ts
   - x-pack/platform/test/functional/apps/lens/open_in_lens/tsvb/config.ts
   - x-pack/platform/test/functional/apps/lens/open_in_lens/agg_based/config.ts
   - x-pack/platform/test/functional/apps/lens/open_in_lens/dashboard/config.ts

.buildkite/ftr_search_serverless_configs.yml

@@ -27,4 +27,6 @@ enabled:
   - x-pack/platform/test/serverless/functional/configs/search/config.group10.ts
   - x-pack/platform/test/serverless/functional/configs/search/config.group11.ts
   - x-pack/platform/test/serverless/functional/configs/search/config.group12.ts
+  - x-pack/platform/test/serverless/functional/configs/search/config.group13.ts
+  - x-pack/platform/test/serverless/functional/configs/search/config.group14.ts
   - x-pack/platform/test/agent_builder_api_integration/configs/config.serverless.ts

.buildkite/ftr_security_serverless_configs.yml

@@ -57,6 +57,8 @@ enabled:
   - x-pack/platform/test/serverless/functional/configs/security/config.group10.ts
   - x-pack/platform/test/serverless/functional/configs/security/config.group11.ts
   - x-pack/platform/test/serverless/functional/configs/security/config.group12.ts
+  - x-pack/platform/test/serverless/functional/configs/security/config.group13.ts
+  - x-pack/platform/test/serverless/functional/configs/security/config.group14.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/detection_engine/actions/trial_license_complete_tier/configs/serverless.config.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/detection_engine/alerts/basic_license_essentials_tier/configs/serverless.config.ts
   - x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/detection_engine/alerts/trial_license_complete_tier/configs/serverless.config.ts

x-pack/platform/test/functional/apps/lens/README.md

@@ -1,7 +1,138 @@
-# What are all these groups?
+# Lens Functional Tests
 
-These tests take a while so they have been broken up into groups with their own `config.ts` and `index.ts` file, causing each of these groups to be independent bundles of tests which can be run on some worker in CI without taking an incredible amount of time.
+This directory contains functional tests for the Lens visualization editor.
 
-Want to change the groups to something more logical? Have fun! Just make sure that each group executes on CI in less than 10 minutes or so. We don't currently have any mechanism for validating this right now, you just need to look at the times in the log output on CI, but we'll be working on tooling for making this information more accessible soon.
+## Test Groups Overview
 
-- Kibana Operations
+| Group | Description | ~Duration |
+|-------|-------------|-----------|
+| group1-6 | Core Lens functionality tests | varies |
+| group7 | ES\|QL tests | ~10-15 min |
+| group8 | LogsDB upgrade scenarios | ~20 min |
+| group9 | TSDB upgrade scenarios | ~15 min |
+| group10 | LogsDB downgrade scenarios | ~20 min |
+| group11 | TSDB downgrade scenarios | ~15 min |
+
+---
+
+## LogsDB & TSDB Scenario-Based Tests
+
+Groups 8-11 use a **scenario-based testing pattern** to verify Lens works correctly with different Elasticsearch index modes and their combinations.
+
+### What is LogsDB?
+
+**LogsDB** is an Elasticsearch index mode optimized for log data. It uses:
+- Synthetic `_source` for storage efficiency
+- Dimension fields (like `host.name`) for efficient grouping
+- Special handling for log-specific patterns
+
+### What is TSDB?
+
+**TSDB** (Time Series Database) is an Elasticsearch index mode for metrics/time series data. It features:
+- Counter and gauge field types with specific aggregation restrictions
+- Downsampling support for data rollup
+- Time series dimensions for efficient querying
+
+---
+
+## Scenario Testing Pattern
+
+### Why Test Multiple Index Combinations?
+
+Real-world Kibana Data Views often span multiple indices of different types:
+- A `logs-*` pattern might match both LogsDB streams and legacy regular indices
+- During migrations, old and new index formats coexist
+- Users may combine different data sources in a single visualization
+
+### The 6 LogsDB Scenarios
+
+Each scenario represents a **Data View configuration** combining different index types:
+
+| # | Scenario | Indexes in Data View | Purpose |
+|---|----------|---------------------|---------|
+| 1 | No additional stream/index | `[logsdb_stream]` | Baseline: Pure LogsDB only |
+| 2 | No additional + no host.name | `[logsdb_stream_no_host]` | LogsDB without dimension field |
+| 3 | + Regular index | `[logsdb_stream, regular_index]` | Mix with standard ES index |
+| 4 | + LogsDB stream | `[logsdb_stream, logsdb_index_2]` | Two LogsDB streams |
+| 5 | + TSDB stream | `[logsdb_stream, tsdb_index]` | Mix with Time Series DB |
+| 6 | + TSDB downsampled | `[logsdb_stream, tsdb_downsampled]` | Mix with downsampled TSDB |
+
+### The 5 TSDB Scenarios
+
+| # | Scenario | Indexes in Data View | Purpose |
+|---|----------|---------------------|---------|
+| 1 | No additional stream/index | `[tsdb_stream]` | Baseline: Pure TSDB only |
+| 2 | + Regular index | `[tsdb_stream, regular_index]` | Mix with standard ES index |
+| 3 | + Downsampled TSDB | `[tsdb_stream, tsdb_downsampled]` | Mix with downsampled data |
+| 4 | + Regular + Downsampled | `[tsdb_stream, regular, downsampled]` | Three-way mix |
+| 5 | + Additional TSDB | `[tsdb_stream, tsdb_index_2]` | Two TSDB streams |
+
+---
+
+## Upgrade vs Downgrade Testing
+
+### Upgrade Scenarios (Groups 8 & 9)
+
+Tests when a regular data stream is **upgraded** to LogsDB/TSDB:
+
+```
+Time: ──────────────────────────────────────────→
+      [Regular stream data] │ [LogsDB/TSDB data]
+                            ↑
+                     Upgrade point
+```
+
+**What's tested**: Lens can visualize data spanning both pre-upgrade (regular) and post-upgrade (specialized) periods.
+
+### Downgrade Scenarios (Groups 10 & 11)
+
+Tests when a LogsDB/TSDB stream is **downgraded** to regular:
+
+```
+Time: ──────────────────────────────────────────→
+      [LogsDB/TSDB data] │ [Regular stream data]
+                         ↑
+                  Downgrade point
+```
+
+**What's tested**: Lens handles transitions when special features are no longer available for new data.
+
+---
+
+## Technical Challenges Being Tested
+
+### 1. Field Type Conflicts
+LogsDB uses `host.name` as a dimension field with special handling. When mixed with regular indices, Lens must handle the field correctly across both.
+
+### 2. Aggregation Compatibility
+TSDB has restrictions (e.g., counter fields can't use `average`). When a Data View spans LogsDB + TSDB, Lens must enforce the stricter rules.
+
+### 3. Downsampled Data
+TSDB downsampling pre-aggregates data, changing which functions are valid (e.g., `median` shows warnings). Mixing this with other index types tests that Lens properly detects and handles these restrictions.
+
+### 4. Time Window Transitions
+Tests verify data is visible when the time picker spans across upgrade/downgrade boundaries.
+
+---
+
+## Test Multiplication
+
+Each scenario runs **5 tests** per block:
+1. Date histogram chart
+2. Date histogram with different date field
+3. Annotation layer
+4. Annotation layer with alternate time field
+5. ES|QL query visualization
+
+**LogsDB total**: `6 scenarios × 5 tests × 2 blocks = 60 tests + 1 smoke test = 61`
+**TSDB total**: `5 scenarios × 2-3 tests × 2 blocks + downsampling + field type tests ≈ 35 tests`
+
+---
+
+## Related Files
+
+- `tsdb_logsdb_helpers.ts` - Shared utilities for scenario setup and test runners
+- `group8/logsdb.ts` - LogsDB upgrade scenarios
+- `group9/tsdb.ts` - TSDB upgrade scenarios  
+- `group10/logsdb_downgrade.ts` - LogsDB downgrade scenarios
+- `group11/tsdb_downgrade.ts` - TSDB downgrade scenarios

x-pack/platform/test/functional/apps/lens/group10/config.ts

@@ -0,0 +1,17 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import type { FtrConfigProviderContext } from '@kbn/test';
+
+export default async function ({ readConfigFile }: FtrConfigProviderContext) {
+  const functionalConfig = await readConfigFile(require.resolve('../../../config.base.ts'));
+
+  return {
+    ...functionalConfig.getAll(),
+    testFiles: [require.resolve('.')],
+  };
+}

x-pack/platform/test/functional/apps/lens/group10/index.ts

@@ -0,0 +1,76 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import type { EsArchiver } from '@kbn/es-archiver';
+import type { FtrProviderContext } from '../../../ftr_provider_context';
+
+export default ({ getService, loadTestFile, getPageObjects }: FtrProviderContext) => {
+  const browser = getService('browser');
+  const log = getService('log');
+  const esArchiver = getService('esArchiver');
+  const kibanaServer = getService('kibanaServer');
+  const { timePicker } = getPageObjects(['timePicker']);
+  const config = getService('config');
+  let remoteEsArchiver;
+
+  describe('lens app - group 10', () => {
+    const esArchive = 'x-pack/platform/test/fixtures/es_archives/logstash_functional';
+    const localIndexPatternString = 'logstash-*';
+    const remoteIndexPatternString = 'ftr-remote:logstash-*';
+    const localFixtures = {
+      lensBasic: 'x-pack/platform/test/functional/fixtures/kbn_archives/lens/lens_basic.json',
+      lensDefault: 'x-pack/platform/test/functional/fixtures/kbn_archives/lens/default',
+    };
+
+    const remoteFixtures = {
+      lensBasic: 'x-pack/platform/test/functional/fixtures/kbn_archives/lens/ccs/lens_basic.json',
+      lensDefault: 'x-pack/platform/test/functional/fixtures/kbn_archives/lens/ccs/default',
+    };
+    let esNode: EsArchiver;
+    let fixtureDirs: {
+      lensBasic: string;
+      lensDefault: string;
+    };
+    let indexPatternString: string;
+    before(async () => {
+      await log.debug('Starting lens before method');
+      await browser.setWindowSize(1280, 1200);
+      await kibanaServer.savedObjects.cleanStandardList();
+      try {
+        config.get('esTestCluster.ccs');
+        remoteEsArchiver = getService('remoteEsArchiver' as 'esArchiver');
+        esNode = remoteEsArchiver;
+        fixtureDirs = remoteFixtures;
+        indexPatternString = remoteIndexPatternString;
+      } catch (error) {
+        esNode = esArchiver;
+        fixtureDirs = localFixtures;
+        indexPatternString = localIndexPatternString;
+      }
+
+      await esNode.load(esArchive);
+      // changing the timepicker default here saves us from having to set it in Discover (~8s)
+      await timePicker.setDefaultAbsoluteRangeViaUiSettings();
+      await kibanaServer.uiSettings.update({
+        defaultIndex: indexPatternString,
+        'dateFormat:tz': 'UTC',
+      });
+      await kibanaServer.importExport.load(fixtureDirs.lensBasic);
+      await kibanaServer.importExport.load(fixtureDirs.lensDefault);
+    });
+
+    after(async () => {
+      await esArchiver.unload(esArchive);
+      await timePicker.resetDefaultAbsoluteRangeViaUiSettings();
+      await kibanaServer.importExport.unload(fixtureDirs.lensBasic);
+      await kibanaServer.importExport.unload(fixtureDirs.lensDefault);
+      await kibanaServer.savedObjects.cleanStandardList();
+    });
+
+    loadTestFile(require.resolve('./logsdb_downgrade'));
+  });
+};