Skip to content

[scout] cross-project-search support#254791

Merged
dmlemeshko merged 29 commits intoelastic:mainfrom
dmlemeshko:scout/cps-support
Mar 11, 2026
Merged

[scout] cross-project-search support#254791
dmlemeshko merged 29 commits intoelastic:mainfrom
dmlemeshko:scout/cps-support

Conversation

@dmlemeshko
Copy link
Copy Markdown
Contributor

@dmlemeshko dmlemeshko commented Feb 24, 2026

Summary

Adds Cross Project Search (CPS) support to Scout's local serverless test infrastructure. When enabled, Scout starts a second ("linked") ES cluster that shares the origin cluster's UIAM service, allowing tests to exercise cross-project search scenarios locally.

How it works

image
  1. A new Scout server config set cps_local/ extends the existing uiam_local/ config with cps: true

  2. When CSP is enabled, the origin ES cluster starts with serverless.cross_project.enabled and remote_cluster_server.enabled flags

  3. After the origin cluster and UIAM are fully ready, a second ES cluster starts with:

  • Its own project ID (MOCK_IDP_UIAM_PROJECT_ID2), cluster name, and isolated object store
  • The same Docker network and UIAM service (no duplicate UIAM containers)
  • Port offset +10 from the origin (HTTP on 9230)
  1. Once the linked cluster is healthy, the origin's operator settings.json is updated to register the linked project endpoint, enabling ES to discover it via /_project/tags
  2. Scout exposes linkedProject fixture for test data ingestion into the linked cluster

Scope of changes

@kbn/es (docker.ts)

  • Refactored SERVERLESS_NODES and SHARED_SERVERLESS_PARAMS into factory functions (getServerlessNodes, getSharedServerlessParams) to support dynamic container names and port offsets
  • Parameterized getOperatorVolume, setupServerlessVolumes, and resolveEsArgs to accept project ID overrides
  • Added runLinkedServerlessCluster() — starts a 3-node linked ES cluster and registers it with the origin's operator settings
  • Updated container cleanup to also handle linked cluster containers

@kbn/es (ServerlessOptions)

  • Added linkedProject?: { projectId: string; port: number } for explicit linked cluster configuration

@kbn/mock-idp-utils

  • Added MOCK_IDP_UIAM_PROJECT_ID2 constant for the linked project

@kbn/scout (config & schema)

  • Added cps boolean to esServerlessOptions schema and ScoutServerConfig type
  • Created cps_local/serverless/security_complete.serverless.config.ts config set
  • Fixed hardcoded https://localhost:9200 in Fleet outputs to use configured ES host/port

@kbn/scout (runtime & fixtures)

  • run_elasticsearch.ts calls runLinkedServerlessCluster after the origin cluster when CSP is enabled
  • ScoutTestConfig extended with linkedProject (hosts + auth) for test access
  • Added linkedEsClient, linkedEsArchiver services and Playwright worker fixture linkedProject to use in tests

How to test:

node scripts/scout start-server --arch serverless --domain security_complete --serverConfigSet cps_local

Check that it starts:

  • Origin ES cluster (3 nodes) on port 9220 with UIAM
  • Linked ES cluster (3 nodes) on port 9230 (connected to the same UIAM)
  • Kibana on port 5620
# Origin cluster
curl -sk https://elastic_serverless:changeme@localhost:9220

# Linked cluster
curl -sk https://elastic_serverless:changeme@localhost:9230

Go to Kibana http://localhost:5620/app/dev_tools#/console and check projects being linked:

GET _project/tags

should return

{
  "origin": {
    "abcdef12345678901234567890123456": {
      "_alias": "local_project",
      "_id": "abcdef12345678901234567890123456",
      "_organization": "org1234567890",
      "_type": "security",
      "env": "local"
    }
  },
  "linked_projects": {
    "fedcba65432109876543210987654321": {
      "_alias": "linked_local_project",
      "_id": "fedcba65432109876543210987654321",
      "_organization": "org1234567890",
      "_type": "security",
      "env": "local"
    }
  }
}

@dmlemeshko dmlemeshko changed the title Scout/cps support [scout] cross-project-search support Feb 24, 2026
@dmlemeshko dmlemeshko added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting labels Feb 26, 2026
dmlemeshko added a commit that referenced this pull request Feb 26, 2026
## Summary

related to #254791

Adds a `dataOnly` mode to `EsArchiver` that enforces data-only ingestion
by rejecting archives containing saved object indices (`.kibana*`) and
removes the requirement for `kbnClient`.

### Motivation:
- Scout tests should use `kbnArchiver` for saved objects, not
`esArchiver`. This change makes that boundary explicit by throwing a
clear error if a Scout test accidentally tries to load .kibana* archives
via esArchiver.
- For Cross Project Search (CPS) support, the linked project's ES
cluster has no Kibana instance, so `esArchiver` must work without
`kbnClient`.

### What changed:
- `EsArchiver` constructor accepts a new `dataOnly?: boolean` option.
When true, `kbnClient` is not required, `emptyKibanaIndex()` throws
error, and `load()/loadIfNeeded()` reject any archive that touches saved
object indices.
- `loadAction` skips the `kbnClient.plugins.getEnabledIds()` call and
saved object migration when `dataOnly` is enabled.
- Scout's `getEsArchiver` and its Playwright fixture now use `dataOnly:
true` without `kbnClient`.
- FTR and Cypress flows are unchanged -- they continue using
`EsArchiver` without dataOnly, with full `kbnClient` support.

---------

Co-authored-by: Cesare de Cal <cesare.decal@elastic.co>
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Feb 26, 2026
…254962)

## Summary

related to elastic#254791

Adds a `dataOnly` mode to `EsArchiver` that enforces data-only ingestion
by rejecting archives containing saved object indices (`.kibana*`) and
removes the requirement for `kbnClient`.

### Motivation:
- Scout tests should use `kbnArchiver` for saved objects, not
`esArchiver`. This change makes that boundary explicit by throwing a
clear error if a Scout test accidentally tries to load .kibana* archives
via esArchiver.
- For Cross Project Search (CPS) support, the linked project's ES
cluster has no Kibana instance, so `esArchiver` must work without
`kbnClient`.

### What changed:
- `EsArchiver` constructor accepts a new `dataOnly?: boolean` option.
When true, `kbnClient` is not required, `emptyKibanaIndex()` throws
error, and `load()/loadIfNeeded()` reject any archive that touches saved
object indices.
- `loadAction` skips the `kbnClient.plugins.getEnabledIds()` call and
saved object migration when `dataOnly` is enabled.
- Scout's `getEsArchiver` and its Playwright fixture now use `dataOnly:
true` without `kbnClient`.
- FTR and Cypress flows are unchanged -- they continue using
`EsArchiver` without dataOnly, with full `kbnClient` support.

---------

Co-authored-by: Cesare de Cal <cesare.decal@elastic.co>
(cherry picked from commit 8500782)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Feb 26, 2026
…254962)

## Summary

related to elastic#254791

Adds a `dataOnly` mode to `EsArchiver` that enforces data-only ingestion
by rejecting archives containing saved object indices (`.kibana*`) and
removes the requirement for `kbnClient`.

### Motivation:
- Scout tests should use `kbnArchiver` for saved objects, not
`esArchiver`. This change makes that boundary explicit by throwing a
clear error if a Scout test accidentally tries to load .kibana* archives
via esArchiver.
- For Cross Project Search (CPS) support, the linked project's ES
cluster has no Kibana instance, so `esArchiver` must work without
`kbnClient`.

### What changed:
- `EsArchiver` constructor accepts a new `dataOnly?: boolean` option.
When true, `kbnClient` is not required, `emptyKibanaIndex()` throws
error, and `load()/loadIfNeeded()` reject any archive that touches saved
object indices.
- `loadAction` skips the `kbnClient.plugins.getEnabledIds()` call and
saved object migration when `dataOnly` is enabled.
- Scout's `getEsArchiver` and its Playwright fixture now use `dataOnly:
true` without `kbnClient`.
- FTR and Cypress flows are unchanged -- they continue using
`EsArchiver` without dataOnly, with full `kbnClient` support.

---------

Co-authored-by: Cesare de Cal <cesare.decal@elastic.co>
(cherry picked from commit 8500782)
@dmlemeshko dmlemeshko marked this pull request as ready for review February 26, 2026 12:07
@dmlemeshko dmlemeshko requested review from a team as code owners February 26, 2026 12:07

test.beforeAll(async ({ linkedProject }) => {
// Load data archive into the linked ES cluster
await linkedProject.esArchiver.loadIfNeeded('path/to/data/archive');
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know these are examples, but since the field is optional, it's probably:

Suggested change
await linkedProject.esArchiver.loadIfNeeded('path/to/data/archive');
await linkedProject?.esArchiver?.loadIfNeeded('path/to/data/archive');

Or we need a null-check.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually forgot to wire linkedProject for ui and api tests, done now.

Those fields are not optional, but I already have a check to throw clear error message in case someone use it for non-CPS setup:

https://github.com/dmlemeshko/kibana/blob/4cfb12ac6940d62851dfd185e42a42a2870b906b/src/platform/packages/shared/kbn-scout/src/playwright/fixtures/scope/worker/linked_es_archiver.ts#L29-L36

];
}

const SHARED_SERVERLESS_PARAMS = getSharedServerlessParams();
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since the function is deterministic, we could just use the function wherever we need these node names, no need to store this in a constant-like variable

const namesCmd = SERVERLESS_NODES.concat(UIAM_CONTAINERS).reduce<string[]>((acc, { name }) => {
acc.push('--filter', `name=${name}`);
const linkedNodes = getServerlessNodes('-linked', 10);
const namesCmd = SERVERLESS_NODES.concat(linkedNodes, UIAM_CONTAINERS).reduce<string[]>(
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: a .flatMap makes for better readability here

@dmlemeshko dmlemeshko requested a review from delanni March 11, 2026 09:45
@elasticmachine
Copy link
Copy Markdown
Contributor

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/es 41 47 +6
@kbn/mock-idp-utils 50 51 +1
@kbn/scout 253 255 +2
@kbn/scout-oblt 259 261 +2
@kbn/scout-search 248 250 +2
@kbn/scout-security 263 265 +2
total +15

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id before after diff
@kbn/scout 35 36 +1
Unknown metric groups

API count

id before after diff
@kbn/es 57 63 +6
@kbn/mock-idp-utils 57 58 +1
@kbn/scout 659 661 +2
@kbn/scout-oblt 650 652 +2
@kbn/scout-search 639 641 +2
@kbn/scout-security 654 656 +2
total +15

History

cc @dmlemeshko

qn895 pushed a commit to qn895/kibana that referenced this pull request Mar 11, 2026
…254962)

## Summary

related to elastic#254791

Adds a `dataOnly` mode to `EsArchiver` that enforces data-only ingestion
by rejecting archives containing saved object indices (`.kibana*`) and
removes the requirement for `kbnClient`.

### Motivation:
- Scout tests should use `kbnArchiver` for saved objects, not
`esArchiver`. This change makes that boundary explicit by throwing a
clear error if a Scout test accidentally tries to load .kibana* archives
via esArchiver.
- For Cross Project Search (CPS) support, the linked project's ES
cluster has no Kibana instance, so `esArchiver` must work without
`kbnClient`.

### What changed:
- `EsArchiver` constructor accepts a new `dataOnly?: boolean` option.
When true, `kbnClient` is not required, `emptyKibanaIndex()` throws
error, and `load()/loadIfNeeded()` reject any archive that touches saved
object indices.
- `loadAction` skips the `kbnClient.plugins.getEnabledIds()` call and
saved object migration when `dataOnly` is enabled.
- Scout's `getEsArchiver` and its Playwright fixture now use `dataOnly:
true` without `kbnClient`.
- FTR and Cypress flows are unchanged -- they continue using
`EsArchiver` without dataOnly, with full `kbnClient` support.

---------

Co-authored-by: Cesare de Cal <cesare.decal@elastic.co>
@dmlemeshko dmlemeshko merged commit c3b16eb into elastic:main Mar 11, 2026
17 checks passed
@dmlemeshko dmlemeshko added backport:all-open Backport to all branches that could still receive a release and removed v9.4.0 backport:skip This PR does not require backporting labels Mar 11, 2026
@kibanamachine
Copy link
Copy Markdown
Contributor

Starting backport for target branches: 8.19, 9.2, 9.3

https://github.com/elastic/kibana/actions/runs/22963579221

@kibanamachine
Copy link
Copy Markdown
Contributor

💔 All backports failed

Status Branch Result
8.19 Backport failed because of merge conflicts

You might need to backport the following PRs to 8.19:
- [Ai4DSoc][Serverless] Update search_ai_lake tier specific roles (#229919)
9.2 Backport failed because of merge conflicts

You might need to backport the following PRs to 9.2:
- [Scout] Improve test target definition and tagging (#251436)
9.3 Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 254791

Questions ?

Please refer to the Backport tool documentation

sorenlouv pushed a commit that referenced this pull request Mar 17, 2026
## Summary

Adds Cross Project Search (CPS) support to Scout's local serverless test
infrastructure. When enabled, Scout starts a second ("linked") ES
cluster that shares the origin cluster's UIAM service, allowing tests to
exercise cross-project search scenarios locally.

### How it works

<img width="1834" height="754" alt="image"
src="https://github.com/user-attachments/assets/12ce2141-3b05-49e5-82ed-dc364332bacd"
/>


1. A new Scout server config set `cps_local/` extends the existing
`uiam_local/` config with `cps: true`

2. When CSP is enabled, the origin ES cluster starts with
`serverless.cross_project.enabled` and `remote_cluster_server.enabled`
flags
3. After the origin cluster and UIAM are fully ready, a second ES
cluster starts with:
- Its own project ID (`MOCK_IDP_UIAM_PROJECT_ID2`), cluster name, and
isolated object store
- The same Docker network and UIAM service (no duplicate UIAM
containers)
- Port offset `+10 `from the origin (HTTP on `9230`)
4. Once the linked cluster is healthy, the origin's operator
settings.json is updated to register the linked project endpoint,
enabling ES to discover it via /_project/tags
5. Scout exposes `linkedProject` fixture for test data ingestion into
the linked cluster

### Scope of changes

`@kbn/es (docker.ts)`
- Refactored `SERVERLESS_NODES` and `SHARED_SERVERLESS_PARAMS` into
factory functions (`getServerlessNodes`, `getSharedServerlessParams`) to
support dynamic container names and port offsets
- Parameterized `getOperatorVolume`, `setupServerlessVolumes`, and
`resolveEsArgs` to accept project ID overrides
- Added `runLinkedServerlessCluster()` — starts a 3-node linked ES
cluster and registers it with the origin's operator settings
- Updated container cleanup to also handle linked cluster containers

`@kbn/es (ServerlessOptions)`
- Added `linkedProject?: { projectId: string; port: number }` for
explicit linked cluster configuration

`@kbn/mock-idp-utils`
- Added `MOCK_IDP_UIAM_PROJECT_ID2` constant for the linked project

`@kbn/scout (config & schema)`
- Added `cps` boolean to `esServerlessOptions` schema and
`ScoutServerConfig` type
- Created `cps_local/serverless/security_complete.serverless.config.ts`
config set
- Fixed hardcoded `https://localhost:9200` in Fleet outputs to use
configured ES host/port

`@kbn/scout (runtime & fixtures)`
- `run_elasticsearch.ts` calls `runLinkedServerlessCluster` after the
origin cluster when CSP is enabled
- `ScoutTestConfig` extended with linkedProject (hosts + auth) for test
access
- Added `linkedEsClient`, `linkedEsArchiver` services and Playwright
worker fixture `linkedProject` to use in tests

### How to test:

```
node scripts/scout start-server --arch serverless --domain security_complete --serverConfigSet cps_local
```

Check that it starts:
- Origin ES cluster (3 nodes) on port `9220` with UIAM
- Linked ES cluster (3 nodes) on port `9230` (connected to the same
UIAM)
- Kibana on port `5620`

```
# Origin cluster
curl -sk https://elastic_serverless:changeme@localhost:9220

# Linked cluster
curl -sk https://elastic_serverless:changeme@localhost:9230
```

Go to Kibana http://localhost:5620/app/dev_tools#/console and check
projects being linked:

```
GET _project/tags
```

should return

```
{
  "origin": {
    "abcdef12345678901234567890123456": {
      "_alias": "local_project",
      "_id": "abcdef12345678901234567890123456",
      "_organization": "org1234567890",
      "_type": "security",
      "env": "local"
    }
  },
  "linked_projects": {
    "fedcba65432109876543210987654321": {
      "_alias": "linked_local_project",
      "_id": "fedcba65432109876543210987654321",
      "_organization": "org1234567890",
      "_type": "security",
      "env": "local"
    }
  }
}
```

---------

Co-authored-by: Cesare de Cal <cesare.decal@elastic.co>
@dmlemeshko
Copy link
Copy Markdown
Contributor Author

💚 All backports created successfully

Status Branch Result
9.3

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

dmlemeshko added a commit to dmlemeshko/kibana that referenced this pull request Mar 17, 2026
## Summary

Adds Cross Project Search (CPS) support to Scout's local serverless test
infrastructure. When enabled, Scout starts a second ("linked") ES
cluster that shares the origin cluster's UIAM service, allowing tests to
exercise cross-project search scenarios locally.

### How it works

<img width="1834" height="754" alt="image"
src="https://github.com/user-attachments/assets/12ce2141-3b05-49e5-82ed-dc364332bacd"
/>

1. A new Scout server config set `cps_local/` extends the existing
`uiam_local/` config with `cps: true`

2. When CSP is enabled, the origin ES cluster starts with
`serverless.cross_project.enabled` and `remote_cluster_server.enabled`
flags
3. After the origin cluster and UIAM are fully ready, a second ES
cluster starts with:
- Its own project ID (`MOCK_IDP_UIAM_PROJECT_ID2`), cluster name, and
isolated object store
- The same Docker network and UIAM service (no duplicate UIAM
containers)
- Port offset `+10 `from the origin (HTTP on `9230`)
4. Once the linked cluster is healthy, the origin's operator
settings.json is updated to register the linked project endpoint,
enabling ES to discover it via /_project/tags
5. Scout exposes `linkedProject` fixture for test data ingestion into
the linked cluster

### Scope of changes

`@kbn/es (docker.ts)`
- Refactored `SERVERLESS_NODES` and `SHARED_SERVERLESS_PARAMS` into
factory functions (`getServerlessNodes`, `getSharedServerlessParams`) to
support dynamic container names and port offsets
- Parameterized `getOperatorVolume`, `setupServerlessVolumes`, and
`resolveEsArgs` to accept project ID overrides
- Added `runLinkedServerlessCluster()` — starts a 3-node linked ES
cluster and registers it with the origin's operator settings
- Updated container cleanup to also handle linked cluster containers

`@kbn/es (ServerlessOptions)`
- Added `linkedProject?: { projectId: string; port: number }` for
explicit linked cluster configuration

`@kbn/mock-idp-utils`
- Added `MOCK_IDP_UIAM_PROJECT_ID2` constant for the linked project

`@kbn/scout (config & schema)`
- Added `cps` boolean to `esServerlessOptions` schema and
`ScoutServerConfig` type
- Created `cps_local/serverless/security_complete.serverless.config.ts`
config set
- Fixed hardcoded `https://localhost:9200` in Fleet outputs to use
configured ES host/port

`@kbn/scout (runtime & fixtures)`
- `run_elasticsearch.ts` calls `runLinkedServerlessCluster` after the
origin cluster when CSP is enabled
- `ScoutTestConfig` extended with linkedProject (hosts + auth) for test
access
- Added `linkedEsClient`, `linkedEsArchiver` services and Playwright
worker fixture `linkedProject` to use in tests

### How to test:

```
node scripts/scout start-server --arch serverless --domain security_complete --serverConfigSet cps_local
```

Check that it starts:
- Origin ES cluster (3 nodes) on port `9220` with UIAM
- Linked ES cluster (3 nodes) on port `9230` (connected to the same
UIAM)
- Kibana on port `5620`

```
# Origin cluster
curl -sk https://elastic_serverless:changeme@localhost:9220

# Linked cluster
curl -sk https://elastic_serverless:changeme@localhost:9230
```

Go to Kibana http://localhost:5620/app/dev_tools#/console and check
projects being linked:

```
GET _project/tags
```

should return

```
{
  "origin": {
    "abcdef12345678901234567890123456": {
      "_alias": "local_project",
      "_id": "abcdef12345678901234567890123456",
      "_organization": "org1234567890",
      "_type": "security",
      "env": "local"
    }
  },
  "linked_projects": {
    "fedcba65432109876543210987654321": {
      "_alias": "linked_local_project",
      "_id": "fedcba65432109876543210987654321",
      "_organization": "org1234567890",
      "_type": "security",
      "env": "local"
    }
  }
}
```

---------

Co-authored-by: Cesare de Cal <cesare.decal@elastic.co>
(cherry picked from commit c3b16eb)

# Conflicts:
#	src/platform/packages/private/kbn-mock-idp-utils/src/constants.ts
#	src/platform/packages/shared/kbn-es/src/utils/docker.ts
#	src/platform/packages/shared/kbn-scout/src/playwright/index.ts
@dmlemeshko dmlemeshko added backport:skip This PR does not require backporting and removed backport:all-open Backport to all branches that could still receive a release labels Mar 17, 2026
jeramysoucy pushed a commit to jeramysoucy/kibana that referenced this pull request Mar 26, 2026
## Summary

Adds Cross Project Search (CPS) support to Scout's local serverless test
infrastructure. When enabled, Scout starts a second ("linked") ES
cluster that shares the origin cluster's UIAM service, allowing tests to
exercise cross-project search scenarios locally.

### How it works

<img width="1834" height="754" alt="image"
src="https://github.com/user-attachments/assets/12ce2141-3b05-49e5-82ed-dc364332bacd"
/>


1. A new Scout server config set `cps_local/` extends the existing
`uiam_local/` config with `cps: true`

2. When CSP is enabled, the origin ES cluster starts with
`serverless.cross_project.enabled` and `remote_cluster_server.enabled`
flags
3. After the origin cluster and UIAM are fully ready, a second ES
cluster starts with:
- Its own project ID (`MOCK_IDP_UIAM_PROJECT_ID2`), cluster name, and
isolated object store
- The same Docker network and UIAM service (no duplicate UIAM
containers)
- Port offset `+10 `from the origin (HTTP on `9230`)
4. Once the linked cluster is healthy, the origin's operator
settings.json is updated to register the linked project endpoint,
enabling ES to discover it via /_project/tags
5. Scout exposes `linkedProject` fixture for test data ingestion into
the linked cluster

### Scope of changes

`@kbn/es (docker.ts)`
- Refactored `SERVERLESS_NODES` and `SHARED_SERVERLESS_PARAMS` into
factory functions (`getServerlessNodes`, `getSharedServerlessParams`) to
support dynamic container names and port offsets
- Parameterized `getOperatorVolume`, `setupServerlessVolumes`, and
`resolveEsArgs` to accept project ID overrides
- Added `runLinkedServerlessCluster()` — starts a 3-node linked ES
cluster and registers it with the origin's operator settings
- Updated container cleanup to also handle linked cluster containers

`@kbn/es (ServerlessOptions)`
- Added `linkedProject?: { projectId: string; port: number }` for
explicit linked cluster configuration

`@kbn/mock-idp-utils`
- Added `MOCK_IDP_UIAM_PROJECT_ID2` constant for the linked project

`@kbn/scout (config & schema)`
- Added `cps` boolean to `esServerlessOptions` schema and
`ScoutServerConfig` type
- Created `cps_local/serverless/security_complete.serverless.config.ts`
config set
- Fixed hardcoded `https://localhost:9200` in Fleet outputs to use
configured ES host/port

`@kbn/scout (runtime & fixtures)`
- `run_elasticsearch.ts` calls `runLinkedServerlessCluster` after the
origin cluster when CSP is enabled
- `ScoutTestConfig` extended with linkedProject (hosts + auth) for test
access
- Added `linkedEsClient`, `linkedEsArchiver` services and Playwright
worker fixture `linkedProject` to use in tests

### How to test:

```
node scripts/scout start-server --arch serverless --domain security_complete --serverConfigSet cps_local
```

Check that it starts:
- Origin ES cluster (3 nodes) on port `9220` with UIAM
- Linked ES cluster (3 nodes) on port `9230` (connected to the same
UIAM)
- Kibana on port `5620`

```
# Origin cluster
curl -sk https://elastic_serverless:changeme@localhost:9220

# Linked cluster
curl -sk https://elastic_serverless:changeme@localhost:9230
```

Go to Kibana http://localhost:5620/app/dev_tools#/console and check
projects being linked:

```
GET _project/tags
```

should return

```
{
  "origin": {
    "abcdef12345678901234567890123456": {
      "_alias": "local_project",
      "_id": "abcdef12345678901234567890123456",
      "_organization": "org1234567890",
      "_type": "security",
      "env": "local"
    }
  },
  "linked_projects": {
    "fedcba65432109876543210987654321": {
      "_alias": "linked_local_project",
      "_id": "fedcba65432109876543210987654321",
      "_organization": "org1234567890",
      "_type": "security",
      "env": "local"
    }
  }
}
```

---------

Co-authored-by: Cesare de Cal <cesare.decal@elastic.co>
dmlemeshko pushed a commit that referenced this pull request Apr 1, 2026
## Summary

A small fix for the `cps_local` config set added in
[#254791](#254791):

- Kibana wasn't getting `--cps.cpsEnabled=true` passed in, so CPS was
never actually enabled for Kibana even though ES was starting in CPS
mode.
- Kibana was originally generating native ES keys, and we need UIAM keys
when ES is running in a UIAM environment, so we need to pass the
`--xpack.alerting.rules.apiKeyType=uiam` flag.

I have also added a function to set the default space on server start
up. When rules run, if they're using the default space, ES requests use
the default NPRE (`kibana_space_default_default`). If it doesn't exist,
rules that should be able to search linked projects will behave like
local-only rules and can warn/fail when trying to access remote indices.
In higher serverless environments, the control plane creates and
maintains these NPREs automatically.

<details><summary>Before</summary>
<p>
<img width="1635" height="553" alt="Screenshot 2026-03-30 at 4 15 28 PM"
src="https://github.com/user-attachments/assets/9b35841a-ff84-4367-bd29-3682fba01de6"
/>
</p>
</details> 

<details><summary>After</summary>
<p>
<img width="1647" height="636" alt="Screenshot 2026-03-30 at 4 35 26 PM"
src="https://github.com/user-attachments/assets/84a2eeba-0051-4d66-b916-986fbf7d4b8b"
/>
</p>
</details>
jeramysoucy pushed a commit to jeramysoucy/kibana that referenced this pull request Apr 1, 2026
## Summary

A small fix for the `cps_local` config set added in
[elastic#254791](elastic#254791):

- Kibana wasn't getting `--cps.cpsEnabled=true` passed in, so CPS was
never actually enabled for Kibana even though ES was starting in CPS
mode.
- Kibana was originally generating native ES keys, and we need UIAM keys
when ES is running in a UIAM environment, so we need to pass the
`--xpack.alerting.rules.apiKeyType=uiam` flag.

I have also added a function to set the default space on server start
up. When rules run, if they're using the default space, ES requests use
the default NPRE (`kibana_space_default_default`). If it doesn't exist,
rules that should be able to search linked projects will behave like
local-only rules and can warn/fail when trying to access remote indices.
In higher serverless environments, the control plane creates and
maintains these NPREs automatically.

<details><summary>Before</summary>
<p>
<img width="1635" height="553" alt="Screenshot 2026-03-30 at 4 15 28 PM"
src="https://github.com/user-attachments/assets/9b35841a-ff84-4367-bd29-3682fba01de6"
/>
</p>
</details> 

<details><summary>After</summary>
<p>
<img width="1647" height="636" alt="Screenshot 2026-03-30 at 4 35 26 PM"
src="https://github.com/user-attachments/assets/84a2eeba-0051-4d66-b916-986fbf7d4b8b"
/>
</p>
</details>
eokoneyo pushed a commit to davismcphee/kibana that referenced this pull request Apr 2, 2026
## Summary

A small fix for the `cps_local` config set added in
[elastic#254791](elastic#254791):

- Kibana wasn't getting `--cps.cpsEnabled=true` passed in, so CPS was
never actually enabled for Kibana even though ES was starting in CPS
mode.
- Kibana was originally generating native ES keys, and we need UIAM keys
when ES is running in a UIAM environment, so we need to pass the
`--xpack.alerting.rules.apiKeyType=uiam` flag.

I have also added a function to set the default space on server start
up. When rules run, if they're using the default space, ES requests use
the default NPRE (`kibana_space_default_default`). If it doesn't exist,
rules that should be able to search linked projects will behave like
local-only rules and can warn/fail when trying to access remote indices.
In higher serverless environments, the control plane creates and
maintains these NPREs automatically.

<details><summary>Before</summary>
<p>
<img width="1635" height="553" alt="Screenshot 2026-03-30 at 4 15 28 PM"
src="https://github.com/user-attachments/assets/9b35841a-ff84-4367-bd29-3682fba01de6"
/>
</p>
</details> 

<details><summary>After</summary>
<p>
<img width="1647" height="636" alt="Screenshot 2026-03-30 at 4 35 26 PM"
src="https://github.com/user-attachments/assets/84a2eeba-0051-4d66-b916-986fbf7d4b8b"
/>
</p>
</details>
paulinashakirova pushed a commit to paulinashakirova/kibana that referenced this pull request Apr 2, 2026
## Summary

A small fix for the `cps_local` config set added in
[elastic#254791](elastic#254791):

- Kibana wasn't getting `--cps.cpsEnabled=true` passed in, so CPS was
never actually enabled for Kibana even though ES was starting in CPS
mode.
- Kibana was originally generating native ES keys, and we need UIAM keys
when ES is running in a UIAM environment, so we need to pass the
`--xpack.alerting.rules.apiKeyType=uiam` flag.

I have also added a function to set the default space on server start
up. When rules run, if they're using the default space, ES requests use
the default NPRE (`kibana_space_default_default`). If it doesn't exist,
rules that should be able to search linked projects will behave like
local-only rules and can warn/fail when trying to access remote indices.
In higher serverless environments, the control plane creates and
maintains these NPREs automatically.

<details><summary>Before</summary>
<p>
<img width="1635" height="553" alt="Screenshot 2026-03-30 at 4 15 28 PM"
src="https://github.com/user-attachments/assets/9b35841a-ff84-4367-bd29-3682fba01de6"
/>
</p>
</details> 

<details><summary>After</summary>
<p>
<img width="1647" height="636" alt="Screenshot 2026-03-30 at 4 35 26 PM"
src="https://github.com/user-attachments/assets/84a2eeba-0051-4d66-b916-986fbf7d4b8b"
/>
</p>
</details>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting release_note:skip Skip the PR/issue when compiling release notes v9.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants