-
Notifications
You must be signed in to change notification settings - Fork 215
CPS tech preview #5498
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
shainaraskas
wants to merge
68
commits into
main
Choose a base branch
from
cps-tech-preview
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
CPS tech preview #5498
Changes from all commits
Commits
Show all changes
68 commits
Select commit
Hold shift + click to select a range
c0ccf4e
CPS configuration docs
marciw d3f0044
Merge branch 'main' of github.com:elastic/docs-content into cps-tech-…
florent-leborgne 8997244
Merge branch 'main' into cps-tech-preview
shainaraskas 45ad524
Merge branch 'main' into cps-tech-preview
nastasha-solomon a08794f
Merge branch 'main' into cps-tech-preview
marciw 2ea5469
CPS config: update + reconcile with E&A CPS
marciw 3338a8d
CPS: Data views automatically list indices from linked projects (#5426)
florent-leborgne f71932c
CPS in Kibana: scope selector and availability matrix (#5423)
florent-leborgne 8bd157b
CPS: Inspector lists linked projects searched by the query (#5425)
florent-leborgne a8c643a
CPS: Add project routing section to ES|QL in Kibana and ES|QL visuali…
florent-leborgne 2a1539a
Merge branch 'main' into cps-tech-preview
florent-leborgne 7406e08
CPS: Dashboard-level scope and panel-level custom scope badge (#5428)
florent-leborgne b6a8786
Fix invalid cross-link for ES|QL SET command
florent-leborgne fda3b8f
Update redirected links to cps-config-link-and-manage.md
florent-leborgne accb294
CPS: Document cross-project search support in Maps (#5429)
florent-leborgne bdaad85
CPS: Document space-level CPS scope for alerting rules (#5427)
florent-leborgne afc7f7e
Fix broken anchors and redirected links in CPS pages
florent-leborgne 6215d6c
Merge branch 'main' into cps-tech-preview
nastasha-solomon b243911
Small tweaks to CPS manage scope page
florent-leborgne aa533ec
Improve manage scope description in CPS overview page
florent-leborgne de3d47b
Merge branch 'main' into cps-tech-preview
florent-leborgne c25ed7a
Adjustments from feedback
florent-leborgne 0964786
Apply suggestions from review
marciw 9073304
Apply suggestions from review
marciw 461081e
Apply suggestions from review
marciw d7a6adf
Apply suggestions from review
marciw 33e208b
Apply suggestions from review
marciw 1a1fd12
Apply suggestions from review
marciw ddb3437
Apply suggestions from review
marciw 81e2dfd
Apply suggestions from review incl fix snippets
marciw 1c699b5
Add notes about using Painless with cross-project search (#5608)
marciw 3d2a109
Port over changes from mw-cps-config
marciw 11b6de8
Address review comments
marciw e62eb41
CPS in Kibana, Observability, and Security: Feature support (#5596)
nastasha-solomon f8f8f91
Merge branch 'main' into cps-tech-preview
nastasha-solomon 1a12d53
updated ref paths in alerting docs
nastasha-solomon 5b9e3d5
re-adding cps-painless-scripting section
nastasha-solomon 571c4ab
avada kedavra!
nastasha-solomon 4309b20
small fixes
marciw 7e7c57c
Update cps-config-impacts-and-limitations.md
marciw 5fcb2b9
Address more review comments
marciw 459ee4a
Fold impacts + limitations into main page
marciw 3cb4c94
Fix see also
marciw 070cd7f
Assorted edits
marciw e929ed5
😵💫
marciw 1fd1d93
Move prog access section + fix inaccurate sentence
marciw fd10244
Update deploy-manage/cross-project-search-config.md
marciw 50ff316
Update deploy-manage/cross-project-search-config/cps-config-link-and-…
marciw f69f128
CPS-related edits to existing pages (#5560)
marciw 12f11d5
fix link
marciw 955e256
Merge branch 'main' into cps-tech-preview
marciw 12848fe
Address comments from review
marciw 09da820
missed some review comments
marciw 06f19ce
Merge branch 'main' into cps-tech-preview
marciw 0b8555a
adds clarification on alert workflow
nastasha-solomon a1aa080
Merge branch 'main' into cps-tech-preview
shainaraskas 5d94357
Add project tags to glossary
marciw 4284eca
CPS: Refactor auth-related stuff and small style edits (#5799)
shainaraskas f20114d
Merge branch 'main' into cps-tech-preview
shainaraskas 5ba0840
Update explore-analyze/cross-project-search/_snippets/cps-security.md
shainaraskas 0b24073
Update warning about custom role access requirements
shainaraskas 2780a08
Update deploy-manage/users-roles/serverless-custom-roles.md
shainaraskas c940503
Merge branch 'main' into cps-tech-preview
shainaraskas 4c2866e
Apply suggestion from review
marciw 13609b3
Apply suggestion from review
marciw d65dce5
Remove stale TODOs
marciw bb2a429
surface sec and obs limitations on core limitations list
shainaraskas 0a7c613
Merge branch 'main' into cps-tech-preview
shainaraskas File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| With {{cps}} ({{cps-init}}), users in your organization can search across multiple {{serverless-full}} projects at once, instead of searching each project individually. When your data is split across projects to organize ownership, use cases, or environments, {{cps}} lets you query all the data from a single place. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| - **Maximum of 20 linked projects:** Each origin project can have up to 20 linked projects. A linked project can be associated with any number of origin projects. | ||
| - **System indices:** Indices such as `.security` and `.fleet-*` are excluded from {{cps}} results by design. | ||
| - **New projects only:** During technical preview, only newly created projects can function as origin projects. | ||
| - **ML and transforms:** During technical preview, ML {{anomaly-jobs}} and transforms are not supported. They continue to run on origin project data only. | ||
| - For {{esql}} limitations specific to {{cps-init}}, refer to [ES|QL with {{cps}}](elasticsearch://reference/query-languages/esql/esql-cross-serverless-projects.md#limitations). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| {{cps-cap}} runs across _origin_ and _linked_ projects within your {{ecloud}} organization: | ||
|
|
||
| - **Origin project:** The base project where you create links and run cross-project searches. | ||
| - **Linked projects:** The projects you connect to the origin project. Data in the linked projects becomes searchable from the origin project. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,143 @@ | ||
| --- | ||
| applies_to: | ||
| stack: unavailable | ||
| serverless: preview | ||
| products: | ||
| - id: cloud-serverless | ||
| navigation_title: "Cross-project search" | ||
| --- | ||
|
|
||
| # Configure {{cps}} [configure-cross-project-search] | ||
|
|
||
| ::::{include} /deploy-manage/_snippets/cps-definition.md | ||
| :::: | ||
|
|
||
| {{cps-cap}} is the {{serverless-short}} equivalent of [{{ccs}}](/explore-analyze/cross-cluster-search.md), with a few differences and enhancements: | ||
|
|
||
| * Setting up {{cps}} doesn't require an understanding of your deployment architecture or complex security configurations. | ||
| * Permissions stay consistent across projects, and you can always adjust scope and access as needed. | ||
| * Searches are performed across projects by default, reducing the need to refactor your queries as you link additional projects. | ||
|
|
||
| This section explains how to set up and manage {{cps}} for your organization, including linking projects, managing user access, and refining scope. For information on using {{cps}}, including syntax and examples, refer to [](/explore-analyze/cross-project-search.md). | ||
|
|
||
| :::{note} | ||
| {{cps-cap}} is available for {{serverless-full}} projects only. For other deployment types, refer to [{{ccs}}](/explore-analyze/cross-cluster-search.md). | ||
| ::: | ||
|
|
||
| ## Key concepts | ||
|
|
||
| ::::{include} /deploy-manage/_snippets/cps-origin-linked-definitions.md | ||
| :::: | ||
|
|
||
| ### Projects and search scope | ||
|
|
||
| Projects are intended to act as logical namespaces for data, not hard boundaries for querying it. You can split data into projects to organize ownership, use cases, or environments, while still expecting to search and analyze that data from a single place. | ||
|
|
||
| ::::{include} /explore-analyze/cross-project-search/_snippets/cps-default-search-behavior.md | ||
shainaraskas marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| :::: | ||
|
|
||
| Administrators can also adjust the search scope by [configuring the {{cps-init}} scope for each space](/deploy-manage/cross-project-search-config/cps-config-access-and-scope.md#cps-default-search-scope). For best results, set this space-level default before you link projects. | ||
|
|
||
| For details about project IDs and aliases used in search expressions, refer to [Project IDs and aliases](/explore-analyze/cross-project-search.md#project-ids-and-aliases). | ||
|
|
||
| ## Before you begin [cps-prerequisites] | ||
|
|
||
| Before you configure {{cps}}, review these prerequisites and best practices: | ||
|
|
||
| - You must be an organization owner or project administrator: | ||
| - **Organization owners** can link any projects within the organization. | ||
| - **Project administrators** must have admin access on both the origin project and each linked project. | ||
| - Your origin and linked projects must meet certain [requirements](#cps-compatibility). | ||
| - Consider the [architecture patterns](#cps-arch) and choose the right linking topology for your organization. | ||
|
|
||
| ### Projects available for linking [cps-compatibility] | ||
|
|
||
| ::::{important} - Origin projects must be new | ||
| :applies_to: serverless: preview | ||
| During technical preview, only newly created projects can be origin projects for {{cps}}. Existing projects can be linked from an origin project, but they can't serve as origin projects themselves. To get started, create a new {{serverless-short}} project and link it to your existing projects. | ||
| :::: | ||
|
|
||
| To be available for linking, projects must meet the following requirements: | ||
|
|
||
| - The origin project and all linked projects must be in the same {{ecloud}} organization. | ||
| - You can link any combination of {{product.elasticsearch}}, {{product.observability}}, and {{product.security}} projects in the same organization. | ||
| - {{sec-serverless}} and {{obs-serverless}} projects require the **Complete** feature tier. Projects on the **Essentials** tier are not compatible with {{cps}}. | ||
|
|
||
| Only compatible projects appear in the [{{cps}} linking wizard](/deploy-manage/cross-project-search-config/cps-config-link-and-manage.md#cps-link-projects). If a project you expected to link to is missing from the list, it might not meet the requirements, or you might not have the necessary [permissions](#cps-compatibility) on the project. | ||
|
|
||
|
|
||
| ## Plan your {{cps-init}} architecture [cps-arch] | ||
shainaraskas marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| When configuring {{cps}}, consider how the {{cps-init}} architecture (or linking pattern) will affect searches, dashboards, and alerting across your organization. {{cps-cap}} supports three patterns, each with a different level of operational risk. | ||
|
|
||
| ### Recommended: Overview project [cps-arch-overview] | ||
|
|
||
| For most deployments, we recommend creating a dedicated **overview project** that can act as an origin project. You can also think of this as a hub-and-spoke model. | ||
|
|
||
| In this architecture, you create a new, empty project and link existing projects to it. You run all cross-project searches from the new overview project, while your actual active projects continue to operate independently. The linked ("spoke") projects are not linked to each other. | ||
|
|
||
|  | ||
|
|
||
| The overview project becomes a central point for broad searches, dashboards, and investigations, without affecting your existing setup. | ||
|
|
||
| ### Other supported patterns | ||
|
|
||
| The overview project model is strongly recommended and appropriate for most {{cps-init}} configurations. These additional patterns are valid, but they involve additional risk and require careful configuration: | ||
|
|
||
| - **Shared data project (N-to-1):** A single project stores data from a shared service (for example, logs). Multiple origin projects link to this central data project. | ||
|
|
||
| The N-to-1 pattern is often used when several teams need to query shared data independently. The main risk is that linking to a shared data project affects searches, dashboards, and alerts in each origin project. If the shared project is a large, active project, the expanded dataset might cause unexpected behavior. If you're using this pattern, make sure to [manage user access](/deploy-manage/cross-project-search-config/cps-config-access-and-scope.md#manage-user-and-api-key-access) and consider [CPS scope](/deploy-manage/cross-project-search-config/cps-config-access-and-scope.md#cps-search-scope). | ||
|
|
||
| - **Data mesh (N-to-N):** Multiple active projects link directly to each other. | ||
|
|
||
| The N-to-N pattern is the most complex and involves the highest risk. After you link projects, all searches, dashboards, and alerting rules in each origin project will query data from every linked project by default, which might make workflows unpredictable. Make sure you check alerting rules, which might be applied to data that the rule was never intended to evaluate. | ||
|
|
||
| ## Configure {{cps-init}} | ||
|
|
||
| After reviewing the architecture patterns, you can configure {{cps-init}} scope and manage linked projects. For best results, complete these tasks in order: | ||
|
|
||
| 1. [Set space scope defaults](/deploy-manage/cross-project-search-config/cps-config-access-and-scope.md#about-cps-init-scope): Configure the default {{cps}} scope for each space that will be used with {{cps}}. | ||
| 1. [Manage user access and programmatic access](/deploy-manage/cross-project-search-config/cps-config-access-and-scope.md): Confirm user roles in both the origin and linked projects, as well as roles granted to [{{ecloud}} API keys](/deploy-manage/api-keys/elastic-cloud-api-keys.md#roles) that will be used with {{cps}}. | ||
| 1. [Link and manage projects](/deploy-manage/cross-project-search-config/cps-config-link-and-manage.md): Link projects in the {{ecloud}} UI, manage linked projects, and unlink projects. | ||
|
|
||
| Make sure to also review the [feature impacts](#cps-feature-impacts) and [limitations](#cps-limitations) of {{cps-init}}. | ||
|
|
||
| ## Feature impacts [cps-feature-impacts] | ||
|
|
||
| When you link projects for {{cps}}, the expanded dataset can affect existing features in the origin project. | ||
|
|
||
| % TODO billing, subscriptions, licensing | ||
|
|
||
| - **Alerts:** By default, rules in the origin project run against the **combined dataset** of the origin and all linked projects. Rules tuned for a single project's data might produce false positives when they evaluate a larger dataset. This is one reason we recommend using a dedicated [overview project](/deploy-manage/cross-project-search-config.md#cps-arch-overview), so that existing rules on data projects are not affected. Make sure to also consider the [default {{cps}} scope](/deploy-manage/cross-project-search-config/cps-config-access-and-scope.md#cps-default-search-scope) for each space, or save explicit project routing on individual rules. | ||
|
|
||
| - **Dashboards and visualizations:** Existing dashboards and visualizations in the origin project will query all linked projects by default. To control this, set the [default {{cps}} scope](/deploy-manage/cross-project-search-config/cps-config-access-and-scope.md#cps-default-search-scope) for each space, or save explicit project routing on individual dashboard panels. | ||
|
|
||
| - **User permissions:** {{cps-cap}} results are filtered by each user's role assignments across projects. Users with different roles will see different results from the same query. Refer to [Manage user access](/deploy-manage/cross-project-search-config/cps-config-access-and-scope.md#manage-user-and-api-key-access). | ||
|
|
||
| - **{{product.painless}} scripting:** The [{{product.painless}} execute API](/explore-analyze/cross-project-search.md#cps-painless-scripting) does not search across linked projects. It resolves index names against the origin project only. You can target a linked project by prefixing the index with the project alias (for example, `projectAlias:myindex`). | ||
|
|
||
| ## Limitations [cps-limitations] | ||
|
|
||
| {{cps-cap}} has the following limitations: | ||
|
|
||
| ::::{include} /deploy-manage/_snippets/cps-limitations-core.md | ||
| :::: | ||
| * Additional limitations apply to Elastic {{observability}} and {{elastic-sec}} projects. | ||
|
|
||
| **{{elastic-sec}} apps** | ||
|
|
||
| :::{include} /explore-analyze/cross-project-search/_snippets/cps-availability-security-apps.md | ||
| ::: | ||
|
|
||
| **Elastic {{observability}} apps** | ||
|
|
||
| {{observability}} apps have limited {{cps-init}} support. The scope selector is not available in {{observability}} apps, and most apps remain scoped to the local project. | ||
|
|
||
| % For specific app details, refer to [{{cps-cap}} in {{observability}}](/solutions/observability/cross-project-search.md). | ||
|
|
||
| ## Using {{cps-init}} | ||
|
|
||
| After you configure {{cps}} and link projects, users can start searching across linked projects from the origin project. For search syntax, scope controls, and examples, refer to the following pages: | ||
|
|
||
| - [{{cps-cap}} overview](/explore-analyze/cross-project-search.md): Learn how to build queries in a {{cps-init}} context, including how to restrict search scope. | ||
| - [](/explore-analyze/cross-project-search/cross-project-search-manage-scope.md): Learn how {{cps-init}} works with compatible {{kib}} apps, including how to adjust search scope. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.