[Fleet] Allow to override data stream type in policies for input packages by jsoriano · Pull Request #214216 · elastic/kibana

jsoriano · 2025-03-12T16:10:22Z

Summary

Allow to select the data stream type when creating policies for input packages. This will allow to benefit from presets related to specific signal types.

It adds the following selector as an advanced option:

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
~~If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list~~
~~This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.~~
Flaky Test Runner was used on any tests changed
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

How to test

Install an input package (input packages are the ones that have type: input in their manifests, such as sql or log packages).
Create a new policy for this input package.
When creating a new policy, open the advanced settings, and select a different data stream type from the default.
Check that an index policy has been created for this package, with the prefix of the selected data stream type.

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

See some risk examples
...

…-fix'

…put-package-ds-type

jsoriano · 2025-04-24T11:52:30Z

Almost everything is working here: the UI element works, its state is persisted in the policy, and the index templates are created as expected. But the type is not set in the agent policy, and agents using the policy don't ingest anything.

For example in this example data_stream should contain type: metrics IIUC:

    streams:
      - id: metrics-log.logs-d43daf00-344f-4378-9a45-ec84501addf9
        data_stream:
          dataset: test
        paths:
          - /tmp/foo*
        ignore_older: 72h

… package policies

jsoriano · 2025-04-24T17:44:10Z

...olicy/create_package_policy_page/components/steps/components/package_policy_input_stream.tsx

+                                {
+                                  id: 'traces',
+                                  label: 'Traces',
+                                },


Is it ok to allow all types for any input package?

In principle I would say so, it is up to the user to know what kind of data they are collecting.

If we think that some packages should be limited to only certain types, we would need to add something in the spec.

jsoriano · 2025-04-24T17:47:22Z

...olicy/create_package_policy_page/components/steps/components/package_policy_input_stream.tsx

+                            <EuiRadioGroup
+                              data-test-subj="packagePolicyDataStreamType"
+                              disabled={isEditPage}
+                              idSelected={packagePolicyInputStream.data_stream.type || 'logs'}


The Package Spec requires a type now in policy templates in input packages.

Should we remove this requirement, now that the type can be configured?

As it is now here, this code will support streams without defined type, and will use logs as default, but older versions of kibana may not work.

Yes we can probably remove this if we bump the max spec version available in kibana

elasticmachine · 2025-04-24T17:47:46Z

Pinging @elastic/fleet (Team:Fleet)

jsoriano · 2025-04-24T17:55:17Z

...olicy/create_package_policy_page/components/steps/components/package_policy_input_stream.tsx

+                              )
+                            }
+                          >
+                            <EuiRadioGroup


It would be probably nice to add some kind of E2E or integration test to check that this radio button works and that the expected index template is created after installing the package and adding a policy.

@juliaElastic @nchaulet wdyt? do you have recommendations about where to start adding such a test?

We probably could have a cypress test here for that https://github.com/nchaulet/kibana/blob/9ec5275d2f6a8ca0ac3a5a37059d5cb22326fa20/x-pack/platform/plugins/shared/fleet/cypress/e2e/package_policy.cy.ts#L161

…put-package-ds-type

…-fix'

…put-package-ds-type

elasticmachine · 2025-05-26T16:40:00Z

💚 Build Succeeded

Buildkite Build
Commit: b4f0a5b

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	1.7MB	1.7MB	+1.5KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	164.5KB	164.6KB	+49.0B

History

💔 Build #302276 failed 30d39f6
💔 Build #302223 failed 916acd2
💔 Build #302087 failed fe56b3b
💔 Build #301815 failed 124fbc5
💛 Build #300409 was flaky 7213359

cc @jsoriano

…ages (elastic#214216) Allow to select the data stream type when creating policies for input packages. It is only possible to select the data stream type when creating policies and not when editing them. The selector is added as an advanced option.

kibanamachine · 2025-06-09T19:42:50Z

Starting backport for target branches: 8.17, 8.18, 8.19

https://github.com/elastic/kibana/actions/runs/15543134795

…ages (elastic#214216) Allow to select the data stream type when creating policies for input packages. It is only possible to select the data stream type when creating policies and not when editing them. The selector is added as an advanced option. (cherry picked from commit 4337d10)

kibanamachine · 2025-06-09T19:49:56Z

💔 Some backports could not be created

Status	Branch	Result
❌	8.17	Backport failed because of merge conflicts
✅	8.18
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 214216

Questions ?

Please refer to the Backport tool documentation

…ut packages (#214216) (#223151) # Backport This will backport the following commits from `main` to `8.18`: - [[Fleet] Allow to override data stream type in policies for input packages (#214216)](#214216)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

…ut packages (#214216) (#223152) # Backport This will backport the following commits from `main` to `8.19`: - [[Fleet] Allow to override data stream type in policies for input packages (#214216)](#214216)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

… for input packages (#214216) (#223151)" (#223247) This change was only intended to be backported to 8.19, backported by mistake to 8.18 in #223151.

jsoriano added 3 commits March 11, 2025 23:04

Add UI components to select ds type

fa8a050

Move UI element to stream editor

0f31982

Fix update of policy input stream

7a90ba3

jsoriano self-assigned this Mar 12, 2025

jsoriano added backport:skip This PR does not require backporting Team:Fleet Team label for Observability Data Collection Fleet team labels Mar 12, 2025

jsoriano and others added 3 commits March 12, 2025 17:16

Remove unused variable

f6376f8

Fix comparison

0b921b9

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

866630a

…-fix'

kpollich mentioned this pull request Mar 31, 2025

[Change Proposal] Support flexible signal type in input packages elastic/package-spec#849

Closed

5 tasks

jsoriano added 8 commits April 2, 2025 13:29

Merge remote-tracking branch 'origin/main' into input-package-ds-type

1489f2f

Merge remote-tracking branch 'origin/main' into input-package-ds-type

10123cc

Use data stream type instead of input type for stream ids

a279a34

Merge remote-tracking branch 'jsoriano/input-package-ds-type' into in…

c580901

…put-package-ds-type

Take data stream user config into account when creating it

2e89a91

Set the data stream type where the dataset is also configured

88772ae

Revert possibly unneeded change

ff1b760

Merge remote-tracking branch 'origin/main' into input-package-ds-type

ebf07cc

Fix data stream override in streams when generating agent inputs from…

698afc9

… package policies

jsoriano force-pushed the input-package-ds-type branch from 486a493 to 698afc9 Compare April 24, 2025 14:58

jsoriano added 2 commits April 24, 2025 19:36

Add unit tests

d07739a

Merge remote-tracking branch 'origin/main' into input-package-ds-type

80a3139

jsoriano added the release_note:feature Makes this part of the condensed release notes label Apr 24, 2025

jsoriano commented Apr 24, 2025

View reviewed changes

jsoriano marked this pull request as ready for review April 24, 2025 17:47

jsoriano requested a review from a team as a code owner April 24, 2025 17:47

jsoriano commented Apr 24, 2025

View reviewed changes

jsoriano and others added 8 commits May 23, 2025 16:43

Linting

d520cbf

Merge remote-tracking branch 'jsoriano/input-package-ds-type' into in…

390fe14

…put-package-ds-type

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

fe56b3b

…-fix'

Fix test file

7695e9a

Merge remote-tracking branch 'jsoriano/input-package-ds-type' into in…

916acd2

…put-package-ds-type

Try again

fb3493a

Merge remote-tracking branch 'origin/main' into input-package-ds-type

30d39f6

One more space

b4f0a5b

jsoriano merged commit 4337d10 into elastic:main May 26, 2025
11 checks passed

jsoriano deleted the input-package-ds-type branch May 26, 2025 16:44

kibanamachine added the v9.1.0 label May 26, 2025

criamico mentioned this pull request May 27, 2025

[Fleet] When removing a inputs type package policy, clean up assets #218582

Merged

3 tasks

jsoriano added backport:prev-major and removed backport:skip This PR does not require backporting labels Jun 9, 2025

kibanamachine mentioned this pull request Jun 9, 2025

[8.18] [Fleet] Allow to override data stream type in policies for input packages (#214216) #223151

Merged

kibanamachine mentioned this pull request Jun 9, 2025

[8.19] [Fleet] Allow to override data stream type in policies for input packages (#214216) #223152

Merged

kibanamachine added the v8.18.3 label Jun 9, 2025

kibanamachine added the v8.19.0 label Jun 9, 2025

jsoriano mentioned this pull request Jul 31, 2025

Update default stack version to 9.1.0 elastic/elastic-package#2792

Merged

jsoriano mentioned this pull request Sep 23, 2025

[Fleet] Custom data_stream type and dataset lost in some input packages #236015

Closed

jsoriano mentioned this pull request Mar 2, 2026

[Fleet] Override of data stream type for input packages is not setting the type in the policy #255486

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Allow to override data stream type in policies for input packages#214216

[Fleet] Allow to override data stream type in policies for input packages#214216
jsoriano merged 50 commits intoelastic:mainfrom
jsoriano:input-package-ds-type

jsoriano commented Mar 12, 2025 •

edited

Loading

Uh oh!

jsoriano commented Apr 24, 2025

Uh oh!

jsoriano Apr 24, 2025 •

edited

Loading

Uh oh!

jsoriano Apr 24, 2025 •

edited

Loading

Uh oh!

nchaulet Apr 24, 2025

Uh oh!

elasticmachine commented Apr 24, 2025

Uh oh!

jsoriano Apr 24, 2025

Uh oh!

nchaulet Apr 24, 2025

Uh oh!

elasticmachine commented May 26, 2025

Uh oh!

Uh oh!

kibanamachine commented Jun 9, 2025

Uh oh!

kibanamachine commented Jun 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

jsoriano commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

How to test

Identify risks

Uh oh!

jsoriano commented Apr 24, 2025

Uh oh!

jsoriano Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsoriano Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nchaulet Apr 24, 2025

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Apr 24, 2025

Uh oh!

jsoriano Apr 24, 2025

Choose a reason for hiding this comment

Uh oh!

nchaulet Apr 24, 2025

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented May 26, 2025

💚 Build Succeeded

Metrics [docs]

Async chunks

Page load bundle

History

Uh oh!

Uh oh!

kibanamachine commented Jun 9, 2025

Uh oh!

kibanamachine commented Jun 9, 2025

💔 Some backports could not be created

Manual backport

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jsoriano commented Mar 12, 2025 •

edited

Loading

jsoriano Apr 24, 2025 •

edited

Loading

jsoriano Apr 24, 2025 •

edited

Loading