-
Notifications
You must be signed in to change notification settings - Fork 8.6k
[Fleet] Allow to override data stream type in policies for input packages #214216
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
50 commits
Select commit
Hold shift + click to select a range
fa8a050
Add UI components to select ds type
jsoriano 0f31982
Move UI element to stream editor
jsoriano 7a90ba3
Fix update of policy input stream
jsoriano f6376f8
Remove unused variable
jsoriano 0b921b9
Fix comparison
jsoriano 866630a
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine 1489f2f
Merge remote-tracking branch 'origin/main' into input-package-ds-type
jsoriano 10123cc
Merge remote-tracking branch 'origin/main' into input-package-ds-type
jsoriano a279a34
Use data stream type instead of input type for stream ids
jsoriano c580901
Merge remote-tracking branch 'jsoriano/input-package-ds-type' into in…
jsoriano 2e89a91
Take data stream user config into account when creating it
jsoriano 88772ae
Set the data stream type where the dataset is also configured
jsoriano ff1b760
Revert possibly unneeded change
jsoriano ebf07cc
Merge remote-tracking branch 'origin/main' into input-package-ds-type
jsoriano 698afc9
Fix data stream override in streams when generating agent inputs from…
jsoriano d07739a
Add unit tests
jsoriano 80a3139
Merge remote-tracking branch 'origin/main' into input-package-ds-type
jsoriano 2661c95
Persist data stream type in variable
jsoriano c040ff0
Linting
jsoriano fde3239
Ensure data stream type cannot be edited
jsoriano 1e7d998
Add cypress test case
jsoriano 97c84dc
Merge remote-tracking branch 'origin/main' into input-package-ds-type
jsoriano ca960b9
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine 68e1a24
Add required field
jsoriano 643d2af
Merge remote-tracking branch 'jsoriano/input-package-ds-type' into in…
jsoriano 1c04b43
Try looking for disabled inputs
jsoriano f0b728a
Add more tests
jsoriano 2434d12
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine 310d593
Fix tests
jsoriano 6323ba4
Merge remote-tracking branch 'jsoriano/input-package-ds-type' into in…
jsoriano f561a81
[CI] Auto-commit changed files from 'node scripts/notice'
kibanamachine e0a1526
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine adb28d1
[CI] Auto-commit changed files from 'node scripts/notice'
kibanamachine c1c9ef5
Try to fix more tests
jsoriano 05a5b4e
Merge remote-tracking branch 'jsoriano/input-package-ds-type' into in…
jsoriano 788c432
Try to fix more errors
jsoriano 44680ad
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine 7213359
Merge branch 'main' into input-package-ds-type
elasticmachine be8d738
Merge branch 'main' into input-package-ds-type
elasticmachine 65ae4d3
Merge remote-tracking branch 'origin/main' into input-package-ds-type
jsoriano 7eb2694
Avoid changing the order of stream subfields
jsoriano 124fbc5
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine d520cbf
Linting
jsoriano 390fe14
Merge remote-tracking branch 'jsoriano/input-package-ds-type' into in…
jsoriano fe56b3b
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine 7695e9a
Fix test file
jsoriano 916acd2
Merge remote-tracking branch 'jsoriano/input-package-ds-type' into in…
jsoriano fb3493a
Try again
jsoriano 30d39f6
Merge remote-tracking branch 'origin/main' into input-package-ds-type
jsoriano b4f0a5b
One more space
jsoriano File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
123 changes: 123 additions & 0 deletions
123
x-pack/platform/plugins/shared/fleet/cypress/e2e/input_packages_real.cy.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| import { | ||
| ADD_INTEGRATION_POLICY_BTN, | ||
| CREATE_PACKAGE_POLICY_SAVE_BTN, | ||
| INTEGRATION_NAME_LINK, | ||
| POLICY_EDITOR, | ||
| } from '../screens/integrations'; | ||
| import { EXISTING_HOSTS_TAB } from '../screens/fleet'; | ||
| import { CONFIRM_MODAL } from '../screens/navigation'; | ||
|
|
||
| import { API_VERSIONS } from '../../common'; | ||
| import { cleanupAgentPolicies } from '../tasks/cleanup'; | ||
| import { login } from '../tasks/login'; | ||
| import { request } from '../tasks/common'; | ||
|
|
||
| const INPUT_TEST_PACKAGE = 'input_package-1.0.0'; | ||
|
|
||
| describe('Input package with custom data stream type', () => { | ||
| beforeEach(() => { | ||
| login(); | ||
| }); | ||
|
|
||
| const agentPolicyId = 'test-input-package-policy'; | ||
| const agentPolicyName = 'Test input package policy'; | ||
| const packagePolicyName = 'input-package-policy'; | ||
| const datasetName = 'logs'; // Default from the package. | ||
| const dataStreamType = 'metrics'; | ||
|
|
||
| before(() => { | ||
| cy.task('installTestPackage', INPUT_TEST_PACKAGE); | ||
|
|
||
| request({ | ||
| method: 'POST', | ||
| url: `/api/fleet/agent_policies`, | ||
| body: { | ||
| id: agentPolicyId, | ||
| name: agentPolicyName, | ||
| description: 'desc', | ||
| namespace: 'default', | ||
| monitoring_enabled: [], | ||
| }, | ||
| headers: { 'kbn-xsrf': 'cypress', 'Elastic-Api-Version': `${API_VERSIONS.public.v1}` }, | ||
| }); | ||
| }); | ||
|
|
||
| after(() => { | ||
| cleanupAgentPolicies(); | ||
| cy.task('uninstallTestPackage', INPUT_TEST_PACKAGE); | ||
| }); | ||
|
|
||
| it('should successfully create a package policy', () => { | ||
| cy.visit(`/app/integrations/detail/${INPUT_TEST_PACKAGE}/overview`); | ||
| cy.getBySel(ADD_INTEGRATION_POLICY_BTN).click(); | ||
|
|
||
| cy.getBySel(POLICY_EDITOR.POLICY_NAME_INPUT).click().clear().type(packagePolicyName); | ||
| cy.getBySel('multiTextInput-paths') | ||
| .find('[data-test-subj="multiTextInputRow-0"]') | ||
| .click() | ||
| .type('/var/log/test.log'); | ||
|
|
||
| cy.getBySel('multiTextInput-tags') | ||
| .find('[data-test-subj="multiTextInputRow-0"]') | ||
| .click() | ||
| .type('tag1'); | ||
|
|
||
| // Select metrics data stream type. | ||
| cy.get('[data-test-subj^="advancedStreamOptionsToggle"]').click(); | ||
| cy.get('[data-test-subj="packagePolicyDataStreamType"') | ||
| .find(`label[for="${dataStreamType}"]`) | ||
| .click(); | ||
|
|
||
| cy.getBySel(EXISTING_HOSTS_TAB).click(); | ||
|
|
||
| cy.getBySel(POLICY_EDITOR.AGENT_POLICY_SELECT).click(); | ||
| cy.getBySel('agentPolicyMultiItem').each(($el) => { | ||
| if ($el.text() === agentPolicyName) { | ||
| $el.trigger('click'); | ||
| } | ||
| }); | ||
| cy.wait(1000); // wait for policy id to be set | ||
| cy.getBySel(CREATE_PACKAGE_POLICY_SAVE_BTN).click(); | ||
|
|
||
| cy.getBySel(CONFIRM_MODAL.CANCEL_BUTTON).click(); | ||
| }); | ||
|
|
||
| it(`${dataStreamType} checkbox should be checked`, () => { | ||
| cy.visit(`/app/integrations/detail/${INPUT_TEST_PACKAGE}/policies`); | ||
|
|
||
| cy.getBySel(INTEGRATION_NAME_LINK).contains(packagePolicyName).click(); | ||
|
|
||
| cy.get('button').contains('Change defaults').click(); | ||
| cy.get('[data-test-subj^="advancedStreamOptionsToggle"]').click(); | ||
| cy.get('[data-test-subj="packagePolicyDataStreamType"') | ||
| .find(`input#${dataStreamType}`) | ||
| .should('be.checked'); | ||
| }); | ||
|
|
||
| it('should not allow to edit data stream type', () => { | ||
| cy.visit(`/app/integrations/detail/${INPUT_TEST_PACKAGE}/policies`); | ||
|
|
||
| cy.getBySel(INTEGRATION_NAME_LINK).contains(packagePolicyName).click(); | ||
|
|
||
| cy.get('button').contains('Change defaults').click(); | ||
| cy.get('[data-test-subj^="advancedStreamOptionsToggle"]').click(); | ||
| cy.get('[data-test-subj="packagePolicyDataStreamType"') | ||
| .find('input') | ||
| .should('have.length', 3) | ||
| .each(($el) => cy.wrap($el).should('be.disabled')); | ||
| }); | ||
|
|
||
| it('has an index template', () => { | ||
| cy.visit(`app/management/data/index_management/templates/${dataStreamType}-${datasetName}`); | ||
|
|
||
| // Check that the index pattern appears in the view. | ||
| cy.get('[data-test-subj="templateDetails"').contains(`${dataStreamType}-${datasetName}-*`); | ||
| }); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -10,10 +10,14 @@ import ReactMarkdown from 'react-markdown'; | |
| import styled from 'styled-components'; | ||
| import { uniq } from 'lodash'; | ||
| import { FormattedMessage } from '@kbn/i18n-react'; | ||
| import { i18n } from '@kbn/i18n'; | ||
| import { | ||
| EuiFlexGrid, | ||
| EuiFlexGroup, | ||
| EuiFlexItem, | ||
| EuiFormRow, | ||
| EuiLink, | ||
| EuiRadioGroup, | ||
| EuiSwitch, | ||
| EuiText, | ||
| EuiSpacer, | ||
|
|
@@ -25,9 +29,12 @@ import { useRouteMatch } from 'react-router-dom'; | |
|
|
||
| import { useQuery } from '@tanstack/react-query'; | ||
|
|
||
| import { DATASET_VAR_NAME } from '../../../../../../../../../common/constants'; | ||
| import { | ||
| DATASET_VAR_NAME, | ||
| DATA_STREAM_TYPE_VAR_NAME, | ||
| } from '../../../../../../../../../common/constants'; | ||
|
|
||
| import { useConfig, sendGetDataStreams } from '../../../../../../../../hooks'; | ||
| import { useConfig, sendGetDataStreams, useStartServices } from '../../../../../../../../hooks'; | ||
|
|
||
| import { | ||
| getRegistryDataStreamAssetBaseName, | ||
|
|
@@ -79,6 +86,8 @@ export const PackagePolicyInputStreamConfig = memo<Props>( | |
| isEditPage, | ||
| totalStreams, | ||
| }) => { | ||
| const { docLinks } = useStartServices(); | ||
|
|
||
| const config = useConfig(); | ||
| const isExperimentalDataStreamSettingsEnabled = | ||
| config.enableExperimental?.includes('experimentalDataStreamSettings') ?? false; | ||
|
|
@@ -98,6 +107,10 @@ export const PackagePolicyInputStreamConfig = memo<Props>( | |
| const customDatasetVar = packagePolicyInputStream.vars?.[DATASET_VAR_NAME]; | ||
| const customDatasetVarValue = customDatasetVar?.value?.dataset || customDatasetVar?.value; | ||
|
|
||
| const customDataStreamTypeVar = packagePolicyInputStream.vars?.[DATA_STREAM_TYPE_VAR_NAME]; | ||
| const customDataStreamTypeVarValue = | ||
| customDataStreamTypeVar?.value || packagePolicyInputStream.data_stream.type || 'logs'; | ||
|
|
||
| const { exists: indexTemplateExists, isLoading: isLoadingIndexTemplate } = | ||
| useIndexTemplateExists( | ||
| getRegistryDataStreamAssetBaseName({ | ||
|
|
@@ -295,7 +308,7 @@ export const PackagePolicyInputStreamConfig = memo<Props>( | |
| })} | ||
|
|
||
| {/* Advanced section */} | ||
| {hasAdvancedOptions && ( | ||
| {(hasAdvancedOptions || packageInfo.type === 'input') && ( | ||
| <Fragment> | ||
| <EuiFlexItem> | ||
| <EuiFlexGroup justifyContent="spaceBetween" alignItems="center"> | ||
|
|
@@ -328,6 +341,75 @@ export const PackagePolicyInputStreamConfig = memo<Props>( | |
| </EuiFlexItem> | ||
| {isShowingAdvanced ? ( | ||
| <> | ||
| {packageInfo.type === 'input' && ( | ||
| <EuiFlexItem> | ||
| <EuiFormRow | ||
| label={ | ||
| <FormattedMessage | ||
| id="xpack.fleet.createPackagePolicy.stepConfigure.packagePolicyDataStreamTypeInputLabel" | ||
| defaultMessage="Data Stream Type" | ||
| /> | ||
| } | ||
| helpText={ | ||
| isEditPage ? ( | ||
| <FormattedMessage | ||
| id="xpack.fleet.createPackagePolicy.stepConfigure.packagePolicyInputOnlyEditDataStreamTypeHelpLabel" | ||
| defaultMessage="The data stream type cannot be changed for this integration. Create a new integration policy to use a different input type." | ||
| /> | ||
| ) : ( | ||
| <FormattedMessage | ||
| id="xpack.fleet.createPackagePolicy.stepConfigure.packagePolicyDataStreamTypeHelpLabel" | ||
| defaultMessage="Select a data stream type for this policy. This setting changes the name of the integration's data stream. {learnMore}." | ||
| values={{ | ||
| learnMore: ( | ||
| <EuiLink | ||
| href={docLinks.links.fleet.datastreamsNamingScheme} | ||
| target="_blank" | ||
| > | ||
| {i18n.translate( | ||
| 'xpack.fleet.createPackagePolicy.stepConfigure.packagePolicyNamespaceHelpLearnMoreLabel', | ||
| { defaultMessage: 'Learn more' } | ||
| )} | ||
| </EuiLink> | ||
| ), | ||
| }} | ||
| /> | ||
| ) | ||
| } | ||
| > | ||
| <EuiRadioGroup | ||
| data-test-subj="packagePolicyDataStreamType" | ||
| disabled={isEditPage} | ||
| idSelected={customDataStreamTypeVarValue} | ||
| options={[ | ||
| { | ||
| id: 'logs', | ||
| label: 'Logs', | ||
| }, | ||
| { | ||
| id: 'metrics', | ||
| label: 'Metrics', | ||
| }, | ||
| { | ||
| id: 'traces', | ||
| label: 'Traces', | ||
| }, | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is it ok to allow all types for any input package? In principle I would say so, it is up to the user to know what kind of data they are collecting. If we think that some packages should be limited to only certain types, we would need to add something in the spec. |
||
| ]} | ||
| onChange={(type: string) => { | ||
| updatePackagePolicyInputStream({ | ||
| vars: { | ||
| ...packagePolicyInputStream.vars, | ||
| [DATA_STREAM_TYPE_VAR_NAME]: { | ||
| type: 'string', | ||
| value: type, | ||
| }, | ||
| }, | ||
| }); | ||
| }} | ||
| /> | ||
| </EuiFormRow> | ||
| </EuiFlexItem> | ||
| )} | ||
| {advancedVars.map((varDef) => { | ||
| if (!packagePolicyInputStream.vars) return null; | ||
| const { name: varName, type: varType } = varDef; | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be probably nice to add some kind of E2E or integration test to check that this radio button works and that the expected index template is created after installing the package and adding a policy.
@juliaElastic @nchaulet wdyt? do you have recommendations about where to start adding such a test?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We probably could have a cypress test here for that https://github.com/nchaulet/kibana/blob/9ec5275d2f6a8ca0ac3a5a37059d5cb22326fa20/x-pack/platform/plugins/shared/fleet/cypress/e2e/package_policy.cy.ts#L161