Skip to content

[Security][Fleet] Install the security_detection_engine package automatically #97191

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rw-access merged 21 commits into from
Apr 20, 2021
Merged

[Security][Fleet] Install the security_detection_engine package automatically #97191

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
124bdc5
Automatically install the security_detection_engine package via fleet
rw-access Apr 14, 2021
35ff135
Merge branch 'master' into autoinstall-security-detection-pkg
kibanamachine Apr 15, 2021
43b52a3
Update dockerImage to include the security_detection_engine package
rw-access Apr 15, 2021
5610c81
Merge branch 'autoinstall-security-detection-pkg' of github.com:rw-ac…
rw-access Apr 15, 2021
c2501a9
Merge branch 'master' into autoinstall-security-detection-pkg
kibanamachine Apr 15, 2021
51147f8
Update api/fleet/setup install test
rw-access Apr 15, 2021
9c48e96
Merge branch 'autoinstall-security-detection-pkg' of github.com:rw-ac…
rw-access Apr 15, 2021
7e21c86
Merge branch 'master' into autoinstall-security-detection-pkg
kibanamachine Apr 15, 2021
340ab0a
Merge remote-tracking branch 'origin/master' into autoinstall-securit…
rw-access Apr 15, 2021
820ef5c
Merge branch 'master' into autoinstall-security-detection-pkg
kibanamachine Apr 16, 2021
287828e
Merge branch 'master' into autoinstall-security-detection-pkg
kibanamachine Apr 16, 2021
6501dad
Merge branch 'master' into autoinstall-security-detection-pkg
kibanamachine Apr 19, 2021
39264b8
Update test data for Endpoint package
rw-access Apr 19, 2021
5d19234
Fix JSON token
rw-access Apr 20, 2021
2dd2fc6
Updae firist json enry in destination_index
rw-access Apr 20, 2021
5515757
Update destination_index structure
rw-access Apr 20, 2021
d95a0ef
Update destination_index structure
rw-access Apr 20, 2021
61057aa
Merge branch 'autoinstall-security-detection-pkg' of github.com:rw-ac…
rw-access Apr 20, 2021
acaa76a
Change KQL query to unblock testing
rw-access Apr 20, 2021
4a06e1f
Restore KQL and fix JSON instead
rw-access Apr 20, 2021
d425a8a
update timestamps to pass tests
kevinlog Apr 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions x-pack/plugins/fleet/common/constants/epm.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ export const requiredPackages = {
System: 'system',
Endpoint: 'endpoint',
ElasticAgent: 'elastic_agent',
SecurityDetectionEngine: 'security_detection_engine',
Copy link
Copy Markdown
Member

@spong spong Apr 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if this dependency is removed in the future -- is the package just gracefully uninstalled and SO's removed on upgrade?

Copy link
Copy Markdown
Contributor Author

@rw-access rw-access Apr 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah you got it.
fleet manages all the SOs, so they will be removed during an upgrade/uninstall gracefully. the detection engine will then just use the rules compiled in

Copy link
Copy Markdown
Contributor

@jen-huang jen-huang Apr 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe I misunderstood the question but if security_detection_engine is removed from this list of required packages in the future, Fleet will not automatically uninstall the package if it's already installed

Copy link
Copy Markdown
Member

@spong spong Apr 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahhh, so we can have lingering SO's from this package then? This would result in the rules installing from these SO's still, or would the fact the package is no-longer present short circuit that logic?

How can they be manually removed -- will it still show up in fleet until it's un-installed, or is there another way to perform this cleanup?

} as const;

// these are currently identical. we can separate if they later diverge
Expand Down
8 changes: 7 additions & 1 deletion x-pack/test/fleet_api_integration/apis/fleet_setup.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,13 @@ export default function (providerContext: FtrProviderContext) {
.map((p: any) => p.name)
.sort();

expect(installedPackages).to.eql(['elastic_agent', 'endpoint', 'fleet_server', 'system']);
expect(installedPackages).to.eql([
'elastic_agent',
'endpoint',
'fleet_server',
'security_detection_engine',
'system',
]);
});
});
}
2 changes: 1 addition & 1 deletion x-pack/test/fleet_api_integration/config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import { defineDockerServersConfig } from '@kbn/test';
// example: https://beats-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fpackage-storage/detail/snapshot/74/pipeline/257#step-302-log-1.
// It should be updated any time there is a new Docker image published for the Snapshot Distribution of the Package Registry.
export const dockerImage =
'docker.elastic.co/package-registry/distribution:c5925eb82898dfc3e879a521871c7383513804c7';
'docker.elastic.co/package-registry/distribution:b6a53ac9300333a4a45f3f7d350c9aed72061a66';

export default async function ({ readConfigFile }: FtrConfigProviderContext) {
const xPackAPITestsConfig = await readConfigFile(require.resolve('../api_integration/config.ts'));
Expand Down
36 changes: 18 additions & 18 deletions x-pack/test/functional/es_archives/endpoint/metadata/api_feature/data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"id": "3KVN2G8BYQH1gtPUuYk7",
"index": "metrics-endpoint.metadata-default",
"source": {
"@timestamp": 1579881969541,
"@timestamp": 1618841405309,
"agent": {
"id": "963b081e-60d1-482c-befd-a5815fa8290f",
"version": "6.6.1",
Expand All @@ -26,7 +26,7 @@
}
},
"event": {
"created": 1579881969541,
"created": 1618841405309,
"id": "32f5fda2-48e4-4fae-b89e-a18038294d14",
"kind": "metric",
"category": [
Expand Down Expand Up @@ -74,7 +74,7 @@
"id": "3aVN2G8BYQH1gtPUuYk7",
"index": "metrics-endpoint.metadata-default",
"source": {
"@timestamp": 1579881969541,
"@timestamp": 1618841405309,
"agent": {
"id": "b3412d6f-b022-4448-8fee-21cc936ea86b",
"version": "6.0.0",
Expand All @@ -96,7 +96,7 @@
}
},
"event": {
"created": 1579881969541,
"created": 1618841405309,
"id": "32f5fda2-48e4-4fae-b89e-a18038294d15",
"kind": "metric",
"category": [
Expand Down Expand Up @@ -143,7 +143,7 @@
"id": "3qVN2G8BYQH1gtPUuYk7",
"index": "metrics-endpoint.metadata-default",
"source": {
"@timestamp": 1579881969541,
"@timestamp": 1618841405309,
"agent": {
"id": "3838df35-a095-4af4-8fce-0b6d78793f2e",
"version": "6.8.0",
Expand All @@ -165,7 +165,7 @@
}
},
"event": {
"created": 1579881969541,
"created": 1618841405309,
"id": "32f5fda2-48e4-4fae-b89e-a18038294d16",
"kind": "metric",
"category": [
Expand Down Expand Up @@ -210,7 +210,7 @@
"id": "36VN2G8BYQH1gtPUuYk7",
"index": "metrics-endpoint.metadata-default",
"source": {
"@timestamp": 1579878369541,
"@timestamp": 1618841405309,
"agent": {
"id": "963b081e-60d1-482c-befd-a5815fa8290f",
"version": "6.6.1",
Expand All @@ -232,7 +232,7 @@
}
},
"event": {
"created": 1579878369541,
"created": 1618841405309,
"id": "32f5fda2-48e4-4fae-b89e-a18038294d18",
"kind": "metric",
"category": [
Expand Down Expand Up @@ -280,7 +280,7 @@
"id": "4KVN2G8BYQH1gtPUuYk7",
"index": "metrics-endpoint.metadata-default",
"source": {
"@timestamp": 1579878369541,
"@timestamp": 1618841405309,
"agent": {
"id": "b3412d6f-b022-4448-8fee-21cc936ea86b",
"version": "6.0.0",
Expand All @@ -302,7 +302,7 @@
}
},
"event": {
"created": 1579878369541,
"created": 1618841405309,
"id": "32f5fda2-48e4-4fae-b89e-a18038294d19",
"kind": "metric",
"category": [
Expand Down Expand Up @@ -348,7 +348,7 @@
"id": "4aVN2G8BYQH1gtPUuYk7",
"index": "metrics-endpoint.metadata-default",
"source": {
"@timestamp": 1579878369541,
"@timestamp": 1618841405309,
"agent": {
"id": "3838df35-a095-4af4-8fce-0b6d78793f2e",
"version": "6.8.0",
Expand All @@ -370,7 +370,7 @@
}
},
"event": {
"created": 1579878369541,
"created": 1618841405309,
"id": "32f5fda2-48e4-4fae-b89e-a18038294d39",
"kind": "metric",
"category": [
Expand Down Expand Up @@ -416,7 +416,7 @@
"id": "4qVN2G8BYQH1gtPUuYk7",
"index": "metrics-endpoint.metadata-default",
"source": {
"@timestamp": 1579874769541,
"@timestamp": 1618841405309,
"agent": {
"id": "963b081e-60d1-482c-befd-a5815fa8290f",
"version": "6.6.1",
Expand All @@ -438,7 +438,7 @@
}
},
"event": {
"created": 1579874769541,
"created": 1618841405309,
"id": "32f5fda2-48e4-4fae-b89e-a18038294d31",
"kind": "metric",
"category": [
Expand Down Expand Up @@ -485,7 +485,7 @@
"id": "46VN2G8BYQH1gtPUuYk7",
"index": "metrics-endpoint.metadata-default",
"source": {
"@timestamp": 1579874769541,
"@timestamp": 1618841405309,
"agent": {
"id": "b3412d6f-b022-4448-8fee-21cc936ea86b",
"version": "6.0.0",
Expand All @@ -507,7 +507,7 @@
}
},
"event": {
"created": 1579874769541,
"created": 1618841405309,
"id": "32f5fda2-48e4-4fae-b89e-a18038294d23",
"kind": "metric",
"category": [
Expand Down Expand Up @@ -553,7 +553,7 @@
"id": "5KVN2G8BYQH1gtPUuYk7",
"index": "metrics-endpoint.metadata-default",
"source": {
"@timestamp": 1579874769541,
"@timestamp": 1618841405309,
"agent": {
"id": "3838df35-a095-4af4-8fce-0b6d78793f2e",
"version": "6.8.0",
Expand All @@ -575,7 +575,7 @@
}
},
"event": {
"created": 1579874769541,
"created": 1618841405309,
"id": "32f5fda2-48e4-4fae-b89e-a18038294d35",
"kind": "metric",
"category": [
Expand Down
Loading