Skip to content

[7.x] [Event Log] Extend ECS event schema with fields needed for Detection Engine (#95067) #95654

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kibanamachine merged 1 commit into from
Mar 29, 2021
Merged

[7.x] [Event Log] Extend ECS event schema with fields needed for Detection Engine (#95067) #95654

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 64 additions & 9 deletions x-pack/plugins/event_log/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,23 @@ actitivies.
## Overview

This plugin provides a persistent log of "events" that can be used by other
plugins to record their processing, for later acccess. Currently it's only
used by the alerts and actions plugins.
plugins to record their processing, for later accces. It is used by:

The "events" are ECS documents, with some custom properties for Kibana, and
alerting-specific properties within those Kibana properties. The number of
ECS fields is limited today, but can be extended fairly easily. We are being
conservative in adding new fields though, to help prevent indexing explosions.
- `alerting` and `actions` plugins
- [work in progress] `security_solution` (detection rules execution log)

The "events" are [ECS documents](https://www.elastic.co/guide/en/ecs/current/index.html)
containing both standard ECS fields and some custom fields for Kibana.

- Standard fields are those which are defined in the ECS specification.
Examples: `@timestamp`, `message`, `event.provider`. The number of ECS fields
supported in Event Log is limited today, but can be extended fairly easily.
We are being conservative in adding new fields though, to help prevent
indexing explosions.
- Custom fields are not part of the ECS spec. We defined a top-level `kibana`
field set where we have some Kibana-specific fields like `kibana.server_uuid`
and `kibana.saved_objects`. Plugins added a few custom fields as well,
for example `kibana.alerting` field set.

A client API is available for other plugins to:

Expand Down Expand Up @@ -47,16 +57,25 @@ The structure of the event documents can be seen in the
generated via a script when the structure changes. See the
[README.md](generated/README.md) for how to change the document structure.

Below is an document in the expected structure, with descriptions of the fields:
Below is a document in the expected structure, with descriptions of the fields:

```js
{
// Base ECS fields.
// https://www.elastic.co/guide/en/ecs/current/ecs-base.html
"@timestamp": "ISO date",
tags: ["tags", "here"],
message: "message for humans here",

// ECS version. This is set by the Event Log and should not be specified
// by a client of Event Log.
// https://www.elastic.co/guide/en/ecs/current/ecs-ecs.html
ecs: {
version: "version of ECS used by the event log",
},

// Event fields. All of them are supported.
// https://www.elastic.co/guide/en/ecs/current/ecs-event.html
event: {
provider: "see below",
action: "see below",
Expand All @@ -65,19 +84,44 @@ Below is an document in the expected structure, with descriptions of the fields:
end: "ISO date of end time for events that capture a duration",
outcome: "success | failure, for events that indicate an outcome",
reason: "additional detail on failure outcome",
// etc
},

// Error fields. All of them are supported.
// https://www.elastic.co/guide/en/ecs/current/ecs-error.html
error: {
message: "an error message, usually associated with outcome: failure",
// etc
},

// Log fields. Only a subset is supported.
// https://www.elastic.co/guide/en/ecs/current/ecs-log.html
log: {
level: "info | warning | any log level keyword you need",
logger: "name of the logger",
},

// Rule fields. All of them are supported.
// https://www.elastic.co/guide/en/ecs/current/ecs-rule.html
rule: {
author: ["Elastic"],
id: "a823fd56-5467-4727-acb1-66809737d943",
// etc
},

// User fields. Only user.name is supported.
// https://www.elastic.co/guide/en/ecs/current/ecs-user.html
user: {
name: "name of Kibana user",
},
kibana: { // custom ECS field

// Custom fields that are not part of ECS.
kibana: {
server_uuid: "UUID of kibana server, for diagnosing multi-Kibana scenarios",
alerting: {
instance_id: "alert instance id, for relevant documents",
action_group_id: "alert action group, for relevant documents",
action_subgroup_id: "alert action subgroup, for relevant documents",
action_subgroup: "alert action subgroup, for relevant documents",
status: "overall alert status, after alert execution",
},
saved_objects: [
Expand Down Expand Up @@ -363,3 +407,14 @@ yarn test:jest x-pack/plugins/event_log --watch

See: [`x-pack/test/plugin_api_integration/test_suites/event_log`](https://github.com/elastic/kibana/tree/master/x-pack/test/plugin_api_integration/test_suites/event_log).

To develop integration tests, first start the test server from the root of the repo:

```sh
node scripts/functional_tests_server --config x-pack/test/plugin_api_integration/config.ts
```

Then start the test runner:

```sh
node scripts/functional_test_runner --config x-pack/test/plugin_api_integration/config.ts --include x-pack/test/plugin_api_integration/test_suites/event_log/index.ts
```
29 changes: 22 additions & 7 deletions x-pack/plugins/event_log/generated/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,26 @@
# Generating event schema

The files in this directory were generated by manually running the script
../scripts/create-schemas.js from the root directory of the repository.
`../scripts/create-schemas.js` from the root directory of the repository.

These files should not be edited by hand.
**These files should not be edited by hand.**

Please follow the following steps:
1. clone the [ECS](https://github.com/elastic/ecs) repo locally so that it resides along side your kibana repo, and checkout the ECS version you wish to support (for example, the `1.6` branch, for version 1.6)
2. In the `x-pack/plugins/event_log/scripts/mappings.js` file you'll want to make th efollowing changes:
1. Update `EcsKibanaExtensionsMappings` to include the mapping of the fields you wish to add.
2. Update `EcsEventLogProperties` to include the fields in the generated mappings.json.
3. cd to the `kibana` root folder and run: `node ./x-pack/plugins/event_log/scripts/create_schemas.js`

1. Clone the [ECS](https://github.com/elastic/ecs) repo locally so that it
resides along side your kibana repo, and checkout the ECS version you wish to
support (for example, the `1.8` branch, for version 1.8).

2. In the `x-pack/plugins/event_log/scripts/mappings.js` file you'll want to
make the following changes:
- Update `EcsCustomPropertyMappings` to include the mapping of the custom
fields you wish to add.
- Update `EcsPropertiesToGenerate` to include the fields in the generated
`mappings.json`.
- Make sure to list all array fields in `EcsEventLogMultiValuedProperties`.

3. Cd to the `kibana` root folder and run:

```sh
node ./x-pack/plugins/event_log/scripts/create_schemas.js
```
181 changes: 169 additions & 12 deletions x-pack/plugins/event_log/generated/mappings.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,17 @@
"@timestamp": {
"type": "date"
},
"message": {
"norms": false,
"type": "text"
},
"tags": {
"ignore_above": 1024,
"type": "keyword",
"meta": {
"isArray": "true"
}
},
"message": {
"norms": false,
"type": "text"
},
"ecs": {
"properties": {
"version": {
Expand All @@ -23,40 +23,197 @@
}
}
},
"error": {
"properties": {
"code": {
"ignore_above": 1024,
"type": "keyword"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"message": {
"norms": false,
"type": "text"
},
"stack_trace": {
"doc_values": false,
"fields": {
"text": {
"norms": false,
"type": "text"
}
},
"ignore_above": 1024,
"index": false,
"type": "keyword"
},
"type": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"event": {
"properties": {
"action": {
"ignore_above": 1024,
"type": "keyword"
},
"provider": {
"category": {
"ignore_above": 1024,
"type": "keyword",
"meta": {
"isArray": "true"
}
},
"code": {
"ignore_above": 1024,
"type": "keyword"
},
"start": {
"created": {
"type": "date"
},
"dataset": {
"ignore_above": 1024,
"type": "keyword"
},
"duration": {
"type": "long"
},
"end": {
"type": "date"
},
"hash": {
"ignore_above": 1024,
"type": "keyword"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"ingested": {
"type": "date"
},
"kind": {
"ignore_above": 1024,
"type": "keyword"
},
"module": {
"ignore_above": 1024,
"type": "keyword"
},
"original": {
"doc_values": false,
"ignore_above": 1024,
"index": false,
"type": "keyword"
},
"outcome": {
"ignore_above": 1024,
"type": "keyword"
},
"provider": {
"ignore_above": 1024,
"type": "keyword"
},
"reason": {
"ignore_above": 1024,
"type": "keyword"
},
"reference": {
"ignore_above": 1024,
"type": "keyword"
},
"risk_score": {
"type": "float"
},
"risk_score_norm": {
"type": "float"
},
"sequence": {
"type": "long"
},
"severity": {
"type": "long"
},
"start": {
"type": "date"
},
"timezone": {
"ignore_above": 1024,
"type": "keyword"
},
"type": {
"ignore_above": 1024,
"type": "keyword",
"meta": {
"isArray": "true"
}
},
"url": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"error": {
"log": {
"properties": {
"message": {
"norms": false,
"type": "text"
"level": {
"ignore_above": 1024,
"type": "keyword"
},
"logger": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"rule": {
"properties": {
"author": {
"ignore_above": 1024,
"type": "keyword",
"meta": {
"isArray": "true"
}
},
"category": {
"ignore_above": 1024,
"type": "keyword"
},
"description": {
"ignore_above": 1024,
"type": "keyword"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"license": {
"ignore_above": 1024,
"type": "keyword"
},
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"reference": {
"ignore_above": 1024,
"type": "keyword"
},
"ruleset": {
"ignore_above": 1024,
"type": "keyword"
},
"uuid": {
"ignore_above": 1024,
"type": "keyword"
},
"version": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
Expand Down Expand Up @@ -101,6 +258,7 @@
}
},
"saved_objects": {
"type": "nested",
"properties": {
"rel": {
"type": "keyword",
Expand All @@ -118,8 +276,7 @@
"type": "keyword",
"ignore_above": 1024
}
},
"type": "nested"
}
}
}
}
Expand Down
Loading