Skip to content

Commit 4966fd3

Browse files
andrewkrohandrewkroh
authored
journald - Add 8.x support (#4037)
Update kibana.constraint to allow 8.1.0+. 8.1.0 is required because elastic/beats#30032 is needed due to a breaking change that was introduced in 8.0.0. Rename message_id to ECS event.code. This field was added in 8.0. From man systemd.journal-fields MESSAGE_ID= A 128-bit message identifier ID for recognizing certain message types, if this is desirable. ...
1 parent 01bc513 commit 4966fd3

File tree

8 files changed

+234
-8
lines changed

8 files changed

+234
-8
lines changed

‎packages/journald/_dev/build/build.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
dependencies:
22
ecs:
3-
reference: git@1.11
3+
reference: git@v8.4.0-rc1

‎packages/journald/_dev/build/docs/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,4 +5,6 @@ The journald input reads the log data and the metadata associated with it.
55

66
The journald input is available on Linux systems with `systemd` installed.
77

8-
{{fields "log"}}
8+
{{event "log"}}
9+
10+
{{fields "log"}}

‎packages/journald/changelog.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,12 @@
11
# newer versions go on top
2+
- version: "0.0.4"
3+
changes:
4+
- description: Update Kibana requirement to support 8.x.
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/4037
7+
- description: Map `message_id` to ECS `event.code`.
8+
type: enhancement
9+
link: https://github.com/elastic/integrations/pull/4037
210
- version: "0.0.3"
311
changes:
412
- description: Add documentation for multi-fields

‎packages/journald/data_stream/log/elasticsearch/ingest_pipeline/default.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,11 @@ processors:
3232
target_field: log.syslog.identifier
3333
ignore_missing: true
3434

35+
- rename:
36+
field: message_id
37+
target_field: event.code
38+
ignore_missing: true
39+
3540
# Cleanup an empty syslog object.
3641
- remove:
3742
if: ctx?.syslog != null && ctx.syslog instanceof Map && ctx.syslog.isEmpty()

‎packages/journald/data_stream/log/fields/ecs.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
- name: ecs.version
22
external: ecs
3+
- name: event.code
4+
external: ecs
35
- name: host.hostname
46
external: ecs
57
- name: host.id

‎packages/journald/data_stream/log/sample_event.json‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
{
2+
"@timestamp": "2020-07-22T13:17:10.012Z",
3+
"agent": {
4+
"ephemeral_id": "27e2a00a-dab2-4790-8d45-29ad272d0392",
5+
"id": "bef8099b-68f6-4621-8089-2229b35a669d",
6+
"name": "docker-fleet-agent",
7+
"type": "filebeat",
8+
"version": "8.3.2"
9+
},
10+
"data_stream": {
11+
"dataset": "journald.log",
12+
"namespace": "ep",
13+
"type": "logs"
14+
},
15+
"ecs": {
16+
"version": "8.0.0"
17+
},
18+
"elastic_agent": {
19+
"id": "bef8099b-68f6-4621-8089-2229b35a669d",
20+
"snapshot": false,
21+
"version": "8.3.2"
22+
},
23+
"event": {
24+
"agent_id_status": "verified",
25+
"code": "ec387f577b844b8fa948f33cad9a75e6",
26+
"created": "2022-08-18T18:14:11.588Z",
27+
"dataset": "journald.log",
28+
"ingested": "2022-08-18T18:14:15Z",
29+
"kind": "event"
30+
},
31+
"host": {
32+
"hostname": "sleipnir",
33+
"id": "505afdafda3b4f33a63749ae39284742"
34+
},
35+
"input": {
36+
"type": "journald"
37+
},
38+
"journald": {
39+
"custom": {
40+
"available": "0",
41+
"available_pretty": "0B",
42+
"current_use": "1023455232",
43+
"current_use_pretty": "976.0M",
44+
"disk_available": "6866636800",
45+
"disk_available_pretty": "6.3G",
46+
"disk_keep_free": "1466253312",
47+
"disk_keep_free_pretty": "1.3G",
48+
"journal_name": "System journal",
49+
"journal_path": "/var/log/journal/505afdafda3b4f33a63749ae39284742",
50+
"limit": "977502208",
51+
"limit_pretty": "932.2M",
52+
"max_use": "977502208",
53+
"max_use_pretty": "932.2M"
54+
},
55+
"gid": 0,
56+
"host": {
57+
"boot_id": "fa3c2e3080dc4cd5be5cb5a43e140d51"
58+
},
59+
"pid": 19317,
60+
"process": {
61+
"capabilities": "25402800cf",
62+
"command_line": "/lib/systemd/systemd-journald",
63+
"executable": "/lib/systemd/systemd-journald",
64+
"name": "systemd-journal"
65+
},
66+
"uid": 0
67+
},
68+
"log": {
69+
"syslog": {
70+
"facility": {
71+
"code": 3
72+
},
73+
"identifier": "systemd-journald",
74+
"priority": 6
75+
}
76+
},
77+
"message": "System journal (/var/log/journal/505afdafda3b4f33a63749ae39284742) is 976.0M, max 932.2M, 0B free.",
78+
"process": {
79+
"args": [
80+
"/lib/systemd/systemd-journald"
81+
],
82+
"args_count": 1,
83+
"command_line": "/lib/systemd/systemd-journald",
84+
"pid": 19317
85+
},
86+
"systemd": {
87+
"cgroup": "/system.slice/systemd-journald.service",
88+
"invocation_id": "7c11cda63635437bafe21c92851618a8",
89+
"slice": "system.slice",
90+
"transport": "driver",
91+
"unit": "systemd-journald.service"
92+
},
93+
"tags": [
94+
"forwarded"
95+
],
96+
"user": {
97+
"group": {
98+
"id": "0"
99+
},
100+
"id": "0"
101+
}
102+
}

‎packages/journald/docs/README.md‎

Lines changed: 111 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,113 @@ The journald input reads the log data and the metadata associated with it.
55

66
The journald input is available on Linux systems with `systemd` installed.
77

8+
An example event for `log` looks as following:
9+
10+
```json
11+
{
12+
"@timestamp": "2020-07-22T13:17:10.012Z",
13+
"agent": {
14+
"ephemeral_id": "27e2a00a-dab2-4790-8d45-29ad272d0392",
15+
"id": "bef8099b-68f6-4621-8089-2229b35a669d",
16+
"name": "docker-fleet-agent",
17+
"type": "filebeat",
18+
"version": "8.3.2"
19+
},
20+
"data_stream": {
21+
"dataset": "journald.log",
22+
"namespace": "ep",
23+
"type": "logs"
24+
},
25+
"ecs": {
26+
"version": "8.0.0"
27+
},
28+
"elastic_agent": {
29+
"id": "bef8099b-68f6-4621-8089-2229b35a669d",
30+
"snapshot": false,
31+
"version": "8.3.2"
32+
},
33+
"event": {
34+
"agent_id_status": "verified",
35+
"code": "ec387f577b844b8fa948f33cad9a75e6",
36+
"created": "2022-08-18T18:14:11.588Z",
37+
"dataset": "journald.log",
38+
"ingested": "2022-08-18T18:14:15Z",
39+
"kind": "event"
40+
},
41+
"host": {
42+
"hostname": "sleipnir",
43+
"id": "505afdafda3b4f33a63749ae39284742"
44+
},
45+
"input": {
46+
"type": "journald"
47+
},
48+
"journald": {
49+
"custom": {
50+
"available": "0",
51+
"available_pretty": "0B",
52+
"current_use": "1023455232",
53+
"current_use_pretty": "976.0M",
54+
"disk_available": "6866636800",
55+
"disk_available_pretty": "6.3G",
56+
"disk_keep_free": "1466253312",
57+
"disk_keep_free_pretty": "1.3G",
58+
"journal_name": "System journal",
59+
"journal_path": "/var/log/journal/505afdafda3b4f33a63749ae39284742",
60+
"limit": "977502208",
61+
"limit_pretty": "932.2M",
62+
"max_use": "977502208",
63+
"max_use_pretty": "932.2M"
64+
},
65+
"gid": 0,
66+
"host": {
67+
"boot_id": "fa3c2e3080dc4cd5be5cb5a43e140d51"
68+
},
69+
"pid": 19317,
70+
"process": {
71+
"capabilities": "25402800cf",
72+
"command_line": "/lib/systemd/systemd-journald",
73+
"executable": "/lib/systemd/systemd-journald",
74+
"name": "systemd-journal"
75+
},
76+
"uid": 0
77+
},
78+
"log": {
79+
"syslog": {
80+
"facility": {
81+
"code": 3
82+
},
83+
"identifier": "systemd-journald",
84+
"priority": 6
85+
}
86+
},
87+
"message": "System journal (/var/log/journal/505afdafda3b4f33a63749ae39284742) is 976.0M, max 932.2M, 0B free.",
88+
"process": {
89+
"args": [
90+
"/lib/systemd/systemd-journald"
91+
],
92+
"args_count": 1,
93+
"command_line": "/lib/systemd/systemd-journald",
94+
"pid": 19317
95+
},
96+
"systemd": {
97+
"cgroup": "/system.slice/systemd-journald.service",
98+
"invocation_id": "7c11cda63635437bafe21c92851618a8",
99+
"slice": "system.slice",
100+
"transport": "driver",
101+
"unit": "systemd-journald.service"
102+
},
103+
"tags": [
104+
"forwarded"
105+
],
106+
"user": {
107+
"group": {
108+
"id": "0"
109+
},
110+
"id": "0"
111+
}
112+
}
113+
```
114+
8115
**Exported fields**
9116

10117
| Field | Description | Type |
@@ -16,6 +123,7 @@ The journald input is available on Linux systems with `systemd` installed.
16123
| data_stream.namespace | Data stream namespace. | constant_keyword |
17124
| data_stream.type | Data stream type. | constant_keyword |
18125
| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword |
126+
| event.code | Identification code for this event, if one exists. Some event sources use event codes to identify messages unambiguously, regardless of message language or wording adjustments over time. An example of this is the Windows Event ID. | keyword |
19127
| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword |
20128
| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword |
21129
| input.type | | keyword |
@@ -56,11 +164,11 @@ The journald input is available on Linux systems with `systemd` installed.
56164
| log.syslog.identifier | Identifier (usually process) contained in the syslog header. | keyword |
57165
| log.syslog.pid | PID contained in the syslog header. | long |
58166
| log.syslog.priority | Syslog numeric priority of the event, if available. According to RFCs 5424 and 3164, the priority is 8 \* facility + severity. This number is therefore expected to contain a value between 0 and 191. | long |
59-
| message | For log events the message field contains the log message, optimized for viewing in a log viewer. For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event. If multiple messages exist, they can be combined into one message. | text |
167+
| message | For log events the message field contains the log message, optimized for viewing in a log viewer. For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event. If multiple messages exist, they can be combined into one message. | match_only_text |
60168
| process.args | Array of process arguments, starting with the absolute path to the executable. May be filtered to protect sensitive information. | keyword |
61169
| process.args_count | Length of the process.args array. This field can be useful for querying or performing bucket analysis on how many arguments were provided to start a process. More arguments may be an indication of suspicious activity. | long |
62-
| process.command_line | Full command line that started the process, including the absolute path to the executable, and all arguments. Some arguments may be filtered to protect sensitive information. | keyword |
63-
| process.command_line.text | Multi-field of `process.command_line`. | text |
170+
| process.command_line | Full command line that started the process, including the absolute path to the executable, and all arguments. Some arguments may be filtered to protect sensitive information. | wildcard |
171+
| process.command_line.text | Multi-field of `process.command_line`. | match_only_text |
64172
| process.pid | Process id. | long |
65173
| systemd.cgroup | The control group path in the systemd hierarchy. | keyword |
66174
| systemd.invocation_id | The invocation ID for the runtime cycle of the unit the message was generated in, as available to processes of the unit in $INVOCATION_ID. | keyword |
@@ -74,4 +182,3 @@ The journald input is available on Linux systems with `systemd` installed.
74182
| tags | List of keywords used to tag each event. | keyword |
75183
| user.group.id | Unique identifier for the group on the system/platform. | keyword |
76184
| user.id | Unique identifier of the user. | keyword |
77-

‎packages/journald/manifest.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,15 @@
11
format_version: 1.0.0
22
name: journald
33
title: "Custom Journald logs"
4-
version: 0.0.3
4+
version: 0.0.4
55
license: basic
66
description: Collect logs from journald with Elastic Agent.
77
type: integration
88
categories:
99
- custom
1010
release: experimental
1111
conditions:
12-
kibana.version: "^7.16.0"
12+
kibana.version: "^7.17.0 || ^8.1.0"
1313
icons:
1414
- src: /img/systemd-logo.svg
1515
title: systemd logo

0 commit comments

Comments
 (0)