diff --git a/CHANGELOG.md b/CHANGELOG.md index e913e32cae..10da17fcaa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,7 @@ All notable changes to this project will be documented in this file based on the * Add `network.community_id` field. #208 * Add fields `geo.country_name` and `geo.region_iso_code`. #214 * Add `event.kind` and `event.outcome`. #242 +* Add `client` and `server` objects and fields. #236 ### Improvements * Improved the definition of the file fields #196 diff --git a/README.md b/README.md index 2ce821dfb8..9be087b0e4 100644 --- a/README.md +++ b/README.md @@ -49,6 +49,7 @@ fully up to date. ECS defines these fields. * [Base fields](#base) * [Agent fields](#agent) + * [Client fields](#client) * [Cloud fields](#cloud) * [Container fields](#container) * [Destination fields](#destination) @@ -67,6 +68,7 @@ ECS defines these fields. * [Operating System fields](#os) * [Process fields](#process) * [Related fields](#related) + * [Server fields](#server) * [Service fields](#service) * [Source fields](#source) * [URL fields](#url) @@ -102,6 +104,21 @@ The agent fields contain the data about the software entity, if any, that collec Examples: In the case of Beats for logs, the agent.name is filebeat. For APM, it is the agent running in the app/service. The agent information does not change if data is sent through queuing systems like Kafka, Redis, or processing systems such as Logstash or APM Server. +## Client fields + +A client is defined as the initiator of a network connection for events regarding sessions, connections, or bidirectional flow records. For TCP events, the client is the initiator of the TCP connection that sends the SYN packet(s). For other protocols, the client is generally the initiator or requestor in the network transaction. Some systems use the term "originator" to refer the client in TCP connections. The client fields describe details about the system acting as the client in the network event. Client fields are usually populated in conjuction with server fields. Client fields are generally not populated for packet-level events. + + +| Field | Description | Level | Type | Example | +|---|---|---|---|---| +| client.ip | IP address of the client.
Can be one or multiple IPv4 or IPv6 addresses. | core | ip | | +| client.port | Port of the client. | core | long | | +| client.mac | MAC address of the client. | core | keyword | | +| client.domain | Client domain. | core | keyword | | +| client.bytes | Bytes sent from the client to the server. | core | long | `184` | +| client.packets | Packets sent from the client to the server. | core | long | `12` | + + ## Cloud fields Fields related to the cloud or infrastructure the events are coming from. @@ -227,7 +244,7 @@ A file is defined as a set of information that has been created on, or has exist Geo fields can carry data about a specific location related to an event or geo information derived from an IP field. -The `geo` fields are expected to be nested at: `destination.geo`, `host.geo`, `observer.geo`, `source.geo`. +The `geo` fields are expected to be nested at: `client.geo`, `destination.geo`, `host.geo`, `observer.geo`, `server.geo`, `source.geo`. Note also that the `geo` fields are not expected to be used directly at the top level. @@ -379,9 +396,7 @@ These fields contain information about a process. These fields can help you corr ## Related fields -This field set is meant to facilitate pivoting around a piece of data. Some pieces of information can be seen in many places in ECS. To facilitate searching for them, append values to their corresponding field in `related.`. - -A concrete example is IP addresses, which can be under host, observer, source, destination, and network.forwarded_ip. If you append all IPs to `related.ip`, you can then search for a given IP trivially, no matter where it appeared, by querying `related.ip:a.b.c.d`. +This field set is meant to facilitate pivoting around a piece of data. Some pieces of information can be seen in many places in ECS. To facilitate searching for them, append values to their corresponding field in `related.`. A concrete example is IP addresses, which can be under host, observer, source, destination, client, server, and network.forwarded_ip. If you append all IPs to `related.ip`, you can then search for a given IP trivially, no matter where it appeared, by querying `related.ip:a.b.c.d`. | Field | Description | Level | Type | Example | @@ -389,6 +404,21 @@ A concrete example is IP addresses, which can be under host, observer, source, d | related.ip | All of the IPs seen on your event. | extended | ip | | +## Server fields + +A Server is defined as the responder in a network connection for events regarding sessions, connections, or bidirectional flow records. For TCP events, the server is the receiver of the initial SYN packet(s) of the TCP connection. For other protocols, the server is generally the responder in the network transaction. Some systems actually use the term "responder" to refer the server in TCP connections. The server fields describe details about the system acting as the server in the network event. Server fields are usually populated in conjunction with client fields. Server fields are generally not populated for packet-level events. + + +| Field | Description | Level | Type | Example | +|---|---|---|---|---| +| server.ip | IP address of the server.
Can be one or multiple IPv4 or IPv6 addresses. | core | ip | | +| server.port | Port of the server. | core | long | | +| server.mac | MAC address of the server. | core | keyword | | +| server.domain | Server domain. | core | keyword | | +| server.bytes | Bytes sent from the server to the client. | core | long | `184` | +| server.packets | Packets sent from the server to the client. | core | long | `12` | + + ## Service fields The service fields describe the service for or from which the data was collected. These fields help you find and correlate logs for a specific service and version. @@ -445,7 +475,7 @@ or The user fields describe information about the user that is relevant to the event. Fields can have one entry or multiple entries. If a user has more than one id, provide an array that includes all of them. -The `user` fields are expected to be nested at: `destination.user`, `host.user`, `source.user`. +The `user` fields are expected to be nested at: `client.user`, `destination.user`, `host.user`, `server.user`, `source.user`. Note also that the `user` fields may be used directly at the top level. diff --git a/fields.yml b/fields.yml index 1f1cb1f9de..3674838487 100644 --- a/fields.yml +++ b/fields.yml @@ -119,6 +119,55 @@ different values which are then freely searchable. If multiple messages exist, they can be combined into one message. + - name: client + title: Client + group: 2 + description: > + A client is defined as the initiator of a network connection for events regarding sessions, connections, or bidirectional flow records. For TCP events, the client is the initiator of the TCP connection that sends the SYN packet(s). For other protocols, the client is generally the initiator or requestor in the network transaction. Some systems use the term "originator" to refer the client in TCP connections. The client fields describe details about the system acting as the client in the network event. Client fields are usually populated in conjuction with server fields. Client fields are generally not populated for packet-level events. + type: group + fields: + + - name: ip + level: core + type: ip + description: > + IP address of the client. + + Can be one or multiple IPv4 or IPv6 addresses. + + - name: port + level: core + type: long + description: > + Port of the client. + + - name: mac + level: core + type: keyword + description: > + MAC address of the client. + + - name: domain + level: core + type: keyword + description: > + Client domain. + + # Metrics + - name: bytes + level: core + type: long + example: 184 + description: > + Bytes sent from the client to the server. + + - name: packets + level: core + type: long + example: 12 + description: > + Packets sent from the client to the server. + - name: cloud title: Cloud group: 2 @@ -613,9 +662,11 @@ reusable: top_level: false expected: + - client - destination - observer - host + - server - source type: group fields: @@ -1151,11 +1202,7 @@ Some pieces of information can be seen in many places in ECS. To facilitate searching for them, append values to their corresponding field in `related.`. - - A concrete example is IP addresses, which can be under host, observer, source, - destination, and network.forwarded_ip. If you append all IPs to - `related.ip`, you can then search for a given IP trivially, - no matter where it appeared, by querying `related.ip:a.b.c.d`. + A concrete example is IP addresses, which can be under host, observer, source, destination, client, server, and network.forwarded_ip. If you append all IPs to `related.ip`, you can then search for a given IP trivially, no matter where it appeared, by querying `related.ip:a.b.c.d`. type: group fields: @@ -1165,6 +1212,55 @@ description: > All of the IPs seen on your event. + - name: server + title: Server + group: 2 + description: > + A Server is defined as the responder in a network connection for events regarding sessions, connections, or bidirectional flow records. For TCP events, the server is the receiver of the initial SYN packet(s) of the TCP connection. For other protocols, the server is generally the responder in the network transaction. Some systems actually use the term "responder" to refer the server in TCP connections. The server fields describe details about the system acting as the server in the network event. Server fields are usually populated in conjunction with client fields. Server fields are generally not populated for packet-level events. + type: group + fields: + + - name: ip + level: core + type: ip + description: > + IP address of the server. + + Can be one or multiple IPv4 or IPv6 addresses. + + - name: port + level: core + type: long + description: > + Port of the server. + + - name: mac + level: core + type: keyword + description: > + MAC address of the server. + + - name: domain + level: core + type: keyword + description: > + Server domain. + + # Metrics + - name: bytes + level: core + type: long + example: 184 + description: > + Bytes sent from the server to the client. + + - name: packets + level: core + type: long + example: 12 + description: > + Packets sent from the server to the client. + - name: service title: Service group: 2 @@ -1404,8 +1500,10 @@ reusable: top_level: true expected: + - client - destination - host + - server - source type: group fields: diff --git a/schema.csv b/schema.csv index bab8decbdc..d4351c2fef 100644 --- a/schema.csv +++ b/schema.csv @@ -8,6 +8,12 @@ agent.id,keyword,core,8a4f500d agent.name,keyword,core,foo agent.type,keyword,core,filebeat agent.version,keyword,core,6.0.0-rc2 +client.bytes,long,core,184 +client.domain,keyword,core, +client.ip,ip,core, +client.mac,keyword,core, +client.packets,long,core,12 +client.port,long,core, cloud.account.id,keyword,extended,666777888999 cloud.availability_zone,keyword,extended,us-east-1c cloud.instance.id,keyword,extended,i-1234567890abcdef0 @@ -120,6 +126,12 @@ process.thread.id,long,extended,4242 process.title,keyword,extended, process.working_directory,keyword,extended,/home/alice related.ip,ip,extended, +server.bytes,long,core,184 +server.domain,keyword,core, +server.ip,ip,core, +server.mac,keyword,core, +server.packets,long,core,12 +server.port,long,core, service.ephemeral_id,keyword,extended,8a4f500f service.id,keyword,core,d37e5ebfe0ae6c4972dbe9f0174a1637bb8247f6 service.name,keyword,core,elasticsearch-metrics diff --git a/schemas/client.yml b/schemas/client.yml new file mode 100644 index 0000000000..e81a741d9d --- /dev/null +++ b/schemas/client.yml @@ -0,0 +1,49 @@ +--- +- name: client + title: Client + group: 2 + description: > + A client is defined as the initiator of a network connection for events regarding sessions, connections, or bidirectional flow records. For TCP events, the client is the initiator of the TCP connection that sends the SYN packet(s). For other protocols, the client is generally the initiator or requestor in the network transaction. Some systems use the term "originator" to refer the client in TCP connections. The client fields describe details about the system acting as the client in the network event. Client fields are usually populated in conjuction with server fields. Client fields are generally not populated for packet-level events. + type: group + fields: + + - name: ip + level: core + type: ip + description: > + IP address of the client. + + Can be one or multiple IPv4 or IPv6 addresses. + + - name: port + level: core + type: long + description: > + Port of the client. + + - name: mac + level: core + type: keyword + description: > + MAC address of the client. + + - name: domain + level: core + type: keyword + description: > + Client domain. + + # Metrics + - name: bytes + level: core + type: long + example: 184 + description: > + Bytes sent from the client to the server. + + - name: packets + level: core + type: long + example: 12 + description: > + Packets sent from the client to the server. diff --git a/schemas/geo.yml b/schemas/geo.yml index 2d339b9f61..8553803f0e 100644 --- a/schemas/geo.yml +++ b/schemas/geo.yml @@ -8,9 +8,11 @@ reusable: top_level: false expected: + - client - destination - observer - host + - server - source type: group fields: diff --git a/schemas/related.yml b/schemas/related.yml index 679156e545..9815aa80dc 100644 --- a/schemas/related.yml +++ b/schemas/related.yml @@ -7,11 +7,7 @@ Some pieces of information can be seen in many places in ECS. To facilitate searching for them, append values to their corresponding field in `related.`. - - A concrete example is IP addresses, which can be under host, observer, source, - destination, and network.forwarded_ip. If you append all IPs to - `related.ip`, you can then search for a given IP trivially, - no matter where it appeared, by querying `related.ip:a.b.c.d`. + A concrete example is IP addresses, which can be under host, observer, source, destination, client, server, and network.forwarded_ip. If you append all IPs to `related.ip`, you can then search for a given IP trivially, no matter where it appeared, by querying `related.ip:a.b.c.d`. type: group fields: diff --git a/schemas/server.yml b/schemas/server.yml new file mode 100644 index 0000000000..504b07300e --- /dev/null +++ b/schemas/server.yml @@ -0,0 +1,49 @@ +--- +- name: server + title: Server + group: 2 + description: > + A Server is defined as the responder in a network connection for events regarding sessions, connections, or bidirectional flow records. For TCP events, the server is the receiver of the initial SYN packet(s) of the TCP connection. For other protocols, the server is generally the responder in the network transaction. Some systems actually use the term "responder" to refer the server in TCP connections. The server fields describe details about the system acting as the server in the network event. Server fields are usually populated in conjunction with client fields. Server fields are generally not populated for packet-level events. + type: group + fields: + + - name: ip + level: core + type: ip + description: > + IP address of the server. + + Can be one or multiple IPv4 or IPv6 addresses. + + - name: port + level: core + type: long + description: > + Port of the server. + + - name: mac + level: core + type: keyword + description: > + MAC address of the server. + + - name: domain + level: core + type: keyword + description: > + Server domain. + + # Metrics + - name: bytes + level: core + type: long + example: 184 + description: > + Bytes sent from the server to the client. + + - name: packets + level: core + type: long + example: 12 + description: > + Packets sent from the server to the client. diff --git a/schemas/user.yml b/schemas/user.yml index 0ae3eca32b..8598a41e0d 100644 --- a/schemas/user.yml +++ b/schemas/user.yml @@ -10,8 +10,10 @@ reusable: top_level: true expected: + - client - destination - host + - server - source type: group fields: diff --git a/template.json b/template.json index f913623fcb..ede14e3e4f 100644 --- a/template.json +++ b/template.json @@ -47,6 +47,30 @@ } } }, + "client": { + "properties": { + "bytes": { + "type": "long" + }, + "domain": { + "ignore_above": 1024, + "type": "keyword" + }, + "ip": { + "type": "ip" + }, + "mac": { + "ignore_above": 1024, + "type": "keyword" + }, + "packets": { + "type": "long" + }, + "port": { + "type": "long" + } + } + }, "cloud": { "properties": { "account": { @@ -579,6 +603,30 @@ } } }, + "server": { + "properties": { + "bytes": { + "type": "long" + }, + "domain": { + "ignore_above": 1024, + "type": "keyword" + }, + "ip": { + "type": "ip" + }, + "mac": { + "ignore_above": 1024, + "type": "keyword" + }, + "packets": { + "type": "long" + }, + "port": { + "type": "long" + } + } + }, "service": { "properties": { "ephemeral_id": {