From 9b4d64dbf0e1104cfcb02cc1594b31f7e81af13f Mon Sep 17 00:00:00 2001 From: jamiehynds <62879768+jamiehynds@users.noreply.github.com> Date: Mon, 5 Oct 2020 10:35:54 +0100 Subject: [PATCH 1/7] Create 0008-email.md --- rfcs/text/0008-email.md | 144 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 144 insertions(+) create mode 100644 rfcs/text/0008-email.md diff --git a/rfcs/text/0008-email.md b/rfcs/text/0008-email.md new file mode 100644 index 0000000000..46256f3d54 --- /dev/null +++ b/rfcs/text/0008-email.md @@ -0,0 +1,144 @@ +# 0008: Email + + +- Stage: **1** +- Date: **Oct 5th 2020** + +This RFC proposes a new top-level field to facilitate email use cases. + + + + + +## Fields + + + +| field | type | description | +| --- | --- | --- | +| `email.action` | keyword | Action take by the source device, e.g. delivered, blocked, quarantined, deleted | +| `email.bcc.address` | keyword | Addresses of Bcc's | +| `email.bcc.domain` | keyword | Domains of the Bcc's | +| `email.cc.address` | keyword | Addresses of Cc's | +| `email.cc.domain` | keyword | Domains of Cc addresses | +| `email.cipher` | keyword | Cipher used e.g. TLS | +| `email.file.count` | value | Number of attachments included in the message | +| `email.file.extension` | keyword | Extensions of attachment, e.g. .zip, .docx | +| `email.file.hash` | keyword | Hash of attachments | +| `email.file.name` | keyword | File name of attachements | +| `email.file.size` | keyword | Total size of all attachements in bytes | +| `email.direction` | keyword | Direction of the message based on the sending and receving domains | +| `email.from.address` | keyword | Senders email address | +| `email.from.domain` | keyword | Senders domain | +| `email.latency` | keyword | The time, in milliseconds, the delivery attempt took | +| `email.message_id` | keyword | Internet message ID of the message | +| `email.process` | keyword | Name of the executable that carried out the transaction, e.g. outlook, sendmail | +| `email.protocol` | keyword | The email protocol used, e.g. SMTP, IMAP | +| `email.reply_to.address` | keyword | Reply-to address | +| `object.return.address` | keyword | The return address for the message | +| `email.size` | keyword | Total size of the message, in bytes, including attachments | +| `email.subject` | keyword | Subject of the message | +| `email.to` | keyword | Recipieint address | +| `email.to.domain` | keyword | Recipient domain | + +## Usage + + + +Email use cases stretch across all three Elastic solutions - Search, Observe, Protect. Whether it's searching for content within email, ensuring email infrastrucure is operational or detecting email based attacks, there are many possibilities for email fields within ECS. + +## Source data + + + +- **Email Analytics**: [Hubspot](https://legacydocs.hubspot.com/docs/methods/email/email_events_overview), Marketo, Salesforce Pardot +- **Email Server**: [O365 Message Tracing](https://docs.microsoft.com/en-us/exchange/monitoring/trace-an-email-message/run-a-message-trace-and-view-results), [Postfix](https://nxlog.co/documentation/nxlog-user-guide/postfix.html) +- **Email Security**: [Barracuda](https://campus.barracuda.com/product/emailsecuritygateway/doc/12193950/syslog-and-the-barracuda-email-security-gateway/), [Forcepoint](https://www.websense.com/content/support/library/email/v85/email_siem/siem_log_map.pdf), [Mimecast](https://www.mimecast.com/tech-connect/documentation/tutorials/understanding-siem-logs/), [Proofpoint](https://help.proofpoint.com/Threat_Insight_Dashboard/API_Documentation/SIEM_API) + + + + + +## Scope of impact + + + +## Concerns + + + + + + + + + +## Real-world implementations + + + +People + +The following are the people that consulted on the contents of this RFC. + +Jamie Hynds | author +TBD | Sponsor + + + + +## References + + + +### RFC Pull Requests + + + +* Stage 0: https://github.com/elastic/ecs/pull/NNN + + From 1a4c2fd0dd886aba17ef126f6ada3d4979d3382b Mon Sep 17 00:00:00 2001 From: P1llus Date: Mon, 9 Nov 2020 17:18:15 +0100 Subject: [PATCH 2/7] updating the RFC based on PR comments --- rfcs/text/0008-email.md | 55 ++++++++++++++++++++++------------------- 1 file changed, 29 insertions(+), 26 deletions(-) diff --git a/rfcs/text/0008-email.md b/rfcs/text/0008-email.md index 46256f3d54..20a8c61957 100644 --- a/rfcs/text/0008-email.md +++ b/rfcs/text/0008-email.md @@ -1,10 +1,10 @@ # 0008: Email -- Stage: **1** +- Stage: **1 (proposal)** - Date: **Oct 5th 2020** -This RFC proposes a new top-level field to facilitate email use cases. +This RFC proposes a new top-level field to facilitate email use cases. +Email specific fields: + | field | type | description | | --- | --- | --- | -| `email.action` | keyword | Action take by the source device, e.g. delivered, blocked, quarantined, deleted | -| `email.bcc.address` | keyword | Addresses of Bcc's | -| `email.bcc.domain` | keyword | Domains of the Bcc's | -| `email.cc.address` | keyword | Addresses of Cc's | -| `email.cc.domain` | keyword | Domains of Cc addresses | -| `email.cipher` | keyword | Cipher used e.g. TLS | -| `email.file.count` | value | Number of attachments included in the message | -| `email.file.extension` | keyword | Extensions of attachment, e.g. .zip, .docx | -| `email.file.hash` | keyword | Hash of attachments | -| `email.file.name` | keyword | File name of attachements | -| `email.file.size` | keyword | Total size of all attachements in bytes | +| `email.bcc.addresses` | wildcard | Addresses of Bcc's | +| `email.cc.addresses` | wildcard | Addresses of Cc's | +| `email.attachments_count` | long | A field outside the flattened structure to control how many attachments are included in the email | +| `email.attachments` | flattened | A flattened field for anything related to attachments. This allows objects being stored with all information for each file when you have multiple attachments | | `email.direction` | keyword | Direction of the message based on the sending and receving domains | -| `email.from.address` | keyword | Senders email address | -| `email.from.domain` | keyword | Senders domain | -| `email.latency` | keyword | The time, in milliseconds, the delivery attempt took | +| `email.sender.address` | wildcard | Senders email address | +| `email.sender.top_level_domain` | keyword | Senders email address | | `email.message_id` | keyword | Internet message ID of the message | -| `email.process` | keyword | Name of the executable that carried out the transaction, e.g. outlook, sendmail | -| `email.protocol` | keyword | The email protocol used, e.g. SMTP, IMAP | -| `email.reply_to.address` | keyword | Reply-to address | -| `object.return.address` | keyword | The return address for the message | +| `email.reply_to.address` | wildcard | Reply-to address | +| `email.return.address` | wildcard | The return address for the message | | `email.size` | keyword | Total size of the message, in bytes, including attachments | -| `email.subject` | keyword | Subject of the message | -| `email.to` | keyword | Recipieint address | -| `email.to.domain` | keyword | Recipient domain | +| `email.subject` | wildcard | Subject of the message | +| `email.recipients.addresses` | keyword | Recipient addresses | +| `email.domains` | keyword | domains related to the email | + + +Other ECS fields used together with email usecases: +| field | description | +| --- | --- | +| `event.duration` | The duration related to the email event. Could be the total duration in Quarantine, how long the email tok to send from source to destination etc | +| `process.name` | When the event is related to a server or client. Does not take MTA into account which is part of a ongoing discussion | +| `network.protocol` | Type of email protocol used | +| `tls.*` | Used for TLS related information for the connection to for example a SMTP server over TLS | + + ## Usage @@ -112,8 +115,8 @@ People The following are the people that consulted on the contents of this RFC. -Jamie Hynds | author -TBD | Sponsor +Marius Iversen | Author +Jamie Hynds | Sponsor -* Stage 0: https://github.com/elastic/ecs/pull/NNN +* Stage 0: https://github.com/elastic/ecs/pull/999 +Current concerns or topics still being discussed from stage 1: + +- Whether we want to add specific fields for email protocols, either as a root field or nested under email.* (SMTP, IMAP, POP etc). +- Need to make sure that the ECS fieldset for email catches all common usecases, for example spam, metrics and deliverables and logging. +- Whether we want to create a new event.category field (email) and which event.type it should be combined with. +- The email RFC will be the first ECS fieldset that uses the flattened datatype (for attachments), need to ensure that there will be major issues related to this. -People +## People The following are the people that consulted on the contents of this RFC. -@p1llus | Author -@jamiehynds | Sponsor +* @p1llus | Author +* @jamiehynds | Sponsor +- [Hubspot](https://legacydocs.hubspot.com/docs/methods/email/email_events_overview) +- [O365 Message Tracing](https://docs.microsoft.com/en-us/exchange/monitoring/trace-an-email-message/run-a-message-trace-and-view-results) +- [Postfix](https://nxlog.co/documentation/nxlog-user-guide/postfix.html) +- [Barracuda](https://campus.barracuda.com/product/emailsecuritygateway/doc/12193950/syslog-and-the-barracuda-email-security-gateway/) +- [Forcepoint](https://www.websense.com/content/support/library/email/v85/email_siem/siem_log_map.pdf) +- [Mimecast](https://www.mimecast.com/tech-connect/documentation/tutorials/understanding-siem-logs/) +- [Proofpoint](https://help.proofpoint.com/Threat_Insight_Dashboard/API_Documentation/SIEM_API) + ### RFC Pull Requests -* Stage 0: https://github.com/elastic/ecs/pull/999 +* Stage 1: https://github.com/elastic/ecs/pull/999 - Stage: **1 (proposal)** -- Date: **Oct 5th 2020** +- Date: **2020-11-30** This RFC proposes a new top-level field to facilitate email use cases.