diff --git a/rfcs/text/0010-email.md b/rfcs/text/0010-email.md index 98e1efc575..b160118eaa 100644 --- a/rfcs/text/0010-email.md +++ b/rfcs/text/0010-email.md @@ -1,32 +1,28 @@ # 0010: Email - -- Stage: **1 (draft)** -- Date: **2021-08-16** +- Stage: **2 (candidate)** +- Date: **2021-12-13** This RFC proposes a new top-level field set to facilitate email use cases, `email.*`. The `email.*` field set adds fields for the sender, recipient, message header fields, and other attributes of an email message typically seen logs produced by mail transfer agent (MTA) and email gateway applications. ## Fields - - ### Email specific fields | field | type | description | | --- | --- | --- | -| `email.from` | keyword | Stores the `from` email address from the RFC5322 `From:` header field. | | `email.origination_timestamp` | date | The date and time the email message was composed. Many email clients will fill this in automatically when the message is sent by a user. | | `email.delivery_timestamp` | date | The date and time the email message was received by the service or client. | -| `email.to` | keyword (array) | The email address(es) of the message recipient(s) | -| `email.subject` | keyword; `.text` text multi-field | A brief summary of the topic of the message | -| `email.cc` | keyword (array) | The email address(es) of the carbon copy (CC) recipient(s) | -| `email.bcc` | keyword (array) | The email address(es) of the blind carbon copy (CC) recipient(s) | +| `email.from.address` | keyword (array) | Stores the `from` email address from the RFC5322 `From:` header field. | +| `email.sender.address` | keyword | When the `from` field contains more than one address or the `sender` and `from` are distinct then this field is populated. | +| `email.to.address` | keyword (array)| The email address of message recipient | +| `email.cc.address` | keyword (array) | The email address of a carbon copy (CC) recipient | +| `email.bcc.address` | keyword (array) | The email address of the blind carbon copy (CC) recipient(s) | +| `email.reply_to.address` | keyword (array) | The address that replies should be delivered to from the RFC 5322 `Reply-To:` header field. | +| `email.subject` | keyword (`.text` text multi-field) | A brief summary of the topic of the message | | `email.content_type` | keyword | Information about how the message is to be displayed. Typically a MIME type | | `email.message_id` | wildcard | Identifier from the RFC5322 `Message-ID:` header field that refers to a particular version of a particular message. | | `email.local_id` | keyword | Unique identifier given to the email by the source (MTA, gateway, etc.) that created the event and is not persistent across hops (for example, the `X-MS-Exchange-Organization-Network-Message-Id` id). | -| `email.reply_to` | keyword | The address that replies should be delivered to from the RFC 5322 `Reply-To:` header field. | | `email.direction` | keyword | Direction of the message based on the sending and receiving domains | | `email.x_mailer` | keyword | What application was used to draft and send the original email. | | `email.attachments` | nested | Nested object of attachments on the email. | @@ -44,18 +40,10 @@ Email events may benefit from an additional ECS allowed event categorization val ## Usage - - -Email use cases stretch across all three Elastic solutions - Search, Observe, Protect. Whether it's searching for content within email, ensuring email infrastructure is operational or detecting email based attacks, there are many possibilities for email fields within ECS. +Email use cases stretch across all three Elastic solutions - Search, Observe, Protect. Whether it's searching for content within email, ensuring email infrastructure is operational, or detecting email-based attacks, there are many possibilities for email fields within ECS. ## Source data - - - **Email Analytics**: [Hubspot](https://legacydocs.hubspot.com/docs/methods/email/email_events_overview), Marketo, Salesforce Pardot - **Email Server**: [O365 Message Tracing](https://docs.microsoft.com/en-us/exchange/monitoring/trace-an-email-message/run-a-message-trace-and-view-results), [Postfix](https://nxlog.co/documentation/nxlog-user-guide/postfix.html) - **Email Security**: [Barracuda](https://campus.barracuda.com/product/emailsecuritygateway/doc/12193950/syslog-and-the-barracuda-email-security-gateway/), [Forcepoint](https://www.websense.com/content/support/library/email/v85/email_siem/siem_log_map.pdf), [Mimecast](https://www.mimecast.com/tech-connect/documentation/tutorials/understanding-siem-logs/), [Proofpoint](https://help.proofpoint.com/Threat_Insight_Dashboard/API_Documentation/SIEM_API) @@ -70,7 +58,7 @@ Stage 2: Included a real world example source document. Ideally this example com ```json { - "EndDate": "2020-11-10T22:12:34.8196921Z", + "EndDate": "2021-11-10T22:12:34.8196921Z", "FromIP": "8.8.8.8", "Index": 25, "MessageId": "\\u003c95689d8d5e7f429390a4e3646eef75e8-JFBVALKQOJXWILKBK4YVA7APGM3DKTLFONZWCZ3FINSW45DFOJ6EAQ2ENFTWK43UL4YTCMBYGIYHYU3NORYA====@microsoft.com\\u003e", @@ -94,12 +82,16 @@ Stage 2: Included a real world example source document. Ideally this example com "@timestamp": 1626984241830, "email": { "timestamp": "2020-11-08T22:12:34.8196921Z", - "from": [ - "o365mc@microsoft.com" - ], - "to": [ - "john@testdomain.onmicrosoft.com" - ], + "from": { + "address": [ + "o365mc@microsoft.com" + ] + }, + "to": { + "address": [ + "john@testdomain.onmicrosoft.com" + ] + }, "subject": "Weekly digest: Microsoft service updates", "message_id": "\\u003c95689d8d5e7f429390a4e3646eef75e8-JFBVALKQOJXWILKBK4YVA7APGM3DKTLFONZWCZ3FINSW45DFOJ6EAQ2ENFTWK43UL4YTCMBYGIYHYU3NORYA====@microsoft.com\\u003e" }, @@ -144,12 +136,16 @@ Stage 2: Included a real world example source document. Ideally this example com "@timestamp": 1626984241830, "email": { "timestamp": "2020-11-10T22:12:34.8196921Z", - "from": [ - "postmaster@testdomain.onmicrosoft.com" - ], - "to": [ - "o365mc@microsoft.com" - ], + "from": { + "address": [ + "postmaster@testdomain.onmicrosoft.com" + ] + }, + "to": { + "address": [ + "o365mc@microsoft.com" + ] + }, "subject": "Undeliverable: Message Center Major Change Update Notification", "message_id": "\\u003c72872e16-f4c2-4eef-a393-e5621748a0ff@AS8P19vMB1605.EURP191.PROD.OUTLOOK.COM\\u003e" }, @@ -169,30 +165,41 @@ Stage 2: Included a real world example source document. Ideally this example com #### Original log ``` -<38>1 2016-06-24T21:00:08Z - ProofpointTAP - MSGBLK [tapmsg@21139 messageTime="2016-06-24T21:18:38.000Z" messageID="20160624211145.62086.mail@evil.zz" recipient="clark.kent@pharmtech.zz, diana.prince@pharmtech.zz" sender="e99d7ed5580193f36a51f597bc2c0210@evil.zz" senderIP="192.0.2.255" phishScore="46" spamScore="4" QID="r2FNwRHF004109" GUID="c26dbea0-80d5-463b-b93c-4e8b708219ce" subject="Please find a totally safe invoice attached." quarantineRule="module.sandbox.threat" quarantineFolder="Attachment Defense" policyRoutes="default_inbound,executives" modulesRun="sandbox,urldefense,spam,pdr" headerFrom="\"A. Badguy\" " headerTo="\"Clark Kent\" ; \"Diana Prince\" " headerCC="\"Bruce Wayne\" " headerReplyTo="null" toAddresses="clark.kent@pharmtech.zz,diana.prince@pharmtech.zz" ccAddresses="bruce.wayne@university-of-education.zz" fromAddress="badguy@evil.zz" replyToAddress="null" clusterId="pharmtech_hosted" messageParts="[{\"contentType\":\"text/plain\",\"disposition\":\"inline\",\"filename\":\"text.txt\",\"md5\":\"008c5926ca861023c1d2a36653fd88e2\",\"oContentType\":\"text/plain\",\"sandboxStatus\":\"unsupported\",\"sha256\":\"85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281\"},{\"contentType\":\"application/pdf\",\"disposition\":\"attached\",\"filename\":\"Invoice for Pharmtech.pdf\",\"md5\":\"5873c7d37608e0d49bcaa6f32b6c731f\",\"oContentType\":\"application/pdf\",\"sandboxStatus\":\"threat\",\"sha256\":\"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca\"}]" xmailer="Spambot v2.5"] +<38>1 2021-06-24T21:00:08Z - ProofpointTAP - MSGBLK [tapmsg@21139 messageTime="2021-06-24T21:18:38.000Z" messageID="20160624211145.62086.mail@evil.zz" recipient="clark.kent@pharmtech.zz, diana.prince@pharmtech.zz" sender="e99d7ed5580193f36a51f597bc2c0210@evil.zz" senderIP="192.0.2.255" phishScore="46" spamScore="4" QID="r2FNwRHF004109" GUID="c26dbea0-80d5-463b-b93c-4e8b708219ce" subject="Please find a totally safe invoice attached." quarantineRule="module.sandbox.threat" quarantineFolder="Attachment Defense" policyRoutes="default_inbound,executives" modulesRun="sandbox,urldefense,spam,pdr" headerFrom="\"A. Badguy\" " headerTo="\"Clark Kent\" ; \"Diana Prince\" " headerCC="\"Bruce Wayne\" " headerReplyTo="null" toAddresses="clark.kent@pharmtech.zz,diana.prince@pharmtech.zz" ccAddresses="bruce.wayne@university-of-education.zz" fromAddress="badguy@evil.zz" replyToAddress="null" clusterId="pharmtech_hosted" messageParts="[{\"contentType\":\"text/plain\",\"disposition\":\"inline\",\"filename\":\"text.txt\",\"md5\":\"008c5926ca861023c1d2a36653fd88e2\",\"oContentType\":\"text/plain\",\"sandboxStatus\":\"unsupported\",\"sha256\":\"85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281\"},{\"contentType\":\"application/pdf\",\"disposition\":\"attached\",\"filename\":\"Invoice for Pharmtech.pdf\",\"md5\":\"5873c7d37608e0d49bcaa6f32b6c731f\",\"oContentType\":\"application/pdf\",\"sandboxStatus\":\"threat\",\"sha256\":\"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca\"}]" xmailer="Spambot v2.5"] ``` #### Mapped event ```json { - "@timestamp": "2016-06-24T21:00:08Z", + "@timestamp": "2021-06-24T21:00:08Z", "email": { - "timestamp": "2016-06-24T21:18:38.000Z", - "message_id": "20160624211145.62086.mail@evil.zz", + "timestamp": "2021-06-24T21:18:38.000Z", + "message_id": "20210624211145.62086.mail@evil.zz", "local_id": "c26dbea0-80d5-463b-b93c-4e8b708219ce", - "to": [ - "clark.kent@pharmtech.zz", - "diana.prince@pharmtech.zz" - ], - "cc": [ - "bruce.wayne@university-of-education.zz" - ], - "from": [ - "badguy@evil.zz" - ], + "to": { + "address": [ + "clark.kent@pharmtech.zz", + "diana.prince@pharmtech.zz" + ] + }, + "cc": { + "address": [ + "bruce.wayne@university-of-education.zz" + ] + }, + "from": { + "address": [ + "badguy@evil.zz" + ] + }, + "sender": { + "address": "e99d7ed5580193f36a51f597bc2c0210@evil.zz" + }, "subject": "Please find a totally safe invoice attached.", - "reply_to": "null", + "reply_to": { + "address": "null" + }, "x_mailer": "Spambot v2.5", "attachments": [ { @@ -215,8 +222,8 @@ Stage 2: Included a real world example source document. Ideally this example com "action": "MSGBLK" }, "source": { - "address": 192.0.2.255, - "ip": 192.0.2.255 + "address": "192.0.2.255", + "ip": "192.0.2.255" } } ``` @@ -226,26 +233,30 @@ Stage 2: Included a real world example source document. Ideally this example com #### Original log ``` -datetime=2017-05-26T16:47:41+0100|aCode=7O7I7MvGP1mj8plHRDuHEA|acc=C0A0|SpamLimit=0|IP=123.123.123.123|Dir=Internal|MsgId=<81ce15$8r2j59@mail01.example.com>|Subject=\message subject\|headerFrom=from@mimecast.com|Sender=from@mimecast.com|Rcpt=auser@mimecast.com|SpamInfo=[]|Act=Acc|TlsVer=TLSv1|Cphr=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA|SpamProcessingDetail={"spf":{"info":"SPF_FAIL","allow":true},"dkim":{"info":"DKIM_UNKNOWN","allow":true}}|SpamScore=1 +datetime=2021-05-26T16:47:41+0100|aCode=7O7I7MvGP1mj8plHRDuHEA|acc=C0A0|SpamLimit=0|IP=123.123.123.123|Dir=Internal|MsgId=<81ce15$8r2j59@mail01.example.com>|Subject=\message subject\|headerFrom=from@mimecast.com|Sender=from@mimecast.com|Rcpt=auser@mimecast.com|SpamInfo=[]|Act=Acc|TlsVer=TLSv1|Cphr=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA|SpamProcessingDetail={"spf":{"info":"SPF_FAIL","allow":true},"dkim":{"info":"DKIM_UNKNOWN","allow":true}}|SpamScore=1 ``` #### Mapped event ```json { - "@timestamp": "2017-05-26T16:47:41+0100", + "@timestamp": "2021-05-26T16:47:41+0100", "source": { - "address": 123.123.123.123, - "ip": 123.123.123.123 + "address": "123.123.123.123", + "ip": "123.123.123.123" }, "email": { "message_id": "<81ce15$8r2j59@mail01.example.com>", - "from": [ - "from@mimecast.com" - ], - "to": [ - "auser@mimecast.com" - ], + "from": { + "address": [ + "from@mimecast.com" + ] + }, + "to": { + "address": [ + "auser@mimecast.com" + ] + }, "subject": "message subject", "direction": "internal" }, @@ -263,42 +274,44 @@ Stage 3: Add more real world example source documents so we have at least 2 tota ## Scope of impact - +This is a new field set, and the changes introduced will not affect existing ECS implementations. -## Concerns +Integrations or other data sources mapping to ECS will need to map their original events to the new fields. - +## Concerns ### Email messages vs. protocols -The fields proposed in this document are focused on the contents of an email message but not on specific fields for email protocols. Do protocols like SMTP, POP3, IMAP, etc. be represented in ECS? +The fields proposed in this document focus on an email message's content but not on specific fields for email protocols. However, should protocols like SMTP, POP3, IMAP, etc., be represented in ECS? For example, users may need to compare the email address from the SMTP (envelope) sender to the `From:` header email address. -### Email metrics and observability use caes +**Resolution**: Focus on email message content in this initial phase. Additional protocol details can be added later on. + +### Email metrics and observability use cases + +Does the initial set of `email` fields need to consider observability and email monitoring use cases, for example, spam, metrics, deliverables, and logging? -Does the initial set of `email` fields need to consider observability and email monitoring use cases, for example spam, metrics, deliverables, and logging. +**Resolution**: This initial field set focuses on email message content. ### Additional event categorization values -Should a new event.category field (email) be created, and, if so, which `event.type` values the `email` category should be combined with? +Should a new event.category field (email) be created, and, if so, which `event.type` values should be used for the `email` category? + +**Resolution**: Propose to add `event.category: email` and make `info` an expected event type for the category. ### Display names -Should the display name be captured separately from the email address for senders and recipients. If so, how do we accomplish this in a document while keeping the 1:1 of a display name to email address. +Should the display name be captured separately from the email address for senders and recipients? If so, how do we accomplish this in a document while keeping the 1:1 of a display name to email address? + +**Resolution**: Initially, this proposal considered using `nested` types to allows arrays of objects containing both the email address and display name for the `to`, `cc`, and `bcc` recipients. However, after more consideration of the limitations to using `nested` fields types and limited support for `nested` fields in Kibana, that decision was reversed. ### Spam processing details Should fields intended to capture details around spam processing like sender policy framework (SPF), domainkeys identified mail (DKIM), or domain-based message authentication, reporting, and conformance (DMARC) be in scope for this proposal as well? +**Resolution**: This initial field set focuses on email message content. + @@ -317,19 +330,6 @@ The following are the people that consulted on the contents of this RFC. * @devonakerr | Co-sponsor - - - ## References @@ -349,3 +349,4 @@ e.g.: * Stage 1 (formerly proposal stage): https://github.com/elastic/ecs/pull/999 * RFC ID correction: https://github.com/elastic/ecs/pull/1157 * Stage 1 (draft): https://github.com/elastic/ecs/pull/1219 +* Stage 2 (candidate): https://github.com/elastic/ecs/pull/1593 diff --git a/rfcs/text/0010/email.yml b/rfcs/text/0010/email.yml index 2d98a88061..5049078d19 100644 --- a/rfcs/text/0010/email.yml +++ b/rfcs/text/0010/email.yml @@ -1,116 +1,210 @@ ---- - name: email title: Email group: 2 - short: Fields describing an email message. + short: Fields describing an email transaction. description: > - Email fields are used for information about an email message header and body. + Fields relating to an email transaction. + + This field set focuses on the email message header, body, and attachments. Network protocols that send and receive + email messages such as SMTP are outside the scope of the `email.*` fields. type: group fields: - - name: to + - name: attachments level: extended - type: keyword - short: Recipient address(es) + type: nested + short: List of objects describing the attachments. description: > - Stores the `to` email address(es). - example: "employee@example.com" + A list of attachment files sent along with an email message. + normalize: - array - - name: from + - name: attachments.file.extension + level: extended + type: keyword + short: Attachment file extension. + description: > + Attachment file extension, excluding the leading dot. + example: "txt" + + - name: attachments.file.mime_type + level: extended + type: keyword + short: MIME type of the attachment file. + description: > + The MIME media type of the attachment. + + This value will typically be extracted from the `Content-Type` MIME header field. + example: "text/plain" + + - name: attachments.file.name + level: extended + type: keyword + short: Name of the attachment file. + description: > + Name of the attachment file including the file extension. + example: "attachment.txt" + + - name: attachments.file.size + level: extended + type: long + short: Attachment file size. + description: > + Attachment file size in bytes. + example: 64329 + + - name: attachments.hash.md5 level: extended type: keyword - short: Sender address + short: MD5 hash of the attachment. description: > - Stores the `from` email address. - example: "administrator@example.com" + MD5 hash of the attachment file. + example: "e25f1c98ffdacf611473af364362ec48" - - name: cc + - name: attachments.hash.sha1 level: extended type: keyword - short: CC recipient(s) + short: SHA-1 hash of the attachment. description: > - The email address(es) of the carbon copy (CC) recipient(s). - example: '["cc.user@example.com", "cc.user2@example.com"]' + SHA-1 hash of the attachment file. + example: "8c1cd40f17109b427e61d4e72ca6d9a4fc8175f3" + + - name: attachments.hash.sha256 + level: extended + type: keyword + short: SHA-256 hash of the attachment. + description: > + SHA-256 hash of the attachment file. + example: "f0366b3559f577d8732f7e9cc343a4960d202e8137dcc42f9783f3963f6abc6a" + + - name: bcc.address + level: extended + type: keyword + short: Email address of BCC recipient + description: > + The email address of BCC recipient + example: "bcc.user1@example.com" normalize: - array - - name: bcc + - name: cc.address level: extended type: keyword - short: BCC recipient(s) + short: Email address of CC recipient description: > - The email address(es) of the blind carbon copy (BCC) recipient(s). - example: '["bcc.user@example.com", "bcc.user2@example.com"]' + The email address of CC recipient + example: "cc.user1@example.com" normalize: - array - - name: reply_to + - name: content_type level: extended type: keyword - short: Address for replies. + short: MIME type of the email message. description: > - The address that replies should be delivered to. - example: "user@example.com" + Information about how the message is to be displayed. - - name: timestamp + Typically a MIME type. + example: "text/plain" + + - name: delivery_timestamp level: extended type: date - short: Date and time message was sent. + short: Date and time when message was delivered. description: > - The date and time that the sender authorized delivery of the message. - - For example, by pressing a "Send" button in their email client. + The date and time when the email message was received by the service or client. example: "2020-11-10T22:12:34.8196921Z" - - name: subject + - name: direction level: extended type: keyword - short: Topic of the message + short: Direction of the message. description: > - A brief summary of the topic of the message. - example: "Status update: email fields progress" - multi_fields: - - type: match_only_text - name: text + The direction of the message based on the sending and receiving domains. + example: inbound - - name: content_type + - name: from.address level: extended type: keyword - short: Content-Type header value + short: The sender's email address. description: > - Information about how the message is to be displayed. Typically a MIME type. - example: "multipart/mixed" + The email address of the sender, typically from the RFC 5322 `From:` header field. + example: "sender@example.com" + normalize: + - array + + - name: local_id + level: extended + type: keyword + short: Unique identifier given by the source. + description: > + Unique identifier given to the email by the source that created the event. + + Identifier is not persistent across hops. + example: "c26dbea0-80d5-463b-b93c-4e8b708219ce" - name: message_id level: extended - type: wildcard - short: Unique identifier of the message + type: keyword + short: Value from the Message-ID header. description: > - Unique identifier for the email message that refers to a particular version of a particular message. + Identifier from the RFC 5322 `Message-ID:` email header that refers to a particular email + message. + example: "<81ce15$8r2j59@mail01.example.com>" - An example would be the identifier found in the `Message-ID` email header. - example: "" + - name: origination_timestamp + level: extended + type: date + short: Date and time the email was composed. + description: > + The date and time the email message was composed. Many email clients will fill in this value + automatically when the message is sent by a user. + example: "2020-11-10T22:12:34.8196921Z" - - name: direction + - name: reply_to.address + level: extended + type: keyword + short: Address replies should be delivered to. + description: > + The address that replies should be delivered to based on the value in the RFC 5322 `Reply-To:` header. + example: "reply.here@example.com" + normalize: + - array + + - name: sender.address level: extended type: keyword - short: Direction of the email message + short: Address of the message sender. description: > - The direction of the message based on the sending and received domains. + Per RFC 5322, specifies the address responsible for the actual transmission of + the message. + + - name: subject + level: extended + type: keyword + short: The subject of the email message. + description: > + A brief summary of the topic of the message. + example: "Please see this important message." + multi_fields: + - type: match_only_text + name: text - Recommended values are: - * `inbound` - From external senders to internal recipients - * `outbound` - From internal senders to external recipients - * `internal` - From internal senders to internal recipients - * `external` - From external senders to external recipients - * `unknown` - Direction is unknown + - name: to.address + level: extended + type: keyword + short: Email address of recipient + description: > + The email address of recipient + example: "user1@example.com" + normalize: + - array - - name: x_mailer - level: extended - type: keyword - short: The value from the X-Mailer header - description: > - The value from the `X-Mailer` header. Value captures what application was used - to draft and send the original email message. + - name: x_mailer + level: extended + type: keyword + short: Application that drafted email. + description: > + The name of the application that was used to draft and send the original email message. + example: "Spambot v2.5"