[RFC] Introduce email field set - stage 2

[ebeahan]

Summary

Continuing onto Stage 2 with this proposal to introduce the email.* field set to the schema.

Stage 2 (Candidate) Criteria:

• Opened pull request for this draft revising the existing proposal
• Completed field definitions
• Included a real-world example source document
• Identifies scope of impact of changes to ingestion mechanisms (e.g., beats/logstash), usage mechanisms (e.g., Kibana applications, detections), and the ECS project (e.g., docs, tooling)
• Subject matter experts weighed in on the technical utility of field definitions in the pull request

Preview of markdown proposal

[ebeahan]

Opening PR to capture any feedback or suggestions around the proposed set of email.* fields.

[wasserman]

In regards to Display Name, you could consider some of these options:

1. Allow for name/address pairs in all email fields instead of just the email address itself.
2. Permit RFC 5322 Email formats like First Last <first.last@example.com>. Then a keyword with a normalizer could index just the emails while still keeping the display names intact. Of course the names wouldn't be searchable.
3. Multi-fields with some variation of this could work and allow for some form of the keyword and text fields.

I personally ran with option 2. I was hoping to leverage the uax_url_email tokenizer, but I had to settle for a simple regex.

"normalizer": {
    "email_normalizer": {
      "type": "custom",
      "char_filter": [
        "email_filter"
      ]
    }
  },
  "char_filter": {
    "email_filter": {
      "type": "pattern_replace",
      "pattern": ".*?<([^>]+)>",
      "replacement": "$1"
    }
  }

[ebeahan]

Proposed fields now include arrays of objects with both the email address and display name for the to, cc, and bcc recipients. The display_name and address fields have been added under email.from.

Field name	Data type
email.from.address	keyword
email.from.display_name	keyword
email.to	nested
email.to.address	keyword
email.to.display_name	keyword
email.subject	keyword
email.cc	nested
email.cc.address	keyword
email.cc.display_name	keyword
email.bcc	nested
email.bcc.address	keyword
email.bcc.display_name	keyword

[djptek]

LGTM!

As an aside, when this is merged I'd like to use it as an example to update the docs around nested type, as the difference between email.from and email.to illustrates this perfectly

[ebeahan]

The limitations around building visualizations using type nested fields make me question if using nested for the various email sender/recipient fields is the best direction.

I'm going to think this over a bit more and iterate on the proposed fields.

[djptek]

@ebeahan re nested type - for the legal use case nested type is not necessary as that would center on *.address

Regarding visualisation of emails, we'd probably need a Chord or Sankey diagram - a quick look in Kibana Issues does not have those on the radar at this point in time though there is a Sankey example using Vega on the Blog - again this would probably center on *.address. so not nested.

There might also be value in running address and display name through ML, that would probably be incompatible with nested type

The only use case off the top of my head where we might want nested might be Spoof detection, however, given the cardinality this would probably be best done using ML for the heavy lifting and then manual inspection of anomalies, so you could work around that

@peasead Do you have a specific use case/query/aggs in mind where we'd need to leverage nested type?

[ebeahan]

@jamiehynds as the sponsor, can you take a look at how the email.* fields proposal is shaping up?

[peasead]

@peasead Do you have a specific use case/query/aggs in mind where we'd need to leverage nested type?

Thanks for your patience.

@djptek I don't have anything specific. I wasn't sure if there'd be a use case to query nested objects independently, but thinking more, I'm not sure that'd be needed.

[wasserman]

FYI, nested was just thought to be a useful way to preserve the relationships between display names and emails. Multi-fields or a normalizer could work too. Ultimately any smart way to not lose the display names in the process since it could be valuable just to be able to see, if nothing else.

[djptek]

LGTM!

[djptek]

Thanks @wasserman

nested was just thought to be a useful way to preserve the relationships between display names and emails

Looking at one use case that leverages this relationship, e.g. checking for spoofing of a known address, we'd need to involve additional data - a table/index defining the a priori relationship between a defined and countable set of address(es) and the legitimate display_name(s) for these address(es). This would require additional logic, including a join against that table/index. Elasticsearch joins are generally best implemented at ingest time rather than query time, so this use case could perhaps be addressed by building this into the ingest pipeline, or by reindexing a subset of data related to specific address(es) of interest.

Conversely, where the relationship between display_name and address is not explicitly defined a priori, there is no upper limit to the number of address(es) in the related use cases so it may be preferable to avoid nested_type to ensure the most performant solution.

[ebeahan]

While implementing the email.* field set into the schema, reusing hash.* at email.attachments.file.hash.* felt like a better approach that aligns with the existing file.hash.* fields and adds a few additional hash fields for any attachment: #1688 (comment).

I'll capture these details fully in the proposal doc during stage 3.