[RFC] Wildcard - stage 3 proposal

[ebeahan]

Summary

Revisions to the stage two wildcard RFC for consideration of advancement for stage three.

Opening this PR with minimal changes. Feedback and concerns as the wildcard changes are adopted can be captured in this PR and incorporated into the proposal as appropriate.

Criteria for consideration

• Opened pull request for this candidate revising the existing draft
• Completed field definitions
• Included multiple real world example source documents
• Existing or newly raised questions and concerns are addressed

Markdown preview of this RFC

[rw-access]

I saw a few typos. These say "leading wildcard" but the example is a trailing wildcard: foo*

ecs/rfcs/text/0001-wildcard-data-type.md

Lines 148 to 149 in e9b9681

	| Leading wildcard queries on low-cardinality fields (foo*) | Fast | Slower (see *3) |
	| Leading wildcard queries on high-cardinality fields (foo* ) | Terrible | Much faster |

Inconsistent escaping, should do \\ for all escaped backslashes.

ecs/rfcs/text/0001-wildcard-data-type.md

Line 182 in e9b9681

* Match under registry path: `registry.path:\\HKLM\\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\*\Debugger`

Similarly, I think these should be escaped with \ not /. I don't think this is valid syntax:

ecs/rfcs/text/0001-wildcard-data-type.md

Line 241 in e9b9681

process.command_line:*f/ foo* and process.command_line:*/b bar*

Worth double checking if this is valid syntax. I'm not 100% sure if \ can escape whitespace.
process.command_line:*f\ foo* and process.command_line:*/b\ bar*

[webmat]

We should remove user.domain, in line with not including host.domain

[webmat]

We could add a link to the wildcard docs in the references https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#wildcard-field-type

[webmat]

Whoa, I just came across this process.thread.name:

"process.thread.name":"TC: 14:05:53 Stage: Initializing Startup subsystem; TeamCity Server (version 2020.1.4 build 78906 started 2020-10-21 14:05:53.422), node id: MAIN_SERVER"

And here I thought people were as imaginative as me, calling their threads "thread1", "thread2", "worker1" and so on.

I think we should make this field wildcard as well.

[ebeahan]

@webmat - I, as well, must not be very imaginative 😂

I've captured process.thread.name in the list.

[webmat]

Just like I mentioned in today's meeting, I think we should make obvious that this RFC is solely about the migration of pre-existing keyword fields to wildcard.

Using wildcard right away on net new fields is totally fine, IMO. WDYT?

[ebeahan]

Yes agreed! I've actually made the suggestion to consider wildcard for net new fields in other proposals, so I'm glad we're aligned 😉 😂

I will adjust the RFC's messaging to make it clear this RFC is documenting the initial proposal focused on existing fields, but any net new fields should consider using wildcard whenever appropriate.

[ebeahan]

I made some adjustments to the title and intro trying to better focus on the migration over wildcard's general presence in ECS: 3f2eaaa (#1015)

I'm happy to flesh this out with more detail if that feels appropriate.

[webmat]

Noted a few things around the recent changes 👍

[ebeahan]

PR to apply wildcard migration changes in progress #1098

[ebeahan]

Including event.original as a candidate field for wildcard may need reconsideration.

The event.original field in ECS is currently keyword with indexing disabled. Valid concerns were raised (#970 (comment)) to continue disabling indexing on event.original to avoid any significant impact on indexing. Some data sources will produce original events that are very, very large.

Wildcard is part of the keyword family. Unlike other keyword data types, it doesn't support disabling either indexing or doc values by design. Trying to set either option to false will be rejected by the API:

PUT wildcard-index
{
  "mappings": {
    "properties": {
      "wildcard": {
        "type": "wildcard",
        "index": false
      }
    }
  }
}

{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "The field [wildcard] cannot have index = false"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "Failed to parse mapping [_doc]: The field [wildcard] cannot have index = false",
    "caused_by" : {
      "type" : "mapper_parsing_exception",
      "reason" : "The field [wildcard] cannot have index = false"
    }
  },
  "status" : 400
}

A simple path may be to omit event.original as a candidate for wildcard and have it remain keyword with index: false.

@webmat @leehinman Any thoughts?

[webmat]

I think we're good on the current list of fields to migrate. This LGTM other than a few small adjustments.

Nitpick: we're missing organization.name in the table :-)

I'm good with reverting event.original to keyword with index: false for now, as the way forward. We could adjust the field definition in #1098 to say it's currently not indexed, and that if users want to override this and make it indexed, they should consider wildcard.

[ebeahan]

Latest round of changes:

• Added missing organization.name to the fields table
• Removed event.original as a wildcard candidate field
• Corrected formatting and incorporated peer suggestions.

[webmat]

This looks good, let's change that date and merge 👍

RFC 0001 is also the first to reach stage 3 🎉 😃