Add syslog protocol fields to event

[kvch]

This PR adds the following fields to ECS:

• event.facility
• event.priority
• event.facility_label
• event.priority_label

[webmat]

It's been bugging me that we didn't have specific support for Syslog yet, so thanks for submitting this :-)

@ruflin @MikePaquette What do you think about one more top level field set? Should we nest this under log? (log.syslog.*)

[webmat]

@kvch Oh, and your Hunglish is fine ;-)

[kvch]

If you don't want to add a new top-level field, I suggest these are added to log.*. These concepts are strongly tied to logging in general. Both values are syslog components specified by syslog RFCs. Also, they are added to log messages by syslog daemons.

[willemdh]

Personally I'd love to see a spot for the syslog pri too, see #36

[ruflin]

We should evaluate if we can fit these into log.*. If not, it seems syslog would be the first one that fits under a new category we could call service or similar like we discussed for protocols in the context of http.

[willemdh]

@ruflin Putting it under log.* is ok for me, as long as it has a place somewhere in ecs.

[webmat]

I agree with adding syslog's priority as well. Any other attributes we should consider?

I prefer to see these syslog details under log. than under service.. We haven't discussed the semantics of how service. would work (at least I don't remember). So in that sense, nesting these details under log. seems much more straightforward.

[willemdh]

Ok, seem most kind of agree it needs to be under log and we need facility, severity and priority. Only choice left is if we should place it directly under log like this:

log.facility
log.priority
log.severity

or, considering these are really syslog specific place them under log.syslog like this:

log.syslog.facility
log.syslog.priority
log.syslog.severity

To me the log.syslog prefix makes more sense. I think there might be other candidats with similar requirements in the future. (such as certain java or .net logging frameworks).

[webmat]

I'd be curious to know which other tool use facility/priority/severity. And are the semantics close enough to Syslog's to map them to the same fields?

For now I have a slight preference for log.syslog.*

[ph]

from my understanding, they are really close the Syslog reality, so I think would prefer to have it under log.syslog.*

[MikePaquette]

Note: outlier position ahead :-)

I think burying these syslog fields down under log.* or syslog.* defeats the purpose of trying to make a common schema. I think they would fit better under event.*

Reason: in ECS, the "event" is the top level entity. "logs" and "metrics" are sub-entities (See event field definition).

Despite the name, "syslog" is just a transport mechanism for events. In fact syslog can be used for any key-value pair, including metrics. For example, Pivotal allows you to send container metrics over the syslog protocol.

So nesting syslog fields below log.* does not seem correct, just as it would not be correct to nest it under metric.* if ECS had a "metric" object (which it doesn't).

So I'd propose that we create new fields for event.priority and event.facility and map as follows:

syslog severity -> event.severity (field already defined by ECS)
syslog priority -> event.priority (proposed new field)
syslog facility -> event.facility (proposed new field)

@webmat ArcSight CEF has a deviceFacility field that would also map to an ECS event.facility
Splunk CIM has several related fields including message_priority which would map to event.priority and severity would would map to event.severity

[willemdh]

Personally I've wondered several times why there still is a log top level object. Isn't a log some sort of event? And why would log. Original not fit into event. Original? And log. Level under event. Level or event. Severity?

[webmat]

@willemdh Yeah currently log. is used for lower level details specific to log files. Filebeat (in 7) will also be nesting a few more details here that aren't ECS specific, like the file offset.

@MikePaquette The reason I was proposing with log.syslog.* was because I want to avoid the event field set to become overcrowded. But you're right that these attributes don't necessarily have to be specific to log files, they are applicable to anything else.

I agree to go with event.[facility|priority|severity].

Our description for these fields should mention the relationship Syslog, but shouldn't limit to Syslog. These values should be populated per event source, and only need to be consistent within that event source (they shouldn't be normalized vs other sources populating the same fields).

[kvch]

I have moved the fields under the namespace event.
The new field names need to be followed up in Beats.

[webmat]

@MikePaquette Revisiting this, I'd like to close this loop soon.

Another worry I have with adding them directly at event.priority, event.priority_label, event.facility and event.facility_label is that we've been moving in the direction of normalizing the content of some of the other event fields.

In that context, what do we do with the Syslog ones? Should we recommend people populate the fields with the Syslog values as is? And what about other event sources that have attributes similar to severity, priority and facility? Should they be recorded there as is, too?

[MikePaquette]

Another worry I have with adding them directly at event.priority, event.priority_label, event.facility and event.facility_label is that we've been moving in the direction of normalizing the content of some of the other event fields.

True for event categorization fields, but I think these don't fall strictly into that category.

In that context, what do we do with the Syslog ones? Should we recommend people populate the fields with the Syslog values as is?

Yes, I think that's fine. Within a syslog transported event stream, the set should be consistent.

And what about other event sources that have attributes similar to severity, priority and facility? Should they be recorded there as is, too?

Yes, but you raise a good question about event.severity however that is beyond the scope of this PR, as it is already an ECS field.

[zlammers]

In addition to the syslog fields mentioned above, what about the syslog timestamp and syslog host?

In many major vendors (Fortinet, Checkpoint, Palo Alto come to mind), there are log aggregators often in the mix, like a FortiAnalyzer, MLM, or Panorama. Forwarded syslog often contains the TS and host of the log aggregator itself, not of the original [observer] host (or if it does, it contains both, where you'd see "TS HOST TS HOST MESSAGE" where the first pair of TS HOST is a log fowarder).

We are currently working on syslog -> ECS parsers for major FW vendors (the above, and Cisco and Juniper), and as ECS was missing fields contained in this PR (plus others), we had decided to just 'store in labels' until we have a more official place (which sounds like it will be at least partially 'event.*' at this time):

labels.syslog_host (keep labels.syslog_host value even if that value is utilized for observer.hostname [notably Cisco, Juniper])
labels.syslog_timestamp (keep labels.syslog_timestamp value even if that value is utilized for @timestamp [notably Cisco, Juniper])
labels.syslog_priority
labels.syslog_facility 
labels.syslog_facility_label
labels.syslog_severity 
labels.syslog_severity_label
labels.syslog_forwarder_host
labels.syslog_forwarder_timestamp

[micoq]

Since the facility, the priority are specific to syslog (in it's header), we use a separate syslog namespace:

syslog.pri: the complete priority code between `<` and `>`)
syslog.facility.code: the numeric identifier of the facility decoded from the priority code
syslog.facility.label: normalized label (`local0`,`kernel`...)
syslog.severity.code: the numeric identifier of the severity decoded from the priority code
syslog.severity.label: normalized label (`emerg`,`warning`...)
syslog.hostname: the hostname in the syslog header
syslog.rfc: which version of the RFC (5424 or 3164)
syslog.date.raw: the raw date/timestamp in the header

We can add some fields to store the tag (the process name):

syslog.tag.process: the name of the process (httpd)
syslog.tag.pid: the pid (if specified: `httpd[4242]`)

However, these fields can be redundant and are not already defined since they are considered as a part of the user message.

[webmat]

@micoq I like your .code and .label layout :-) However since we already have event.severity defined in ECS, I'm not sure we should introduce a breaking change to move it to event.severity.code, unfortunately.

So I think adding the following fields is straightforward:

• event.priority
• event.facility
• event.facility_label

There's two issues I still see here:

1. I'm not sure I would add event.severity_label: values expected here are equivalent to what we have in log.level right now (debug, info, warn, error, etc). I think I would rather adjust the description of event.severity to say its corresponding label must be stored in log.level. Or perhaps we could instead deprecate log.level and move it under event. eventually?

2. This PR is adding priority_label, but I'm not sure there's such a thing as priority label? severity and facility both have a corresponding _label, but priority doesn't, afaik.

What do you folks think?

[MikePaquette]

Fields LGTM
PR Approved with suggested changes to definitions.

[MikePaquette]

Enhance this definition to something like: "The facility or process from which the event originated. Typically associated with syslog facility for events adhering to RFC 5424 or RFC 3164."

[MikePaquette]

Enhance this definition to something like: "The priority of the event. Typically associated with syslog priority for events adhering to RFC 5424 or RFC 3164."

[MikePaquette]

Rather than "Human readable", can we add "text-based" or something similar? Humans can read numbers too :-)

[MikePaquette]

Rather than "Human readable", can we add "text-based" or something similar? Humans can read numbers too :-)

[webmat]

The Informational example here is an example of a severity label (which maps to log.level in ECS).

I don't think there is such a thing as a priority label in Syslog. The priority number is the one made up of 8*facility + severity.

I think we need to remove this field.

[webmat]

Just opened a new PR to potentially replace this one. See details and rationale over there 👉#525 :-)