Feature request: more keys or dynamic keys

[dwoo4dwoo]

Hello,

The v0.14 fluentd MSI is great, much easier than the approach on v0.12, many thanks!

We've been comparing winlogbeat to fluent-plugin-windows-eventlog. One big disadvantage to fluentd is the small key/field list. Our particular use case involves the information in the event_data element for security events.

For now, I guess we're stuck with winlogbeat, but I might be able to contribute a PR in the future. Maybe a more dynamic XML to JSON approach would be feasible such that keys/fields are dynamically generated?

[repeatedly]

Which fields do you want? Sorry, I'm not familiar with winlogbeat.
We use https://github.com/djberg96/win32-eventlog library. If your wanted keys can be read from windows event, we can support it.

[dwoo4dwoo]

Hi @repeatedly,

Sorry for the delay in my response.

Which fields do you want?

We're interested in the EventData. Start > eventvwr.msc > Windows Logs > Security > Details. I've included a detailed Logon category entry below.

We use https://github.com/djberg96/win32-eventlog library

Hmm, if I understand this correctly, we need the event data from EVENTLOGRECORD :

https://msdn.microsoft.com/en-us/library/windows/desktop/aa363650(v=vs.85).aspx
https://msdn.microsoft.com/en-us/library/windows/desktop/aa363646(v=vs.85).aspx

This unfortunately is not consumed by win32-eventlog:

https://github.com/djberg96/win32-eventlog/blob/e555f0fab0a3c592e75473aaea90d60538d867c2/lib/win32/eventlog.rb#L519

It looks like someone tried to extend this in https://github.com/djberg96/win32-eventlog/pull/21/files; however, as you can see it's been languishing since June 2016.

David

<EventData>
  <Data Name="SubjectUserSid">S-1-5-18</Data> 
  <Data Name="SubjectUserName">POSSUM$</Data> 
  <Data Name="SubjectDomainName">FOOBAR</Data> 
  <Data Name="SubjectLogonId">0x3e7</Data> 
  <Data Name="TargetUserSid">S-1-5-18</Data> 
  <Data Name="TargetUserName">SYSTEM</Data> 
  <Data Name="TargetDomainName">NT AUTHORITY</Data> 
  <Data Name="TargetLogonId">0x3e7</Data> 
  <Data Name="LogonType">5</Data> 
  <Data Name="LogonProcessName">Advapi</Data> 
  <Data Name="AuthenticationPackageName">Negotiate</Data> 
  <Data Name="WorkstationName">-</Data> 
  <Data Name="LogonGuid">{00000000-0000-0000-0000-000000000000}</Data> 
  <Data Name="TransmittedServices">-</Data> 
  <Data Name="LmPackageName">-</Data> 
  <Data Name="KeyLength">0</Data> 
  <Data Name="ProcessId">0x34c</Data> 
  <Data Name="ProcessName">C:\Windows\System32\services.exe</Data> 
  <Data Name="IpAddress">-</Data> 
  <Data Name="IpPort">-</Data> 
  <Data Name="ImpersonationLevel">%%1833</Data> 
  <Data Name="RestrictedAdminMode">-</Data> 
  <Data Name="TargetOutboundUserName">-</Data> 
  <Data Name="TargetOutboundDomainName">-</Data> 
  <Data Name="VirtualAccount">%%1843</Data> 
  <Data Name="TargetLinkedLogonId">0x0</Data> 
  <Data Name="ElevatedToken">%%1842</Data> 
</EventData>

[igorpeshansky]

Doesn't win32-eventlog retrieve EventData as string_inserts? I thought I added that support here in #7...

[dwoo4dwoo]

Hi @igorpeshansky,

Thanks a lot for your reply. I reviewed your PR and it seems to leverage https://github.com/djberg96/win32-eventlog/blob/ffi2/lib/win32/eventlog.rb#L580. If I understand this correctly ... an array of insertion strings is returned:

• https://www.eventsentry.com/blog/2008/04/event-log-message-files-the-de.html
• https://msdn.microsoft.com/en-us/library/windows/desktop/aa363651%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396

I upgraded fluent-plugin-windows-eventlog to v0.2.1 and I now see the insertion strings:

"string_inserts": [
  "S-1-5-18",
  "EC2AMAZ-GBRRAPG$",
  "WORKGROUP",
  "0x3e7",
  ....

However, they aren't terribly useful, because they have no structure. It's a shame that FormatMessage (?) wasn't used to generate the keys. The description element has a similar problem in that's it's just a big blob of text (new lines added for readability):

 "description": "An account was successfully logged on.\r\n\r\n
 Subject:\r\n\t
 Security ID:\t\tS-1-5-18\r\n\t
 Account Name:\t\tEC2AMAZ-GBRRAPG$\r\n\t
 Account Domain:\t\tWORKGROUP\r\n\t
 Logon ID:\t\t0x3e7\r\n\r\n
 [ ... snip ... ]

As I mentioned earlier, for now we ended up going with winlogbeat, because out of the box the data is well structured. It seems they chose to go down the XML path: https://github.com/elastic/beats/blob/c821b84cf55f88778c9702a60aea52c52d5643d7/winlogbeat/eventlog/wineventlog.go#L80. If you keep following you'll eventually reach https://msdn.microsoft.com/en-us/library/windows/desktop/aa385471%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396. I don't know what the answer is here ... but the unstructured data is a show stopper for us.

David

[TaLoN1x]

Tryied to implement fluentd for windows environment and hit the same problem.

[repeatedly]

parse_description true is not fit for you?

https://github.com/fluent/fluent-plugin-windows-eventlog#parse_description-details

[TaLoN1x]

string_inserts and description fields are not structure, that makes pattern analysis and event correlation very complicated, requiring additional parsing, processing etc.

If there would be just a way to structure them... Basically Logs collected by Fluentd windows version cannot be processed by most SIEM, IPS/IDS solutions without additional adapter or extensive parsing rules.

I do understand that it's mostly win32-eventlog problem though...
I would say, that win32-eventlog is unfortunately weak on windows events. In addition to this problem there is still very severe issue: chef/win32-eventlog#25

We've tried to fork and fix win32-eventlog, but the effort looks to be just too big... :(
win32-eventlog is neither actively supported.

Sorry, this comment might be very emotional. I was trying to get it working for last 4 months, without any good success.

[TaLoN1x]

If I am correct with my digging the problem consists in parsing of following structures:
https://docs.microsoft.com/en-us/windows/desktop/api/winnt/ns-winnt-_eventlogrecord
https://docs.microsoft.com/en-us/windows/desktop/api/winnt/ns-winnt-security_descriptor
https://docs.microsoft.com/en-us/windows/desktop/api/winnt/ns-winnt-cache_descriptor

[repeatedly]

string_inserts and description fields are not structure,

With parse_description true, description values are converted into proper keys. See my link.
I'm not familiar with Windows, so if description doesn't have all events or values are delimited by non popular separater, parse_description is not fit.
Yeah maybe, need to improve win32-eventlog gem to get all fields.