parse_description in windows_eventlog2 is broken

[ichintu]

I think parse_description true is broken. My config:

<source>
  @type windows_eventlog2
  @id windows_eventlog2
  channels Windows PowerShell,Microsoft-Windows-Sysmon/Operational,Security
  tag winevt.raw
  render_as_xml true
  parse_description true
  read_existing_events false
  <storage>
    @type local
    persistent true
    path C:\opt\td-agent\winlog.json
  </storage>
</source>

<match winevt.raw>
  @type file
  path C:\logs_out\fluentd\winlog
  # compress gzip
  <format>
    @type json
  </format>
  <buffer>
    timekey 5m
    timekey_use_utc true
    timekey_wait 1m
  </buffer>
</match>

What i am expecting is something that looks like this:

{
  "ProviderName": "Microsoft-Windows-Security-Auditing",
  "ProviderGUID": "{D441060A-9695-472B-90BC-24DCA9D503A4}",
  "EventID": "4798",
  "Qualifiers": "",
  "Level": "0",
  "Task": "13824",
  "Opcode": "0",
  "Keywords": "0x8020000000000000",
  "TimeCreated": "2019-06-19T03:10:01.982940200Z",
  "EventRecordID": "87028",
  "ActivityID": "",
  "RelatedActivityID": "{2599DE71-2F70-44AD-9DC8-C5FF2AE8D1EF}",
  "ThreadID": "16888",
  "Channel": "Security",
  "Computer": "DESKTOP-TEST",
  "UserID": "",
  "Version": "0",
  "DescriptionTitle": "A user's local group membership was enumerated.",
  "subject.security_id": "S-X-Y-Z",
  "subject.account_name": "DESKTOP-TEST$",
  "subject.account_domain": "WORKGROUP",
  "subject.logon_id": "0x3e7",
  "user.security_id": "S-XXX-YYY-ZZZ",
  "user.account_name": "Administrator",
  "user.account_domain": "DESKTOP-TEST",
  "process_information.process_id": "0xbac",
  "process_information.process_name": "C:\\Windows\\System32\\svchost.exe"
}

However what I am getting is:

{
  "ProviderName": "Microsoft-Windows-Sysmon",
  "ProviderGUID": "{5770385f-c22a-43e0-bf4c-06f5698ffbd9}",
  "EventID": "1",
  "Qualifiers": "",
  "Level": "4",
  "Task": "1",
  "Opcode": "0",
  "Keywords": "0x8000000000000000",
  "TimeCreated": "2023-03-06T18:15:15.560133900Z",
  "EventRecordID": "7154",
  "ActivityID": "",
  "RelatedActivityID": "",
  "ProcessID": "2936",
  "ThreadID": "4880",
  "Channel": "Microsoft-Windows-Sysmon/Operational",
  "Computer": "W10-SANDY-001",
  "UserID": "S-1-5-18",
  "Version": "5",
  "DescriptionTitle": "Process Create:\r\nRuleName: \r\nUtcTime: 2023-03-06 18:15:15.558\r\nProcessGuid: {84faa657-2db3-6406-0000-0010e59d5202}\r\nProcessId: 1336\r\nImage: C:\\opt\\td-agent\\bin\\ruby.exe\r\nFileVersion: 2.7.6p219\r\nDescription: Ruby interpreter (CUI) 2.7.6p219 [x64-mingw32]\r\nProduct: Ruby interpreter 2.7.6p219 [x64-mingw32]\r\nCompany: http://www.ruby-lang.org/\r\nOriginalFileName: ruby.exe\r\nCommandLine: C:\\opt\\td-agent\\bin\\ruby.exe -Eascii-8bit:ascii-8bit -h\r\nCurrentDirectory: c:\\opt\\td-agent\\\r\nUser: W10-SANDY-001\\Johnny Douche\r\nLogonGuid: {84faa657-1673-6405-0000-0020b4830100}\r\nLogonId: 0x183B4\r\nTerminalSessionId: 1\r\nIntegrityLevel: High\r\nHashes: MD5=949C42C68A780AB3AB2EE4730B56230D,SHA256=4815AC3DA6906A52031D0A21314493F66635D717125D644CFB3784396838F7DC\r\nParentProcessGuid: {84faa657-2dae-6406-0000-0010bf854f02}\r\nParentProcessId: 5988\r\nParentImage: C:\\opt\\td-agent\\bin\\ruby.exe\r\nParentCommandLine: \"C:\\opt\\td-agent\\bin\\ruby.exe\"  \"C:\\opt\\td-agent\\bin\\fluentd\" -c etc\\td-agent\\td-agent.conf"
}

Am I doing something wrong or something broken with the parser?

[ichintu]

I wanted to share of what happens when i set parse_description false
so the configuration is definitely trying to parse but its not parsing properly.

{
  "ProviderName": "Microsoft-Windows-Sysmon",
  "ProviderGUID": "{5770385f-c22a-43e0-bf4c-06f5698ffbd9}",
  "EventID": "1",
  "Qualifiers": "",
  "Level": "4",
  "Task": "1",
  "Opcode": "0",
  "Keywords": "0x8000000000000000",
  "TimeCreated": "2023-03-06T17:32:19.497163500Z",
  "EventRecordID": "7137",
  "ActivityID": "",
  "RelatedActivityID": "",
  "ProcessID": "2936",
  "ThreadID": "4880",
  "Channel": "Microsoft-Windows-Sysmon/Operational",
  "Computer": "W10-SANDY-001",
  "UserID": "S-1-5-18",
  "Version": "5",
  "Description": "Process Create:\r\nRuleName: \r\nUtcTime: 2023-03-06 17:32:19.495\r\nProcessGuid: {84faa657-23a3-6406-0000-0010d577d201}\r\nProcessId: 7668\r\nImage: C:\\opt\\td-agent\\bin\\ruby.exe\r\nFileVersion: 2.7.6p219\r\nDescription: Ruby interpreter (CUI) 2.7.6p219 [x64-mingw32]\r\nProduct: Ruby interpreter 2.7.6p219 [x64-mingw32]\r\nCompany: http://www.ruby-lang.org/\r\nOriginalFileName: ruby.exe\r\nCommandLine: \"C:\\opt\\td-agent\\bin\\ruby.exe\"  \"C:\\opt\\td-agent\\bin\\fluentd\" -c etc\\td-agent\\td-agent.conf\r\nCurrentDirectory: c:\\opt\\td-agent\\\r\nUser: W10-SANDY-001\\Johnny Douche\r\nLogonGuid: {84faa657-1673-6405-0000-0020b4830100}\r\nLogonId: 0x183B4\r\nTerminalSessionId: 1\r\nIntegrityLevel: High\r\nHashes: MD5=949C42C68A780AB3AB2EE4730B56230D,SHA256=4815AC3DA6906A52031D0A21314493F66635D717125D644CFB3784396838F7DC\r\nParentProcessGuid: {84faa657-02a1-6406-0000-0010ffd05401}\r\nParentProcessId: 1204\r\nParentImage: C:\\Windows\\SysWOW64\\cmd.exe\r\nParentCommandLine: \"C:\\Windows\\SysWOW64\\cmd.exe\" /k \"C:\\opt\\td-agent\\td-agent-prompt.bat\"",
  "EventData": [
    "",
    "2023-03-06 17:32:19.495",
    "{84FAA657-23A3-6406-0000-0010D577D201}",
    "7668",
    "C:\\opt\\td-agent\\bin\\ruby.exe",
    "2.7.6p219",
    "Ruby interpreter (CUI) 2.7.6p219 [x64-mingw32]",
    "Ruby interpreter 2.7.6p219 [x64-mingw32]",
    "http://www.ruby-lang.org/",
    "ruby.exe",
    "\"C:\\opt\\td-agent\\bin\\ruby.exe\"  \"C:\\opt\\td-agent\\bin\\fluentd\" -c etc\\td-agent\\td-agent.conf",
    "c:\\opt\\td-agent\\",
    "W10-SANDY-001\\Johnny Douche",
    "{84FAA657-1673-6405-0000-0020B4830100}",
    "0x00000000000183b4",
    "1",
    "High",
    "MD5=949C42C68A780AB3AB2EE4730B56230D,SHA256=4815AC3DA6906A52031D0A21314493F66635D717125D644CFB3784396838F7DC",
    "{84FAA657-02A1-6406-0000-0010FFD05401}",
    "1204",
    "C:\\Windows\\SysWOW64\\cmd.exe",
    "\"C:\\Windows\\SysWOW64\\cmd.exe\" /k \"C:\\opt\\td-agent\\td-agent-prompt.bat\""
  ]
}

[AmeOnCoffee]

Same issue here, using Fluentd and Loki. doesnt seem to be fixed

[landon-lengyel]

So I've gotten this to work, but it's using some undocumented options (mainly the winevt_sax)
I unfortunately don't remember how I learned about this option. Probably some forum somewhere.

Obviously I'm using the YAML format, you'll have to convert to the XML:

config:
  - source:
      $type: windows_eventlog2
      #channels: security,system,application
      subscribe:
        channels: security,system,application
        #read_existing_events: true
      tag: winevt
      storage:
        persistent: true
        path: C:\opt\fluent\etc\fluent\winevt.json
      # Break out EventData into it's own fields
      parse_description: true
      preserve_qualifiers_on_hash: true
      parse:
        $type: winevt_sax
        preserve_qualifiers: true

With this option I get what you were describing, where EventData is broken out into it's own fields.

[BlakeHensleyy]

@landon-lengyel Out of curiousity, what do you use EventData for if there aren't fields assigned to them? In all of my usage EventData is only able to extract the field values.

[landon-lengyel]

@BlakeHensleyy Are you referring to the field names? FluentD does extract them for me with that config I provided.
For example, with Security EventID 4648 FluentD has extracted the following keys for me that were stored in EventData:

• subject.logon_guid
• subject.security_id
• process_information.process_name
• subject.account_domain

And so forth. It does seem to rename them in the sense that it's not the identical name that appears in Event Viewer > Details > XML View. I'm not sure where it gets the names it chooses.

[BlakeHensleyy]

@landon-lengyel I see what you mean now. So these fields are parsed from the description, for EventID 4648 the description is:

A logon was attempted using explicit credentials.

Subject:
   Security ID:  WIN-R9H529RIO4Y\Administrator
   Account Name:  Administrator
   Account Domain:  WIN-R9H529RIO4Y
   Logon ID:  0x1ba0e
   Logon GUID:  {00000000-0000-0000-0000-000000000000}
Account Whose Credentials Were Used:
   Account Name:  [[email protected]](mailto:[email protected])
   Account Domain:  WIN-R9H529RIO4Y
   Logon GUID:  {00000000-0000-0000-0000-000000000000}
Target Server:
   Target Server Name: sp01.IceMAIL.com
   Additional Information: sp01.IceMAIL.com
Process Information:
   Process ID:  0x77c
   Process Name:  C:\Program Files\Internet Explorer\iexplore.exe
Network Information:
   Network Address: -
   Port:   -

The fields and field values are then parsed from the description. For example with the category(parent key) being Subject, the child keys for Subject is:

• subject.security_id
• subject.account_name
• subject.account_domain
• subject.logon_id
• subject.logon_guid

From what I can tell there isn't a way to actually get the XML field names, just the values which are stored in the EventData field.

[landon-lengyel]

I see, I must have misunderstood it but that does make sense. Thank you for finding that!
I think that meets most of my needs. Still unfortunate that it can't gather that additional data, for those who need it.

I know when I experimented with getting that EventData in PowerShell and C# it was a pain.