Winlogbeat misses some events with Forwarded Events

[dandrestor]

I am running Winlogbeat 5.2.0 x64 on Windows 2012 R2. This windows machine (henceforth "the collector") is receiving forwarded event logs from various sources. I have isolated one remote machine, sending only one event log (Security), for testing purposes, in a separate event log on the collector.

Winlogbeat is dropping some of these events. I have noticed that the number of events written by Winlogbeat is consistently about 95% of the actual number of events received through Windows Event Forwarding. I see no errors in the Winlogbeat logs.

I understand this description is not very detailed, but I'm not sure what other information I could provide. If anybody more knowledgeable picks this up however, I'm happy to contribute with any testing that might be required.

[dandrestor]

Possibly related: https://discuss.elastic.co/t/winlogbeat-ignores-registry-file-with-forwarded-events/76601/5

[andrewkroh]

I believe that the problem is with the way Winlogbeat stores the "read pointer" for the Forwarded Events. We need to make this enhancement to fix it.

[paddibr]

same error here on Windows Server 2012 R2 with forwarded events

[andrewkroh]

One workaround until this is fixed is to to use Logstash with fingerprint filter to create the _id field based on the event's @timestamp + computer_name + log_name + record_number. (@timestamp comes from the event record and not the time Winlogbeat read the event so it's safe to use in a fingerprint.)

Then elasticsearch will deduplicate the data because each unique event has only one possible _id.

[dandrestor]

@andrewkroh I saw you changed the title, and wanted to double-check that we are speaking about the same thing here. I opened this issue not because Winlogbeat will lose events when starting up (resuming), but rather during normal operation. The issue with the resume we discussed earlier is possibly related, but not the same problem.
If these are in fact two separate problems, maybe it's worth changing the title again to something more suggestive. Also, would your workaround still apply in this case?

Edit: spelling

[PeterZuge]

I am wondering on the status of this bug (Winlogbeat doesn't resume correctly with Forwarded Events #3731). I can see it is still marked open, but I was hoping it might be picked up on a roadmap or has a date the bug would be likely fixed.

I want to use the builtin Windows Eventlog Collection capabilities for collecting workstation events, and then use WinLogBeat to ship them over to our ELK stack. This bug would directly impact that (though I will look into the fingerprint filter suggestion).

Much thanks to everyone involved in this project. WinLogBeat/FileBeat really are amazingly useful utilities!

[dandrestor]

@PeterZuge we changed to nxlog for the time being.

[dandrestor]

Hi, I was wondering what's the status of this bug? This makes Winlogbeat unsuitable for production use pretty much, and forced us to look to other agents such as nxlog.

[andrewkroh]

With the changes made in #6150 Winlogbeat should now be able to persist and resume state correctly for the ForwardedEvents log. This will be released in v6.3.0.

If anyone wants to test the changes before v6.3.0 is released you can use the snapshot build. If you test please leave some feedback here.

[uncletimmy3]

I have the same problem.
I tried to use the snapshot build, but it didn't help me. Winlogbeat is still dropping events from one of my DC.

[andrewkroh]

@uncletimmy3 Could you please explain a bit more about the conditions under which events were dropped during your testing?

[uncletimmy3]

@andrewkroh
Six domain controllers (DCs) send their events from security log to a windows event collector(WEC). DC names (may be it matters): BSTDC1, BSTDC2, SD1DC1, SD1DC2, SD2DC1, SD2DC2. Event rate from each DC is approximately equal to 40000-50000 events per hour. WEC puts events in ForwardedEvents log.
When I start or restart winlogbeat, winlogbeat works correctly, all events appear in Elastic/Kibana and output file, which I've created for test. After winlogbeat run for a while, new events from SD2DC2 don't appear at all in Elastic/Kibana and output file.

Winlogbeat conf:
winlogbeat.event_logs:
- name: ForwardedEvents
ignore_older: 2h
tags: ["sysevents"]
output.logstash:
hosts: ["vm-logstash1:5066"]
max_retries: -1
logging.level: info
logging.to_files: true
logging.files:
name: winlogbeat.log
keepfiles: 5
rotateeverybytes: 20971520

output.file:
path: "C:/etc/winlogbeat-7.0.0/output"
filename: winlogbeat_output
rotate_every_kb: 524288
number_of_files: 7

Subscription conf:
>wecutil gs "DC Security Log"
Subscription Id: DC Security Log
SubscriptionType: SourceInitiated
Description:
Enabled: true
Uri: http://schemas.microsoft.com/wbem/wsman/1/windows/EventLog
ConfigurationMode: MinLatency
DeliveryMode: Push
DeliveryMaxLatencyTime: 30000
HeartbeatInterval: 3600000
Query: *
ReadExistingEvents: false
TransportName: HTTP
ContentFormat: RenderedText
Locale: ru-RU
LogFile: ForwardedEvents
PublisherName: Microsoft-Windows-EventCollector
AllowedIssuerCAList:
AllowedSubjectList:
DeniedSubjectList:
AllowedSourceDomainComputers: O:NSG:BAD:P(A;;GA;;;DD)S:

[uncletimmy3]

My bad... I tried to collect events only from SD2DC2, and nothing changed. Probably there are some problems with SD2DC2

[uncletimmy3]

It's strange but everything began to work normally when I deleted "ignore_older:" option.
Time is synchronized between domain controllers

[andrewkroh]

@uncletimmy3 Interesting. That's the first time I've heard of an issue with ignore_older. This feature creates a subscription using a XML query. You can do this in the Windows Event Viewer by creating a Filter and selecting the XML tab. I wonder if you see the same thing in a event viewer? Our XML query looks like

<QueryList>
  <Query Id="0">
    <Select Path="ForwardedEvents">*[System[TimeCreated[timediff(@SystemTime) <= 3600000]]]</Select>
  </Query>
</QueryList>

where the time is given in milliseconds (3600000ms = 1h).

[uncletimmy3]

@andrewkroh That's right. I checked code in query.go and saw XML query, beacause of it I had said it was strange in previous comment. I treid to test xml-filter through event viewer and powershell, both were fine. I think there is some bugs in event viewer api.