Update winlogbeat-options.asciidoc #2919

gmoskovicz · 2016-11-02T12:06:01Z

Fix for #1491.

tsg · 2016-11-02T12:09:02Z

LGTM, waiting for reviews from @andrewkroh and @dedemorton.

tsg · 2016-11-02T12:11:58Z

winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc

+
+More than 22 event id's should be handled with multiple monitors to the same name:
+
+--------------------------------------------------------------------------------


you can add [source,yaml] on this to get syntax highlighting.

tsg · 2016-11-02T12:12:38Z

winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc

@@ -149,6 +149,24 @@ winlogbeat.event_logs:
    event_id: 4624, 4625, 4700-4800, -4735
 --------------------------------------------------------------------------------

+[WARNING]


This is not the usual way to create warnings, but it might be difficult in this case because you have code section in it. I'll wait for @dedemorton to suggest something.

This is valid asciidoc tagging. You could add [source,yaml] above the dashed line that starts the example to format the code correctly.

Keeping [WARNING] and added [source,yaml]

andrewkroh

I think we should add this note about the limit of 22 event IDs. But the suggested work-around could have some unintended consequences with respect to state persistence. It would work, but I would be hesitant to recommend it.

The two event log consumers defined for the "Security" log will both persist their state in the registry file under "Security" key. The record ID of the event that is read last will be persisted. This could cause a problem if the read offsets for the consumers varied because at startup they will both resume of the last persisted state potentially causing some events to be missed.

We could probably implement a workaround in the query builder to split the query after 22 "event sources". (The term "event sources" is vague so I would need to do some API testing to figure out how conditionals are factored into the limit of 22.) Then only a single consumer would be started for the "Security" log.

andrewkroh · 2016-11-02T13:04:21Z

winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc

+=======================================
+There is a limit on the query depth to prevent the possibility of a stack overflow from deep recursive queries. This limit is actually set to 22 filters.
+
+More than 22 event id's should be handled with multiple monitors to the same name:


andrewkroh · 2016-11-02T13:04:50Z

winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc

@@ -149,6 +149,24 @@ winlogbeat.event_logs:
    event_id: 4624, 4625, 4700-4800, -4735
 --------------------------------------------------------------------------------

+[WARNING]
+=======================================
+There is a limit on the query depth to prevent the possibility of a stack overflow from deep recursive queries. This limit is actually set to 22 filters.


actually could be dropped.

We need to avoid repeating content from other websites verbatim (the content is likely copyrighted). This sentence is taken directly from the Microsoft website: "There is a limit on the query depth to prevent the possibility of a stack overflow from deep recursive queries."

It would be better anyhow to state this limitation in terms of what the user specifies in the Winlogbeat config. How about something like this:

You can specify up to 22 event IDs. Specifying more than 22 event IDs may produce deep recursive queries that could result in a stack overflow. For more information, see https://support.microsoft.com/en-us/kb/970453.

I didn't suggest edits to the text about the workaround because it sounds like you are still discussing the best approach to solving this issue.

gmoskovicz · 2016-11-02T14:07:11Z

@andrewkroh

I am happy to remove the workaround, but if we add this in the documentation, then it means that we need to suggest a workaround. Do you want me to remove the workaround? What other workaround do you see here? If we do what you propose , do we still need this to be added in the documentation?

andrewkroh · 2016-11-02T14:41:52Z

@gmoskovicz, I think another possible workaround would be to use a processor to do the filtering.

I can spend some time on investigating the change to the query builder. If I can implement that then we don't need this in the documentation. But since it's a problem now for 5.0.0 we should merge this PR with a workaround that uses a processor. Can you see if you can setup a processor to do the desired filtering?

gmoskovicz · 2016-11-02T14:56:02Z

I'll work with this and get back here.

gmoskovicz · 2016-11-02T16:41:29Z

@andrewkroh

Works, but the configuration gets larger since you need a huge OR with equal conditions:

processors:
 - drop_event:
     when:
       equals:
         event_id: 50037

gmoskovicz

I've changed all the suggested text.

gmoskovicz · 2016-11-03T13:37:12Z

@andrewkroh @tsg Can i have final review of this? Given that there is no consensus around the workaround, i've suggested one using multiple processors and/or multiple IDs at the input level.

andrewkroh · 2016-11-03T15:00:26Z

winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc

+  - name: Security
+    event_id: 4624, 4625, 4700-4800, -4735 ....
+
+processors:


Here's what I'm thinking on this topic.

If you specify more that 22 event IDs to include or 22 event IDs to exclude, Windows will prevent Winlogbeat from reading the event log because it limits the number of conditions that can be used in an event log query. If this occurs a similar warning as shown below will be logged by Winlogbeat, and it will continue processing data from other event logs. For more information, see https://support.microsoft.com/en-us/kb/970453. `WARN EventLog[Application] Open() error. No events will be read from this source. The specified query is invalid.` If you have more than 22 event IDs, you can workaround this Windows limitation by using a drop_event[drop-event] processor to do the filtering after Winlogbeat has received the events from Windows. The filter shown below is equivalent to `event_id: 903, 1024, 4624` but can be expanded beyond 22 event IDs. [source,yaml] -------------------------------------------------------------------------------- processors: - drop_event.when.and: - equals.log_name: Security - not.or: - equals.event_id: 903 - equals.event_id: 1024 - equals.event_id: 4624 --------------------------------------------------------------------------------

andrewkroh · 2016-11-03T21:34:18Z

winlogbeat/docs/reference/configuration/winlogbeat-options.asciidoc

+
+[source,yaml]
+--------------------------------------------------------------------------------
+winlogbeat.event_logs:


I investigated automatically splitting up the event IDs across two queries. It gets complicated due to the interactions with the suppress filters (e.g. -1, -2). I think we should wait until #1491 (comment) is implemented and in the meantime use the drop_event processor as a workaround to either include or exclude events.

I don't want to recommend creating duplicate Security event log consumers because it can potentially result in data loss in some conditions. And in the future we may want to disallow duplicate names to prevent the aforementioned condition (but that's TBD).

gmoskovicz · 2016-11-04T11:20:41Z

@andrewkroh sounds good for me! I've updated it to the latest version

tsg · 2016-11-04T14:12:16Z

Belated LGTM :)

Add a note and workaround to the Winlogbeat docs about the limit of 22 event IDs in a query. And provide a workaround for elastic#1491 using filters.

Add a note and workaround to the Winlogbeat docs about the limit of 22 event IDs in a query. And provide a workaround for #1491 using filters.

Add a note and workaround to the Winlogbeat docs about the limit of 22 event IDs in a query. And provide a workaround for elastic#1491 using filters.

Update winlogbeat-options.asciidoc

e938693

tsg added docs Winlogbeat needs_backport PR is waiting to be backported to other branches. review labels Nov 2, 2016

tsg reviewed Nov 2, 2016

View reviewed changes

andrewkroh reviewed Nov 2, 2016

View reviewed changes

gmoskovicz commented Nov 3, 2016

View reviewed changes

Update winlogbeat-options.asciidoc

92b8426

andrewkroh reviewed Nov 3, 2016

View reviewed changes

Final fix

a39ba72

andrewkroh merged commit 1f86779 into 5.0 Nov 4, 2016

gmoskovicz deleted the doc-event-id-filter-fix branch November 4, 2016 13:52

dedemorton mentioned this pull request Dec 21, 2016

Forwardport: Update winlogbeat-options.asciidoc (cherrypick #2919 into master) #3230

Merged

dedemorton mentioned this pull request Dec 21, 2016

Forwardport: Update winlogbeat-options.asciidoc (cherrypicks #2919 into 5.1) #3231

Merged

dedemorton removed the needs_backport PR is waiting to be backported to other branches. label Dec 21, 2016

ruflin pushed a commit that referenced this pull request Dec 21, 2016

Update winlogbeat-options.asciidoc (#2919) (#3231)

64c5dfa

Add a note and workaround to the Winlogbeat docs about the limit of 22 event IDs in a query. And provide a workaround for #1491 using filters.

andrewkroh pushed a commit that referenced this pull request Dec 21, 2016

Update winlogbeat-options.asciidoc (#2919) (#3230)

d69c07a

Add a note and workaround to the Winlogbeat docs about the limit of 22 event IDs in a query. And provide a workaround for #1491 using filters.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update winlogbeat-options.asciidoc #2919

Update winlogbeat-options.asciidoc #2919

gmoskovicz commented Nov 2, 2016

tsg commented Nov 2, 2016

tsg Nov 2, 2016

gmoskovicz Nov 3, 2016

tsg Nov 2, 2016

dedemorton Nov 2, 2016

gmoskovicz Nov 3, 2016

andrewkroh left a comment

andrewkroh Nov 2, 2016

gmoskovicz Nov 3, 2016

andrewkroh Nov 2, 2016

dedemorton Nov 2, 2016

gmoskovicz Nov 3, 2016

gmoskovicz commented Nov 2, 2016 •

edited

Loading

andrewkroh commented Nov 2, 2016

gmoskovicz commented Nov 2, 2016

gmoskovicz commented Nov 2, 2016

gmoskovicz left a comment

gmoskovicz commented Nov 3, 2016

andrewkroh Nov 3, 2016

andrewkroh Nov 3, 2016

gmoskovicz commented Nov 4, 2016

tsg commented Nov 4, 2016


		More than 22 event id's should be handled with multiple monitors to the same name:

		--------------------------------------------------------------------------------

Update winlogbeat-options.asciidoc #2919

Update winlogbeat-options.asciidoc #2919

Conversation

gmoskovicz commented Nov 2, 2016

tsg commented Nov 2, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gmoskovicz commented Nov 2, 2016 • edited Loading

andrewkroh commented Nov 2, 2016

gmoskovicz commented Nov 2, 2016

gmoskovicz commented Nov 2, 2016

gmoskovicz left a comment

Choose a reason for hiding this comment

gmoskovicz commented Nov 3, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gmoskovicz commented Nov 4, 2016

tsg commented Nov 4, 2016

gmoskovicz commented Nov 2, 2016 •

edited

Loading