scl: add parser for netskope logs

[bazsi]

Signed-off-by: Balazs Scheidler [email protected]

[kira-syslogng]

Build SUCCESS

[bazsi]

That is coming from default-network-drivers(). Theres no point in creating a separate source for all device types.

…

On Thu, Feb 7, 2019, 17:01 László Szemere ***@***.*** wrote: ***@***.**** commented on this pull request. ------------------------------ In scl/netskope/plugin.conf <#2541 (comment)>: > @@ -0,0 +1,22 @@ + + +# sample log +# <134>{"count": 1, "supporting_data": {"data_values": ["x.x.x.x", ***@***.***"], "data_type": "user"}, "organization_unit": "domain/domain/Domain Users/Enterprise Users", "severity_level": 2, "category": null, "timestamp": 1547421943, "_insertion_epoch_timestamp": 1547421943, "ccl": "unknown", "user": ***@***.***", "audit_log_event": "Login Successful", "ur_normalized": ***@***.***", "_id": "936289", "type": "admin_audit_logs", "appcategory": null} + +# assumed to be running with flags(no-parse), e.g. the entire incoming log Can we also assume that netskope logs are coming over network? In this case we can "include the flags(no-parse) assumption" by creating a whole source block instead of a parser. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2541 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AArldjjX_IMZdNExp5dhMozLNoMDX5I6ks5vLE3mgaJpZM4anlfQ> .

[kira-syslogng]

Build SUCCESS

[gaborznagy]

Please include the new scl in these files too:

• packaging/debian/syslog-ng-core.install
• scl/CMakeLists.txt

[gaborznagy]

Can you please link a reference document for netskope log format? I have not found any with a quick search.
It seems that "<134>" is hard coded to every message, and _insertion_epoch_timestamp is a fingerprint as well.

[bazsi]

Both of these are true observations. This is a heuristics to fix the logs coming from Netskope and the format is potentially not documented and can change any time.

…

On Fri, Feb 8, 2019 at 10:22 AM Gábor Nagy ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In scl/netskope/plugin.conf <#2541 (comment)>: > + +# assumed to be running with flags(no-parse), e.g. the entire incoming log +# message is in $MSG +block parser netskope-parser(prefix(".netscope.")) { + channel { + rewrite { + subst("^<[0-9]+>", ""); + }; + parser { + json-parser(prefix(`prefix`)); + }; + }; +}; + +application netskope[syslog-raw] { + filter { message("<134>{" type(string) flags(prefix)) and message("_insertion_epoch_timestamp"); }; Can you please link a reference document for netskope log format? I have not found any with a quick search. It seems that "<134>" is hard coded to every message, and _insertion_epoch_timestamp is a fingerprint as well. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2541 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AArldrFIu_OxXniyWKZA5VIzYh6c6EJSks5vLUG8gaJpZM4anlfQ> .

[kira-syslogng]

Build SUCCESS

[gaborznagy]

Both of these are true observations. This is a heuristics to fix the logs coming from Netskope and the format is potentially not documented and can change any time.

Then is it worth to include something fragile into default-network-drivers via app-parser?
How are we gonna test that it's broken?

It sound to me, that the netskope-parser is enough.
I know, Cisco is the same: it uses heuristics and the format can broke as well, but integrating that many things by default that can broke easily seems to be hard to maintain.

[bazsi]

Then is it worth to include something fragile into default-network-drivers via app-parser?
How are we gonna test that it's broken?

Well, we won't. These are "as is" contributions that anyone is welcome to adapt to their own liking and PRs are appreciated. Bugreports are also appreciated, which we should handle on a best effort basis.

Usually though, if there's a problem, we already have a log sample to fix the parser with. So in fact, these kind of bugreports are a good source of getting log samples :)

And these kind of support levels are pretty standard, log formats are breaking everywhere, still the value that we deliver is that we compiled them to make it easy to use and then make it easy to fix/upgrade.

[bazsi]

@MrAnno was nice enough to fix the outstanding issues which I have added to this branch.

Removing wip

[kira-syslogng]

Build SUCCESS