[sonicwall] Add new fields to pipeline by jhaugh0 · Pull Request #2729 · elastic/integrations

jhaugh0 · 2022-02-22T18:37:11Z

What does this PR do?

Adds new fields to be parsed from sonicwall logs

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Example logs to test against:

10.0.0.1 id=firewall sn=123456789 time="2022-02-22 18:24:30 UTC" fw=10.0.0.2 pri=6 c=262144 gcat=6 m=98 msg="Connection Opened" src=10.0.0.3:52379:X0 natSrc=10.0.0.2:48245 dst=8.8.8.8:443:X1 natDst=8.8.8.8:443 usr="Unknown (SSO failed)" proto=tcp/https sent=52 app=49177 appName='General HTTPS' n=123456789 fw_action="NA" dpi=0

10.0.0.1 id=firewall sn=123456789 time="2022-02-22 18:29:37 UTC" fw=10.0.0.2 pri=6 c=1024 gcat=2 m=97 msg="Web site hit" srcMac=12:34:56:78:90:ab src=10.0.0.3:64828:X0 srcZone=Trusted natSrc=10.0.0.2:47621 dstMac=ab:09:87:65:43:21 dst=8.8.8.8:443:X1 dstZone=Untrusted natDst=8.8.8.8:443 usr="Unknown (SSO failed)" proto=tcp/https sent=3523 rcvd=14226 app=7927 dstname=chat-pa.clients6.google.com arg=/ code=29 Category="Search Engines and Portals" note="Policy: cfsZonePolicy0, Info: 6148 " n=123456789 fw_action="NA" dpi=1

10.0.0.1 id=firewall sn=2CB8ED17E180 time="2022-02-22 18:34:21 UTC" fw=10.0.0.2 pri=6 c=1024 gcat=2 m=97 msg="Web site hit" srcMac=12:34:56:78:90:ab src=10.0.0.3:49217:X0 srcZone=Trusted natSrc=10.0.0.2:53466 dstMac=ab:09:87:65:43:21 dst=8.8.8.8:443:X1 dstZone=Untrusted natDst=8.8.8.8:443 usr="Unknown (SSO failed)" proto=tcp/https sent=2079 rcvd=6642 app=7927 dstname=seg.ad.gt arg=/ code=15 Category="Business and Economy" note="Policy: cfsZonePolicy0, Info: 6148 " n=123456789 fw_action="NA" dpi=1

Related issues

Screenshots

elasticmachine · 2022-02-22T18:42:40Z

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-03-31T08:05:24.803+0000
Duration: 3 min 14 sec

Steps errors

Expand to view the steps failures

`Google Storage Download`

Took 0 min 0 sec . View more details here

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-02-22T18:43:29Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh · 2022-02-24T00:58:39Z

/test

andrewkroh · 2022-02-24T01:02:29Z

The test cases should be added in https://github.com/elastic/integrations/tree/main/packages/sonicwall/data_stream/firewall/_dev/test/pipeline. See https://github.com/elastic/elastic-package/blob/main/docs/howto/pipeline_testing.md for info on how to update the "expected" files.

jhaugh0 · 2022-02-24T18:28:13Z

The test cases should be added in https://github.com/elastic/integrations/tree/main/packages/sonicwall/data_stream/firewall/_dev/test/pipeline. See https://github.com/elastic/elastic-package/blob/main/docs/howto/pipeline_testing.md for info on how to update the "expected" files.

This is updated now

andrewkroh · 2022-02-24T19:42:29Z

/test

andrewkroh · 2022-02-24T20:04:52Z

packages/sonicwall/data_stream/firewall/_dev/test/pipeline/test-general.log-expected.json

            "tags": [
                "preserve_original_event"
            ]
+        },


These 3 sample logs should go into a file like:

diff --git a/packages/sonicwall/data_stream/firewall/_dev/test/pipeline/test-issue2729.log b/packages/sonicwall/data_stream/firewall/_dev/test/pipeline/test-issue2729.log new file mode 100644 index 000000000..0a2758170 --- /dev/null +++ b/packages/sonicwall/data_stream/firewall/_dev/test/pipeline/test-issue2729.log @@ -0,0 +1,3 @@ +10.0.0.1 id=firewall sn=123456789 time="2022-02-22 18:24:30 UTC" fw=10.0.0.2 pri=6 c=262144 gcat=6 m=98 msg="Connection Opened" src=10.0.0.3:52379:X0 natSrc=10.0.0.2:48245 dst=8.8.8.8:443:X1 natDst=8.8.8.8:443 usr="Unknown (SSO failed)" proto=tcp/https sent=52 app=49177 appName='General HTTPS' n=123456789 fw_action="NA" dpi=0 +10.0.0.1 id=firewall sn=123456789 time="2022-02-22 18:29:37 UTC" fw=10.0.0.2 pri=6 c=1024 gcat=2 m=97 msg="Web site hit" srcMac=12:34:56:78:90:ab src=10.0.0.3:64828:X0 srcZone=Trusted natSrc=10.0.0.2:47621 dstMac=ab:09:87:65:43:21 dst=8.8.8.8:443:X1 dstZone=Untrusted natDst=8.8.8.8:443 usr="Unknown (SSO failed)" proto=tcp/https sent=3523 rcvd=14226 app=7927 dstname=chat-pa.clients6.google.com arg=/ code=29 Category="Search Engines and Portals" note="Policy: cfsZonePolicy0, Info: 6148 " n=123456789 fw_action="NA" dpi=1 +10.0.0.1 id=firewall sn=2CB8ED17E180 time="2022-02-22 18:34:21 UTC" fw=10.0.0.2 pri=6 c=1024 gcat=2 m=97 msg="Web site hit" srcMac=12:34:56:78:90:ab src=10.0.0.3:49217:X0 srcZone=Trusted natSrc=10.0.0.2:53466 dstMac=ab:09:87:65:43:21 dst=8.8.8.8:443:X1 dstZone=Untrusted natDst=8.8.8.8:443 usr="Unknown (SSO failed)" proto=tcp/https sent=2079 rcvd=6642 app=7927 dstname=seg.ad.gt arg=/ code=15 Category="Business and Economy" note="Policy: cfsZonePolicy0, Info: 6148 " n=123456789 fw_action="NA" dpi=1

then you would use the tooling (elastic-package test pipeline -g) to generate the "-expected.json" file.

andrewkroh · 2022-02-24T20:07:26Z

packages/sonicwall/data_stream/firewall/fields/fields.yml

+          type: keyword
+        - name: user
+          type: keyword
+      - name: pipeline


This file is malformed. Looks like indentation around this location. Please use elastic-package check to check for problems.

fixed. Indented too far

andrewkroh · 2022-02-24T20:08:48Z

packages/sonicwall/data_stream/firewall/elasticsearch/ingest_pipeline/default.yml

-  # IP Geolocation Lookup
+      ignore_failure: false
+      pattern_definitions:
+        "SONICWALL_TIME" : "%{SONICWALL_TIME_WITH_TIMEZONE}?%{SONICWALL_TIME_NO_TIMEZONE}?"


Remove the spaces between the key and the colon.

andrewkroh · 2022-02-24T20:10:51Z

packages/sonicwall/data_stream/firewall/elasticsearch/ingest_pipeline/default.yml

+      patterns:
+        - "c=%{NUMBER:sonicwall.event.category_legacy:long}"
+      ignore_failure: true
+   - grok:


Indentation is off. Please test the pipeline with elastic-package test pipeline. The test will fail if the pipeline is invalid.

fixed

I'm having it fail on test-generated.log, but after a look through those logs I do not believe they are properly formatted logs from a sonicwall. Not sure if too much identifying information was removed, or just improperly removed/replaced

andrewkroh · 2022-02-24T20:11:09Z

packages/sonicwall/data_stream/firewall/elasticsearch/ingest_pipeline/default.yml

+      value: "{{sonicwall.destination_name}}"
+  - set:
+      field: _source.ingest_time
+      value: "{{_ingest_timestamp}}


Missing a closing quote.

efd6 · 2022-03-04T01:14:15Z

/test

efd6 · 2022-03-09T03:08:23Z

packages/sonicwall/data_stream/firewall/elasticsearch/ingest_pipeline/default.yml

+  - grok:
+      description: "Pull header information"
+      field: message
+      patterns:
+        - "%{IP:event.provider:ip}%{SPACE}id=%{NOTSPACE:observer.hostname} sn=%{DATA:observer.serial_number} time=.%{SONICWALL_TIME:_tmp.timestamp}. (?:fw=%{IP:observer.egress.ip})?(?: pri=%{NUMBER:event.severity:long})? %{GREEDYDATA:message}"
+        - "%{IP:event.provider:ip}%{SPACE}id=%{NOTSPACE:observer.hostname} sn=%{DATA:observer.serial_number} time=\\\"%{SONICWALL_TIME:_tmp.timestamp}\\\" (?:fw=%{IP:observer.egress.ip})?(?: pri=%{NUMBER:event.severity:long})? %{GREEDYDATA:message}"
      ignore_missing: true
-  # IP Geolocation Lookup
+      ignore_failure: false
+      pattern_definitions:
+        SONICWALL_TIME: '%{SONICWALL_TIME_WITH_TIMEZONE}?%{SONICWALL_TIME_NO_TIMEZONE}?'
+        SONICWALL_TIME_WITH_TIMEZONE: '%{YEAR}-%{NONNEGINT}-%{NONNEGINT} %{TIME} UTC'
+        SONICWALL_TIME_NO_TIMEZONE: '%{YEAR}-%{NONNEGINT}-%{NONNEGINT} %{TIME}'
+      on_failure:
+        [
+          {
+            "append":
+              {
+                "field": "error.message",
+                "value": "{{ _ingest.on_failure_message }} grok failure on header",
+              },
+          },
+        ]


I'd suggest that you simplify this so that you just collect the timestamp and leading IP address with grok and leave the rest of the line to be processed later by grabbing it as greedy data.

--- description: Pipeline for Sonicwall-FW processors: - set: field: event.module value: "sonicwall" - set: field: ecs.version value: '8.0.0' - rename: field: message target_field: event.original ignore_missing: true - grok: description: "Pull header information" field: event.original patterns: - "^%{DATA}%{TIMESTAMP:event.created} %{IP:observer.ip:ip} %{GREEDYDATA:message}$" - "^%{DATA}%{IP:observer.ip:ip} %{GREEDYDATA:message}$" pattern_definitions: TIMESTAMP: "%{MONTH} +%{MONTHDAY} %{TIME}" # split Sonicwall fields - kv: field: message field_split: " (?=[a-zA-Z0-9_]+=)" value_split: "=" prefix: "sonicwall." ignore_missing: true ignore_failure: false trim_value: "\""

This will give you the observer IP and the creation time if it exists, and it will then leave you with the remainder of the line to parse.

From looking at the log lines, these are all key-value pairs with field separated by space and key/val separated by =, so the kv processor will happily consume those (with a little help since some of the values have spaces; the field_split regex handles that). With that all of the grok processors below can go away.

This will leave you finally with event that look like this

{ "ecs": { "version": "8.0.0" }, "event": { "created": "Jan 3 13:45:50", "module": "sonicwall", "original": "Jan 3 13:45:50 192.168.5.1 id=firewall sn=000SERIAL time=\"2007-01-03 14:48:21\" fw=1.128.3.4 pri=6 c=262144 m=98 msg=\"Connection Opened\" n=23427 src=192.168.6.10:28503:WAN dst=192.168.5.10:53:LAN proto=tcp/dns" }, "message": "id=firewall sn=000SERIAL time=\"2007-01-03 14:48:21\" fw=1.128.3.4 pri=6 c=262144 m=98 msg=\"Connection Opened\" n=23427 src=192.168.6.10:28503:WAN dst=192.168.5.10:53:LAN proto=tcp/dns", "observer": { "ip": "192.168.5.1" }, "sonicwall": { "c": "262144", "dst": "192.168.5.10:53:LAN", "fw": "1.128.3.4", "id": "firewall", "m": "98", "msg": "Connection Opened", "n": "23427", "pri": "6", "proto": "tcp/dns", "sn": "000SERIAL", "src": "192.168.6.10:28503:WAN", "time": "2007-01-03 14:48:21" }, "tags": [ "preserve_original_event" ] }, { "ecs": { "version": "8.0.0" }, "event": { "module": "sonicwall", "original": "10.0.0.1 id=firewall sn=123456789 time=\"2022-02-22 18:24:30 UTC\" fw=10.0.0.2 pri=6 c=262144 gcat=6 m=98 msg=\"Connection Opened\" src=10.0.0.3:52379:X0 natSrc=10.0.0.2:48245 dst=216.160.83.61:443:X1 natDst=216.160.83.61:443 usr=\"Unknown (SSO failed)\" proto=tcp/https sent=52 app=49177 appName='General HTTPS' n=123456789 fw_action=\"NA\" dpi=0" }, "message": "id=firewall sn=123456789 time=\"2022-02-22 18:24:30 UTC\" fw=10.0.0.2 pri=6 c=262144 gcat=6 m=98 msg=\"Connection Opened\" src=10.0.0.3:52379:X0 natSrc=10.0.0.2:48245 dst=216.160.83.61:443:X1 natDst=216.160.83.61:443 usr=\"Unknown (SSO failed)\" proto=tcp/https sent=52 app=49177 appName='General HTTPS' n=123456789 fw_action=\"NA\" dpi=0", "observer": { "ip": "10.0.0.1" }, "sonicwall": { "app": "49177", "appName": "'General HTTPS'", "c": "262144", "dpi": "0", "dst": "216.160.83.61:443:X1", "fw": "10.0.0.2", "fw_action": "NA", "gcat": "6", "id": "firewall", "m": "98", "msg": "Connection Opened", "n": "123456789", "natDst": "216.160.83.61:443", "natSrc": "10.0.0.2:48245", "pri": "6", "proto": "tcp/https", "sent": "52", "sn": "123456789", "src": "10.0.0.3:52379:X0", "time": "2022-02-22 18:24:30 UTC", "usr": "Unknown (SSO failed)" }, "tags": [ "preserve_original_event" ] },

So all that is now needed is to appropriately rename the fields mapping them to the right place in the ECS and giving them the right types with convert processors, and some cases dissect (for IPs, which can be enriched through geoip and community_id) and date.

There is a caveat to this, this does not work with the generated log. I suspect that some of the lines there are not valid, so in the first instance delete that file so at least you can have something that works. If there are valid lines in that sample set, they can be added back later, adjusting the processors as needed.

Also, you can delete the script processors in data_stream/firewall/agent/stream/stream.yml.hbs, data_stream/firewall/agent/stream/tcp.yml.hbs and data_stream/firewall/agent/stream/udp.yml.hbs since we are now doing the work here.

Just extracting the time/IP first is a good idea, less to break on. It needs to be reworked a bit from what you posted though as it seems like different firmware versions have different header formats. Also using rsyslog to write logs to disk first, which I do, by default prepends the sender IP address, which could be different from the IP in the log if they're on different subnets. For example:
10.0.0.1 id=firewall sn=123456789 time="2022-02-22 18:29:37 UTC" fw=10.0.0.2
Where 10.0.0.1 is the IP on the subnet as the rsyslog host, and 10.0.02 is the DMZ IP of the sonicwall. I recognize I might be a bit more of a rare case for that use though. What are your thoughts on that?

EDIT: I just realized yeah, your suggestion works for all of the other examples. It looks like they were prepended from rsyslog or similar as well. I should have more coffee before replying... In any case, I wasn't completely wrong. Doing a tcpdump on the logs coming in confirmed at least on my sonicwall's config, the actual log starts at id=firewall

It's been a bit since I initially wrote up this version, but I think I initially tried using KV and found some logs that broke it. I'm going to dig in some more and see if I can pull those up, or maybe it was nothing.

efd6 · 2022-03-09T03:26:04Z

packages/sonicwall/data_stream/firewall/fields/fields.yml

+    - name: hostname
+      type: keyword
+- name: ingest_time
+  type: date


Sorry, forgot to mention. This needs to have its final newline added in. Running elastic-package format will do this for you.

Ran the pipeline overnight and found a few more bugs which are now all fixed. Also added another log that caused the issues to the test list

efd6 · 2022-03-14T21:38:17Z

/test

efd6

It would be helpful for future maintenance if logically related sections were split from each other with vertical whitespace and with comment annotation, but this is looking good.

packages/sonicwall/changelog.yml

packages/sonicwall/data_stream/firewall/elasticsearch/ingest_pipeline/default.yml

efd6 · 2022-03-15T00:50:19Z

packages/sonicwall/data_stream/firewall/elasticsearch/ingest_pipeline/default.yml

+      patterns:
+        - '^%{IP:event.provider:IP} +%{GREEDYDATA:message}$'
+        - >-
+          ^%{DATA}%{TIMESTAMP:event.provided} %{IP:event.provider:IP}
+          %{GREEDYDATA:message}$


I don't think the multi-line string improves clarity here and is arguably more fragile to un-noticed damage in the future.

Suggested change

patterns:

- '^%{IP:event.provider:IP} +%{GREEDYDATA:message}$'

- >-

^%{DATA}%{TIMESTAMP:event.provided} %{IP:event.provider:IP}

%{GREEDYDATA:message}$

patterns:

- "^%{DATA}%{TIMESTAMP:event.created} %{IP:observer.ip:ip} %{GREEDYDATA:message}$"

- "^%{DATA}%{IP:observer.ip:ip} %{GREEDYDATA:message}$"

Missed that, I just used a json > yaml converter from doing testing in kibana. Looks like that added that.

Just had to add an extra '+' to keep the message field from starting with a whitespace and breaking the kv proccessor

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

adriansr · 2022-05-17T11:33:27Z

Hi @jhaugh0

Thanks for your contribution.

After discussing this we've decided it's better to create a brand new integration and deprecate the existing one. I've created a new integration for SonicWall (#3365). Uses some of your ideas and logs, will give you proper credit.

jamiehynds · 2022-05-19T10:47:07Z

Closing as per @adriansr comment above. Thanks again @jhaugh0!

jhaugh0 added 2 commits February 22, 2022 12:40

Update default.yml

b0fdf2d

Update fields.yml

94b3ec9

jamiehynds added the Team:Security-External Integrations label Feb 22, 2022

jamiehynds linked an issue Feb 23, 2022 that may be closed by this pull request

Sonicwall #1207

Closed

15 tasks

andrewkroh added the Integration:sonicwall_firewall SonicWall Firewall label Feb 24, 2022

andrewkroh added the enhancement New feature or request label Feb 24, 2022

andrewkroh changed the title ~~Add new fields to sonicwall pipeline~~ [sonicwall] Add new fields to pipeline Feb 24, 2022

jhaugh0 added 4 commits February 24, 2022 13:14

Update test-general.log-expected.json

3e9af42

Update test-general.log-expected.json

17e6948

Update test-general.log-expected.json

a19a237

Update changelog.yml

c9bd252

andrewkroh requested changes Feb 24, 2022

View reviewed changes

jhaugh0 added 2 commits February 28, 2022 16:04

fixed formatting, added fields

1c55398

readded the desination geo processor that was accidentally removed

1c9ab02

efd6 self-assigned this Mar 9, 2022

efd6 reviewed Mar 9, 2022

View reviewed changes

Pipeline now based on KV

b737d19

jhaugh0 requested a review from a team as a code owner March 10, 2022 20:41

Fixed some bugs from some less common log types

4d6a9b3

Ran the pipeline overnight and found a few more bugs which are now all fixed. Also added another log that caused the issues to the test list

efd6 reviewed Mar 15, 2022

View reviewed changes

Update packages/sonicwall/changelog.yml

cc8fe0b

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

Added descriptions, changed initial grok processor

319a11f

This was referenced Apr 21, 2022

[Filebeat] Dissect Parsing Error with Sonicwall Module elastic/beats#24124

Closed

[Filebeat] Sonicwall Module don't show all ecs fields elastic/beats#25469

Closed

adriansr mentioned this pull request May 17, 2022

New integration for SonicWall firewall #3365

Merged

8 tasks

jamiehynds closed this May 19, 2022

Conversation

jhaugh0 commented Feb 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Feb 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❕ Build Aborted

Build stats

Steps errors

Google Storage Download

🤖 GitHub comments

Uh oh!

elasticmachine commented Feb 22, 2022

Uh oh!

andrewkroh commented Feb 24, 2022

Uh oh!

andrewkroh commented Feb 24, 2022

Uh oh!

jhaugh0 commented Feb 24, 2022

Uh oh!

andrewkroh commented Feb 24, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 commented Mar 4, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jhaugh0 Mar 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 commented Mar 14, 2022

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adriansr commented May 17, 2022

Uh oh!

jamiehynds commented May 19, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jhaugh0 commented Feb 22, 2022 •

edited

Loading

elasticmachine commented Feb 22, 2022 •

edited

Loading

`Google Storage Download`

jhaugh0 Mar 9, 2022 •

edited

Loading