Skip to content

[Sophos] Update Sophos pipelines for new fields #2163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andrewkroh merged 8 commits into from
Apr 18, 2022
Merged

[Sophos] Update Sophos pipelines for new fields #2163

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
bcb96c1
Update Sophos pipelines for new fields
legoguy1000 Nov 16, 2021
0262008
update pipelines after rebase
legoguy1000 Mar 31, 2022
10c9467
Merge branch 'main' into 2158-sophos
andrewkroh Apr 6, 2022
7d00de9
version: 1.2.3
andrewkroh Apr 6, 2022
1df73f2
updates from comments
legoguy1000 Apr 13, 2022
be5af5c
Merge branch '2158-sophos' of https://github.com/legoguy1000/integrat…
legoguy1000 Apr 13, 2022
e4879b6
Updates per comments
legoguy1000 Apr 18, 2022
9e25bad
Update readme
andrewkroh Apr 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions packages/sophos/_dev/deploy/docker/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,25 +7,25 @@ services:
- ${SERVICE_LOGS_DIR}:/var/log
command: /bin/sh -c "cp /sample_logs/* /var/log/"
sophos-utm-udp:
image: akroh/stream:v0.2.0
image: docker.elastic.co/observability/stream:v0.7.0
volumes:
- ./sample_logs:/sample_logs:ro
entrypoint: /bin/bash
command: -c "/stream log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9549 -p=udp /sample_logs/sophos-utm-*.log"
sophos-utm-tcp:
image: akroh/stream:v0.2.0
image: docker.elastic.co/observability/stream:v0.7.0
volumes:
- ./sample_logs:/sample_logs:ro
entrypoint: /bin/bash
command: -c "/stream log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9549 -p=tcp /sample_logs/sophos-utm-*.log"
sophos-xg-udp:
image: akroh/stream:v0.2.0
image: docker.elastic.co/observability/stream:v0.7.0
volumes:
- ./sample_logs:/sample_logs:ro
entrypoint: /bin/bash
command: -c "/stream log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9549 -p=udp /sample_logs/sophos-xg*.log"
sophos-xg-tcp:
image: akroh/stream:v0.2.0
image: docker.elastic.co/observability/stream:v0.7.0
volumes:
- ./sample_logs:/sample_logs:ro
entrypoint: /bin/bash
Expand Down
5 changes: 5 additions & 0 deletions packages/sophos/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.2.3"
changes:
- description: Update pipelines to parse new fields
type: enhancement
link: https://github.com/elastic/integrations/pull/2163
- version: "1.2.2"
changes:
- description: Add documentation for multi-fields
Expand Down
2 changes: 0 additions & 2 deletions packages/sophos/data_stream/utm/_dev/test/pipeline/test-common-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
dynamic_fields:
event.ingested: ".*"
fields:
tags:
- preserve_original_event
300 changes: 0 additions & 300 deletions packages/sophos/data_stream/utm/_dev/test/pipeline/test-generated.log-expected.json
View file
Open in desktop

Large diffs are not rendered by default.

3 changes: 0 additions & 3 deletions packages/sophos/data_stream/utm/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,6 @@ description: Pipeline for Astaro Security Gateway

processors:
# ECS event.ingested
- set:
field: event.ingested
value: '{{_ingest.timestamp}}'
- set:
field: ecs.version
value: '8.0.0'
Expand Down
2 changes: 0 additions & 2 deletions ...st/pipeline/test-sophos-xg.log-config.yml → ..._dev/test/pipeline/test-common-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
dynamic_fields:
event.ingested: ".*"
fields:
tags:
- preserve_original_event
Expand Down
6 changes: 3 additions & 3 deletions packages/sophos/data_stream/xg/_dev/test/pipeline/test-sophos-xg.log
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@
<30>device="SFW" date=2018-06-06 time=12:53:39 timezone="IST" device_name="SG430" device_id=S4000806149EE49 log_id=041123413012 log_type="Anti-Spam" log_component="SMTP" log_subtype="Dos" status="" priority=Warning fw_rule_id=0 user_name="" av_policy_name="None" from_email_address="" to_email_address="" email_subject="" mailid="" mailsize=0 spamaction="TMPREJECT" reason="SMTP DoS" src_domainname="" dst_domainname="" src_ip=10.198.16.121 src_country_code=R1 dst_ip=10.198.17.121 dst_country_code=R1 protocol="TCP" src_port=60392 dst_port=25 sent_bytes=0 recv_bytes=0 quarantine_reason="Other"
<30>device="SFW" date=2018-06-06 time=12:56:53 timezone="IST" device_name="SG430" device_id=S4000806149EE49 log_id=041102413014 log_type="Anti-Spam" log_component="SMTP" log_subtype="Denied" status="" priority=Warning fw_rule_id=0 user_name="" av_policy_name="postman" from_email_address="pankhil1@postman.local" to_email_address="pankhil@postman. local" email_subject="Fwd: test sand" mailid="c0000008-1528270010" mailsize=419835 spamaction="DROP" reason="Email is marked Malicious by Sophos Sandstorm." src_domainname="postman.local" dst_domainname="" src_ip=10.198.16.121 src_country_code=R1 dst_ip=10.198.17.121 dst_country_code=R1 protocol="TCP" src_port=60608 dst_port=25 sent_bytes=0 recv_bytes=0
<30>device="SFW" date=2017-01-31 time=18:31:11 timezone="IST" device_name="CR750iNG-XP" device_id=C44313350024-P29PUA log_id=041207414001 log_type="Anti-Spam" log_component="POP3" log_subtype="Spam" status="" priority=Warning fw_rule_id=0 user_name="gaurav" av_policy_name="GauravPatel" from_email_address="gaurav1@iview.com" to_email_address="gaurav2@iview. com" email_subject="RPD Spam Test: Spam" mailid="<2a2dd5d4-1a30-617b-27b1-7961ad07cf07@iview.com>" mailsize=574 spamaction="Accept" reason="" src_domainname=" iview.com" dst_domainname="iview.com" src_ip=10.198.47.71 src_country_code=R1 dst_ip=10.198.233.61 dst_country_code=R1 protocol="TCP" src_port=22333 dst_port=110 sent_bytes=0 recv_bytes=0 quarantine_reason="Other"
<30>device="SFW" date=2020-05-18 time=14:38:33 timezone="CEST" device_name="XG230" device_id=1234567890123457 log_id=030906208001 log_type="Anti-Virus" log_component="HTTP" log_subtype="Virus" status="" priority=Critical fw_rule_id=2 user_name="" iap=13 av_policy_name="" virus="Sandstorm" url="http://sophostest.com/Sandstorm/SBTestFile1.pdf" domainname="sophostest.com" src_ip=175.16.199.1 src_country_code=R1 dst_ip=175.16.199.1 dst_country_code=USA protocol="TCP" src_port=57695 dst_port=80 sent_bytes=550 recv_bytes=1616 user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36" status_code=403
<30>device="SFW" date=2020-05-18 time=14:38:34 timezone="CEST" device_name="XG230" device_id=1234567890123456 log_id=030906208001 log_type="Anti-Virus" log_component="HTTP" log_subtype="Virus" status="" priority=Critical fw_rule_id=2 user_name="" iap=13 av_policy_name="" virus="EICAR-AV-Test" url="http://sophostest.com/eicar/index.html" domainname="sophostest.com" src_ip=175.16.199.1 src_country_code=R1 dst_ip=175.16.199.1 dst_country_code=USA protocol="TCP" src_port=57835 dst_port=80 sent_bytes=541 recv_bytes=553 user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36" status_code=403
<30>device="SFW" date=2020-05-18 time=14:38:33 timezone="CEST" device_name="XG230" device_id=1234567890123457 log_id=030906208001 log_type="Anti-Virus" log_component="HTTP" log_subtype="Virus" status="" priority=Critical fw_rule_id=2 user_name="" iap=13 av_policy_name="" virus="Sandstorm" url="http://sophostest.com/Sandstorm/SBTestFile1.pdf" domainname="sophostest.com" src_ip=175.16.199.1 src_country_code=R1 dst_ip=175.16.199.1 dst_country_code=USA protocol="TCP" src_port=57695 dst_port=80 sent_bytes=550 recv_bytes=1616 user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/889.160.20.156 Safari/537.36" status_code=403
<30>device="SFW" date=2020-05-18 time=14:38:34 timezone="CEST" device_name="XG230" device_id=1234567890123456 log_id=030906208001 log_type="Anti-Virus" log_component="HTTP" log_subtype="Virus" status="" priority=Critical fw_rule_id=2 user_name="" iap=13 av_policy_name="" virus="EICAR-AV-Test" url="http://sophostest.com/eicar/index.html" domainname="sophostest.com" src_ip=175.16.199.1 src_country_code=R1 dst_ip=175.16.199.1 dst_country_code=USA protocol="TCP" src_port=57835 dst_port=80 sent_bytes=541 recv_bytes=553 user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/889.160.20.156 Safari/537.36" status_code=403
<30>device="SFW" date=2020-05-18 time=14:38:35 timezone="CEST" device_name="XG230" device_id=1234567890123457 log_id=031106210001 log_type="Anti-Virus" log_component="SMTP" log_subtype="Virus" status="" priority=Critical fw_rule_id=22 user_name="" av_policy_name="default-smtp-av" from_email_address="info@farasamed.com" to_email_address="info@elastic-user.local" subject="ZAHLUNG (PROFORMA INVOICE)" mailid="<20200520004312.Horde.lEUeVf2I6PwO5K5TtMndnC7@webmail.sevengayr" mailsize=2254721 virus="TR/AD.AgentTesla.eaz" filename="" quarantine="" src_domainname="farasamed.com" dst_domainname="" src_ip=175.16.199.1 src_country_code=DEU dst_ip=175.16.199.1 dst_country_code=DEU protocol="TCP" src_port=56336 dst_port=25 sent_bytes=0 recv_bytes=0 quarantine_reason="Infected"
<30>device="SFW" date=2020-05-18 time=14:38:36 timezone="CEST" device_name="XG230" device_id=1234567890123456 log_id=031106210001 log_type="Anti-Virus" log_component="SMTP" log_subtype="Virus" status="" priority=Critical fw_rule_id=22 user_name="" av_policy_name="default-smtp-av" from_email_address="spedizioni@divella.it" to_email_address="info@elastic-user.local" subject="Re: NEW PRO-FORMA INVOICE" mailid="<20200519072944.AFCA295AF2A037A6@divella.it>" mailsize=537457 virus="Mal/BredoZp-B" filename="" quarantine="" src_domainname="divella.it" dst_domainname="" src_ip=175.16.199.1 src_country_code=USA dst_ip=175.16.199.1 dst_country_code=DEU protocol="TCP" src_port=54693 dst_port=25 sent_bytes=0 recv_bytes=0 quarantine_reason="Infected"
<30>device="SFW" date=2018-06-06 time=10:51:29 timezone="IST" device_name="SG430" device_id=S4000806149EE49 log_id=036106211001 log_type="Anti-Virus" log_component="POPS" log_subtype="Virus" status="" priority=Critical fw_rule_id=0 user_name="" av_policy_name="None" from_email_address="pankhil@postman.local" to_email_address="pankhil@postman.local" subject="EICAR" mailid="<a5c35e4b-1198-d0eb-0763-c0d5af3c817e@postman.local>" mailsize=0 virus="EICAR-AV-Test" filename="" quarantine="" src_domainname="postman.local" dst_domainname="" src_ip=10.198.16.121 src_country_code=R1 dst_ip=10.198.234.240 dst_country_code=R1 protocol="TCP" src_port=56653 dst_port=995 sent_bytes=0 recv_bytes=0 quarantine_reason="Other"
Expand Down Expand Up @@ -87,5 +87,5 @@
<30>device="SFW" date=2020-05-20 time=18:03:31 timezone="IST" device_name="XG230" device_id=1234567890123457 log_id=075000617071 log_type="WAF" log_component="Web Application Firewall" priority=Information user_name="-" server=- sourceip=175.16.199.1 localip=175.16.199.1 ws_protocol="HTTP/1.0" url=/ querystring="" cookie="-" referer="-" method=GET httpstatus=403 reason="WAF Anomaly" extra="Inbound Anomaly Score Exceeded (Total Score: 7, SQLi=, XSS=): Last Matched Message: Request Missing a User Agent Header" contenttype="text/html" useragent="-" host=175.16.199.1 responsetime=608 bytessent=5353 bytesrcv=295 fw_rule_id=3
<30>device="SFW" date=2017-02-01 time=14:17:35 timezone="IST" device_name="SG115" device_id=S110016E28BA631 log_id=106025618011 log_type="Wireless Protection" log_component="Wireless Protection" log_subtype="Information" priority=Information ap=A40024A636F7862 ssid=SPIDIGO2015 clients_conn_SSID=2
<30>device="SFW" date=2017-02-01 time=14:19:47 timezone="IST" device_name="SG115" device_id=S110016E28BA631 log_id=106025618011 log_type="Wireless Protection" log_component="Wireless Protection" log_subtype="Information" priority=Information ap=A40024A636F7862 ssid=SPIDIGO2015 clients_conn_SSID=3
<01>Feb 11 13:12:45 _gateway device="SFW" date=2021-02-11 time=13:12:45 timezone="CET" device_name="XG210" device_id=dem-dev log_id=010101600001 log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" status="Allow" priority=Information duration=0 fw_rule_id=9 nat_rule_id=16 policy_type=1 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=0 application="" application_risk=0 application_technology="" application_category="" vlan_id="" ether_type=Unknown (0x0000) bridge_name="" bridge_display_name="" in_interface="Port2.109" in_display_interface="CD21-IPs_WAN" out_interface="Port5.200" out_display_interface="Port5" src_mac=11:22:33:44:55:66 dst_mac=66:55:44:33:22:11 src_ip=1.128.3.4 src_country_code=ESP dst_ip=175.16.199.1 dst_country_code=GB protocol="TCP" src_port=33370 dst_port=443 sent_pkts=0 recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip=216.160.83.57 tran_src_port=0 tran_dst_ip=216.160.83.61 tran_dst_port=0 srczonetype="WAN" srczone="WAN" dstzonetype="DMZ" dstzone="Zone 9" dir_disp="" connevent="Start" connid="3933925696" vconnid="" hb_health="No Heartbeat" message="" appresolvedby="Signature" app_is_cloud=0
<01>Feb 11 13:12:45 _gateway device="SFW" date=2021-02-11 time=13:12:45 timezone="CET" device_name="XG210" device_id=dem-dev log_id=010101600001 log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" status="Allow" priority=Information duration=0 fw_rule_id=9 nat_rule_id=16 policy_type=1 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=0 application="" application_risk=0 application_technology="" application_category="" vlan_id="" ether_type=Unknown (0x0000) bridge_name="" bridge_display_name="" in_interface="Port2.109" in_display_interface="CD21-IPs_WAN" out_interface="Port5.200" out_display_interface="Port5" src_mac=11:22:33:44:55:66 dst_mac=66:55:44:33:22:11 src_ip=89.160.20.156 src_country_code=ESP dst_ip=175.16.199.1 dst_country_code=GB protocol="TCP" src_port=33370 dst_port=443 sent_pkts=0 recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip=216.160.83.57 tran_src_port=0 tran_dst_ip=216.160.83.61 tran_dst_port=0 srczonetype="WAN" srczone="WAN" dstzonetype="DMZ" dstzone="Zone 9" dir_disp="" connevent="Start" connid="3933925696" vconnid="" hb_health="No Heartbeat" message="" appresolvedby="Signature" app_is_cloud=0
<01>device="SFW" date=2020-06-05 time=03:45:23 timezone="CEST" device_name="SF01V" device_id=SFDemo-ta-vm-55 log_id=010101600001 log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" status="Allow" priority=Information duration=0 fw_rule_id=5 nat_rule_id=2 policy_type=1 user_name="" user_gp="" iap=13 ips_policy_id=0 appfilter_policy_id=0 application="" application_risk=0 application_technology="" application_category="" vlan_id="" ether_type=Unknown (0x0000) bridge_name="" bridge_display_name="" in_interface="Port2" in_display_interface="Port2" out_interface="Port1" out_display_interface="Port1" src_mac=00:50:56:99:51:94 dst_mac=00:50:56:99:3D:AC src_ip=10.146.13.30 src_country_code= dst_ip=10.8.142.181 dst_country_code= protocol="TCP" src_port=45294 dst_port=443 sent_pkts=0 recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip=10.8.13.110 tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="LAN" srczone="LAN" dstzonetype="WAN" dstzone="WAN" dir_disp="" connevent="Start" connid="2674291981" vconnid="" hb_health="No Heartbeat"message="" appresolvedby="Signature" app_is_cloud=0 log_occurrence=1
Loading