Skip to content

Commit

Permalink
[panw] Parse URL for threat-file message types (elastic#11730)
Browse files Browse the repository at this point in the history 
It's been observed that with threat-file events, the URL may be placed in a "FUTURE_USE" field. This adds support for parsing this field to URL, if it appears this is a URL in the file sub_type.

There isn't any PAN-OS documentation on this usage, so this change is based on actual observed events.
  • Loading branch information
@mjwolf
mjwolf authored Nov 18, 2024
1 parent 70b41e1 commit 60fcb22
Show file tree
Hide file tree
Showing 5 changed files with 183 additions and 1 deletion.
5 changes: 5 additions & 0 deletions packages/panw/changelog.yml
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "4.1.0"
changes:
- description: Parse URL from threat-file event type
type: enhancement
link: https://github.com/elastic/integrations/pull/11730
- version: "4.0.4"
changes:
- description: Parse threat-vulnerability properly
Expand Down
1 change: 1 addition & 0 deletions packages/panw/data_stream/panos/_dev/test/pipeline/test-panw-panos-threat-sample.log
View file
Original file line number Diff line number Diff line change
Expand Up @@ -211,3 +211,4 @@ Apr 9 20:43:30 AC-PA5250 1,2024/04/09 20:43:29,123456789012,THREAT,virus,2561,20
<14>Aug 19 13:58:32 fw1034.example.io 1,2024/08/19 13:58:32,007951000353454,THREAT,vulnerability,2562,2024/08/19 13:58:32,10.71.208.15,10.68.15.198,0.0.0.0,0.0.0.0,SectorProxy Browsing my3-user,,,web-browsing,vsys1,interconnect,proxy,ethernet1/2,ethernet1/3,HOST-LOGCOLLECTOR,2024/08/19 13:58:32,577801,1,18830,8097,0,0,0x1102000,tcp,alert,"fpt2.example.com/Clear.HTML?ctx=Ls1.0&wl=True&session_id=425e6d9b-6193-46d4-b5a2-35ee68e38086&id=2fd84f97-783d-43d4-977f-1a9f6446550e&w=8DCC056FF6F008C&tkt=H3ihr9e92IdW6yd1ZgQ9S9GE/yxCfNn1WRJjtpTkl7bZYNRP0G8iRDJKxj9TXCkOCRKWke9IXcntwxu7pJ00Kju4SIo+ydc3O2XOKak2s+umUnAnqOU9xu81Pv6fr1zV2m0HeGUeEPpayNxh9AVB9lnwfYsQh0ENPN98KUyDCmN77AEdVF7P5Njwlc2nOpzkVKgP4vmPeQd/o32ZFgpNPNp2AS2MV2oUfqU6TL9lfhTdFQZLw+jek7u4uMmuUFdtYYLULZAPshhlHwnTi9mcdsP9624GSOhWC/mtNEFNWEzEmg6VbgmWtHq6GMPiAOFr&CustomerId=02C58649-E822-405B-B6C3-17A7509D2FCC",Potential HTML Evasion Technique Detected in HTTP Response(91883),ua-generic,low,server-to-client,7395705320518981763,0x8000000000000000,10.0.0.0-10.255.255.255,10.0.0.0-10.255.255.255,,,0,,,1,,,,,,,,0,850,852,0,0,,fw1034,,,,,0,,0,2024/08/19 13:58:26,N/A,protocol-anomaly,AppThreat-8883-8920,0x0,0,4294967295,,,c1b9f945-e213-4cdf-b77d-2700446a3baf,805901,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0,2024-08-19T13:58:32.880+00:00,,,,internet-utility,general-internet,browser-based,4,"used-by-malware,able-to-transfer-file,has-known-vulnerability,tunnel-other-application,pervasive-use",,web-browsing,no,no,
<14>Aug 19 13:58:31 fw0096.example.io 1,2024/08/19 13:58:31,019901001188,THREAT,scan,2562,2024/08/19 13:58:31,10.48.12.171,10.190.160.25,0.0.0.0,0.0.0.0,,,,not-applicable,vsys2,interconnect,public,ae2.1349,,HOST-LOGCOLLECTOR,2024/08/19 13:58:31,0,1,41526,443,0,0,0x2000,tcp,alert,,SCAN: Host Sweep(8002),any,medium,client-to-server,7361590532514024944,0x8000000000000000,10.0.0.0-10.255.255.255,European Union,,,0,,,0,,,,,,,,0,15,23,0,0,az1_vsys_internet,fw0096,,,,,0,,0,,N/A,scan,AppThreat-0-0,0x0,0,4294967295,,,,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0,2024-08-19T13:58:31.761+00:00,,,0,unknown,unknown,unknown,1,,,not-applicable,no,no,
<14>Aug 19 13:58:32 fw1034.example.io 1,2024/08/19 13:58:32,007951000353454,THREAT,vulnerability,2562,2024/08/19 13:58:32,10.71.208.15,10.68.15.198,0.0.0.0,0.0.0.0,SectorProxy Browsing my3-user,,,web-browsing,vsys1,interconnect,proxy,ethernet1/2,ethernet1/3,HOST-LOGCOLLECTOR,2024/08/19 13:58:32,577801,1,18830,8097,0,0,0x1102000,tcp,alert,"shadow",Potential HTML Evasion Technique Detected in HTTP Response(91883),ua-generic,low,server-to-client,7395705320518981763,0x8000000000000000,10.0.0.0-10.255.255.255,10.0.0.0-10.255.255.255,,,0,,,1,,,,,,,,0,850,852,0,0,,fw1034,,,,,0,,0,2024/08/19 13:58:26,N/A,protocol-anomaly,AppThreat-8883-8920,0x0,0,4294967295,,,c1b9f945-e213-4cdf-b77d-2700446a3baf,805901,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0,2024-08-19T13:58:32.880+00:00,,,,internet-utility,general-internet,browser-based,4,"used-by-malware,able-to-transfer-file,has-known-vulnerability,tunnel-other-application,pervasive-use",,web-browsing,no,no,
<14>Nov 06 14:11:30 pa555 1,2024/11/06 14:11:30,0000000000001,THREAT,file,2562,2024/11/06 14:11:30,192.168.1.2,10.71.208.15,0.0.0.0,0.0.0.0,file download test rule,contoso\\steve,,web-browsing,vsys1,HOMENET,EXTNET,ethernet1/2,ethernet1/1,log-profile1,2024/11/06 14:11:30,994313,2,37268,443,0,0,0x1002000,tcp,alert,"elastic-agent.exe",Windows Executable (EXE)(52020),computer-and-internet-info,low,server-to-client,7367538158076100804,0x8000000000000000,192.168.0.0-192.168.255.255,United States,,,0,,,1,,,,,,,,0,199,479,0,0,,pa555,artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.15.3+build202411051926-windows-x86_64.zip,,,,7213055707168598,,0,2024/11/06 14:11:30,N/A,N/A,AppThreat-8911-9049,0x0,0,4294967295,,,88e69ca4-8783-4b7c-9982-f73ec6f1a83c,1679420,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0,2024-11-06T14:11:30.036-05:00,,,,internet-utility,generate-internet,browser-based,2,"used-bymalware,able-to-transfer-file,has-known-vulnerability,tunnel-other-application,pervasive-use",,web-browsing,no,no,
170 changes: 170 additions & 0 deletions ...panw/data_stream/panos/_dev/test/pipeline/test-panw-panos-threat-sample.log-expected.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36871,6 +36871,176 @@
"tags": [
"preserve_original_event"
]
},
{
"@timestamp": "2024-11-07T04:41:30.036+09:30",
"destination": {
"domain": "artifacts.elastic.co",
"geo": {
"name": "United States"
},
"ip": "10.71.208.15",
"port": 443
},
"ecs": {
"version": "8.11.0"
},
"event": {
"action": "file_match",
"category": [
"intrusion_detection",
"threat",
"network"
],
"kind": "alert",
"original": "<14>Nov 06 14:11:30 pa555 1,2024/11/06 14:11:30,0000000000001,THREAT,file,2562,2024/11/06 14:11:30,192.168.1.2,10.71.208.15,0.0.0.0,0.0.0.0,file download test rule,contoso\\\\steve,,web-browsing,vsys1,HOMENET,EXTNET,ethernet1/2,ethernet1/1,log-profile1,2024/11/06 14:11:30,994313,2,37268,443,0,0,0x1002000,tcp,alert,\"elastic-agent.exe\",Windows Executable (EXE)(52020),computer-and-internet-info,low,server-to-client,7367538158076100804,0x8000000000000000,192.168.0.0-192.168.255.255,United States,,,0,,,1,,,,,,,,0,199,479,0,0,,pa555,artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.15.3+build202411051926-windows-x86_64.zip,,,,7213055707168598,,0,2024/11/06 14:11:30,N/A,N/A,AppThreat-8911-9049,0x0,0,4294967295,,,88e69ca4-8783-4b7c-9982-f73ec6f1a83c,1679420,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0,2024-11-06T14:11:30.036-05:00,,,,internet-utility,generate-internet,browser-based,2,\"used-bymalware,able-to-transfer-file,has-known-vulnerability,tunnel-other-application,pervasive-use\",,web-browsing,no,no,",
"outcome": "success",
"severity": 4,
"timezone": "+09:30",
"type": [
"allowed"
]
},
"file": {
"name": "elastic-agent.exe"
},
"http": {
"version": "2"
},
"labels": {
"ssl_decrypted": true,
"temporary_match": true
},
"log": {
"level": "low"
},
"message": "192.168.1.2,10.71.208.15,0.0.0.0,0.0.0.0,file download test rule,contoso\\\\steve,,web-browsing,vsys1,HOMENET,EXTNET,ethernet1/2,ethernet1/1,log-profile1,2024/11/06 14:11:30,994313,2,37268,443,0,0,0x1002000,tcp,alert,\"elastic-agent.exe\",Windows Executable (EXE)(52020),computer-and-internet-info,low,server-to-client,7367538158076100804,0x8000000000000000,192.168.0.0-192.168.255.255,United States,,,0,,,1,,,,,,,,0,199,479,0,0,,pa555,artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.15.3+build202411051926-windows-x86_64.zip,,,,7213055707168598,,0,2024/11/06 14:11:30,N/A,N/A,AppThreat-8911-9049,0x0,0,4294967295,,,88e69ca4-8783-4b7c-9982-f73ec6f1a83c,1679420,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0,2024-11-06T14:11:30.036-05:00,,,,internet-utility,generate-internet,browser-based,2,\"used-bymalware,able-to-transfer-file,has-known-vulnerability,tunnel-other-application,pervasive-use\",,web-browsing,no,no,",
"network": {
"application": "web-browsing",
"community_id": "1:sA1Tr5zVqaZQn7PEngH9gMb+2Tg=",
"direction": "outbound",
"transport": "tcp",
"type": "ipv4"
},
"observer": {
"egress": {
"interface": {
"name": "ethernet1/1"
},
"zone": "EXTNET"
},
"hostname": "pa555",
"ingress": {
"interface": {
"name": "ethernet1/2"
},
"zone": "HOMENET"
},
"product": "PAN-OS",
"serial_number": "0000000000001",
"type": "firewall",
"vendor": "Palo Alto Networks"
},
"panw": {
"panos": {
"action": "alert",
"action_flags": "0x8000000000000000",
"application": {
"category": "generate-internet",
"characteristics": "used-bymalware,able-to-transfer-file,has-known-vulnerability,tunnel-other-application,pervasive-use",
"is_saas": "no",
"is_sanctioned": "no",
"risk_level": 2,
"sub_category": "internet-utility",
"technology": "browser-based",
"tunneled": "web-browsing"
},
"content_version": "AppThreat-8911-9049",
"device_group_hierarchy1": "199",
"device_group_hierarchy2": "479",
"device_group_hierarchy3": "0",
"device_group_hierarchy4": "0",
"flow_id": "994313",
"generated_time": "2024-11-06T14:11:30.000+09:30",
"high_resolution_timestamp": "2024-11-07T04:41:30.036+09:30",
"http2_connection": "1679420",
"imsi": "7213055707168598",
"log_profile": "log-profile1",
"logged_time": "2024-11-06T14:11:30.000+09:30",
"parent_session": {
"id": "0",
"start_time": "2024-11-06T14:11:30.000+09:30"
},
"partial_hash": "0",
"payload_protocol_id": "4294967295",
"received_time": "2024-11-06T14:11:30.000+09:30",
"repeat_count": 2,
"ruleset": "file download test rule",
"sctp": {
"assoc_id": "0"
},
"sequence_number": "7367538158076100804",
"sub_type": "file",
"threat": {
"id": "52020",
"name": "Windows Executable (EXE)"
},
"threat_category": "N/A",
"tunnel_type": "N/A",
"type": "THREAT",
"url": {
"category": "computer-and-internet-info"
},
"url_idx": "1",
"virtual_sys": "vsys1",
"wildfire": {
"report_id": "0"
}
}
},
"related": {
"hosts": [
"pa555"
],
"ip": [
"192.168.1.2",
"10.71.208.15"
],
"user": [
"steve"
]
},
"rule": {
"name": "file download test rule",
"uuid": "88e69ca4-8783-4b7c-9982-f73ec6f1a83c"
},
"session": {
"start_time": "2024-11-06T14:11:30.000+09:30"
},
"source": {
"geo": {
"name": "192.168.0.0-192.168.255.255"
},
"ip": "192.168.1.2",
"port": 37268,
"user": {
"domain": "contoso",
"name": "steve"
}
},
"tags": [
"preserve_original_event"
],
"url": {
"domain": "artifacts.elastic.co",
"extension": "zip",
"original": "artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.15.3+build202411051926-windows-x86_64.zip",
"path": "/downloads/beats/elastic-agent/elastic-agent-8.15.3+build202411051926-windows-x86_64.zip"
},
"user": {
"domain": "contoso",
"name": "steve"
}
}
]
}
6 changes: 6 additions & 0 deletions packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/threat.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,12 @@ processors:
tag: set_vulnerability_url
field: url.original
copy_from: panw.panos.misc
# It's been observed that URL is in future_use3 field when sub_type == 'file', but theres' no documentation on this usage
- set:
if: 'ctx.panw?.panos?.sub_type == "file" && ctx._temp_?.future_use3 instanceof String && ctx._temp_.future_use3.contains("/")'
tag: set_url_from_file
field: url.original
copy_from: _temp_.future_use3
# Crude implementation of `uri_parts` as its not working well due to lack of scheme.
# When the scheme of the URL is absent, this script parses the URL in `ctx.panw.panos.misc` into components namely
# `url.original`, `url.domain`, `url.port`, `url.path`, `url.query`, `url.extension`
Expand Down
2 changes: 1 addition & 1 deletion packages/panw/manifest.yml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: panw
title: Palo Alto Next-Gen Firewall
version: "4.0.4"
version: "4.1.0"
description: Collect logs from Palo Alto next-gen firewalls with Elastic Agent.
type: integration
format_version: "3.0.3"
Expand Down

0 comments on commit 60fcb22

Please sign in to comment.