diff --git a/packages/zscaler_zia/_dev/build/docs/README.md b/packages/zscaler_zia/_dev/build/docs/README.md index a54cd87b0e7..43d3ca89c65 100644 --- a/packages/zscaler_zia/_dev/build/docs/README.md +++ b/packages/zscaler_zia/_dev/build/docs/README.md @@ -1,18 +1,15 @@ # Zscaler ZIA -This integration is for Zscaler Internet Access logs. It can be used -to receive logs sent by NSS log server on respective TCP ports. +This integration is for Zscaler Internet Access logs. It can be used to receive logs sent by NSS feeds on TCP port or Cloud NSS on HTTP Endpoint input methods. -The log message is expected to be in JSON format. The data is mapped to -ECS fields where applicable and the remaining fields are written under -`zscaler_zia..*`. +The log message is expected to be in JSON format. The data is mapped to ECS fields where applicable and the remaining fields are written under `zscaler_zia..*`. -## Setup steps +## Steps for setting up NSS Feeds 1. Enable the integration with the TCP input. -2. Configure the Zscaler NSS Server and NSS Feeds to send logs to the Elastic Agent that is running this integration. See [_Add NSS Server_](https://help.zscaler.com/zia/adding-nss-servers) and [_Add NSS Feeds_](https://help.zscaler.com/zia/adding-nss-feeds). Use the IP address hostname of the Elastic Agent as the 'NSS Feed SIEM IP Address/FQDN', and use the listening port of the Elastic Agent as the 'SIEM TCP Port' on the _Add NSS Feed_ configuration screen. To configure Zscalar NSS Server and NSS Feeds follow the following steps. +2. Configure the Zscaler NSS Server and NSS Feeds to send logs to the Elastic Agent that is running this integration. See [_Add NSS Server_](https://help.zscaler.com/zia/adding-nss-servers) and [_Add NSS Feeds_](https://help.zscaler.com/zia/adding-nss-feeds). Use the IP address hostname of the Elastic Agent as the 'NSS Feed SIEM IP Address/FQDN', and use the listening port of the Elastic Agent as the 'SIEM TCP Port' on the _Add NSS Feed_ configuration screen. To configure Zscaler NSS Server and NSS Feeds follow the following steps. - In the ZIA Admin Portal, add an NSS Server. - - Log in to the ZIA Admin Portal using your admin account. If you're unable to log in, contact Support. + - Log in to the ZIA Admin Portal using your admin account. - Add an NSS server. Refer to Adding NSS Servers to set up an [_Add NSS Server_](https://help.zscaler.com/zia/adding-nss-servers) for Web and/or Firewall. - Verify that the state of the NSS Server is healthy. - In the ZIA Admin Portal, go to Administration > Nanolog Streaming Service > NSS Servers. @@ -27,10 +24,32 @@ ECS fields where applicable and the remaining fields are written under - **Firewall**: 9012 - **Tunnel**: 9013 - **Web**: 9014 - - **Feed Output Type**: Select Custom paste the appropriate response format as follows: - ![NSS feeds setup image](../img/nss_feeds.png?raw=true) - -3. *Please make sure to use the given response formats.* + - **Feed Output Type**: Select Custom in Feed output type and paste the appropriate response format in Feed output format as follows: + ![NSS Feeds setup image](../img/nss_feeds.png?raw=true) + +## Steps for setting up Cloud NSS Feeds + +1. Enable the integration with the HTTP Endpoint input. +2. Configure the Zscaler Cloud NSS Feeds to send logs to the Elastic Agent that is running this integration. Provide API URL to send logs to the Elastic Agent. To configure Zscaler Cloud NSS Feeds follow the following steps. + - In the ZIA Admin Portal, add a Cloud NSS Feed. + - Log in to the ZIA Admin Portal using your admin account. + - Add a Cloud NSS Feed. Refer to [_Add Cloud NSS Feed_](https://help.zscaler.com/zia/adding-cloud-nss-feeds). + - In the ZIA Admin Portal, go to Administration > Nanolog Streaming Service > Cloud NSS Feeds. + - Give Feed Name, change status to Enabled. + - Select NSS Type. + - Change SIEM Type to other. + - Add an API URL. + - Default ports: + - **DNS**: 9556 + - **Firewall**: 9557 + - **Tunnel**: 9558 + - **Web**: 9559 + - Select JSON as feed output type. + - Add appropriate HTTP headers. + ![Cloud NSS Feeds setup image](../img/cloud_nss_feeds.png?raw=true) +3. Repeat step 2 for each log type. + +**Please make sure to use the given response formats for NSS and Cloud NSS Feeds.** ## Compatibility @@ -40,7 +59,7 @@ This package has been tested against `Zscaler Internet Access version 6.1` ### Alerts -Default port: _9010_ +- Default port (NSS Feed): _9010_ Vendor documentation: https://help.zscaler.com/zia/about-alerts @@ -56,7 +75,8 @@ Sample Response: ### DNS Log -Default port: _9011_ +- Default port (NSS Feed): _9011_ +- Default port (Cloud NSS Feed): _9556_ Vendor documentation: https://help.zscaler.com/zia/nss-feed-output-format-dns-logs @@ -72,7 +92,8 @@ Sample Response: ### Firewall Log -Default port: _9012_ +- Default port (NSS Feed): _9012_ +- Default port (Cloud NSS Feed): _9557_ Vendor documentation: https://help.zscaler.com/zia/nss-feed-output-format-firewall-logs @@ -88,7 +109,8 @@ Sample Response: ### Tunnel Log -Default port: _9013_ +- Default port (NSS Feed): _9013_ +- Default port (Cloud NSS Feed): _9558_ Vendor documentation: https://help.zscaler.com/zia/nss-feed-output-format-tunnel-logs @@ -117,8 +139,9 @@ Sample Response: ### Web Log -Default port: _9014_ -Add characters **"** and **\\** in **feed escape character** while configuring Web Log. +- Default port (NSS Feed): _9014_ +- Default port (Cloud NSS Feed): _9559_ +- Add characters **"** and **\\** in **feed escape character** while configuring Web Log. ![Escape feed setup image](../img/escape_feed.png?raw=true) Vendor documentation: https://help.zscaler.com/zia/nss-feed-output-format-web-logs diff --git a/packages/zscaler_zia/_dev/deploy/docker/docker-compose.yml b/packages/zscaler_zia/_dev/deploy/docker/docker-compose.yml index f76b2156e6b..307924c28e7 100644 --- a/packages/zscaler_zia/_dev/deploy/docker/docker-compose.yml +++ b/packages/zscaler_zia/_dev/deploy/docker/docker-compose.yml @@ -1,32 +1,68 @@ version: '2.3' services: zscaler-zia-alerts-tcp: - image: docker.elastic.co/observability/stream:v0.6.2 + image: docker.elastic.co/observability/stream:v0.7.0 volumes: - ./sample_logs:/sample_logs:ro entrypoint: /bin/bash command: -c "/stream log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9010 -p=tcp /sample_logs/alerts.log" zscaler-zia-dns-tcp: - image: docker.elastic.co/observability/stream:v0.6.2 + image: docker.elastic.co/observability/stream:v0.7.0 volumes: - ./sample_logs:/sample_logs:ro entrypoint: /bin/bash command: -c "/stream log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9011 -p=tcp /sample_logs/dns.log" zscaler-zia-firewall-tcp: - image: docker.elastic.co/observability/stream:v0.6.2 + image: docker.elastic.co/observability/stream:v0.7.0 volumes: - ./sample_logs:/sample_logs:ro entrypoint: /bin/bash command: -c "/stream log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9012 -p=tcp /sample_logs/firewall.log" zscaler-zia-tunnel-tcp: - image: docker.elastic.co/observability/stream:v0.6.2 + image: docker.elastic.co/observability/stream:v0.7.0 volumes: - ./sample_logs:/sample_logs:ro entrypoint: /bin/bash command: -c "/stream log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9013 -p=tcp /sample_logs/tunnel.log" zscaler-zia-web-tcp: - image: docker.elastic.co/observability/stream:v0.6.2 + image: docker.elastic.co/observability/stream:v0.7.0 volumes: - ./sample_logs:/sample_logs:ro entrypoint: /bin/bash command: -c "/stream log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9014 -p=tcp /sample_logs/web.log" + zscaler-zia-dns-http-endpoint: + image: docker.elastic.co/observability/stream:v0.7.0 + volumes: + - ./sample_logs:/sample_logs:ro + environment: + - STREAM_PROTOCOL=webhook + - STREAM_ADDR=http://elastic-agent:9556/ + - STREAM_WEBHOOK_HEADER=Content-Type=application/ndjson + command: log --start-signal=SIGHUP --delay=5s /sample_logs/dns-http_endpoint.log + zscaler-zia-firewall-http-endpoint: + image: docker.elastic.co/observability/stream:v0.7.0 + volumes: + - ./sample_logs:/sample_logs:ro + environment: + - STREAM_PROTOCOL=webhook + - STREAM_ADDR=http://elastic-agent:9557/ + - STREAM_WEBHOOK_HEADER=Content-Type=application/ndjson + command: log --start-signal=SIGHUP --delay=5s /sample_logs/firewall-http_endpoint.log + zscaler-zia-tunnel-http-endpoint: + image: docker.elastic.co/observability/stream:v0.7.0 + volumes: + - ./sample_logs:/sample_logs:ro + environment: + - STREAM_PROTOCOL=webhook + - STREAM_ADDR=http://elastic-agent:9558/ + - STREAM_WEBHOOK_HEADER=Content-Type=application/ndjson + command: log --start-signal=SIGHUP --delay=5s /sample_logs/tunnel-http_endpoint.log + zscaler-zia-web-http-endpoint: + image: docker.elastic.co/observability/stream:v0.7.0 + volumes: + - ./sample_logs:/sample_logs:ro + environment: + - STREAM_PROTOCOL=webhook + - STREAM_ADDR=http://elastic-agent:9559/ + - STREAM_WEBHOOK_HEADER=Content-Type=application/ndjson + command: log --start-signal=SIGHUP --delay=5s /sample_logs/web-http_endpoint.log diff --git a/packages/zscaler_zia/_dev/deploy/docker/sample_logs/dns-http_endpoint.log b/packages/zscaler_zia/_dev/deploy/docker/sample_logs/dns-http_endpoint.log new file mode 100644 index 00000000000..d4ec642282d --- /dev/null +++ b/packages/zscaler_zia/_dev/deploy/docker/sample_logs/dns-http_endpoint.log @@ -0,0 +1 @@ +{"sourcetype":"zscalernss-dns","input": {"type": "http_endpoint"}, "event":{"location":"Unknown","deviceowner":"NA","devicehostname":"NA","dns_req":"Unknown","resaction":"None","durationms":"34000","category":"Other","resrulelabel":"None","dns_reqtype":"NotFound","dns_resp":"NotFound","department":"Unknown","user":"Unknown","reqaction":"None","datetime":"Tue Dec 31 02:22:22 2021","srv_dip":"0.0.0.0","clt_sip":"0.0.0.0","reqrulelabel":"None","srv_dport":"0"}} diff --git a/packages/zscaler_zia/_dev/deploy/docker/sample_logs/dns.log b/packages/zscaler_zia/_dev/deploy/docker/sample_logs/dns.log index 747c8eb690b..3064437fdc3 100644 --- a/packages/zscaler_zia/_dev/deploy/docker/sample_logs/dns.log +++ b/packages/zscaler_zia/_dev/deploy/docker/sample_logs/dns.log @@ -1 +1 @@ -{ "sourcetype" : "zscalernss-dns", "event" :{"datetime":"Fri Dec 17 07:27:54 2021","user":"some_user@example.com","department":"Unknown","location":"TestLoc%20DB","reqaction":"REQ_ALLOW","resaction":"Some Response Action","reqrulelabel":"Access%20Blocked","resrulelabel":"None","dns_reqtype":"Some type","dns_req":"example.com","dns_resp":"Some response string","srv_dport":"8080","durationms":"123456","clt_sip":"81.2.69.193","srv_dip":"81.2.69.144","category":"Professional Services","deviceowner":"Owner77","devicehostname":"Machine9000"}} +{ "sourcetype" : "zscalernss-dns", "event" :{"datetime":"Fri Dec 17 07:27:54 2021","user":"some_user@example.com","department":"Unknown","location":"TestLoc%20DB","reqaction":"REQ_ALLOW","resaction":"Some Response Action","reqrulelabel":"Access%20Blocked","resrulelabel":"None","dns_reqtype":"Some type","dns_req":"example.com","dns_resp":"Some response string","srv_dport":"8080","durationms":"123456","clt_sip":"89.160.20.112","srv_dip":"89.160.20.156","category":"Professional Services","deviceowner":"Owner77","devicehostname":"Machine9000"}} diff --git a/packages/zscaler_zia/_dev/deploy/docker/sample_logs/firewall-http_endpoint.log b/packages/zscaler_zia/_dev/deploy/docker/sample_logs/firewall-http_endpoint.log new file mode 100644 index 00000000000..3c472cc3b4b --- /dev/null +++ b/packages/zscaler_zia/_dev/deploy/docker/sample_logs/firewall-http_endpoint.log @@ -0,0 +1 @@ +{"sourcetype":"zscalernss-fw", "event":{"durationms":"0","avgduration":"0","sdip":"0.0.0.0","aggregate":"No","department":"Unknown","nwapp":"NotAvailable","proto":"IP","datetime":"Tue Dec 31 02:22:22 2022","nwsvc":"None","dnat":"No","threatcat":"None","cdport":"120","duration":"0","ipcat":"Other","deviceowner":"NA","csip":"0.0.0.0","devicehostname":"NA","csport":"0","tunsport":"0","destcountry":"NA","rulelabel":"None","locationname":"Unknown","action":"OutOfRange","stateful":"Yes","outbytes":"0","inbytes":"0","ssport":"0","user":"Unknown","tuntype":"OutOfRange","numsessions":"1","ssip":"0.0.0.0","threatname":"None","ipsrulelabel":"None","tsip":"0.0.0.0","sdport":"0","cdip":"0.0.0.0"}} diff --git a/packages/zscaler_zia/_dev/deploy/docker/sample_logs/tunnel-http_endpoint.log b/packages/zscaler_zia/_dev/deploy/docker/sample_logs/tunnel-http_endpoint.log new file mode 100644 index 00000000000..61ae92fd0ba --- /dev/null +++ b/packages/zscaler_zia/_dev/deploy/docker/sample_logs/tunnel-http_endpoint.log @@ -0,0 +1 @@ +{"sourcetype":"zscalernss-tunnel", "event":{"location":"Unknown","sourceport":"0","rxbytes":"0","tunneltype":"GRE","dpdrec":"0","destinationip":"0.0.0.0","recordid":"7083020000000007968","datetime":"Tue Dec 31 08:08:08 2021","sourceip":"0.0.0.0","txbytes":"0","Recordtype":"Tunnel Samples","user":"Unknown"}} diff --git a/packages/zscaler_zia/_dev/deploy/docker/sample_logs/web-http_endpoint.log b/packages/zscaler_zia/_dev/deploy/docker/sample_logs/web-http_endpoint.log new file mode 100644 index 00000000000..329456f3000 --- /dev/null +++ b/packages/zscaler_zia/_dev/deploy/docker/sample_logs/web-http_endpoint.log @@ -0,0 +1 @@ +{ "sourcetype" : "zscalernss-web", "event" :{"time":"2021-12-31 08:08:08","login":"test@example.com","proto":"HTTP_PROXY","eurl":"www.example.com","action":"Blocked","appname":"General Browsing","appclass":"General Browsing","reqsize":"600","respsize":"65","stime":"0","ctime":"0","urlclass":"Business Use","urlsupercat":"Information Technology","urlcat":"Web Search","malwarecat":"None","threatname":"None","riskscore":"0","dlpeng":"None","dlpdict":"None","location":"Test DB","dept":"Unknown","cip":"1.128.3.4","sip":"1.128.3.4","reqmethod":"CONNECT","respcode":"200","ua":"Windows Microsoft Windows 10 Pro ZTunnel/1.0","ereferer":"None","ruletype":"FwFilter","rulelabel":"Zscaler Proxy Traffic","contenttype":"Other","unscannabletype":"None","deviceowner":"administrator1","devicehostname":"TestMachine35"}} diff --git a/packages/zscaler_zia/changelog.yml b/packages/zscaler_zia/changelog.yml index a977c5ffc2c..e314cbf50d8 100644 --- a/packages/zscaler_zia/changelog.yml +++ b/packages/zscaler_zia/changelog.yml @@ -1,4 +1,9 @@ # newer versions go on top +- version: "2.0.0" + changes: + - description: Added input for Cloud NSS using HTTP Endpoint input type. + type: enhancement + link: https://github.com/elastic/integrations/pull/3111 - version: "0.2.0" changes: - description: Update ECS to 8.2 @@ -6,12 +11,12 @@ link: https://github.com/elastic/integrations/pull/2781 - version: "0.1.3" changes: - - description: Updated the image file reference in README file + - description: Updated the image file reference in README file. type: enhancement link: https://github.com/elastic/integrations/pull/3038 - version: "0.1.2" changes: - - description: Add documentation for multi-fields + - description: Add documentation for multi-fields. type: enhancement link: https://github.com/elastic/integrations/pull/2916 - version: "0.1.1" @@ -21,6 +26,6 @@ link: https://github.com/elastic/integrations/pull/2773 - version: "0.1.0" changes: - - description: Initial draft of the package + - description: Initial draft of the package. type: enhancement link: https://github.com/elastic/integrations/pull/2459 diff --git a/packages/zscaler_zia/data_stream/alerts/_dev/test/pipeline/test-alerts.log b/packages/zscaler_zia/data_stream/alerts/_dev/test/pipeline/test-alerts.log index 564e8e7178c..0ee51313e0d 100644 --- a/packages/zscaler_zia/data_stream/alerts/_dev/test/pipeline/test-alerts.log +++ b/packages/zscaler_zia/data_stream/alerts/_dev/test/pipeline/test-alerts.log @@ -1,3 +1,3 @@ -<114>Dec 10 14:04:28 [175.16.199.1] ZscalerNSS: Zscaler cloud configuration connection to 175.16.199.1:443 lost and unavailable for the past 2325.00 minutes -<114>Dec 10 13:40:32 [81.2.69.193] ZscalerNSS: SIEM Feed connection "DNS Logs Feed" to 81.2.69.193:9012 lost and unavailable for the past 2440.00 minutes -<114>Dec 10 13:44:07 [81.2.69.193] Hey, that's a new type of alert. Isn't it? +<114>Dec 31 12:01:04 [175.16.199.1] ZscalerNSS: Zscaler cloud configuration connection to 175.16.199.1:443 lost and unavailable for the past 2325.00 minutes +<114>Dec 31 13:02:05 [81.2.69.193] ZscalerNSS: SIEM Feed connection "DNS Logs Feed" to 81.2.69.193:9012 lost and unavailable for the past 2440.00 minutes +<114>Dec 31 14:03:06 [81.2.69.193] Hey, that's a new type of alert. Isn't it? diff --git a/packages/zscaler_zia/data_stream/alerts/_dev/test/pipeline/test-alerts.log-expected.json b/packages/zscaler_zia/data_stream/alerts/_dev/test/pipeline/test-alerts.log-expected.json index ff32b989cb4..54b51a184ea 100644 --- a/packages/zscaler_zia/data_stream/alerts/_dev/test/pipeline/test-alerts.log-expected.json +++ b/packages/zscaler_zia/data_stream/alerts/_dev/test/pipeline/test-alerts.log-expected.json @@ -1,7 +1,7 @@ { "expected": [ { - "@timestamp": "2022-12-10T14:04:28.000Z", + "@timestamp": "2022-12-31T12:01:04.000Z", "destination": { "address": "175.16.199.1", "ip": "175.16.199.1", @@ -11,7 +11,7 @@ "version": "8.2.0" }, "event": { - "original": "\u003c114\u003eDec 10 14:04:28 [175.16.199.1] ZscalerNSS: Zscaler cloud configuration connection to 175.16.199.1:443 lost and unavailable for the past 2325.00 minutes" + "original": "\u003c114\u003eDec 31 12:01:04 [175.16.199.1] ZscalerNSS: Zscaler cloud configuration connection to 175.16.199.1:443 lost and unavailable for the past 2325.00 minutes" }, "log": { "syslog": { @@ -34,7 +34,7 @@ } }, { - "@timestamp": "2022-12-10T13:40:32.000Z", + "@timestamp": "2022-12-31T13:02:05.000Z", "destination": { "address": "81.2.69.193", "ip": "81.2.69.193", @@ -44,7 +44,7 @@ "version": "8.2.0" }, "event": { - "original": "\u003c114\u003eDec 10 13:40:32 [81.2.69.193] ZscalerNSS: SIEM Feed connection \"DNS Logs Feed\" to 81.2.69.193:9012 lost and unavailable for the past 2440.00 minutes" + "original": "\u003c114\u003eDec 31 13:02:05 [81.2.69.193] ZscalerNSS: SIEM Feed connection \"DNS Logs Feed\" to 81.2.69.193:9012 lost and unavailable for the past 2440.00 minutes" }, "log": { "syslog": { @@ -68,7 +68,7 @@ } }, { - "@timestamp": "2022-12-10T13:44:07.000Z", + "@timestamp": "2022-12-31T14:03:06.000Z", "destination": { "address": "81.2.69.193", "ip": "81.2.69.193" @@ -77,7 +77,7 @@ "version": "8.2.0" }, "event": { - "original": "\u003c114\u003eDec 10 13:44:07 [81.2.69.193] Hey, that's a new type of alert. Isn't it?" + "original": "\u003c114\u003eDec 31 14:03:06 [81.2.69.193] Hey, that's a new type of alert. Isn't it?" }, "log": { "syslog": { diff --git a/packages/zscaler_zia/data_stream/alerts/agent/stream/tcp.yml.hbs b/packages/zscaler_zia/data_stream/alerts/agent/stream/tcp.yml.hbs index 49e3770cb1d..6910573304d 100644 --- a/packages/zscaler_zia/data_stream/alerts/agent/stream/tcp.yml.hbs +++ b/packages/zscaler_zia/data_stream/alerts/agent/stream/tcp.yml.hbs @@ -1,4 +1,3 @@ -tcp: host: "{{listen_address}}:{{listen_port}}" tags: {{#if preserve_original_event}} diff --git a/packages/zscaler_zia/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml b/packages/zscaler_zia/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml index ecf2838641e..f29273dfca2 100644 --- a/packages/zscaler_zia/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml +++ b/packages/zscaler_zia/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml @@ -23,12 +23,12 @@ processors: target_field: destination.ip type: ip ignore_failure: true - - append: + - append: field: related.ip value: "{{{destination.ip}}}" if: ctx?.destination?.ip != null ignore_failure: true - - date: + - date: field: _tmp.timestamp target_field: '@timestamp' ignore_failure: true @@ -37,8 +37,8 @@ processors: - MMM dd HH:mm:ss - MMM d HH:mm:ss - ISO8601 - - remove: - field: + - remove: + field: - _tmp ignore_missing: true - remove: diff --git a/packages/zscaler_zia/data_stream/alerts/manifest.yml b/packages/zscaler_zia/data_stream/alerts/manifest.yml index 1bd14b8c53d..6dec78145d8 100644 --- a/packages/zscaler_zia/data_stream/alerts/manifest.yml +++ b/packages/zscaler_zia/data_stream/alerts/manifest.yml @@ -4,7 +4,7 @@ streams: - input: tcp template_path: tcp.yml.hbs title: Zscaler Internet Access Alerts - description: Collect Zscaler Internet Access Alerts using tcp input + description: Collect Zscaler Internet Access Alerts using TCP Input. vars: - name: listen_port type: integer @@ -27,7 +27,7 @@ streams: required: true show_user: true title: Preserve original event - description: Preserves a raw copy of the original event, added to the field `event.original` + description: Preserves a raw copy of the original event, added to the field `event.original`. type: bool multi: false default: false diff --git a/packages/zscaler_zia/data_stream/alerts/sample_event.json b/packages/zscaler_zia/data_stream/alerts/sample_event.json index ee7917801a3..22f6aca8890 100644 --- a/packages/zscaler_zia/data_stream/alerts/sample_event.json +++ b/packages/zscaler_zia/data_stream/alerts/sample_event.json @@ -1,12 +1,11 @@ { "@timestamp": "2022-12-10T13:40:32.000Z", "agent": { - "ephemeral_id": "8c093fcf-fb2f-4baa-b794-40edb011194d", - "hostname": "docker-fleet-agent", - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "ephemeral_id": "b7f77db9-92fe-4935-8387-b2cb545bcfc6", + "id": "638019f9-173e-4c24-9e28-64b128c92162", "name": "docker-fleet-agent", "type": "filebeat", - "version": "7.16.2" + "version": "8.1.2" }, "data_stream": { "dataset": "zscaler_zia.alerts", @@ -22,21 +21,21 @@ "version": "8.2.0" }, "elastic_agent": { - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "id": "638019f9-173e-4c24-9e28-64b128c92162", "snapshot": false, - "version": "7.16.2" + "version": "8.1.2" }, "event": { "agent_id_status": "verified", "dataset": "zscaler_zia.alerts", - "ingested": "2022-02-04T06:31:25Z" + "ingested": "2022-04-13T17:21:34Z" }, "input": { "type": "tcp" }, "log": { "source": { - "address": "172.21.0.7:32902" + "address": "1.128.3.4:32902" }, "syslog": { "priority": 114 diff --git a/packages/zscaler_zia/data_stream/dns/_dev/test/pipeline/test-dns-http_endpoint.log b/packages/zscaler_zia/data_stream/dns/_dev/test/pipeline/test-dns-http_endpoint.log new file mode 100644 index 00000000000..d4ec642282d --- /dev/null +++ b/packages/zscaler_zia/data_stream/dns/_dev/test/pipeline/test-dns-http_endpoint.log @@ -0,0 +1 @@ +{"sourcetype":"zscalernss-dns","input": {"type": "http_endpoint"}, "event":{"location":"Unknown","deviceowner":"NA","devicehostname":"NA","dns_req":"Unknown","resaction":"None","durationms":"34000","category":"Other","resrulelabel":"None","dns_reqtype":"NotFound","dns_resp":"NotFound","department":"Unknown","user":"Unknown","reqaction":"None","datetime":"Tue Dec 31 02:22:22 2021","srv_dip":"0.0.0.0","clt_sip":"0.0.0.0","reqrulelabel":"None","srv_dport":"0"}} diff --git a/packages/zscaler_zia/data_stream/dns/_dev/test/pipeline/test-dns-http_endpoint.log-expected.json b/packages/zscaler_zia/data_stream/dns/_dev/test/pipeline/test-dns-http_endpoint.log-expected.json new file mode 100644 index 00000000000..a893ab0c5b6 --- /dev/null +++ b/packages/zscaler_zia/data_stream/dns/_dev/test/pipeline/test-dns-http_endpoint.log-expected.json @@ -0,0 +1,80 @@ +{ + "expected": [ + { + "@timestamp": "2021-12-31T02:22:22.000Z", + "destination": { + "ip": "0.0.0.0", + "port": 0 + }, + "dns": { + "answers": { + "name": "NotFound" + }, + "question": { + "name": "Unknown", + "type": "NotFound" + } + }, + "ecs": { + "version": "8.2.0" + }, + "event": { + "category": [ + "network" + ], + "duration": 34000000000, + "kind": "event", + "original": "{\"sourcetype\":\"zscalernss-dns\",\"input\": {\"type\": \"http_endpoint\"}, \"event\":{\"location\":\"Unknown\",\"deviceowner\":\"NA\",\"devicehostname\":\"NA\",\"dns_req\":\"Unknown\",\"resaction\":\"None\",\"durationms\":\"34000\",\"category\":\"Other\",\"resrulelabel\":\"None\",\"dns_reqtype\":\"NotFound\",\"dns_resp\":\"NotFound\",\"department\":\"Unknown\",\"user\":\"Unknown\",\"reqaction\":\"None\",\"datetime\":\"Tue Dec 31 02:22:22 2021\",\"srv_dip\":\"0.0.0.0\",\"clt_sip\":\"0.0.0.0\",\"reqrulelabel\":\"None\",\"srv_dport\":\"0\"}}", + "type": [ + "info" + ] + }, + "network": { + "protocol": "dns" + }, + "related": { + "hosts": [ + "NA" + ], + "ip": [ + "0.0.0.0" + ] + }, + "source": { + "ip": "0.0.0.0" + }, + "tags": [ + "preserve_original_event" + ], + "user": { + "email": "Unknown", + "name": "NA" + }, + "zscaler_zia": { + "dns": { + "department": "Unknown", + "dom": { + "category": "Other" + }, + "duration": { + "milliseconds": 34000 + }, + "hostname": "NA", + "location": "Unknown", + "request": { + "action": "None", + "rule": { + "label": "None" + } + }, + "response": { + "action": "None", + "rule": { + "label": "None" + } + } + } + } + } + ] +} \ No newline at end of file diff --git a/packages/zscaler_zia/data_stream/dns/_dev/test/pipeline/test-dns.log b/packages/zscaler_zia/data_stream/dns/_dev/test/pipeline/test-dns.log index 747c8eb690b..1d49ab95b7e 100644 --- a/packages/zscaler_zia/data_stream/dns/_dev/test/pipeline/test-dns.log +++ b/packages/zscaler_zia/data_stream/dns/_dev/test/pipeline/test-dns.log @@ -1 +1 @@ -{ "sourcetype" : "zscalernss-dns", "event" :{"datetime":"Fri Dec 17 07:27:54 2021","user":"some_user@example.com","department":"Unknown","location":"TestLoc%20DB","reqaction":"REQ_ALLOW","resaction":"Some Response Action","reqrulelabel":"Access%20Blocked","resrulelabel":"None","dns_reqtype":"Some type","dns_req":"example.com","dns_resp":"Some response string","srv_dport":"8080","durationms":"123456","clt_sip":"81.2.69.193","srv_dip":"81.2.69.144","category":"Professional Services","deviceowner":"Owner77","devicehostname":"Machine9000"}} +{ "sourcetype" : "zscalernss-dns", "event" :{"datetime":"Fri Dec 31 01:11:11 2021","user":"some_user@example.com","department":"Unknown","location":"TestLoc%20DB","reqaction":"REQ_ALLOW","resaction":"Some Response Action","reqrulelabel":"Access%20Blocked","resrulelabel":"None","dns_reqtype":"Some type","dns_req":"example.com","dns_resp":"Some response string","srv_dport":"8080","durationms":"123456","clt_sip":"89.160.20.112","srv_dip":"89.160.20.156","category":"Professional Services","deviceowner":"Owner77","devicehostname":"Machine9000"}} diff --git a/packages/zscaler_zia/data_stream/dns/_dev/test/pipeline/test-dns.log-expected.json b/packages/zscaler_zia/data_stream/dns/_dev/test/pipeline/test-dns.log-expected.json index 89acc126516..21b35962c29 100644 --- a/packages/zscaler_zia/data_stream/dns/_dev/test/pipeline/test-dns.log-expected.json +++ b/packages/zscaler_zia/data_stream/dns/_dev/test/pipeline/test-dns.log-expected.json @@ -1,21 +1,28 @@ { "expected": [ { - "@timestamp": "2021-12-17T07:27:54.000Z", - "client": { + "@timestamp": "2021-12-31T01:11:11.000Z", + "destination": { + "as": { + "number": 29518, + "organization": { + "name": "Bredband2 AB" + } + }, "geo": { - "city_name": "London", + "city_name": "Linköping", "continent_name": "Europe", - "country_iso_code": "GB", - "country_name": "United Kingdom", + "country_iso_code": "SE", + "country_name": "Sweden", "location": { - "lat": 51.5142, - "lon": -0.0931 + "lat": 58.4167, + "lon": 15.6167 }, - "region_iso_code": "GB-ENG", - "region_name": "England" + "region_iso_code": "SE-E", + "region_name": "Östergötland County" }, - "ip": "81.2.69.193" + "ip": "89.160.20.156", + "port": 8080 }, "dns": { "answers": { @@ -30,37 +37,48 @@ "version": "8.2.0" }, "event": { - "category": "network", + "category": [ + "network" + ], + "duration": 123456000000, "kind": "event", - "original": "{ \"sourcetype\" : \"zscalernss-dns\", \"event\" :{\"datetime\":\"Fri Dec 17 07:27:54 2021\",\"user\":\"some_user@example.com\",\"department\":\"Unknown\",\"location\":\"TestLoc%20DB\",\"reqaction\":\"REQ_ALLOW\",\"resaction\":\"Some Response Action\",\"reqrulelabel\":\"Access%20Blocked\",\"resrulelabel\":\"None\",\"dns_reqtype\":\"Some type\",\"dns_req\":\"example.com\",\"dns_resp\":\"Some response string\",\"srv_dport\":\"8080\",\"durationms\":\"123456\",\"clt_sip\":\"81.2.69.193\",\"srv_dip\":\"81.2.69.144\",\"category\":\"Professional Services\",\"deviceowner\":\"Owner77\",\"devicehostname\":\"Machine9000\"}}", + "original": "{ \"sourcetype\" : \"zscalernss-dns\", \"event\" :{\"datetime\":\"Fri Dec 31 01:11:11 2021\",\"user\":\"some_user@example.com\",\"department\":\"Unknown\",\"location\":\"TestLoc%20DB\",\"reqaction\":\"REQ_ALLOW\",\"resaction\":\"Some Response Action\",\"reqrulelabel\":\"Access%20Blocked\",\"resrulelabel\":\"None\",\"dns_reqtype\":\"Some type\",\"dns_req\":\"example.com\",\"dns_resp\":\"Some response string\",\"srv_dport\":\"8080\",\"durationms\":\"123456\",\"clt_sip\":\"89.160.20.112\",\"srv_dip\":\"89.160.20.156\",\"category\":\"Professional Services\",\"deviceowner\":\"Owner77\",\"devicehostname\":\"Machine9000\"}}", "type": [ "info" ] }, + "network": { + "protocol": "dns" + }, "related": { "hosts": [ "Machine9000" ], "ip": [ - "81.2.69.193", - "81.2.69.144" + "89.160.20.112", + "89.160.20.156" ] }, - "server": { + "source": { + "as": { + "number": 29518, + "organization": { + "name": "Bredband2 AB" + } + }, "geo": { - "city_name": "London", + "city_name": "Linköping", "continent_name": "Europe", - "country_iso_code": "GB", - "country_name": "United Kingdom", + "country_iso_code": "SE", + "country_name": "Sweden", "location": { - "lat": 51.5142, - "lon": -0.0931 + "lat": 58.4167, + "lon": 15.6167 }, - "region_iso_code": "GB-ENG", - "region_name": "England" + "region_iso_code": "SE-E", + "region_name": "Östergötland County" }, - "ip": "81.2.69.144", - "port": 8080 + "ip": "89.160.20.112" }, "tags": [ "preserve_original_event" diff --git a/packages/zscaler_zia/data_stream/dns/_dev/test/system/test-http-endpoint-config.yml b/packages/zscaler_zia/data_stream/dns/_dev/test/system/test-http-endpoint-config.yml new file mode 100644 index 00000000000..fc0df672c48 --- /dev/null +++ b/packages/zscaler_zia/data_stream/dns/_dev/test/system/test-http-endpoint-config.yml @@ -0,0 +1,8 @@ +service: zscaler-zia-dns-http-endpoint +service_notify_signal: SIGHUP +input: http_endpoint +vars: + listen_address: 0.0.0.0 +data_stream: + vars: + listen_port: 9556 diff --git a/packages/zscaler_zia/data_stream/dns/_dev/test/system/test-default-config.yml b/packages/zscaler_zia/data_stream/dns/_dev/test/system/test-tcp-config.yml similarity index 92% rename from packages/zscaler_zia/data_stream/dns/_dev/test/system/test-default-config.yml rename to packages/zscaler_zia/data_stream/dns/_dev/test/system/test-tcp-config.yml index aceefd2a166..c92d298cbf5 100644 --- a/packages/zscaler_zia/data_stream/dns/_dev/test/system/test-default-config.yml +++ b/packages/zscaler_zia/data_stream/dns/_dev/test/system/test-tcp-config.yml @@ -1,5 +1,6 @@ service: zscaler-zia-dns-tcp service_notify_signal: SIGHUP +input: tcp vars: listen_address: 0.0.0.0 data_stream: diff --git a/packages/zscaler_zia/data_stream/dns/agent/stream/http_endpoint.yml.hbs b/packages/zscaler_zia/data_stream/dns/agent/stream/http_endpoint.yml.hbs new file mode 100644 index 00000000000..443fe325f7c --- /dev/null +++ b/packages/zscaler_zia/data_stream/dns/agent/stream/http_endpoint.yml.hbs @@ -0,0 +1,21 @@ +listen_address: {{listen_address}} +listen_port: {{listen_port}} +content_type: "" +preserve_original_event: true +tags: +{{#if preserve_original_event}} + - preserve_original_event +{{/if}} +{{#each tags as |tag|}} + - {{tag}} +{{/each}} +{{#contains "forwarded" tags}} +publisher_pipeline.disable_host: true +{{/contains}} +{{#if ssl}} +ssl: {{ssl}} +{{/if}} +{{#if processors}} +processors: +{{processors}} +{{/if}} diff --git a/packages/zscaler_zia/data_stream/dns/agent/stream/tcp.yml.hbs b/packages/zscaler_zia/data_stream/dns/agent/stream/tcp.yml.hbs index 030459f2582..bc587e50a3a 100644 --- a/packages/zscaler_zia/data_stream/dns/agent/stream/tcp.yml.hbs +++ b/packages/zscaler_zia/data_stream/dns/agent/stream/tcp.yml.hbs @@ -1,4 +1,3 @@ -tcp: host: "{{listen_address}}:{{listen_port}}" tags: {{#if preserve_original_event}} diff --git a/packages/zscaler_zia/data_stream/dns/elasticsearch/ingest_pipeline/default.yml b/packages/zscaler_zia/data_stream/dns/elasticsearch/ingest_pipeline/default.yml index d6361516e52..db3c13014f7 100644 --- a/packages/zscaler_zia/data_stream/dns/elasticsearch/ingest_pipeline/default.yml +++ b/packages/zscaler_zia/data_stream/dns/elasticsearch/ingest_pipeline/default.yml @@ -11,10 +11,16 @@ processors: - json: field: event.original target_field: resp + ignore_failure: true + - remove: + field: json + if: ctx?.input?.type == 'http_endpoint' + ignore_missing: true - rename: field: resp.event target_field: json - - remove: + ignore_missing: true + - remove: field: resp ignore_missing: true - date: @@ -25,10 +31,25 @@ processors: - E MMM dd HH:mm:ss yyyy - E MMM d HH:mm:ss yyyy - E MMM d HH:mm:ss yyyy + - yyyy-mm-dd HH:mm:ss + - date: + field: json.time + target_field: "@timestamp" + ignore_failure: true + formats: + - E MMM dd HH:mm:ss yyyy + - E MMM d HH:mm:ss yyyy + - E MMM d HH:mm:ss yyyy + - yyyy-MM-dd HH:mm:ss - remove: - field: json.datetime + field: + - json.time + - json.datetime ignore_missing: true - set: + field: network.protocol + value: dns + - append: field: event.category value: network - set: @@ -37,78 +58,94 @@ processors: - append: field: event.type value: info - - rename: + - rename: field: json.clt_sip - target_field: client.ip + target_field: source.ip ignore_missing: true - geoip: - field: client.ip - target_field: client.geo + field: source.ip + target_field: source.geo ignore_missing: true - geoip: database_file: GeoLite2-ASN.mmdb - field: client.ip - target_field: client.as + field: source.ip + target_field: source.as properties: - asn - organization_name ignore_missing: true + - rename: + field: source.as.asn + target_field: source.as.number + ignore_missing: true + - rename: + field: source.as.organization_name + target_field: source.as.organization.name + ignore_missing: true - append: field: related.ip - value: "{{{client.ip}}}" - if: ctx?.client?.ip != null + value: "{{{source.ip}}}" + if: ctx?.source?.ip != null allow_duplicates: false ignore_failure: true - - rename: + - rename: field: json.dns_resp target_field: dns.answers.name ignore_missing: true - - rename: + - rename: field: json.dns_req target_field: dns.question.name ignore_missing: true - - rename: + - rename: field: json.dns_reqtype target_field: dns.question.type ignore_missing: true - - rename: + - rename: field: json.srv_dip - target_field: server.ip + target_field: destination.ip ignore_missing: true - geoip: - field: server.ip - target_field: server.geo + field: destination.ip + target_field: destination.geo ignore_missing: true - geoip: database_file: GeoLite2-ASN.mmdb - field: server.ip - target_field: server.as + field: destination.ip + target_field: destination.as properties: - asn - organization_name ignore_missing: true + - rename: + field: destination.as.asn + target_field: destination.as.number + ignore_missing: true + - rename: + field: destination.as.organization_name + target_field: destination.as.organization.name + ignore_missing: true - append: field: related.ip - value: "{{{server.ip}}}" - if: ctx?.server?.ip != null + value: "{{{destination.ip}}}" + if: ctx?.destination?.ip != null allow_duplicates: false ignore_failure: true - - convert: + - convert: field: json.srv_dport - target_field: server.port + target_field: destination.port type: long ignore_failure: true - remove: field: json.srv_dport ignore_missing: true - - urldecode: + - urldecode: field: json.user target_field: user.email ignore_missing: true - - remove: + - remove: field: json.user ignore_missing: true - - rename: + - rename: field: json.deviceowner target_field: user.name ignore_missing: true @@ -116,14 +153,14 @@ processors: field: json.department target_field: zscaler_zia.dns.department ignore_missing: true - - remove: + - remove: field: json.department ignore_missing: true - urldecode: field: json.location target_field: zscaler_zia.dns.location ignore_missing: true - - remove: + - remove: field: json.location ignore_missing: true - rename: @@ -138,14 +175,14 @@ processors: field: json.reqrulelabel target_field: zscaler_zia.dns.request.rule.label ignore_missing: true - - remove: + - remove: field: json.reqrulelabel ignore_missing: true - urldecode: field: json.resrulelabel target_field: zscaler_zia.dns.response.rule.label ignore_missing: true - - remove: + - remove: field: json.resrulelabel ignore_missing: true - convert: @@ -195,7 +232,12 @@ processors: for (Map.Entry m : ctx.json.entrySet()) { ctx.zscaler_zia.dns[m.getKey()] = m.getValue(); } - - remove: + - script: + lang: painless + if: ctx?.zscaler_zia?.dns?.duration?.milliseconds != null + source: | + ctx.event.duration = ctx?.zscaler_zia?.dns?.duration?.milliseconds * 1000000; + - remove: field: json ignore_failure: true - remove: diff --git a/packages/zscaler_zia/data_stream/dns/fields/ecs.yml b/packages/zscaler_zia/data_stream/dns/fields/ecs.yml index f1d44f3f15e..3a983be9735 100644 --- a/packages/zscaler_zia/data_stream/dns/fields/ecs.yml +++ b/packages/zscaler_zia/data_stream/dns/fields/ecs.yml @@ -1,21 +1,25 @@ - external: ecs - name: client.geo.city_name + name: destination.as.number - external: ecs - name: client.geo.continent_name + name: destination.as.organization.name - external: ecs - name: client.geo.country_iso_code + name: destination.geo.city_name - external: ecs - name: client.geo.country_name -- description: Longitude and latitude. - level: core - name: client.geo.location - type: geo_point + name: destination.geo.continent_name - external: ecs - name: client.geo.region_iso_code + name: destination.geo.country_iso_code - external: ecs - name: client.geo.region_name + name: destination.geo.country_name - external: ecs - name: client.ip + name: destination.geo.location +- external: ecs + name: destination.geo.region_iso_code +- external: ecs + name: destination.geo.region_name +- external: ecs + name: destination.ip +- external: ecs + name: destination.port - external: ecs name: dns.answers.name - external: ecs @@ -25,31 +29,33 @@ - external: ecs name: ecs.version - external: ecs - name: related.ip + name: event.duration +- external: ecs + name: network.protocol - external: ecs name: related.hosts - external: ecs - name: server.geo.city_name + name: related.ip - external: ecs - name: server.geo.continent_name + name: source.as.number - external: ecs - name: server.geo.country_iso_code + name: source.as.organization.name - external: ecs - name: server.geo.country_name -- description: Longitude and latitude. - level: core - name: server.geo.location - type: geo_point + name: source.bytes - external: ecs - name: server.geo.region_iso_code + name: source.geo.city_name - external: ecs - name: server.geo.region_name + name: source.geo.continent_name - external: ecs - name: server.ip + name: source.geo.country_iso_code - external: ecs - name: server.port + name: source.geo.country_name - external: ecs - name: source.bytes + name: source.geo.location +- external: ecs + name: source.geo.region_iso_code +- external: ecs + name: source.geo.region_name - external: ecs name: source.ip - external: ecs diff --git a/packages/zscaler_zia/data_stream/dns/manifest.yml b/packages/zscaler_zia/data_stream/dns/manifest.yml index a47e838e8c3..f50fa29323f 100644 --- a/packages/zscaler_zia/data_stream/dns/manifest.yml +++ b/packages/zscaler_zia/data_stream/dns/manifest.yml @@ -4,7 +4,7 @@ streams: - input: tcp template_path: tcp.yml.hbs title: Zscaler Internet Access DNS Logs - description: Collect Zscaler Internet Access DNS logs using tcp input + description: Collect Zscaler Internet Access DNS logs using TCP Input. vars: - name: listen_port type: integer @@ -27,7 +27,45 @@ streams: required: true show_user: true title: Preserve original event - description: Preserves a raw copy of the original event, added to the field `event.original` + description: Preserves a raw copy of the original event, added to the field `event.original`. + type: bool + multi: false + default: false + - name: processors + type: yaml + title: Processors + multi: false + required: false + show_user: false + description: >- + Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details. + - input: http_endpoint + template_path: http_endpoint.yml.hbs + title: Zscaler Internet Access DNS Logs + description: Collect Zscaler Internet Access DNS logs via HTTP Endpoint Input. + vars: + - name: listen_port + type: integer + title: Listen Port + description: The port number to listen on. + multi: false + required: true + show_user: true + default: 9556 + - name: tags + type: text + title: Tags + multi: true + required: true + show_user: false + default: + - forwarded + - zscaler_zia-dns + - name: preserve_original_event + required: true + show_user: true + title: Preserve original event + description: Preserves a raw copy of the original event, added to the field `event.original`. type: bool multi: false default: false diff --git a/packages/zscaler_zia/data_stream/dns/sample_event.json b/packages/zscaler_zia/data_stream/dns/sample_event.json index b668ffd2eca..ae3effed418 100644 --- a/packages/zscaler_zia/data_stream/dns/sample_event.json +++ b/packages/zscaler_zia/data_stream/dns/sample_event.json @@ -1,32 +1,38 @@ { "@timestamp": "2021-12-17T07:27:54.000Z", "agent": { - "ephemeral_id": "d288c261-b8db-45af-99c0-a673c3c6d8e1", - "hostname": "docker-fleet-agent", - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "ephemeral_id": "88d27df6-beee-4299-bf35-56742db35e98", + "id": "f6f3ddbc-7ab7-4a74-aeeb-152405dea56f", "name": "docker-fleet-agent", "type": "filebeat", - "version": "7.16.2" + "version": "8.1.2" }, - "client": { + "data_stream": { + "dataset": "zscaler_zia.dns", + "namespace": "ep", + "type": "logs" + }, + "destination": { + "as": { + "number": 29518, + "organization": { + "name": "Bredband2 AB" + } + }, "geo": { - "city_name": "London", + "city_name": "Linköping", "continent_name": "Europe", - "country_iso_code": "GB", - "country_name": "United Kingdom", + "country_iso_code": "SE", + "country_name": "Sweden", "location": { - "lat": 51.5142, - "lon": -0.0931 + "lat": 58.4167, + "lon": 15.6167 }, - "region_iso_code": "GB-ENG", - "region_name": "England" + "region_iso_code": "SE-E", + "region_name": "Östergötland County" }, - "ip": "81.2.69.193" - }, - "data_stream": { - "dataset": "zscaler_zia.dns", - "namespace": "ep", - "type": "logs" + "ip": "89.160.20.156", + "port": 8080 }, "dns": { "answers": { @@ -41,15 +47,18 @@ "version": "8.2.0" }, "elastic_agent": { - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "id": "f6f3ddbc-7ab7-4a74-aeeb-152405dea56f", "snapshot": false, - "version": "7.16.2" + "version": "8.1.2" }, "event": { "agent_id_status": "verified", - "category": "network", + "category": [ + "network" + ], "dataset": "zscaler_zia.dns", - "ingested": "2022-02-04T06:32:56Z", + "duration": 123456000000, + "ingested": "2022-04-20T06:45:24Z", "kind": "event", "type": [ "info" @@ -60,33 +69,41 @@ }, "log": { "source": { - "address": "172.21.0.7:54202" + "address": "1.128.3.4:32902" } }, + "network": { + "protocol": "dns" + }, "related": { "hosts": [ "Machine9000" ], "ip": [ - "81.2.69.193", - "81.2.69.144" + "89.160.20.112", + "89.160.20.156" ] }, - "server": { + "source": { + "as": { + "number": 29518, + "organization": { + "name": "Bredband2 AB" + } + }, "geo": { - "city_name": "London", + "city_name": "Linköping", "continent_name": "Europe", - "country_iso_code": "GB", - "country_name": "United Kingdom", + "country_iso_code": "SE", + "country_name": "Sweden", "location": { - "lat": 51.5142, - "lon": -0.0931 + "lat": 58.4167, + "lon": 15.6167 }, - "region_iso_code": "GB-ENG", - "region_name": "England" + "region_iso_code": "SE-E", + "region_name": "Östergötland County" }, - "ip": "81.2.69.144", - "port": 8080 + "ip": "89.160.20.112" }, "tags": [ "forwarded", diff --git a/packages/zscaler_zia/data_stream/firewall/_dev/test/pipeline/test-firewall-http_endpoint.log b/packages/zscaler_zia/data_stream/firewall/_dev/test/pipeline/test-firewall-http_endpoint.log new file mode 100644 index 00000000000..c5948c64707 --- /dev/null +++ b/packages/zscaler_zia/data_stream/firewall/_dev/test/pipeline/test-firewall-http_endpoint.log @@ -0,0 +1 @@ +{"sourcetype":"zscalernss-fw", "event":{"durationms":"1234","avgduration":"1234","sdip":"0.0.0.0","aggregate":"No","department":"Unknown","nwapp":"NotAvailable","proto":"IP","datetime":"Tue Dec 31 02:22:22 2022","nwsvc":"None","dnat":"No","threatcat":"None","cdport":"120","duration":"1","ipcat":"Other","deviceowner":"NA","csip":"0.0.0.0","devicehostname":"NA","csport":"123","tunsport":"0","destcountry":"NA","rulelabel":"None","locationname":"Unknown","action":"OutOfRange","stateful":"Yes","outbytes":"0","inbytes":"0","ssport":"0","user":"Unknown","tuntype":"OutOfRange","numsessions":"1","ssip":"0.0.0.0","threatname":"None","ipsrulelabel":"None","tsip":"0.0.0.0","sdport":"456","cdip":"0.0.0.0"}} diff --git a/packages/zscaler_zia/data_stream/firewall/_dev/test/pipeline/test-firewall-http_endpoint.log-expected.json b/packages/zscaler_zia/data_stream/firewall/_dev/test/pipeline/test-firewall-http_endpoint.log-expected.json new file mode 100644 index 00000000000..85fa703c017 --- /dev/null +++ b/packages/zscaler_zia/data_stream/firewall/_dev/test/pipeline/test-firewall-http_endpoint.log-expected.json @@ -0,0 +1,100 @@ +{ + "expected": [ + { + "@timestamp": "2022-12-31T02:22:22.000Z", + "destination": { + "bytes": 0, + "geo": { + "country_name": "NA" + }, + "ip": "0.0.0.0", + "port": 456 + }, + "ecs": { + "version": "8.2.0" + }, + "event": { + "action": "outofrange", + "category": [ + "network" + ], + "duration": 1234000000, + "kind": "event", + "original": "{\"sourcetype\":\"zscalernss-fw\", \"event\":{\"durationms\":\"1234\",\"avgduration\":\"1234\",\"sdip\":\"0.0.0.0\",\"aggregate\":\"No\",\"department\":\"Unknown\",\"nwapp\":\"NotAvailable\",\"proto\":\"IP\",\"datetime\":\"Tue Dec 31 02:22:22 2022\",\"nwsvc\":\"None\",\"dnat\":\"No\",\"threatcat\":\"None\",\"cdport\":\"120\",\"duration\":\"1\",\"ipcat\":\"Other\",\"deviceowner\":\"NA\",\"csip\":\"0.0.0.0\",\"devicehostname\":\"NA\",\"csport\":\"123\",\"tunsport\":\"0\",\"destcountry\":\"NA\",\"rulelabel\":\"None\",\"locationname\":\"Unknown\",\"action\":\"OutOfRange\",\"stateful\":\"Yes\",\"outbytes\":\"0\",\"inbytes\":\"0\",\"ssport\":\"0\",\"user\":\"Unknown\",\"tuntype\":\"OutOfRange\",\"numsessions\":\"1\",\"ssip\":\"0.0.0.0\",\"threatname\":\"None\",\"ipsrulelabel\":\"None\",\"tsip\":\"0.0.0.0\",\"sdport\":\"456\",\"cdip\":\"0.0.0.0\"}}", + "type": [ + "info" + ] + }, + "host": { + "hostname": "NA" + }, + "network": { + "application": "NotAvailable", + "protocol": "none", + "transport": "ip" + }, + "related": { + "ip": [ + "0.0.0.0" + ] + }, + "rule": { + "name": [ + "None" + ] + }, + "source": { + "bytes": 0, + "ip": "0.0.0.0", + "port": 123 + }, + "tags": [ + "preserve_original_event" + ], + "user": { + "email": "Unknown", + "name": "NA" + }, + "zscaler_zia": { + "firewall": { + "aggregate": "No", + "client": { + "destination": { + "ip": "0.0.0.0", + "port": 120 + } + }, + "department": "Unknown", + "duration": { + "avg": 1234, + "milliseconds": 1234 + }, + "ip_category": "Other", + "location": { + "name": "Unknown" + }, + "nat": "No", + "server": { + "source": { + "ip": "0.0.0.0", + "port": 0 + } + }, + "session": { + "count": 1.0 + }, + "stateful": "Yes", + "threat": { + "category": "None", + "name": "None" + }, + "tunnel": { + "ip": "0.0.0.0", + "port": 0, + "type": "OutOfRange" + } + } + } + } + ] +} \ No newline at end of file diff --git a/packages/zscaler_zia/data_stream/firewall/_dev/test/pipeline/test-firewall.log b/packages/zscaler_zia/data_stream/firewall/_dev/test/pipeline/test-firewall.log index b044d16f17e..c6ab96b1171 100644 --- a/packages/zscaler_zia/data_stream/firewall/_dev/test/pipeline/test-firewall.log +++ b/packages/zscaler_zia/data_stream/firewall/_dev/test/pipeline/test-firewall.log @@ -1 +1 @@ -{ "sourcetype" : "zscalernss-fw", "event" :{"datetime":"Fri Dec 17 07:27:54 2021","user":"some_user@example.com","department":"Unknown","locationname":"TestLoc%20DB","cdport":443,"csport":55018,"sdport":443,"ssport":0,"csip":"0.0.0.0","cdip":"0.0.0.0","ssip":"0.0.0.0","sdip":"0.0.0.0","tsip":"0.0.0.0","tunsport":0,"tuntype":"ZscalerClientConnector","action":"Drop","dnat":"No","stateful":"Yes","aggregate":"No","nwsvc":"HTTPS","nwapp":"http","proto":"TCP","ipcat":"Test Name","destcountry":"Ireland","avgduration":486,"rulelabel":"Access%20Blocked","inbytes":19052,"outbytes":1734,"duration":0,"durationms":486,"numsessions":1,"ipsrulelabel":"None","threatcat":"None","threatname":"None","deviceowner":"admin77","devicehostname":"Machine9000"}} +{ "sourcetype" : "zscalernss-fw", "event" :{"datetime":"Fri Dec 31 07:07:07 2021","user":"some_user@example.com","department":"Unknown","locationname":"TestLoc%20DB","cdport":443,"csport":55018,"sdport":443,"ssport":0,"csip":"0.0.0.0","cdip":"0.0.0.0","ssip":"0.0.0.0","sdip":"0.0.0.0","tsip":"0.0.0.0","tunsport":0,"tuntype":"ZscalerClientConnector","action":"Drop","dnat":"No","stateful":"Yes","aggregate":"No","nwsvc":"HTTPS","nwapp":"http","proto":"TCP","ipcat":"Test Name","destcountry":"Ireland","avgduration":486,"rulelabel":"Access%20Blocked","inbytes":19052,"outbytes":1734,"duration":4,"durationms":4861,"numsessions":1,"ipsrulelabel":"None","threatcat":"None","threatname":"None","deviceowner":"admin77","devicehostname":"Machine9000"}} diff --git a/packages/zscaler_zia/data_stream/firewall/_dev/test/pipeline/test-firewall.log-expected.json b/packages/zscaler_zia/data_stream/firewall/_dev/test/pipeline/test-firewall.log-expected.json index 9deec3338d7..c6aca3f6086 100644 --- a/packages/zscaler_zia/data_stream/firewall/_dev/test/pipeline/test-firewall.log-expected.json +++ b/packages/zscaler_zia/data_stream/firewall/_dev/test/pipeline/test-firewall.log-expected.json @@ -1,25 +1,36 @@ { "expected": [ { - "@timestamp": "2021-12-17T07:27:54.000Z", - "client": { - "bytes": 1734 + "@timestamp": "2021-12-31T07:07:07.000Z", + "destination": { + "bytes": 19052, + "geo": { + "country_name": "Ireland" + }, + "ip": "0.0.0.0", + "port": 443 }, "ecs": { "version": "8.2.0" }, "event": { "action": "drop", - "category": "network", + "category": [ + "network" + ], + "duration": 4861000000, "kind": "event", - "original": "{ \"sourcetype\" : \"zscalernss-fw\", \"event\" :{\"datetime\":\"Fri Dec 17 07:27:54 2021\",\"user\":\"some_user@example.com\",\"department\":\"Unknown\",\"locationname\":\"TestLoc%20DB\",\"cdport\":443,\"csport\":55018,\"sdport\":443,\"ssport\":0,\"csip\":\"0.0.0.0\",\"cdip\":\"0.0.0.0\",\"ssip\":\"0.0.0.0\",\"sdip\":\"0.0.0.0\",\"tsip\":\"0.0.0.0\",\"tunsport\":0,\"tuntype\":\"ZscalerClientConnector\",\"action\":\"Drop\",\"dnat\":\"No\",\"stateful\":\"Yes\",\"aggregate\":\"No\",\"nwsvc\":\"HTTPS\",\"nwapp\":\"http\",\"proto\":\"TCP\",\"ipcat\":\"Test Name\",\"destcountry\":\"Ireland\",\"avgduration\":486,\"rulelabel\":\"Access%20Blocked\",\"inbytes\":19052,\"outbytes\":1734,\"duration\":0,\"durationms\":486,\"numsessions\":1,\"ipsrulelabel\":\"None\",\"threatcat\":\"None\",\"threatname\":\"None\",\"deviceowner\":\"admin77\",\"devicehostname\":\"Machine9000\"}}", - "type": "info" + "original": "{ \"sourcetype\" : \"zscalernss-fw\", \"event\" :{\"datetime\":\"Fri Dec 31 07:07:07 2021\",\"user\":\"some_user@example.com\",\"department\":\"Unknown\",\"locationname\":\"TestLoc%20DB\",\"cdport\":443,\"csport\":55018,\"sdport\":443,\"ssport\":0,\"csip\":\"0.0.0.0\",\"cdip\":\"0.0.0.0\",\"ssip\":\"0.0.0.0\",\"sdip\":\"0.0.0.0\",\"tsip\":\"0.0.0.0\",\"tunsport\":0,\"tuntype\":\"ZscalerClientConnector\",\"action\":\"Drop\",\"dnat\":\"No\",\"stateful\":\"Yes\",\"aggregate\":\"No\",\"nwsvc\":\"HTTPS\",\"nwapp\":\"http\",\"proto\":\"TCP\",\"ipcat\":\"Test Name\",\"destcountry\":\"Ireland\",\"avgduration\":486,\"rulelabel\":\"Access%20Blocked\",\"inbytes\":19052,\"outbytes\":1734,\"duration\":4,\"durationms\":4861,\"numsessions\":1,\"ipsrulelabel\":\"None\",\"threatcat\":\"None\",\"threatname\":\"None\",\"deviceowner\":\"admin77\",\"devicehostname\":\"Machine9000\"}}", + "type": [ + "info" + ] }, "host": { "hostname": "Machine9000" }, "network": { "application": "http", + "community_id": "1:hQwW1HWTOUYlk7y4+T2D+UPDU1c=", "protocol": "https", "transport": "tcp" }, @@ -34,11 +45,10 @@ "None" ] }, - "server": { - "bytes": 19052, - "geo": { - "country_name": "Ireland" - } + "source": { + "bytes": 1734, + "ip": "0.0.0.0", + "port": 55018 }, "tags": [ "preserve_original_event" @@ -54,17 +64,12 @@ "destination": { "ip": "0.0.0.0", "port": 443 - }, - "source": { - "ip": "0.0.0.0", - "port": 55018 } }, "department": "Unknown", "duration": { "avg": 486, - "milliseconds": 486, - "seconds": 0 + "milliseconds": 4861 }, "ip_category": "Test Name", "location": { @@ -72,17 +77,13 @@ }, "nat": "No", "server": { - "destination": { - "ip": "0.0.0.0", - "port": 443 - }, "source": { "ip": "0.0.0.0", "port": 0 } }, "session": { - "count": 1 + "count": 1.0 }, "stateful": "Yes", "threat": { diff --git a/packages/zscaler_zia/data_stream/firewall/_dev/test/system/test-http-endpoint-config.yml b/packages/zscaler_zia/data_stream/firewall/_dev/test/system/test-http-endpoint-config.yml new file mode 100644 index 00000000000..f51683b6d4f --- /dev/null +++ b/packages/zscaler_zia/data_stream/firewall/_dev/test/system/test-http-endpoint-config.yml @@ -0,0 +1,8 @@ +service: zscaler-zia-firewall-http-endpoint +service_notify_signal: SIGHUP +input: http_endpoint +vars: + listen_address: 0.0.0.0 +data_stream: + vars: + listen_port: 9557 diff --git a/packages/zscaler_zia/data_stream/firewall/_dev/test/system/test-default-config.yml b/packages/zscaler_zia/data_stream/firewall/_dev/test/system/test-tcp-config.yml similarity index 100% rename from packages/zscaler_zia/data_stream/firewall/_dev/test/system/test-default-config.yml rename to packages/zscaler_zia/data_stream/firewall/_dev/test/system/test-tcp-config.yml diff --git a/packages/zscaler_zia/data_stream/firewall/agent/stream/http_endpoint.yml.hbs b/packages/zscaler_zia/data_stream/firewall/agent/stream/http_endpoint.yml.hbs new file mode 100644 index 00000000000..443fe325f7c --- /dev/null +++ b/packages/zscaler_zia/data_stream/firewall/agent/stream/http_endpoint.yml.hbs @@ -0,0 +1,21 @@ +listen_address: {{listen_address}} +listen_port: {{listen_port}} +content_type: "" +preserve_original_event: true +tags: +{{#if preserve_original_event}} + - preserve_original_event +{{/if}} +{{#each tags as |tag|}} + - {{tag}} +{{/each}} +{{#contains "forwarded" tags}} +publisher_pipeline.disable_host: true +{{/contains}} +{{#if ssl}} +ssl: {{ssl}} +{{/if}} +{{#if processors}} +processors: +{{processors}} +{{/if}} diff --git a/packages/zscaler_zia/data_stream/firewall/agent/stream/tcp.yml.hbs b/packages/zscaler_zia/data_stream/firewall/agent/stream/tcp.yml.hbs index 030459f2582..bc587e50a3a 100644 --- a/packages/zscaler_zia/data_stream/firewall/agent/stream/tcp.yml.hbs +++ b/packages/zscaler_zia/data_stream/firewall/agent/stream/tcp.yml.hbs @@ -1,4 +1,3 @@ -tcp: host: "{{listen_address}}:{{listen_port}}" tags: {{#if preserve_original_event}} diff --git a/packages/zscaler_zia/data_stream/firewall/elasticsearch/ingest_pipeline/default.yml b/packages/zscaler_zia/data_stream/firewall/elasticsearch/ingest_pipeline/default.yml index 37e03b1b8d9..1648f4b32f0 100644 --- a/packages/zscaler_zia/data_stream/firewall/elasticsearch/ingest_pipeline/default.yml +++ b/packages/zscaler_zia/data_stream/firewall/elasticsearch/ingest_pipeline/default.yml @@ -11,19 +11,25 @@ processors: - json: field: event.original target_field: resp + ignore_failure: true + - remove: + field: json + if: ctx?.input?.type == 'http_endpoint' + ignore_missing: true - rename: field: resp.event target_field: json - - remove: + ignore_missing: true + - remove: field: resp ignore_missing: true - - set: + - append: field: event.category value: network - set: field: event.kind value: event - - set: + - append: field: event.type value: info - date: @@ -34,41 +40,57 @@ processors: - E MMM dd HH:mm:ss yyyy - E MMM d HH:mm:ss yyyy - E MMM d HH:mm:ss yyyy - - remove: - field: json.datetime + - yyyy-mm-dd HH:mm:ss + - date: + field: json.time + target_field: "@timestamp" ignore_failure: true - - rename: + formats: + - E MMM dd HH:mm:ss yyyy + - E MMM d HH:mm:ss yyyy + - E MMM d HH:mm:ss yyyy + - yyyy-MM-dd HH:mm:ss + - remove: + field: + - json.time + - json.datetime + ignore_missing: true + - convert: + field: json.outbytes + target_field: source.bytes + type: long + ignore_failure: true + - remove: field: json.outbytes - target_field: client.bytes ignore_missing: true - - rename: + - rename: field: json.devicehostname target_field: host.hostname ignore_missing: true - - rename: + - rename: field: json.nwapp target_field: network.application ignore_missing: true - - rename: + - rename: field: json.nwsvc target_field: network.protocol ignore_missing: true - lowercase: field: network.protocol ignore_missing: true - - rename: + - rename: field: json.proto target_field: network.transport ignore_missing: true - lowercase: field: network.transport ignore_missing: true - - append: + - append: field: rule.name value: "{{{json.rulelabel}}}" if: ctx?.json?.rulelabel != null allow_duplicates: false - - append: + - append: field: rule.name value: "{{{json.ipsrulelabel}}}" if: ctx?.json?.ipsrulelabel != null @@ -82,54 +104,74 @@ processors: - remove: field: json.ipsrulelabel ignore_missing: true - - rename: + - convert: + field: json.inbytes + target_field: destination.bytes + type: long + ignore_failure: true + - remove: field: json.inbytes - target_field: server.bytes ignore_missing: true - - rename: + - rename: field: json.destcountry - target_field: server.geo.country_name + target_field: destination.geo.country_name ignore_missing: true - - urldecode: + - urldecode: field: json.user target_field: user.email ignore_missing: true - remove: field: json.user ignore_missing: true - - rename: + - rename: field: json.deviceowner target_field: user.name ignore_missing: true - - urldecode: + - urldecode: field: json.department target_field: zscaler_zia.firewall.department ignore_missing: true - remove: field: json.department ignore_missing: true - - urldecode: + - urldecode: field: json.locationname target_field: zscaler_zia.firewall.location.name ignore_missing: true - remove: field: json.locationname ignore_missing: true - - rename: + - convert: field: json.cdport target_field: zscaler_zia.firewall.client.destination.port + type: long + ignore_failure: true + - remove: + field: json.cdport ignore_missing: true - - rename: + - convert: + field: json.csport + target_field: source.port + type: long + ignore_failure: true + - remove: field: json.csport - target_field: zscaler_zia.firewall.client.source.port ignore_missing: true - - rename: + - convert: + field: json.sdport + target_field: destination.port + type: long + ignore_failure: true + - remove: field: json.sdport - target_field: zscaler_zia.firewall.server.destination.port ignore_missing: true - - rename: + - convert: field: json.ssport target_field: zscaler_zia.firewall.server.source.port + type: long + ignore_failure: true + - remove: + field: json.ssport ignore_missing: true - append: field: related.ip @@ -137,9 +179,9 @@ processors: if: ctx?.json?.csip != null allow_duplicates: false ignore_failure: true - - rename: + - rename: field: json.csip - target_field: zscaler_zia.firewall.client.source.ip + target_field: source.ip ignore_missing: true - append: field: related.ip @@ -147,7 +189,7 @@ processors: if: ctx?.json?.cdip != null allow_duplicates: false ignore_failure: true - - rename: + - rename: field: json.cdip target_field: zscaler_zia.firewall.client.destination.ip ignore_missing: true @@ -157,7 +199,7 @@ processors: if: ctx?.json?.ssip != null allow_duplicates: false ignore_failure: true - - rename: + - rename: field: json.ssip target_field: zscaler_zia.firewall.server.source.ip ignore_missing: true @@ -167,9 +209,9 @@ processors: if: ctx?.json?.sdip != null allow_duplicates: false ignore_failure: true - - rename: + - rename: field: json.sdip - target_field: zscaler_zia.firewall.server.destination.ip + target_field: destination.ip ignore_missing: true - append: field: related.ip @@ -177,65 +219,87 @@ processors: if: ctx?.json?.tsip != null allow_duplicates: false ignore_failure: true - - rename: + - rename: field: json.tsip target_field: zscaler_zia.firewall.tunnel.ip ignore_missing: true - - rename: + - convert: field: json.tunsport target_field: zscaler_zia.firewall.tunnel.port + type: long + ignore_failure: true + - remove: + field: json.tunsport ignore_missing: true - - rename: + - rename: field: json.tuntype target_field: zscaler_zia.firewall.tunnel.type ignore_missing: true - - rename: + - rename: field: json.action target_field: event.action ignore_missing: true - - lowercase: + - lowercase: field: event.action ignore_missing: true - - rename: + - rename: field: json.dnat target_field: zscaler_zia.firewall.nat ignore_missing: true - - rename: + - rename: field: json.stateful target_field: zscaler_zia.firewall.stateful ignore_missing: true - - rename: + - rename: field: json.aggregate target_field: zscaler_zia.firewall.aggregate ignore_missing: true - - rename: + - rename: field: json.ipcat target_field: zscaler_zia.firewall.ip_category ignore_missing: true - - rename: + - convert: field: json.avgduration + type: long target_field: zscaler_zia.firewall.duration.avg + ignore_failure: true + - remove: + field: + - json.avgduration + - json.duration ignore_missing: true - - rename: - field: json.duration - target_field: zscaler_zia.firewall.duration.seconds - ignore_missing: true - - rename: + - convert: field: json.durationms target_field: zscaler_zia.firewall.duration.milliseconds + type: long + ignore_failure: true + - remove: + field: json.durationms ignore_missing: true - - rename: + - convert: field: json.numsessions target_field: zscaler_zia.firewall.session.count + type: double + ignore_failure: true + - remove: + field: json.numsessions ignore_missing: true - - rename: + - rename: field: json.threatcat target_field: zscaler_zia.firewall.threat.category ignore_missing: true - - rename: + - rename: field: json.threatname target_field: zscaler_zia.firewall.threat.name ignore_missing: true + - community_id: + source_ip: source.ip + source_port: source.port + destination_ip: destination.ip + destination_port: destination.port + transport: network.transport + target_field: network.community_id + ignore_failure: true - script: description: Drops null/empty values recursively lang: painless @@ -261,9 +325,19 @@ processors: for (Map.Entry m : ctx.json.entrySet()) { ctx.zscaler_zia.firewall[m.getKey()] = m.getValue(); } - - remove: + - script: + lang: painless + if: ctx?.zscaler_zia?.firewall?.duration?.milliseconds != null + source: | + ctx.event.duration = ctx?.zscaler_zia?.firewall?.duration?.milliseconds * 1000000; + - remove: field: json ignore_missing: true + - remove: + field: event.original + if: "ctx?.tags == null || !(ctx.tags.contains('preserve_original_event'))" + ignore_failure: true + ignore_missing: true on_failure: - set: field: error.message diff --git a/packages/zscaler_zia/data_stream/firewall/fields/ecs.yml b/packages/zscaler_zia/data_stream/firewall/fields/ecs.yml index de7ee2cff2c..711d967f427 100644 --- a/packages/zscaler_zia/data_stream/firewall/fields/ecs.yml +++ b/packages/zscaler_zia/data_stream/firewall/fields/ecs.yml @@ -1,13 +1,23 @@ - external: ecs - name: client.bytes + name: destination.bytes +- external: ecs + name: destination.geo.country_name +- external: ecs + name: destination.ip +- external: ecs + name: destination.port - external: ecs name: ecs.version - external: ecs name: event.action +- external: ecs + name: event.duration - external: ecs name: host.hostname - external: ecs name: network.application +- external: ecs + name: network.community_id - external: ecs name: network.protocol - external: ecs @@ -17,9 +27,11 @@ - external: ecs name: rule.name - external: ecs - name: server.bytes + name: source.bytes +- external: ecs + name: source.ip - external: ecs - name: server.geo.country_name + name: source.port - external: ecs name: tags - external: ecs diff --git a/packages/zscaler_zia/data_stream/firewall/fields/fields.yml b/packages/zscaler_zia/data_stream/firewall/fields/fields.yml index c8b4137da0e..49a98954aa8 100644 --- a/packages/zscaler_zia/data_stream/firewall/fields/fields.yml +++ b/packages/zscaler_zia/data_stream/firewall/fields/fields.yml @@ -19,43 +19,21 @@ type: group fields: - name: port - type: double + type: long description: | Client destination port. For aggregated sessions, this is the client destination port of the last session in the aggregate. - name: ip type: keyword description: | Client destination IP address. For aggregated sessions, this is the client destination IP address of the last session in the aggregate. - - name: source - type: group - fields: - - name: port - type: double - description: | - Client source port. For aggregated sessions, this is the client source port of the last session in the aggregate. - - name: ip - type: keyword - description: | - Client source IP address. For aggregated sessions, this is the client source IP address of the last session in the aggregate. - name: server type: group fields: - - name: destination - type: group - fields: - - name: port - type: double - description: | - Server destination port. For aggregated sessions, this is the server destination port of the last session in the aggregate. - - name: ip - type: keyword - description: | - Server Destination IP address. For aggregated sessions, this is the server destination IP address of the last session in the aggregate. - name: source type: group fields: - name: port - type: double + type: long description: | Server source port. For aggregated sessions, this is the server source port of the last session in the aggregate. - name: ip @@ -70,7 +48,7 @@ description: | Tunnel IP address of the client (source). For aggregated sessions, this is the client's tunnel IP address corresponding to the last session in the aggregate. - name: port - type: double + type: long description: | Tunnel port on the client side. For aggregated sessions, this is the client's tunnel port corresponding to the last session in the aggregate. - name: type @@ -93,15 +71,15 @@ type: group fields: - name: avg - type: double + type: long description: | Average session duration, in milliseconds, if the sessions were aggregated. - name: seconds - type: double + type: long description: | - Session or request duration in seconds. + Average session duration, in milliseconds, if the sessions were aggregated. - name: milliseconds - type: double + type: long description: | Session or request duration in milliseconds. - name: session diff --git a/packages/zscaler_zia/data_stream/firewall/manifest.yml b/packages/zscaler_zia/data_stream/firewall/manifest.yml index 6da5481a668..eee2e7be93f 100644 --- a/packages/zscaler_zia/data_stream/firewall/manifest.yml +++ b/packages/zscaler_zia/data_stream/firewall/manifest.yml @@ -4,7 +4,7 @@ streams: - input: tcp template_path: tcp.yml.hbs title: Zscaler Internet Access Firewall Logs - description: Collect Zscaler Internet Access Firewall Logs using tcp input + description: Collect Zscaler Internet Access Firewall Logs using TCP Input. vars: - name: listen_port type: integer @@ -27,7 +27,45 @@ streams: required: true show_user: true title: Preserve original event - description: Preserves a raw copy of the original event, added to the field `event.original` + description: Preserves a raw copy of the original event, added to the field `event.original`. + type: bool + multi: false + default: false + - name: processors + type: yaml + title: Processors + multi: false + required: false + show_user: false + description: >- + Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details. + - input: http_endpoint + template_path: http_endpoint.yml.hbs + title: Zscaler Internet Access Firewall Logs + description: Collect Zscaler Internet Access Firewall logs via HTTP Endpoint Input. + vars: + - name: listen_port + type: integer + title: Listen Port + description: The port number to listen on. + multi: false + required: true + show_user: true + default: 9557 + - name: tags + type: text + title: Tags + multi: true + required: true + show_user: false + default: + - forwarded + - zscaler_zia-firewall + - name: preserve_original_event + required: true + show_user: true + title: Preserve original event + description: Preserves a raw copy of the original event, added to the field `event.original`. type: bool multi: false default: false diff --git a/packages/zscaler_zia/data_stream/firewall/sample_event.json b/packages/zscaler_zia/data_stream/firewall/sample_event.json index 7af4124c4aa..1d138bf5253 100644 --- a/packages/zscaler_zia/data_stream/firewall/sample_event.json +++ b/packages/zscaler_zia/data_stream/firewall/sample_event.json @@ -1,38 +1,46 @@ { - "@timestamp": "2021-12-17T07:27:54.000Z", + "@timestamp": "2021-12-31T07:08:09.000Z", "agent": { - "ephemeral_id": "41987f90-74dc-4b4b-9936-4347028cf558", - "hostname": "docker-fleet-agent", - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "ephemeral_id": "2c292e52-b6ea-4ca0-bfc7-692dadde1a7d", + "id": "f6f3ddbc-7ab7-4a74-aeeb-152405dea56f", "name": "docker-fleet-agent", "type": "filebeat", - "version": "7.16.2" - }, - "client": { - "bytes": 1734 + "version": "8.1.2" }, "data_stream": { "dataset": "zscaler_zia.firewall", "namespace": "ep", "type": "logs" }, + "destination": { + "bytes": 19052, + "geo": { + "country_name": "Ireland" + }, + "ip": "0.0.0.0", + "port": 443 + }, "ecs": { "version": "8.2.0" }, "elastic_agent": { - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "id": "f6f3ddbc-7ab7-4a74-aeeb-152405dea56f", "snapshot": false, - "version": "7.16.2" + "version": "8.1.2" }, "event": { "action": "drop", "agent_id_status": "verified", - "category": "network", + "category": [ + "network" + ], "dataset": "zscaler_zia.firewall", - "ingested": "2022-02-04T06:34:17Z", + "duration": 486000000, + "ingested": "2021-12-31T05:06:07Z", "kind": "event", - "original": "{ \"sourcetype\" : \"zscalernss-fw\", \"event\" :{\"datetime\":\"Fri Dec 17 07:27:54 2021\",\"user\":\"some_user@example.com\",\"department\":\"Unknown\",\"locationname\":\"TestLoc%20DB\",\"cdport\":443,\"csport\":55018,\"sdport\":443,\"ssport\":0,\"csip\":\"0.0.0.0\",\"cdip\":\"0.0.0.0\",\"ssip\":\"0.0.0.0\",\"sdip\":\"0.0.0.0\",\"tsip\":\"0.0.0.0\",\"tunsport\":0,\"tuntype\":\"ZscalerClientConnector\",\"action\":\"Drop\",\"dnat\":\"No\",\"stateful\":\"Yes\",\"aggregate\":\"No\",\"nwsvc\":\"HTTPS\",\"nwapp\":\"http\",\"proto\":\"TCP\",\"ipcat\":\"Test Name\",\"destcountry\":\"Ireland\",\"avgduration\":486,\"rulelabel\":\"Access%20Blocked\",\"inbytes\":19052,\"outbytes\":1734,\"duration\":0,\"durationms\":486,\"numsessions\":1,\"ipsrulelabel\":\"None\",\"threatcat\":\"None\",\"threatname\":\"None\",\"deviceowner\":\"admin77\",\"devicehostname\":\"Machine9000\"}}", - "type": "info" + "type": [ + "info" + ] }, "host": { "hostname": "Machine9000" @@ -42,11 +50,12 @@ }, "log": { "source": { - "address": "172.21.0.7:58194" + "address": "1.128.3.4:43634" } }, "network": { "application": "http", + "community_id": "1:hQwW1HWTOUYlk7y4+T2D+UPDU1c=", "protocol": "https", "transport": "tcp" }, @@ -61,11 +70,10 @@ "None" ] }, - "server": { - "bytes": 19052, - "geo": { - "country_name": "Ireland" - } + "source": { + "bytes": 1734, + "ip": "0.0.0.0", + "port": 55018 }, "tags": [ "forwarded", @@ -82,17 +90,12 @@ "destination": { "ip": "0.0.0.0", "port": 443 - }, - "source": { - "ip": "0.0.0.0", - "port": 55018 } }, "department": "Unknown", "duration": { "avg": 486, - "milliseconds": 486, - "seconds": 0 + "milliseconds": 486 }, "ip_category": "Test Name", "location": { @@ -100,10 +103,6 @@ }, "nat": "No", "server": { - "destination": { - "ip": "0.0.0.0", - "port": 443 - }, "source": { "ip": "0.0.0.0", "port": 0 diff --git a/packages/zscaler_zia/data_stream/tunnel/_dev/test/pipeline/test-tunnel-http_endpoint.log b/packages/zscaler_zia/data_stream/tunnel/_dev/test/pipeline/test-tunnel-http_endpoint.log new file mode 100644 index 00000000000..51866c5af57 --- /dev/null +++ b/packages/zscaler_zia/data_stream/tunnel/_dev/test/pipeline/test-tunnel-http_endpoint.log @@ -0,0 +1 @@ +{"sourcetype":"zscalernss-tunnel", "event":{"location":"Unknown","sourceport":"0","rxbytes":"0","tunneltype":"GRE","dpdrec":"0","destinationip":"0.0.0.0","recordid":"7083029673927507968","datetime":"Tue Dec 31 08:08:08 2021","sourceip":"0.0.0.0","txbytes":"0","Recordtype":"Tunnel Samples","user":"Unknown"}} diff --git a/packages/zscaler_zia/data_stream/tunnel/_dev/test/pipeline/test-tunnel-http_endpoint.log-expected.json b/packages/zscaler_zia/data_stream/tunnel/_dev/test/pipeline/test-tunnel-http_endpoint.log-expected.json new file mode 100644 index 00000000000..0dbb202cad3 --- /dev/null +++ b/packages/zscaler_zia/data_stream/tunnel/_dev/test/pipeline/test-tunnel-http_endpoint.log-expected.json @@ -0,0 +1,61 @@ +{ + "expected": [ + { + "@timestamp": "2021-12-31T08:08:08.000Z", + "destination": { + "bytes": 0, + "ip": "0.0.0.0" + }, + "ecs": { + "version": "8.2.0" + }, + "event": { + "category": [ + "network" + ], + "id": "7083029673927507968", + "kind": "event", + "original": "{\"sourcetype\":\"zscalernss-tunnel\", \"event\":{\"location\":\"Unknown\",\"sourceport\":\"0\",\"rxbytes\":\"0\",\"tunneltype\":\"GRE\",\"dpdrec\":\"0\",\"destinationip\":\"0.0.0.0\",\"recordid\":\"7083029673927507968\",\"datetime\":\"Tue Dec 31 08:08:08 2021\",\"sourceip\":\"0.0.0.0\",\"txbytes\":\"0\",\"Recordtype\":\"Tunnel Samples\",\"user\":\"Unknown\"}}", + "type": [ + "info" + ] + }, + "network": { + "community_id": "1:y8Yi03w0LBfVdMLE1UG7vvaUt5w=", + "iana_number": "47", + "transport": "gre" + }, + "related": { + "ip": [ + "0.0.0.0" + ], + "user": [ + "Unknown" + ] + }, + "source": { + "bytes": 0, + "ip": "0.0.0.0", + "port": 0 + }, + "tags": [ + "preserve_original_event" + ], + "user": { + "name": "Unknown" + }, + "zscaler_zia": { + "tunnel": { + "action": { + "type": "Tunnel Samples" + }, + "dpd_packets": "0", + "location": { + "name": "Unknown" + }, + "type": "GRE" + } + } + } + ] +} \ No newline at end of file diff --git a/packages/zscaler_zia/data_stream/tunnel/_dev/test/pipeline/test-tunnel.log b/packages/zscaler_zia/data_stream/tunnel/_dev/test/pipeline/test-tunnel.log index fc0a18ba331..2d626b1882d 100644 --- a/packages/zscaler_zia/data_stream/tunnel/_dev/test/pipeline/test-tunnel.log +++ b/packages/zscaler_zia/data_stream/tunnel/_dev/test/pipeline/test-tunnel.log @@ -1,3 +1,3 @@ -{ "sourcetype" : "zscalernss-tunnel", "event" : {"datetime":"Thu Dec 30 11:20:12 2021","Recordtype":"IPSec Phase2","tunneltype":"IPSEC IKEV 1","user":"81.2.69.145","location":"some-location","sourceip":"81.2.69.145","destinationip":"81.2.69.143","sourceport":"0","sourceportstart":"0","destinationportstart":"0","srcipstart":"81.2.69.145","srcipend":"81.2.69.145","destinationipstart":"81.2.69.143","destinationipend":"81.2.69.143","lifetime":"3600","ikeversion":"1","lifebytes":"0","spi":"123456789","algo":"AES","authentication":"HMAC-SHA-1","authtype":"None","protocol":"Any","tunnelprotocol":"ESP","policydirection":"Inbound SA Policy","recordid":"1111111111111111111"}} -{ "sourcetype" : "zscalernss-tunnel", "event" : {"datetime":"Thu Dec 30 11:40:27 2021","Recordtype":"IPSec Phase1","tunneltype":"IPSEC IKEV 2","user":"81.2.69.145","location":"some-location","sourceip":"81.2.69.145","destinationip":"81.2.69.143","sourceport":"500","destinationport":"500","lifetime":"0","ikeversion":"2","spi_in":"00000000000000000000","spi_out":"11111111111111111111","algo":"AES-CBS","authentication":"HMAC-SHA1-96","authtype":"PSK","recordid":"1111111111111111111"}} -{ "sourcetype" : "zscalernss-tunnel", "event" : {"datetime":"Thu Dec 30 11:40:27 2021","Recordtype":"Tunnel Event","tunneltype":"IPSec IKEv2","user":"81.2.69.145","location":"some-location","sourceip":"81.2.69.145","destinationip":"81.2.69.143","sourceport":"500","event":"IPsec tunnel is up","eventreason":"None","recordid":"1111111111111111111"}} +{ "sourcetype" : "zscalernss-tunnel", "event" : {"datetime":"Thu Dec 31 11:11:11 2021","Recordtype":"IPSec Phase2","tunneltype":"IPSEC IKEV 1","user":"81.2.69.145","location":"some-location","sourceip":"81.2.69.145","destinationip":"81.2.69.143","sourceport":"0","sourceportstart":"0","destinationportstart":"0","srcipstart":"81.2.69.145","srcipend":"81.2.69.145","destinationipstart":"81.2.69.143","destinationipend":"81.2.69.143","lifetime":"3600","ikeversion":"1","lifebytes":"0","spi":"123456789","algo":"AES","authentication":"HMAC-SHA-1","authtype":"None","protocol":"Any","tunnelprotocol":"ESP","policydirection":"Inbound SA Policy","recordid":"1111111111111111111"}} +{ "sourcetype" : "zscalernss-tunnel", "event" : {"datetime":"Thu Dec 31 11:11:11 2021","Recordtype":"IPSec Phase1","tunneltype":"IPSEC IKEV 2","user":"81.2.69.145","location":"some-location","sourceip":"81.2.69.145","destinationip":"81.2.69.143","sourceport":"500","destinationport":"500","lifetime":"0","ikeversion":"2","spi_in":"00000000000000000000","spi_out":"11111111111111111111","algo":"AES-CBS","authentication":"HMAC-SHA1-96","authtype":"PSK","recordid":"1111111111111111111"}} +{ "sourcetype" : "zscalernss-tunnel", "event" : {"datetime":"Thu Dec 31 11:11:11 2021","Recordtype":"Tunnel Event","tunneltype":"IPSec IKEv2","user":"81.2.69.145","location":"some-location","sourceip":"81.2.69.145","destinationip":"81.2.69.143","sourceport":"500","event":"IPsec tunnel is up","eventreason":"None","recordid":"1111111111111111111"}} diff --git a/packages/zscaler_zia/data_stream/tunnel/_dev/test/pipeline/test-tunnel.log-expected.json b/packages/zscaler_zia/data_stream/tunnel/_dev/test/pipeline/test-tunnel.log-expected.json index 005d8cff8e4..e32d8473920 100644 --- a/packages/zscaler_zia/data_stream/tunnel/_dev/test/pipeline/test-tunnel.log-expected.json +++ b/packages/zscaler_zia/data_stream/tunnel/_dev/test/pipeline/test-tunnel.log-expected.json @@ -1,7 +1,7 @@ { "expected": [ { - "@timestamp": "2021-12-30T11:20:12.000Z", + "@timestamp": "2021-12-31T11:11:11.000Z", "destination": { "ip": "81.2.69.143" }, @@ -9,14 +9,19 @@ "version": "8.2.0" }, "event": { - "category": "network", + "category": [ + "network" + ], "id": "1111111111111111111", "kind": "event", - "original": "{ \"sourcetype\" : \"zscalernss-tunnel\", \"event\" : {\"datetime\":\"Thu Dec 30 11:20:12 2021\",\"Recordtype\":\"IPSec Phase2\",\"tunneltype\":\"IPSEC IKEV 1\",\"user\":\"81.2.69.145\",\"location\":\"some-location\",\"sourceip\":\"81.2.69.145\",\"destinationip\":\"81.2.69.143\",\"sourceport\":\"0\",\"sourceportstart\":\"0\",\"destinationportstart\":\"0\",\"srcipstart\":\"81.2.69.145\",\"srcipend\":\"81.2.69.145\",\"destinationipstart\":\"81.2.69.143\",\"destinationipend\":\"81.2.69.143\",\"lifetime\":\"3600\",\"ikeversion\":\"1\",\"lifebytes\":\"0\",\"spi\":\"123456789\",\"algo\":\"AES\",\"authentication\":\"HMAC-SHA-1\",\"authtype\":\"None\",\"protocol\":\"Any\",\"tunnelprotocol\":\"ESP\",\"policydirection\":\"Inbound SA Policy\",\"recordid\":\"1111111111111111111\"}}", + "original": "{ \"sourcetype\" : \"zscalernss-tunnel\", \"event\" : {\"datetime\":\"Thu Dec 31 11:11:11 2021\",\"Recordtype\":\"IPSec Phase2\",\"tunneltype\":\"IPSEC IKEV 1\",\"user\":\"81.2.69.145\",\"location\":\"some-location\",\"sourceip\":\"81.2.69.145\",\"destinationip\":\"81.2.69.143\",\"sourceport\":\"0\",\"sourceportstart\":\"0\",\"destinationportstart\":\"0\",\"srcipstart\":\"81.2.69.145\",\"srcipend\":\"81.2.69.145\",\"destinationipstart\":\"81.2.69.143\",\"destinationipend\":\"81.2.69.143\",\"lifetime\":\"3600\",\"ikeversion\":\"1\",\"lifebytes\":\"0\",\"spi\":\"123456789\",\"algo\":\"AES\",\"authentication\":\"HMAC-SHA-1\",\"authtype\":\"None\",\"protocol\":\"Any\",\"tunnelprotocol\":\"ESP\",\"policydirection\":\"Inbound SA Policy\",\"recordid\":\"1111111111111111111\"}}", "type": [ "info" ] }, + "network": { + "transport": "ipsec ikev 1" + }, "related": { "ip": [ "81.2.69.143", @@ -87,7 +92,7 @@ } }, { - "@timestamp": "2021-12-30T11:40:27.000Z", + "@timestamp": "2021-12-31T11:11:11.000Z", "destination": { "ip": "81.2.69.143", "port": 500 @@ -96,14 +101,19 @@ "version": "8.2.0" }, "event": { - "category": "network", + "category": [ + "network" + ], "id": "1111111111111111111", "kind": "event", - "original": "{ \"sourcetype\" : \"zscalernss-tunnel\", \"event\" : {\"datetime\":\"Thu Dec 30 11:40:27 2021\",\"Recordtype\":\"IPSec Phase1\",\"tunneltype\":\"IPSEC IKEV 2\",\"user\":\"81.2.69.145\",\"location\":\"some-location\",\"sourceip\":\"81.2.69.145\",\"destinationip\":\"81.2.69.143\",\"sourceport\":\"500\",\"destinationport\":\"500\",\"lifetime\":\"0\",\"ikeversion\":\"2\",\"spi_in\":\"00000000000000000000\",\"spi_out\":\"11111111111111111111\",\"algo\":\"AES-CBS\",\"authentication\":\"HMAC-SHA1-96\",\"authtype\":\"PSK\",\"recordid\":\"1111111111111111111\"}}", + "original": "{ \"sourcetype\" : \"zscalernss-tunnel\", \"event\" : {\"datetime\":\"Thu Dec 31 11:11:11 2021\",\"Recordtype\":\"IPSec Phase1\",\"tunneltype\":\"IPSEC IKEV 2\",\"user\":\"81.2.69.145\",\"location\":\"some-location\",\"sourceip\":\"81.2.69.145\",\"destinationip\":\"81.2.69.143\",\"sourceport\":\"500\",\"destinationport\":\"500\",\"lifetime\":\"0\",\"ikeversion\":\"2\",\"spi_in\":\"00000000000000000000\",\"spi_out\":\"11111111111111111111\",\"algo\":\"AES-CBS\",\"authentication\":\"HMAC-SHA1-96\",\"authtype\":\"PSK\",\"recordid\":\"1111111111111111111\"}}", "type": [ "info" ] }, + "network": { + "transport": "ipsec ikev 2" + }, "related": { "ip": [ "81.2.69.143", @@ -151,7 +161,7 @@ } }, { - "@timestamp": "2021-12-30T11:40:27.000Z", + "@timestamp": "2021-12-31T11:11:11.000Z", "destination": { "ip": "81.2.69.143" }, @@ -160,15 +170,20 @@ }, "event": { "action": "IPsec tunnel is up", - "category": "network", + "category": [ + "network" + ], "id": "1111111111111111111", "kind": "event", - "original": "{ \"sourcetype\" : \"zscalernss-tunnel\", \"event\" : {\"datetime\":\"Thu Dec 30 11:40:27 2021\",\"Recordtype\":\"Tunnel Event\",\"tunneltype\":\"IPSec IKEv2\",\"user\":\"81.2.69.145\",\"location\":\"some-location\",\"sourceip\":\"81.2.69.145\",\"destinationip\":\"81.2.69.143\",\"sourceport\":\"500\",\"event\":\"IPsec tunnel is up\",\"eventreason\":\"None\",\"recordid\":\"1111111111111111111\"}}", + "original": "{ \"sourcetype\" : \"zscalernss-tunnel\", \"event\" : {\"datetime\":\"Thu Dec 31 11:11:11 2021\",\"Recordtype\":\"Tunnel Event\",\"tunneltype\":\"IPSec IKEv2\",\"user\":\"81.2.69.145\",\"location\":\"some-location\",\"sourceip\":\"81.2.69.145\",\"destinationip\":\"81.2.69.143\",\"sourceport\":\"500\",\"event\":\"IPsec tunnel is up\",\"eventreason\":\"None\",\"recordid\":\"1111111111111111111\"}}", "reason": "None", "type": [ "info" ] }, + "network": { + "transport": "ipsec ikev2" + }, "related": { "ip": [ "81.2.69.143", diff --git a/packages/zscaler_zia/data_stream/tunnel/_dev/test/system/test-http-endpoint-config.yml b/packages/zscaler_zia/data_stream/tunnel/_dev/test/system/test-http-endpoint-config.yml new file mode 100644 index 00000000000..8e51858e804 --- /dev/null +++ b/packages/zscaler_zia/data_stream/tunnel/_dev/test/system/test-http-endpoint-config.yml @@ -0,0 +1,8 @@ +service: zscaler-zia-tunnel-http-endpoint +service_notify_signal: SIGHUP +input: http_endpoint +vars: + listen_address: 0.0.0.0 +data_stream: + vars: + listen_port: 9558 diff --git a/packages/zscaler_zia/data_stream/tunnel/_dev/test/system/test-default-config.yml b/packages/zscaler_zia/data_stream/tunnel/_dev/test/system/test-tcp-config.yml similarity index 100% rename from packages/zscaler_zia/data_stream/tunnel/_dev/test/system/test-default-config.yml rename to packages/zscaler_zia/data_stream/tunnel/_dev/test/system/test-tcp-config.yml diff --git a/packages/zscaler_zia/data_stream/tunnel/agent/stream/http_endpoint.yml.hbs b/packages/zscaler_zia/data_stream/tunnel/agent/stream/http_endpoint.yml.hbs new file mode 100644 index 00000000000..443fe325f7c --- /dev/null +++ b/packages/zscaler_zia/data_stream/tunnel/agent/stream/http_endpoint.yml.hbs @@ -0,0 +1,21 @@ +listen_address: {{listen_address}} +listen_port: {{listen_port}} +content_type: "" +preserve_original_event: true +tags: +{{#if preserve_original_event}} + - preserve_original_event +{{/if}} +{{#each tags as |tag|}} + - {{tag}} +{{/each}} +{{#contains "forwarded" tags}} +publisher_pipeline.disable_host: true +{{/contains}} +{{#if ssl}} +ssl: {{ssl}} +{{/if}} +{{#if processors}} +processors: +{{processors}} +{{/if}} diff --git a/packages/zscaler_zia/data_stream/tunnel/agent/stream/tcp.yml.hbs b/packages/zscaler_zia/data_stream/tunnel/agent/stream/tcp.yml.hbs index 030459f2582..bc587e50a3a 100644 --- a/packages/zscaler_zia/data_stream/tunnel/agent/stream/tcp.yml.hbs +++ b/packages/zscaler_zia/data_stream/tunnel/agent/stream/tcp.yml.hbs @@ -1,4 +1,3 @@ -tcp: host: "{{listen_address}}:{{listen_port}}" tags: {{#if preserve_original_event}} diff --git a/packages/zscaler_zia/data_stream/tunnel/elasticsearch/ingest_pipeline/default.yml b/packages/zscaler_zia/data_stream/tunnel/elasticsearch/ingest_pipeline/default.yml index 2aa4e04f47e..3391e5bf916 100644 --- a/packages/zscaler_zia/data_stream/tunnel/elasticsearch/ingest_pipeline/default.yml +++ b/packages/zscaler_zia/data_stream/tunnel/elasticsearch/ingest_pipeline/default.yml @@ -11,9 +11,15 @@ processors: - json: field: event.original target_field: resp + ignore_failure: true + - remove: + field: json + if: ctx?.input?.type == 'http_endpoint' + ignore_missing: true - rename: field: resp.event target_field: json + ignore_missing: true - remove: field: resp ignore_missing: true @@ -25,10 +31,22 @@ processors: - E MMM dd HH:mm:ss yyyy - E MMM d HH:mm:ss yyyy - E MMM d HH:mm:ss yyyy + - yyyy-mm-dd HH:mm:ss + - date: + field: json.time + target_field: "@timestamp" + ignore_failure: true + formats: + - E MMM dd HH:mm:ss yyyy + - E MMM d HH:mm:ss yyyy + - E MMM d HH:mm:ss yyyy + - yyyy-MM-dd HH:mm:ss - remove: - field: json.datetime + field: + - json.time + - json.datetime ignore_missing: true - - set: + - append: field: event.category value: network - set: @@ -53,7 +71,7 @@ processors: field: json.destinationip target_field: destination.ip ignore_missing: true - - append: + - append: field: related.ip value: "{{{destination.ip}}}" if: ctx?.destination?.ip != null @@ -71,7 +89,7 @@ processors: field: json.sourceip target_field: source.ip ignore_missing: true - - append: + - append: field: related.ip value: "{{{source.ip}}}" if: ctx?.source?.ip != null @@ -89,26 +107,34 @@ processors: field: json.user target_field: user.name ignore_missing: true - - remove: + - remove: field: json.user ignore_missing: true - - append: + - append: field: related.user value: "{{{user.name}}}" if: ctx?.user?.name != null allow_duplicates: false ignore_failure: true - - rename: + - convert: field: json.rxbytes target_field: destination.bytes + type: long + ignore_missing: true + - remove: + field: json.rxbytes ignore_missing: true - rename: field: json.rxpackets target_field: destination.packets ignore_missing: true - - rename: + - convert: field: json.txbytes target_field: source.bytes + type: long + ignore_missing: true + - remove: + field: json.txbytes ignore_missing: true - rename: field: json.txpackets @@ -122,7 +148,7 @@ processors: field: json.location target_field: zscaler_zia.tunnel.location.name ignore_missing: true - - remove: + - remove: field: json.location ignore_missing: true - convert: @@ -130,7 +156,7 @@ processors: target_field: zscaler_zia.tunnel.life.time type: long ignore_missing: true - - remove: + - remove: field: json.lifetime ignore_missing: true - convert: @@ -138,7 +164,7 @@ processors: target_field: zscaler_zia.tunnel.ike.version type: integer ignore_missing: true - - remove: + - remove: field: json.ikeversion ignore_missing: true - rename: @@ -165,6 +191,18 @@ processors: field: json.tunneltype target_field: zscaler_zia.tunnel.type ignore_missing: true + - set: + field: network.transport + copy_from: zscaler_zia.tunnel.type + ignore_failure: true + - lowercase: + field: network.transport + ignore_missing: true + - set: + field: network.iana_number + value: "47" + if: ctx?.network?.transport == "gre" + ignore_failure: true - rename: field: json.vendorname target_field: zscaler_zia.tunnel.vendor.name @@ -174,7 +212,7 @@ processors: target_field: zscaler_zia.tunnel.source.start.port type: long ignore_missing: true - - remove: + - remove: field: json.sourceportstart ignore_missing: true - convert: @@ -182,14 +220,14 @@ processors: target_field: zscaler_zia.tunnel.destination.start.port type: long ignore_missing: true - - remove: + - remove: field: json.destinationportstart ignore_missing: true - rename: field: json.srcipstart target_field: zscaler_zia.tunnel.source.start.ip ignore_missing: true - - append: + - append: field: related.ip value: "{{{zscaler_zia.tunnel.source.start.ip}}}" if: ctx?.zscaler_zia?.tunnel?.source?.start?.ip != null @@ -199,7 +237,7 @@ processors: field: json.srcipend target_field: zscaler_zia.tunnel.source.end.ip ignore_missing: true - - append: + - append: field: related.ip value: "{{{zscaler_zia.tunnel.source.end.ip}}}" if: ctx?.zscaler_zia?.tunnel?.source?.end?.ip != null @@ -209,7 +247,7 @@ processors: field: json.destinationipstart target_field: zscaler_zia.tunnel.destination.start.ip ignore_missing: true - - append: + - append: field: related.ip value: "{{{zscaler_zia.tunnel.destination.start.ip}}}" if: ctx?.zscaler_zia?.tunnel?.destination?.start?.ip != null @@ -219,7 +257,7 @@ processors: field: json.destinationipend target_field: zscaler_zia.tunnel.destination.end.ip ignore_missing: true - - append: + - append: field: related.ip value: "{{{zscaler_zia.tunnel.destination.end.ip}}}" if: ctx?.zscaler_zia?.tunnel?.destination?.end?.ip != null @@ -253,6 +291,8 @@ processors: field: json.dpdrec target_field: zscaler_zia.tunnel.dpd_packets ignore_missing: true + - community_id: + ignore_failure: true - script: description: Drops null/empty values recursively lang: painless @@ -278,7 +318,7 @@ processors: for (Map.Entry m : ctx.json.entrySet()) { ctx.zscaler_zia.tunnel[m.getKey()] = m.getValue(); } - - remove: + - remove: field: json ignore_missing: true - remove: diff --git a/packages/zscaler_zia/data_stream/tunnel/fields/ecs.yml b/packages/zscaler_zia/data_stream/tunnel/fields/ecs.yml index eb68acfb87f..3e24464163c 100644 --- a/packages/zscaler_zia/data_stream/tunnel/fields/ecs.yml +++ b/packages/zscaler_zia/data_stream/tunnel/fields/ecs.yml @@ -14,6 +14,12 @@ name: event.id - external: ecs name: event.reason +- external: ecs + name: network.community_id +- external: ecs + name: network.iana_number +- external: ecs + name: network.transport - external: ecs name: related.ip - external: ecs diff --git a/packages/zscaler_zia/data_stream/tunnel/manifest.yml b/packages/zscaler_zia/data_stream/tunnel/manifest.yml index 477b606f29d..d987ddda474 100644 --- a/packages/zscaler_zia/data_stream/tunnel/manifest.yml +++ b/packages/zscaler_zia/data_stream/tunnel/manifest.yml @@ -4,7 +4,7 @@ streams: - input: tcp template_path: tcp.yml.hbs title: Zscaler Internet Access Tunnel Logs - description: Collect Zscaler Internet Access Tunnel Logs using tcp input + description: Collect Zscaler Internet Access Tunnel Logs using TCP Input. vars: - name: listen_port type: integer @@ -27,7 +27,45 @@ streams: required: true show_user: true title: Preserve original event - description: Preserves a raw copy of the original event, added to the field `event.original` + description: Preserves a raw copy of the original event, added to the field `event.original`. + type: bool + multi: false + default: false + - name: processors + type: yaml + title: Processors + multi: false + required: false + show_user: false + description: >- + Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details. + - input: http_endpoint + template_path: http_endpoint.yml.hbs + title: Zscaler Internet Access Tunnel Logs + description: Collect Zscaler Internet Access Tunnel logs via HTTP Endpoint Input. + vars: + - name: listen_port + type: integer + title: Listen Port + description: The port number to listen on. + multi: false + required: true + show_user: true + default: 9558 + - name: tags + type: text + title: Tags + multi: true + required: true + show_user: false + default: + - forwarded + - zscaler_zia-tunnel + - name: preserve_original_event + required: true + show_user: true + title: Preserve original event + description: Preserves a raw copy of the original event, added to the field `event.original`. type: bool multi: false default: false diff --git a/packages/zscaler_zia/data_stream/tunnel/sample_event.json b/packages/zscaler_zia/data_stream/tunnel/sample_event.json index b3aa970bfcf..83379d7618b 100644 --- a/packages/zscaler_zia/data_stream/tunnel/sample_event.json +++ b/packages/zscaler_zia/data_stream/tunnel/sample_event.json @@ -1,12 +1,11 @@ { - "@timestamp": "2021-12-30T11:20:12.000Z", + "@timestamp": "2021-12-31T11:12:13.000Z", "agent": { - "ephemeral_id": "63ac98b6-0ff6-4943-820e-8505eff15937", - "hostname": "docker-fleet-agent", - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "ephemeral_id": "b187ac54-dab8-4e34-b72d-36772d818767", + "id": "f6f3ddbc-7ab7-4a74-aeeb-152405dea56f", "name": "docker-fleet-agent", "type": "filebeat", - "version": "7.16.2" + "version": "8.1.2" }, "data_stream": { "dataset": "zscaler_zia.tunnel", @@ -20,16 +19,18 @@ "version": "8.2.0" }, "elastic_agent": { - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "id": "f6f3ddbc-7ab7-4a74-aeeb-152405dea56f", "snapshot": false, - "version": "7.16.2" + "version": "8.1.2" }, "event": { "agent_id_status": "verified", - "category": "network", + "category": [ + "network" + ], "dataset": "zscaler_zia.tunnel", "id": "1111111111111111111", - "ingested": "2022-02-04T06:36:16Z", + "ingested": "2021-12-31T05:06:07Z", "kind": "event", "type": [ "info" @@ -40,9 +41,12 @@ }, "log": { "source": { - "address": "172.21.0.7:44374" + "address": "1.128.3.4:58370" } }, + "network": { + "transport": "ipsec ikev 1" + }, "related": { "ip": [ "81.2.69.143", diff --git a/packages/zscaler_zia/data_stream/web/_dev/test/pipeline/test-web-http_endpoint.log b/packages/zscaler_zia/data_stream/web/_dev/test/pipeline/test-web-http_endpoint.log new file mode 100644 index 00000000000..c57f9e3211a --- /dev/null +++ b/packages/zscaler_zia/data_stream/web/_dev/test/pipeline/test-web-http_endpoint.log @@ -0,0 +1 @@ +{ "sourcetype" : "zscalernss-web", "event" :{"time":"2021-12-31 08:08:08","login":"test@example.com","proto":"HTTP_PROXY","eurl":"www.example.com","action":"Blocked","appname":"General Browsing","appclass":"General Browsing","reqsize":"600","respsize":"65","stime":"0","ctime":"0","urlclass":"Business Use","urlsupercat":"Information Technology","urlcat":"Web Search","malwarecat":"None","threatname":"None","riskscore":"0","dlpeng":"None","dlpdict":"None","location":"Test DB","dept":"Unknown","cip":"81.2.69.193","sip":"81.2.69.145","reqmethod":"CONNECT","respcode":"200","ua":"Windows Microsoft Windows 10 Pro ZTunnel/1.0","ereferer":"None","ruletype":"FwFilter","rulelabel":"Zscaler Proxy Traffic","contenttype":"Other","unscannabletype":"None","deviceowner":"administrator1","devicehostname":"TestMachine35"}} diff --git a/packages/zscaler_zia/data_stream/web/_dev/test/pipeline/test-web-http_endpoint.log-expected.json b/packages/zscaler_zia/data_stream/web/_dev/test/pipeline/test-web-http_endpoint.log-expected.json new file mode 100644 index 00000000000..de31572f0e5 --- /dev/null +++ b/packages/zscaler_zia/data_stream/web/_dev/test/pipeline/test-web-http_endpoint.log-expected.json @@ -0,0 +1,119 @@ +{ + "expected": [ + { + "@timestamp": "2021-12-31T08:08:08.000Z", + "destination": { + "ip": "81.2.69.145" + }, + "ecs": { + "version": "8.2.0" + }, + "event": { + "action": "blocked", + "category": [ + "web" + ], + "kind": "event", + "original": "{ \"sourcetype\" : \"zscalernss-web\", \"event\" :{\"time\":\"2021-12-31 08:08:08\",\"login\":\"test@example.com\",\"proto\":\"HTTP_PROXY\",\"eurl\":\"www.example.com\",\"action\":\"Blocked\",\"appname\":\"General Browsing\",\"appclass\":\"General Browsing\",\"reqsize\":\"600\",\"respsize\":\"65\",\"stime\":\"0\",\"ctime\":\"0\",\"urlclass\":\"Business Use\",\"urlsupercat\":\"Information Technology\",\"urlcat\":\"Web Search\",\"malwarecat\":\"None\",\"threatname\":\"None\",\"riskscore\":\"0\",\"dlpeng\":\"None\",\"dlpdict\":\"None\",\"location\":\"Test DB\",\"dept\":\"Unknown\",\"cip\":\"81.2.69.193\",\"sip\":\"81.2.69.145\",\"reqmethod\":\"CONNECT\",\"respcode\":\"200\",\"ua\":\"Windows Microsoft Windows 10 Pro ZTunnel/1.0\",\"ereferer\":\"None\",\"ruletype\":\"FwFilter\",\"rulelabel\":\"Zscaler Proxy Traffic\",\"contenttype\":\"Other\",\"unscannabletype\":\"None\",\"deviceowner\":\"administrator1\",\"devicehostname\":\"TestMachine35\"}}", + "risk_score": 0, + "type": [ + "info" + ] + }, + "http": { + "request": { + "bytes": 600, + "method": "CONNECT", + "mime_type": "Other", + "referrer": "None" + }, + "response": { + "bytes": 65, + "status_code": 200 + } + }, + "network": { + "protocol": "http_proxy" + }, + "related": { + "hosts": [ + "TestMachine35" + ], + "ip": [ + "81.2.69.193", + "81.2.69.145" + ] + }, + "rule": { + "name": "Zscaler Proxy Traffic", + "ruleset": "FwFilter" + }, + "source": { + "nat": { + "ip": "81.2.69.193" + }, + "user": { + "name": "administrator1" + } + }, + "tags": [ + "preserve_original_event" + ], + "url": { + "extension": "com", + "original": "www.example.com", + "path": "www.example.com" + }, + "user": { + "email": "test@example.com" + }, + "user_agent": { + "device": { + "name": "Other" + }, + "name": "Other", + "original": "Windows Microsoft Windows 10 Pro ZTunnel/1.0", + "os": { + "full": "Windows 10", + "name": "Windows", + "version": "10" + } + }, + "zscaler_zia": { + "web": { + "app": { + "class": "General Browsing", + "name": "General Browsing" + }, + "ctime": 0, + "department": "Unknown", + "device": { + "hostname": "TestMachine35" + }, + "dpl": { + "dictionaries": "None", + "engine": "None" + }, + "location": "Test DB", + "malware": { + "category": "None" + }, + "stime": 0, + "threat": { + "name": "None" + }, + "unscannable": { + "type": "None" + }, + "url": { + "category": { + "sub": "Web Search", + "super": "Information Technology" + }, + "class": "Business Use" + } + } + } + } + ] +} \ No newline at end of file diff --git a/packages/zscaler_zia/data_stream/web/_dev/test/pipeline/test-web.log b/packages/zscaler_zia/data_stream/web/_dev/test/pipeline/test-web.log index c2a207b118a..58446934f84 100644 --- a/packages/zscaler_zia/data_stream/web/_dev/test/pipeline/test-web.log +++ b/packages/zscaler_zia/data_stream/web/_dev/test/pipeline/test-web.log @@ -1,5 +1,5 @@ -{ "sourcetype" : "zscalernss-web", "event" :{"time":"Fri Dec 17 07:04:57 2021","login":"test@example.com","proto":"HTTP_PROXY","eurl":"www.example.com","action":"Blocked","appname":"General Browsing","appclass":"General Browsing","reqsize":"600","respsize":"65","stime":"0","ctime":"0","urlclass":"Business Use","urlsupercat":"Information Technology","urlcat":"Web Search","malwarecat":"None","threatname":"None","riskscore":"0","dlpeng":"None","dlpdict":"None","location":"Test DB","dept":"Unknown","cip":"81.2.69.193","sip":"81.2.69.145","reqmethod":"CONNECT","respcode":"200","ua":"Windows Microsoft Windows 10 Pro ZTunnel/1.0","ereferer":"None","ruletype":"FwFilter","rulelabel":"Zscaler Proxy Traffic","contenttype":"Other","unscannabletype":"None","deviceowner":"administrator1","devicehostname":"TestMachine35"}} -{ "sourcetype" : "zscalernss-web", "event" :{"time":"Fri Dec 17 07:06:18 2021","login":"test@example.com","proto":"HTTPS","eurl":"www.example.com.com/join/","action":"Blocked","appname":"General Browsing","appclass":"General Browsing","reqsize":"218","respsize":"14230","stime":"0","ctime":"0","urlclass":"Business Use","urlsupercat":"Business and Economy","urlcat":"Corporate Marketing","malwarecat":"None","threatname":"None","riskscore":"0","dlpeng":"None","dlpdict":"None","location":"Test DB","dept":"Unknown","cip":"81.2.69.193","sip":"89.160.20.156","reqmethod":"POST","respcode":"403","ua":"Microsoft-Delivery-Optimization/10.0","ereferer":"None","ruletype":"SSLPol","rulelabel":"SSL_1","contenttype":"Other","unscannabletype":"None","deviceowner":"administrator1","devicehostname":"TestMachine35"}} -{ "sourcetype" : "zscalernss-web", "event" :{"time":"Fri Dec 17 07:05:35 2021","login":"test@example.com","proto":"HTTP_PROXY","eurl":"www.example.com","action":"Blocked","appname":"General Browsing","appclass":"General Browsing","reqsize":"600","respsize":"65","stime":"0","ctime":"0","urlclass":"Business Use","urlsupercat":"Information Technology","urlcat":"Web Search","malwarecat":"None","threatname":"None","riskscore":"0","dlpeng":"None","dlpdict":"None","location":"Test DB","dept":"Unknown","cip":"81.2.69.193","sip":"89.160.20.112","reqmethod":"CONNECT","respcode":"200","ua":"Windows Microsoft Windows 10 Pro ZTunnel/1.0","ereferer":"None","ruletype":"FwFilter","rulelabel":"Zscaler Proxy Traffic","contenttype":"Other","unscannabletype":"None","deviceowner":"administrator1","devicehostname":"TestMachine35"}} -{ "sourcetype" : "zscalernss-web", "event" :{"time":"Fri Dec 17 07:05:35 2021","login":"test@example.com","proto":"HTTP_PROXY","eurl":"www.example.com","action":"Blocked","appname":"General Browsing","appclass":"General Browsing","reqsize":"555","respsize":"65","stime":"0","ctime":"0","urlclass":"Business Use","urlsupercat":"Information Technology","urlcat":"Web Search","malwarecat":"None","threatname":"None","riskscore":"0","dlpeng":"None","dlpdict":"None","location":"Test DB","dept":"Unknown","cip":"81.2.69.193","sip":"81.2.69.144","reqmethod":"CONNECT","respcode":"200","ua":"Windows Microsoft Windows 10 Pro ZTunnel/1.0","ereferer":"None","ruletype":"FwFilter","rulelabel":"Zscaler Proxy Traffic","contenttype":"Other","unscannabletype":"None","deviceowner":"administrator1","devicehostname":"TestMachine35"}} -{ "sourcetype" : "zscalernss-web", "event" :{"time":"Fri Dec 17 07:37:28 2021","login":"test@example.com","proto":"HTTPS","eurl":"www.example.com.com/params?version=10.0.19041.1266&user=65792&Id=1","action":"Blocked","appname":"General Browsing","appclass":"General Browsing","reqsize":"297","respsize":"14135","stime":"0","ctime":"0","urlclass":"Business Use","urlsupercat":"Business and Economy","urlcat":"Corporate Marketing","malwarecat":"None","threatname":"None","riskscore":"0","dlpeng":"None","dlpdict":"None","location":"Test DB","dept":"Unknown","cip":"81.2.69.193","sip":"81.2.69.143","reqmethod":"GET","respcode":"403","ua":"Microsoft-Delivery-Optimization/10.0","ereferer":"None","ruletype":"FwFilter","rulelabel":"Access Blocked","contenttype":"Other","unscannabletype":"None","deviceowner":"administrator1","devicehostname":"TestMachine35"}} +{ "sourcetype" : "zscalernss-web", "event" :{"time":"Fri Dec 31 07:07:07 2021","login":"test@example.com","proto":"HTTP_PROXY","eurl":"www.example.com","action":"Blocked","appname":"General Browsing","appclass":"General Browsing","reqsize":"600","respsize":"65","stime":"0","ctime":"0","urlclass":"Business Use","urlsupercat":"Information Technology","urlcat":"Web Search","malwarecat":"None","threatname":"None","riskscore":"0","dlpeng":"None","dlpdict":"None","location":"Test DB","dept":"Unknown","cip":"81.2.69.193","sip":"81.2.69.145","reqmethod":"CONNECT","respcode":"200","ua":"Windows Microsoft Windows 10 Pro ZTunnel/1.0","ereferer":"None","ruletype":"FwFilter","rulelabel":"Zscaler Proxy Traffic","contenttype":"Other","unscannabletype":"None","deviceowner":"administrator1","devicehostname":"TestMachine35"}} +{ "sourcetype" : "zscalernss-web", "event" :{"time":"Fri Dec 31 07:07:07 2021","login":"test@example.com","proto":"HTTPS","eurl":"www.example.com.com/join/","action":"Blocked","appname":"General Browsing","appclass":"General Browsing","reqsize":"218","respsize":"14230","stime":"0","ctime":"0","urlclass":"Business Use","urlsupercat":"Business and Economy","urlcat":"Corporate Marketing","malwarecat":"None","threatname":"None","riskscore":"0","dlpeng":"None","dlpdict":"None","location":"Test DB","dept":"Unknown","cip":"81.2.69.193","sip":"89.160.20.156","reqmethod":"POST","respcode":"403","ua":"Microsoft-Delivery-Optimization/10.0","ereferer":"None","ruletype":"SSLPol","rulelabel":"SSL_1","contenttype":"Other","unscannabletype":"None","deviceowner":"administrator1","devicehostname":"TestMachine35"}} +{ "sourcetype" : "zscalernss-web", "event" :{"time":"Fri Dec 31 07:07:07 2021","login":"test@example.com","proto":"HTTP_PROXY","eurl":"www.example.com","action":"Blocked","appname":"General Browsing","appclass":"General Browsing","reqsize":"600","respsize":"65","stime":"0","ctime":"0","urlclass":"Business Use","urlsupercat":"Information Technology","urlcat":"Web Search","malwarecat":"None","threatname":"None","riskscore":"0","dlpeng":"None","dlpdict":"None","location":"Test DB","dept":"Unknown","cip":"81.2.69.193","sip":"89.160.20.112","reqmethod":"CONNECT","respcode":"200","ua":"Windows Microsoft Windows 10 Pro ZTunnel/1.0","ereferer":"None","ruletype":"FwFilter","rulelabel":"Zscaler Proxy Traffic","contenttype":"Other","unscannabletype":"None","deviceowner":"administrator1","devicehostname":"TestMachine35"}} +{ "sourcetype" : "zscalernss-web", "event" :{"time":"Fri Dec 31 07:07:07 2021","login":"test@example.com","proto":"HTTP_PROXY","eurl":"www.example.com","action":"Blocked","appname":"General Browsing","appclass":"General Browsing","reqsize":"555","respsize":"65","stime":"0","ctime":"0","urlclass":"Business Use","urlsupercat":"Information Technology","urlcat":"Web Search","malwarecat":"None","threatname":"None","riskscore":"0","dlpeng":"None","dlpdict":"None","location":"Test DB","dept":"Unknown","cip":"81.2.69.193","sip":"81.2.69.144","reqmethod":"CONNECT","respcode":"200","ua":"Windows Microsoft Windows 10 Pro ZTunnel/1.0","ereferer":"None","ruletype":"FwFilter","rulelabel":"Zscaler Proxy Traffic","contenttype":"Other","unscannabletype":"None","deviceowner":"administrator1","devicehostname":"TestMachine35"}} +{ "sourcetype" : "zscalernss-web", "event" :{"time":"Fri Dec 31 07:07:07 2021","login":"test@example.com","proto":"HTTPS","eurl":"www.example.com.com/params?version=10.0.19041.1266&user=65792&Id=1","action":"Blocked","appname":"General Browsing","appclass":"General Browsing","reqsize":"297","respsize":"14135","stime":"0","ctime":"0","urlclass":"Business Use","urlsupercat":"Business and Economy","urlcat":"Corporate Marketing","malwarecat":"None","threatname":"None","riskscore":"0","dlpeng":"None","dlpdict":"None","location":"Test DB","dept":"Unknown","cip":"81.2.69.193","sip":"81.2.69.143","reqmethod":"GET","respcode":"403","ua":"Microsoft-Delivery-Optimization/10.0","ereferer":"None","ruletype":"FwFilter","rulelabel":"Access Blocked","contenttype":"Other","unscannabletype":"None","deviceowner":"administrator1","devicehostname":"TestMachine35"}} diff --git a/packages/zscaler_zia/data_stream/web/_dev/test/pipeline/test-web.log-expected.json b/packages/zscaler_zia/data_stream/web/_dev/test/pipeline/test-web.log-expected.json index 60dd7bde1fd..0526a059941 100644 --- a/packages/zscaler_zia/data_stream/web/_dev/test/pipeline/test-web.log-expected.json +++ b/packages/zscaler_zia/data_stream/web/_dev/test/pipeline/test-web.log-expected.json @@ -1,24 +1,20 @@ { "expected": [ { - "@timestamp": "2021-12-17T07:04:57.000Z", - "client": { - "ip": "81.2.69.193", - "nat": { - "ip": "81.2.69.145" - }, - "user": { - "name": "administrator1" - } + "@timestamp": "2021-12-31T07:07:07.000Z", + "destination": { + "ip": "81.2.69.145" }, "ecs": { "version": "8.2.0" }, "event": { "action": "blocked", - "category": "web", + "category": [ + "web" + ], "kind": "event", - "original": "{ \"sourcetype\" : \"zscalernss-web\", \"event\" :{\"time\":\"Fri Dec 17 07:04:57 2021\",\"login\":\"test@example.com\",\"proto\":\"HTTP_PROXY\",\"eurl\":\"www.example.com\",\"action\":\"Blocked\",\"appname\":\"General Browsing\",\"appclass\":\"General Browsing\",\"reqsize\":\"600\",\"respsize\":\"65\",\"stime\":\"0\",\"ctime\":\"0\",\"urlclass\":\"Business Use\",\"urlsupercat\":\"Information Technology\",\"urlcat\":\"Web Search\",\"malwarecat\":\"None\",\"threatname\":\"None\",\"riskscore\":\"0\",\"dlpeng\":\"None\",\"dlpdict\":\"None\",\"location\":\"Test DB\",\"dept\":\"Unknown\",\"cip\":\"81.2.69.193\",\"sip\":\"81.2.69.145\",\"reqmethod\":\"CONNECT\",\"respcode\":\"200\",\"ua\":\"Windows Microsoft Windows 10 Pro ZTunnel/1.0\",\"ereferer\":\"None\",\"ruletype\":\"FwFilter\",\"rulelabel\":\"Zscaler Proxy Traffic\",\"contenttype\":\"Other\",\"unscannabletype\":\"None\",\"deviceowner\":\"administrator1\",\"devicehostname\":\"TestMachine35\"}}", + "original": "{ \"sourcetype\" : \"zscalernss-web\", \"event\" :{\"time\":\"Fri Dec 31 07:07:07 2021\",\"login\":\"test@example.com\",\"proto\":\"HTTP_PROXY\",\"eurl\":\"www.example.com\",\"action\":\"Blocked\",\"appname\":\"General Browsing\",\"appclass\":\"General Browsing\",\"reqsize\":\"600\",\"respsize\":\"65\",\"stime\":\"0\",\"ctime\":\"0\",\"urlclass\":\"Business Use\",\"urlsupercat\":\"Information Technology\",\"urlcat\":\"Web Search\",\"malwarecat\":\"None\",\"threatname\":\"None\",\"riskscore\":\"0\",\"dlpeng\":\"None\",\"dlpdict\":\"None\",\"location\":\"Test DB\",\"dept\":\"Unknown\",\"cip\":\"81.2.69.193\",\"sip\":\"81.2.69.145\",\"reqmethod\":\"CONNECT\",\"respcode\":\"200\",\"ua\":\"Windows Microsoft Windows 10 Pro ZTunnel/1.0\",\"ereferer\":\"None\",\"ruletype\":\"FwFilter\",\"rulelabel\":\"Zscaler Proxy Traffic\",\"contenttype\":\"Other\",\"unscannabletype\":\"None\",\"deviceowner\":\"administrator1\",\"devicehostname\":\"TestMachine35\"}}", "risk_score": 0, "type": [ "info" @@ -52,6 +48,14 @@ "name": "Zscaler Proxy Traffic", "ruleset": "FwFilter" }, + "source": { + "nat": { + "ip": "81.2.69.193" + }, + "user": { + "name": "administrator1" + } + }, "tags": [ "preserve_original_event" ], @@ -112,24 +116,20 @@ } }, { - "@timestamp": "2021-12-17T07:06:18.000Z", - "client": { - "ip": "81.2.69.193", - "nat": { - "ip": "89.160.20.156" - }, - "user": { - "name": "administrator1" - } + "@timestamp": "2021-12-31T07:07:07.000Z", + "destination": { + "ip": "89.160.20.156" }, "ecs": { "version": "8.2.0" }, "event": { "action": "blocked", - "category": "web", + "category": [ + "web" + ], "kind": "event", - "original": "{ \"sourcetype\" : \"zscalernss-web\", \"event\" :{\"time\":\"Fri Dec 17 07:06:18 2021\",\"login\":\"test@example.com\",\"proto\":\"HTTPS\",\"eurl\":\"www.example.com.com/join/\",\"action\":\"Blocked\",\"appname\":\"General Browsing\",\"appclass\":\"General Browsing\",\"reqsize\":\"218\",\"respsize\":\"14230\",\"stime\":\"0\",\"ctime\":\"0\",\"urlclass\":\"Business Use\",\"urlsupercat\":\"Business and Economy\",\"urlcat\":\"Corporate Marketing\",\"malwarecat\":\"None\",\"threatname\":\"None\",\"riskscore\":\"0\",\"dlpeng\":\"None\",\"dlpdict\":\"None\",\"location\":\"Test DB\",\"dept\":\"Unknown\",\"cip\":\"81.2.69.193\",\"sip\":\"89.160.20.156\",\"reqmethod\":\"POST\",\"respcode\":\"403\",\"ua\":\"Microsoft-Delivery-Optimization/10.0\",\"ereferer\":\"None\",\"ruletype\":\"SSLPol\",\"rulelabel\":\"SSL_1\",\"contenttype\":\"Other\",\"unscannabletype\":\"None\",\"deviceowner\":\"administrator1\",\"devicehostname\":\"TestMachine35\"}}", + "original": "{ \"sourcetype\" : \"zscalernss-web\", \"event\" :{\"time\":\"Fri Dec 31 07:07:07 2021\",\"login\":\"test@example.com\",\"proto\":\"HTTPS\",\"eurl\":\"www.example.com.com/join/\",\"action\":\"Blocked\",\"appname\":\"General Browsing\",\"appclass\":\"General Browsing\",\"reqsize\":\"218\",\"respsize\":\"14230\",\"stime\":\"0\",\"ctime\":\"0\",\"urlclass\":\"Business Use\",\"urlsupercat\":\"Business and Economy\",\"urlcat\":\"Corporate Marketing\",\"malwarecat\":\"None\",\"threatname\":\"None\",\"riskscore\":\"0\",\"dlpeng\":\"None\",\"dlpdict\":\"None\",\"location\":\"Test DB\",\"dept\":\"Unknown\",\"cip\":\"81.2.69.193\",\"sip\":\"89.160.20.156\",\"reqmethod\":\"POST\",\"respcode\":\"403\",\"ua\":\"Microsoft-Delivery-Optimization/10.0\",\"ereferer\":\"None\",\"ruletype\":\"SSLPol\",\"rulelabel\":\"SSL_1\",\"contenttype\":\"Other\",\"unscannabletype\":\"None\",\"deviceowner\":\"administrator1\",\"devicehostname\":\"TestMachine35\"}}", "risk_score": 0, "type": [ "info" @@ -163,6 +163,14 @@ "name": "SSL_1", "ruleset": "SSLPol" }, + "source": { + "nat": { + "ip": "81.2.69.193" + }, + "user": { + "name": "administrator1" + } + }, "tags": [ "preserve_original_event" ], @@ -218,24 +226,20 @@ } }, { - "@timestamp": "2021-12-17T07:05:35.000Z", - "client": { - "ip": "81.2.69.193", - "nat": { - "ip": "89.160.20.112" - }, - "user": { - "name": "administrator1" - } + "@timestamp": "2021-12-31T07:07:07.000Z", + "destination": { + "ip": "89.160.20.112" }, "ecs": { "version": "8.2.0" }, "event": { "action": "blocked", - "category": "web", + "category": [ + "web" + ], "kind": "event", - "original": "{ \"sourcetype\" : \"zscalernss-web\", \"event\" :{\"time\":\"Fri Dec 17 07:05:35 2021\",\"login\":\"test@example.com\",\"proto\":\"HTTP_PROXY\",\"eurl\":\"www.example.com\",\"action\":\"Blocked\",\"appname\":\"General Browsing\",\"appclass\":\"General Browsing\",\"reqsize\":\"600\",\"respsize\":\"65\",\"stime\":\"0\",\"ctime\":\"0\",\"urlclass\":\"Business Use\",\"urlsupercat\":\"Information Technology\",\"urlcat\":\"Web Search\",\"malwarecat\":\"None\",\"threatname\":\"None\",\"riskscore\":\"0\",\"dlpeng\":\"None\",\"dlpdict\":\"None\",\"location\":\"Test DB\",\"dept\":\"Unknown\",\"cip\":\"81.2.69.193\",\"sip\":\"89.160.20.112\",\"reqmethod\":\"CONNECT\",\"respcode\":\"200\",\"ua\":\"Windows Microsoft Windows 10 Pro ZTunnel/1.0\",\"ereferer\":\"None\",\"ruletype\":\"FwFilter\",\"rulelabel\":\"Zscaler Proxy Traffic\",\"contenttype\":\"Other\",\"unscannabletype\":\"None\",\"deviceowner\":\"administrator1\",\"devicehostname\":\"TestMachine35\"}}", + "original": "{ \"sourcetype\" : \"zscalernss-web\", \"event\" :{\"time\":\"Fri Dec 31 07:07:07 2021\",\"login\":\"test@example.com\",\"proto\":\"HTTP_PROXY\",\"eurl\":\"www.example.com\",\"action\":\"Blocked\",\"appname\":\"General Browsing\",\"appclass\":\"General Browsing\",\"reqsize\":\"600\",\"respsize\":\"65\",\"stime\":\"0\",\"ctime\":\"0\",\"urlclass\":\"Business Use\",\"urlsupercat\":\"Information Technology\",\"urlcat\":\"Web Search\",\"malwarecat\":\"None\",\"threatname\":\"None\",\"riskscore\":\"0\",\"dlpeng\":\"None\",\"dlpdict\":\"None\",\"location\":\"Test DB\",\"dept\":\"Unknown\",\"cip\":\"81.2.69.193\",\"sip\":\"89.160.20.112\",\"reqmethod\":\"CONNECT\",\"respcode\":\"200\",\"ua\":\"Windows Microsoft Windows 10 Pro ZTunnel/1.0\",\"ereferer\":\"None\",\"ruletype\":\"FwFilter\",\"rulelabel\":\"Zscaler Proxy Traffic\",\"contenttype\":\"Other\",\"unscannabletype\":\"None\",\"deviceowner\":\"administrator1\",\"devicehostname\":\"TestMachine35\"}}", "risk_score": 0, "type": [ "info" @@ -269,6 +273,14 @@ "name": "Zscaler Proxy Traffic", "ruleset": "FwFilter" }, + "source": { + "nat": { + "ip": "81.2.69.193" + }, + "user": { + "name": "administrator1" + } + }, "tags": [ "preserve_original_event" ], @@ -329,24 +341,20 @@ } }, { - "@timestamp": "2021-12-17T07:05:35.000Z", - "client": { - "ip": "81.2.69.193", - "nat": { - "ip": "81.2.69.144" - }, - "user": { - "name": "administrator1" - } + "@timestamp": "2021-12-31T07:07:07.000Z", + "destination": { + "ip": "81.2.69.144" }, "ecs": { "version": "8.2.0" }, "event": { "action": "blocked", - "category": "web", + "category": [ + "web" + ], "kind": "event", - "original": "{ \"sourcetype\" : \"zscalernss-web\", \"event\" :{\"time\":\"Fri Dec 17 07:05:35 2021\",\"login\":\"test@example.com\",\"proto\":\"HTTP_PROXY\",\"eurl\":\"www.example.com\",\"action\":\"Blocked\",\"appname\":\"General Browsing\",\"appclass\":\"General Browsing\",\"reqsize\":\"555\",\"respsize\":\"65\",\"stime\":\"0\",\"ctime\":\"0\",\"urlclass\":\"Business Use\",\"urlsupercat\":\"Information Technology\",\"urlcat\":\"Web Search\",\"malwarecat\":\"None\",\"threatname\":\"None\",\"riskscore\":\"0\",\"dlpeng\":\"None\",\"dlpdict\":\"None\",\"location\":\"Test DB\",\"dept\":\"Unknown\",\"cip\":\"81.2.69.193\",\"sip\":\"81.2.69.144\",\"reqmethod\":\"CONNECT\",\"respcode\":\"200\",\"ua\":\"Windows Microsoft Windows 10 Pro ZTunnel/1.0\",\"ereferer\":\"None\",\"ruletype\":\"FwFilter\",\"rulelabel\":\"Zscaler Proxy Traffic\",\"contenttype\":\"Other\",\"unscannabletype\":\"None\",\"deviceowner\":\"administrator1\",\"devicehostname\":\"TestMachine35\"}}", + "original": "{ \"sourcetype\" : \"zscalernss-web\", \"event\" :{\"time\":\"Fri Dec 31 07:07:07 2021\",\"login\":\"test@example.com\",\"proto\":\"HTTP_PROXY\",\"eurl\":\"www.example.com\",\"action\":\"Blocked\",\"appname\":\"General Browsing\",\"appclass\":\"General Browsing\",\"reqsize\":\"555\",\"respsize\":\"65\",\"stime\":\"0\",\"ctime\":\"0\",\"urlclass\":\"Business Use\",\"urlsupercat\":\"Information Technology\",\"urlcat\":\"Web Search\",\"malwarecat\":\"None\",\"threatname\":\"None\",\"riskscore\":\"0\",\"dlpeng\":\"None\",\"dlpdict\":\"None\",\"location\":\"Test DB\",\"dept\":\"Unknown\",\"cip\":\"81.2.69.193\",\"sip\":\"81.2.69.144\",\"reqmethod\":\"CONNECT\",\"respcode\":\"200\",\"ua\":\"Windows Microsoft Windows 10 Pro ZTunnel/1.0\",\"ereferer\":\"None\",\"ruletype\":\"FwFilter\",\"rulelabel\":\"Zscaler Proxy Traffic\",\"contenttype\":\"Other\",\"unscannabletype\":\"None\",\"deviceowner\":\"administrator1\",\"devicehostname\":\"TestMachine35\"}}", "risk_score": 0, "type": [ "info" @@ -380,6 +388,14 @@ "name": "Zscaler Proxy Traffic", "ruleset": "FwFilter" }, + "source": { + "nat": { + "ip": "81.2.69.193" + }, + "user": { + "name": "administrator1" + } + }, "tags": [ "preserve_original_event" ], @@ -440,24 +456,20 @@ } }, { - "@timestamp": "2021-12-17T07:37:28.000Z", - "client": { - "ip": "81.2.69.193", - "nat": { - "ip": "81.2.69.143" - }, - "user": { - "name": "administrator1" - } + "@timestamp": "2021-12-31T07:07:07.000Z", + "destination": { + "ip": "81.2.69.143" }, "ecs": { "version": "8.2.0" }, "event": { "action": "blocked", - "category": "web", + "category": [ + "web" + ], "kind": "event", - "original": "{ \"sourcetype\" : \"zscalernss-web\", \"event\" :{\"time\":\"Fri Dec 17 07:37:28 2021\",\"login\":\"test@example.com\",\"proto\":\"HTTPS\",\"eurl\":\"www.example.com.com/params?version=10.0.19041.1266\u0026user=65792\u0026Id=1\",\"action\":\"Blocked\",\"appname\":\"General Browsing\",\"appclass\":\"General Browsing\",\"reqsize\":\"297\",\"respsize\":\"14135\",\"stime\":\"0\",\"ctime\":\"0\",\"urlclass\":\"Business Use\",\"urlsupercat\":\"Business and Economy\",\"urlcat\":\"Corporate Marketing\",\"malwarecat\":\"None\",\"threatname\":\"None\",\"riskscore\":\"0\",\"dlpeng\":\"None\",\"dlpdict\":\"None\",\"location\":\"Test DB\",\"dept\":\"Unknown\",\"cip\":\"81.2.69.193\",\"sip\":\"81.2.69.143\",\"reqmethod\":\"GET\",\"respcode\":\"403\",\"ua\":\"Microsoft-Delivery-Optimization/10.0\",\"ereferer\":\"None\",\"ruletype\":\"FwFilter\",\"rulelabel\":\"Access Blocked\",\"contenttype\":\"Other\",\"unscannabletype\":\"None\",\"deviceowner\":\"administrator1\",\"devicehostname\":\"TestMachine35\"}}", + "original": "{ \"sourcetype\" : \"zscalernss-web\", \"event\" :{\"time\":\"Fri Dec 31 07:07:07 2021\",\"login\":\"test@example.com\",\"proto\":\"HTTPS\",\"eurl\":\"www.example.com.com/params?version=10.0.19041.1266\u0026user=65792\u0026Id=1\",\"action\":\"Blocked\",\"appname\":\"General Browsing\",\"appclass\":\"General Browsing\",\"reqsize\":\"297\",\"respsize\":\"14135\",\"stime\":\"0\",\"ctime\":\"0\",\"urlclass\":\"Business Use\",\"urlsupercat\":\"Business and Economy\",\"urlcat\":\"Corporate Marketing\",\"malwarecat\":\"None\",\"threatname\":\"None\",\"riskscore\":\"0\",\"dlpeng\":\"None\",\"dlpdict\":\"None\",\"location\":\"Test DB\",\"dept\":\"Unknown\",\"cip\":\"81.2.69.193\",\"sip\":\"81.2.69.143\",\"reqmethod\":\"GET\",\"respcode\":\"403\",\"ua\":\"Microsoft-Delivery-Optimization/10.0\",\"ereferer\":\"None\",\"ruletype\":\"FwFilter\",\"rulelabel\":\"Access Blocked\",\"contenttype\":\"Other\",\"unscannabletype\":\"None\",\"deviceowner\":\"administrator1\",\"devicehostname\":\"TestMachine35\"}}", "risk_score": 0, "type": [ "info" @@ -491,6 +503,14 @@ "name": "Access Blocked", "ruleset": "FwFilter" }, + "source": { + "nat": { + "ip": "81.2.69.193" + }, + "user": { + "name": "administrator1" + } + }, "tags": [ "preserve_original_event" ], diff --git a/packages/zscaler_zia/data_stream/web/_dev/test/system/test-http-endpoint-config.yml b/packages/zscaler_zia/data_stream/web/_dev/test/system/test-http-endpoint-config.yml new file mode 100644 index 00000000000..d281dfc2f41 --- /dev/null +++ b/packages/zscaler_zia/data_stream/web/_dev/test/system/test-http-endpoint-config.yml @@ -0,0 +1,8 @@ +service: zscaler-zia-web-http-endpoint +service_notify_signal: SIGHUP +input: http_endpoint +vars: + listen_address: 0.0.0.0 +data_stream: + vars: + listen_port: 9559 diff --git a/packages/zscaler_zia/data_stream/web/_dev/test/system/test-default-config.yml b/packages/zscaler_zia/data_stream/web/_dev/test/system/test-tcp-config.yml similarity index 100% rename from packages/zscaler_zia/data_stream/web/_dev/test/system/test-default-config.yml rename to packages/zscaler_zia/data_stream/web/_dev/test/system/test-tcp-config.yml diff --git a/packages/zscaler_zia/data_stream/web/agent/stream/http_endpoint.yml.hbs b/packages/zscaler_zia/data_stream/web/agent/stream/http_endpoint.yml.hbs new file mode 100644 index 00000000000..443fe325f7c --- /dev/null +++ b/packages/zscaler_zia/data_stream/web/agent/stream/http_endpoint.yml.hbs @@ -0,0 +1,21 @@ +listen_address: {{listen_address}} +listen_port: {{listen_port}} +content_type: "" +preserve_original_event: true +tags: +{{#if preserve_original_event}} + - preserve_original_event +{{/if}} +{{#each tags as |tag|}} + - {{tag}} +{{/each}} +{{#contains "forwarded" tags}} +publisher_pipeline.disable_host: true +{{/contains}} +{{#if ssl}} +ssl: {{ssl}} +{{/if}} +{{#if processors}} +processors: +{{processors}} +{{/if}} diff --git a/packages/zscaler_zia/data_stream/web/agent/stream/tcp.yml.hbs b/packages/zscaler_zia/data_stream/web/agent/stream/tcp.yml.hbs index 030459f2582..bc587e50a3a 100644 --- a/packages/zscaler_zia/data_stream/web/agent/stream/tcp.yml.hbs +++ b/packages/zscaler_zia/data_stream/web/agent/stream/tcp.yml.hbs @@ -1,4 +1,3 @@ -tcp: host: "{{listen_address}}:{{listen_port}}" tags: {{#if preserve_original_event}} diff --git a/packages/zscaler_zia/data_stream/web/elasticsearch/ingest_pipeline/default.yml b/packages/zscaler_zia/data_stream/web/elasticsearch/ingest_pipeline/default.yml index 2d160638d3c..aa47aedd596 100644 --- a/packages/zscaler_zia/data_stream/web/elasticsearch/ingest_pipeline/default.yml +++ b/packages/zscaler_zia/data_stream/web/elasticsearch/ingest_pipeline/default.yml @@ -11,10 +11,16 @@ processors: - json: field: event.original target_field: resp + ignore_failure: true + - remove: + field: json + if: ctx?.input?.type == 'http_endpoint' + ignore_missing: true - rename: field: resp.event target_field: json - - remove: + ignore_missing: true + - remove: field: resp ignore_missing: true - date: @@ -25,10 +31,22 @@ processors: - E MMM dd HH:mm:ss yyyy - E MMM d HH:mm:ss yyyy - E MMM d HH:mm:ss yyyy - - remove: - field: json.time + - yyyy-MM-dd HH:mm:ss + - date: + field: json.datetime + target_field: "@timestamp" + ignore_failure: true + formats: + - E MMM dd HH:mm:ss yyyy + - E MMM d HH:mm:ss yyyy + - E MMM d HH:mm:ss yyyy + - yyyy-MM-dd HH:mm:ss + - remove: + field: + - json.time + - json.datetime ignore_missing: true - - set: + - append: field: event.category value: web - set: @@ -39,27 +57,28 @@ processors: value: info - rename: field: json.cip - target_field: client.ip + target_field: source.nat.ip + if: ctx?.json?.cip != ctx?.json?.cintip ignore_missing: true - append: field: related.ip - value: "{{{client.ip}}}" - if: ctx?.client?.ip != null + value: "{{{source.nat.ip}}}" + if: ctx?.source?.nat?.ip != null allow_duplicates: false ignore_failure: true - rename: field: json.sip - target_field: client.nat.ip + target_field: destination.ip ignore_missing: true - append: field: related.ip - value: "{{{client.nat.ip}}}" - if: ctx?.client?.nat?.ip != null + value: "{{{destination.ip}}}" + if: ctx?.destination?.ip != null allow_duplicates: false ignore_failure: true - rename: field: json.deviceowner - target_field: client.user.name + target_field: source.user.name ignore_missing: true - convert: field: json.reqsize @@ -94,7 +113,7 @@ processors: target_field: http.response.status_code type: long ignore_failure: true - - remove: + - remove: field: json.respcode ignore_missing: true - rename: @@ -103,7 +122,7 @@ processors: ignore_missing: true - lowercase: field: network.protocol - ignore_missing: true + ignore_missing: true - rename: field: json.rulelabel target_field: rule.name @@ -116,7 +135,7 @@ processors: field: json.eurl remove_if_successful: true on_failure: - - set: + - set: field: url.original value: "{{{json.eurl}}}" if: ctx?.json?.eurl != null @@ -220,7 +239,7 @@ processors: field: json.devicehostname target_field: zscaler_zia.web.device.hostname ignore_missing: true - - append: + - append: field: related.hosts value: "{{{zscaler_zia.web.device.hostname}}}" if: ctx?.zscaler_zia?.web?.device?.hostname != null @@ -232,12 +251,12 @@ processors: ignore_missing: true - rename: field: json.cintip - target_field: zscaler_zia.web.client.internet.ip + target_field: source.ip ignore_missing: true - append: field: related.ip - value: "{{{zscaler_zia.web.client.internet.ip}}}" - if: ctx?.zscaler_zia?.web?.client?.internet?.ip != null + value: "{{{source.ip}}}" + if: ctx?.source?.ip != null allow_duplicates: false ignore_failure: true - rename: diff --git a/packages/zscaler_zia/data_stream/web/fields/ecs.yml b/packages/zscaler_zia/data_stream/web/fields/ecs.yml index 8e88de3f3f6..f25be184205 100644 --- a/packages/zscaler_zia/data_stream/web/fields/ecs.yml +++ b/packages/zscaler_zia/data_stream/web/fields/ecs.yml @@ -1,9 +1,5 @@ - external: ecs - name: client.ip -- external: ecs - name: client.nat.ip -- external: ecs - name: client.user.name + name: destination.ip - external: ecs name: ecs.version - external: ecs @@ -32,6 +28,12 @@ name: rule.name - external: ecs name: rule.ruleset +- external: ecs + name: source.ip +- external: ecs + name: source.nat.ip +- external: ecs + name: source.user.name - external: ecs name: tags - external: ecs @@ -54,8 +56,6 @@ name: url.scheme - external: ecs name: url.username -- external: ecs - name: user.email - external: ecs name: user_agent.device.name - external: ecs @@ -70,3 +70,5 @@ name: user_agent.os.version - external: ecs name: user_agent.version +- external: ecs + name: user.email diff --git a/packages/zscaler_zia/data_stream/web/fields/fields.yml b/packages/zscaler_zia/data_stream/web/fields/fields.yml index fcfdd76d8db..d85e5a570e6 100644 --- a/packages/zscaler_zia/data_stream/web/fields/fields.yml +++ b/packages/zscaler_zia/data_stream/web/fields/fields.yml @@ -16,10 +16,6 @@ type: keyword description: | Indicates whether the transaction was throttled due to a configured bandwidth policy. - - name: client.internet.ip - type: keyword - description: | - The client Internet (NATted Public) IP address. This is different from the cip value if the internal IP address is visible. Otherwise, same as cip. - name: ctime type: long description: | diff --git a/packages/zscaler_zia/data_stream/web/manifest.yml b/packages/zscaler_zia/data_stream/web/manifest.yml index c7ec897dbb8..c9137ffacee 100644 --- a/packages/zscaler_zia/data_stream/web/manifest.yml +++ b/packages/zscaler_zia/data_stream/web/manifest.yml @@ -4,7 +4,7 @@ streams: - input: tcp template_path: tcp.yml.hbs title: Zscaler Internet Access Web Logs - description: Collect Zscaler Internet Access Web Logs using tcp input + description: Collect Zscaler Internet Access Web Logs using TCP input. vars: - name: listen_port type: integer @@ -27,7 +27,45 @@ streams: required: true show_user: true title: Preserve original event - description: Preserves a raw copy of the original event, added to the field `event.original` + description: Preserves a raw copy of the original event, added to the field `event.original`. + type: bool + multi: false + default: false + - name: processors + type: yaml + title: Processors + multi: false + required: false + show_user: false + description: >- + Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details. + - input: http_endpoint + template_path: http_endpoint.yml.hbs + title: Zscaler Internet Access Web Logs + description: Collect Zscaler Internet Access Web logs via HTTP Endpoint Input. + vars: + - name: listen_port + type: integer + title: Listen Port + description: The port number to listen on. + multi: false + required: true + show_user: true + default: 9559 + - name: tags + type: text + title: Tags + multi: true + required: true + show_user: false + default: + - forwarded + - zscaler_zia-web + - name: preserve_original_event + required: true + show_user: true + title: Preserve original event + description: Preserves a raw copy of the original event, added to the field `event.original`. type: bool multi: false default: false diff --git a/packages/zscaler_zia/data_stream/web/sample_event.json b/packages/zscaler_zia/data_stream/web/sample_event.json index 061a72cef5a..6c8fd65b273 100644 --- a/packages/zscaler_zia/data_stream/web/sample_event.json +++ b/packages/zscaler_zia/data_stream/web/sample_event.json @@ -1,41 +1,36 @@ { "@timestamp": "2021-12-17T07:04:57.000Z", "agent": { - "ephemeral_id": "ced1fd2e-2f17-4f67-b8b1-d38a1920abbb", - "hostname": "docker-fleet-agent", - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "ephemeral_id": "6f164483-9eb8-4219-bb09-cd2ff3532390", + "id": "f6f3ddbc-7ab7-4a74-aeeb-152405dea56f", "name": "docker-fleet-agent", "type": "filebeat", - "version": "7.16.2" - }, - "client": { - "ip": "81.2.69.193", - "nat": { - "ip": "81.2.69.145" - }, - "user": { - "name": "administrator1" - } + "version": "8.1.2" }, "data_stream": { "dataset": "zscaler_zia.web", "namespace": "ep", "type": "logs" }, + "destination": { + "ip": "81.2.69.145" + }, "ecs": { "version": "8.2.0" }, "elastic_agent": { - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "id": "f6f3ddbc-7ab7-4a74-aeeb-152405dea56f", "snapshot": false, - "version": "7.16.2" + "version": "8.1.2" }, "event": { "action": "blocked", "agent_id_status": "verified", - "category": "web", + "category": [ + "web" + ], "dataset": "zscaler_zia.web", - "ingested": "2022-02-04T08:48:57Z", + "ingested": "2021-12-31T05:06:07Z", "kind": "event", "risk_score": 0, "type": [ @@ -59,7 +54,7 @@ }, "log": { "source": { - "address": "172.21.0.7:48722" + "address": "1.128.3.4:37608" } }, "network": { @@ -78,6 +73,14 @@ "name": "Zscaler Proxy Traffic", "ruleset": "FwFilter" }, + "source": { + "nat": { + "ip": "81.2.69.193" + }, + "user": { + "name": "administrator1" + } + }, "tags": [ "forwarded", "zscaler_zia-web" diff --git a/packages/zscaler_zia/docs/README.md b/packages/zscaler_zia/docs/README.md index 232a1a46b12..acb933e22c8 100644 --- a/packages/zscaler_zia/docs/README.md +++ b/packages/zscaler_zia/docs/README.md @@ -1,18 +1,15 @@ # Zscaler ZIA -This integration is for Zscaler Internet Access logs. It can be used -to receive logs sent by NSS log server on respective TCP ports. +This integration is for Zscaler Internet Access logs. It can be used to receive logs sent by NSS feeds on TCP port or Cloud NSS on HTTP Endpoint input methods. -The log message is expected to be in JSON format. The data is mapped to -ECS fields where applicable and the remaining fields are written under -`zscaler_zia..*`. +The log message is expected to be in JSON format. The data is mapped to ECS fields where applicable and the remaining fields are written under `zscaler_zia..*`. -## Setup steps +## Steps for setting up NSS Feeds 1. Enable the integration with the TCP input. -2. Configure the Zscaler NSS Server and NSS Feeds to send logs to the Elastic Agent that is running this integration. See [_Add NSS Server_](https://help.zscaler.com/zia/adding-nss-servers) and [_Add NSS Feeds_](https://help.zscaler.com/zia/adding-nss-feeds). Use the IP address hostname of the Elastic Agent as the 'NSS Feed SIEM IP Address/FQDN', and use the listening port of the Elastic Agent as the 'SIEM TCP Port' on the _Add NSS Feed_ configuration screen. To configure Zscalar NSS Server and NSS Feeds follow the following steps. +2. Configure the Zscaler NSS Server and NSS Feeds to send logs to the Elastic Agent that is running this integration. See [_Add NSS Server_](https://help.zscaler.com/zia/adding-nss-servers) and [_Add NSS Feeds_](https://help.zscaler.com/zia/adding-nss-feeds). Use the IP address hostname of the Elastic Agent as the 'NSS Feed SIEM IP Address/FQDN', and use the listening port of the Elastic Agent as the 'SIEM TCP Port' on the _Add NSS Feed_ configuration screen. To configure Zscaler NSS Server and NSS Feeds follow the following steps. - In the ZIA Admin Portal, add an NSS Server. - - Log in to the ZIA Admin Portal using your admin account. If you're unable to log in, contact Support. + - Log in to the ZIA Admin Portal using your admin account. - Add an NSS server. Refer to Adding NSS Servers to set up an [_Add NSS Server_](https://help.zscaler.com/zia/adding-nss-servers) for Web and/or Firewall. - Verify that the state of the NSS Server is healthy. - In the ZIA Admin Portal, go to Administration > Nanolog Streaming Service > NSS Servers. @@ -27,10 +24,32 @@ ECS fields where applicable and the remaining fields are written under - **Firewall**: 9012 - **Tunnel**: 9013 - **Web**: 9014 - - **Feed Output Type**: Select Custom paste the appropriate response format as follows: - ![NSS feeds setup image](../img/nss_feeds.png?raw=true) + - **Feed Output Type**: Select Custom in Feed output type and paste the appropriate response format in Feed output format as follows: + ![NSS Feeds setup image](../img/nss_feeds.png?raw=true) -3. *Please make sure to use the given response formats.* +## Steps for setting up Cloud NSS Feeds + +1. Enable the integration with the HTTP Endpoint input. +2. Configure the Zscaler Cloud NSS Feeds to send logs to the Elastic Agent that is running this integration. Provide API URL to send logs to the Elastic Agent. To configure Zscaler Cloud NSS Feeds follow the following steps. + - In the ZIA Admin Portal, add a Cloud NSS Feed. + - Log in to the ZIA Admin Portal using your admin account. + - Add a Cloud NSS Feed. Refer to [_Add Cloud NSS Feed_](https://help.zscaler.com/zia/adding-cloud-nss-feeds). + - In the ZIA Admin Portal, go to Administration > Nanolog Streaming Service > Cloud NSS Feeds. + - Give Feed Name, change status to Enabled. + - Select NSS Type. + - Change SIEM Type to other. + - Add an API URL. + - Default ports: + - **DNS**: 9556 + - **Firewall**: 9557 + - **Tunnel**: 9558 + - **Web**: 9559 + - Select JSON as feed output type. + - Add appropriate HTTP headers. + ![Cloud NSS Feeds setup image](../img/cloud_nss_feeds.png?raw=true) +3. Repeat step 2 for each log type. + +**Please make sure to use the given response formats for NSS and Cloud NSS Feeds.** ## Compatibility @@ -40,7 +59,7 @@ This package has been tested against `Zscaler Internet Access version 6.1` ### Alerts -Default port: _9010_ +- Default port (NSS Feed): _9010_ Vendor documentation: https://help.zscaler.com/zia/about-alerts @@ -56,7 +75,8 @@ Sample Response: ### DNS Log -Default port: _9011_ +- Default port (NSS Feed): _9011_ +- Default port (Cloud NSS Feed): _9556_ Vendor documentation: https://help.zscaler.com/zia/nss-feed-output-format-dns-logs @@ -72,7 +92,8 @@ Sample Response: ### Firewall Log -Default port: _9012_ +- Default port (NSS Feed): _9012_ +- Default port (Cloud NSS Feed): _9557_ Vendor documentation: https://help.zscaler.com/zia/nss-feed-output-format-firewall-logs @@ -88,7 +109,8 @@ Sample Response: ### Tunnel Log -Default port: _9013_ +- Default port (NSS Feed): _9013_ +- Default port (Cloud NSS Feed): _9558_ Vendor documentation: https://help.zscaler.com/zia/nss-feed-output-format-tunnel-logs @@ -117,8 +139,9 @@ Sample Response: ### Web Log -Default port: _9014_ -Add characters **"** and **\\** in **feed escape character** while configuring Web Log. +- Default port (NSS Feed): _9014_ +- Default port (Cloud NSS Feed): _9559_ +- Add characters **"** and **\\** in **feed escape character** while configuring Web Log. ![Escape feed setup image](../img/escape_feed.png?raw=true) Vendor documentation: https://help.zscaler.com/zia/nss-feed-output-format-web-logs @@ -198,12 +221,11 @@ An example event for `alerts` looks as following: { "@timestamp": "2022-12-10T13:40:32.000Z", "agent": { - "ephemeral_id": "8c093fcf-fb2f-4baa-b794-40edb011194d", - "hostname": "docker-fleet-agent", - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "ephemeral_id": "b7f77db9-92fe-4935-8387-b2cb545bcfc6", + "id": "638019f9-173e-4c24-9e28-64b128c92162", "name": "docker-fleet-agent", "type": "filebeat", - "version": "7.16.2" + "version": "8.1.2" }, "data_stream": { "dataset": "zscaler_zia.alerts", @@ -219,21 +241,21 @@ An example event for `alerts` looks as following: "version": "8.2.0" }, "elastic_agent": { - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "id": "638019f9-173e-4c24-9e28-64b128c92162", "snapshot": false, - "version": "7.16.2" + "version": "8.1.2" }, "event": { "agent_id_status": "verified", "dataset": "zscaler_zia.alerts", - "ingested": "2022-02-04T06:31:25Z" + "ingested": "2022-04-13T17:21:34Z" }, "input": { "type": "tcp" }, "log": { "source": { - "address": "172.21.0.7:32902" + "address": "1.128.3.4:32902" }, "syslog": { "priority": 114 @@ -265,14 +287,6 @@ An example event for `alerts` looks as following: | Field | Description | Type | |---|---|---| | @timestamp | Event timestamp. | date | -| client.geo.city_name | City name. | keyword | -| client.geo.continent_name | Name of the continent. | keyword | -| client.geo.country_iso_code | Country ISO code. | keyword | -| client.geo.country_name | Country name. | keyword | -| client.geo.location | Longitude and latitude. | geo_point | -| client.geo.region_iso_code | Region ISO code. | keyword | -| client.geo.region_name | Region name. | keyword | -| client.ip | IP address of the client (IPv4 or IPv6). | ip | | cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | cloud.availability_zone | Availability zone in which this host is running. | keyword | | cloud.image.id | Image ID for the cloud instance. | keyword | @@ -289,11 +303,24 @@ An example event for `alerts` looks as following: | data_stream.dataset | Data stream dataset. | constant_keyword | | data_stream.namespace | Data stream namespace. | constant_keyword | | data_stream.type | Data stream type. | constant_keyword | +| destination.as.number | Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet. | long | +| destination.as.organization.name | Organization name. | keyword | +| destination.as.organization.name.text | Multi-field of `destination.as.organization.name`. | match_only_text | +| destination.geo.city_name | City name. | keyword | +| destination.geo.continent_name | Name of the continent. | keyword | +| destination.geo.country_iso_code | Country ISO code. | keyword | +| destination.geo.country_name | Country name. | keyword | +| destination.geo.location | Longitude and latitude. | geo_point | +| destination.geo.region_iso_code | Region ISO code. | keyword | +| destination.geo.region_name | Region name. | keyword | +| destination.ip | IP address of the destination (IPv4 or IPv6). | ip | +| destination.port | Port of the destination. | long | | dns.answers.name | The domain name to which this resource record pertains. If a chain of CNAME is being resolved, each answer's `name` should be the one that corresponds with the answer's `data`. It should not simply be the original `question.name` repeated. | keyword | | dns.question.name | The name being queried. If the name field contains non-printable characters (below 32 or above 126), those characters should be represented as escaped base 10 integers (\DDD). Back slashes and quotes should be escaped. Tabs, carriage returns, and line feeds should be converted to \t, \r, and \n respectively. | keyword | | dns.question.type | The type of record being queried. | keyword | | ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword | | event.dataset | Event dataset | constant_keyword | +| event.duration | Duration of the event in nanoseconds. If event.start and event.end are known this value should be the difference between the end and start time. | long | | event.module | Event module | constant_keyword | | host.architecture | Operating system architecture. | keyword | | host.containerized | If the host is a container. | boolean | @@ -315,18 +342,20 @@ An example event for `alerts` looks as following: | input.type | Input type | keyword | | log.offset | Log offset | long | | log.source.address | Source address from which the log event was read / sent from. | keyword | +| network.protocol | In the OSI Model this would be the Application Layer protocol. For example, `http`, `dns`, or `ssh`. The field value must be normalized to lowercase for querying. | keyword | | related.hosts | All hostnames or other host identifiers seen on your event. Example identifiers include FQDNs, domain names, workstation names, or aliases. | keyword | | related.ip | All of the IPs seen on your event. | ip | -| server.geo.city_name | City name. | keyword | -| server.geo.continent_name | Name of the continent. | keyword | -| server.geo.country_iso_code | Country ISO code. | keyword | -| server.geo.country_name | Country name. | keyword | -| server.geo.location | Longitude and latitude. | geo_point | -| server.geo.region_iso_code | Region ISO code. | keyword | -| server.geo.region_name | Region name. | keyword | -| server.ip | IP address of the server (IPv4 or IPv6). | ip | -| server.port | Port of the server. | long | +| source.as.number | Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet. | long | +| source.as.organization.name | Organization name. | keyword | +| source.as.organization.name.text | Multi-field of `source.as.organization.name`. | match_only_text | | source.bytes | Bytes sent from the source to the destination. | long | +| source.geo.city_name | City name. | keyword | +| source.geo.continent_name | Name of the continent. | keyword | +| source.geo.country_iso_code | Country ISO code. | keyword | +| source.geo.country_name | Country name. | keyword | +| source.geo.location | Longitude and latitude. | geo_point | +| source.geo.region_iso_code | Region ISO code. | keyword | +| source.geo.region_name | Region name. | keyword | | source.ip | IP address of the source (IPv4 or IPv6). | ip | | source.packets | Packets sent from the source to the destination. | long | | source.port | Port of the source. | long | @@ -351,32 +380,38 @@ An example event for `dns` looks as following: { "@timestamp": "2021-12-17T07:27:54.000Z", "agent": { - "ephemeral_id": "d288c261-b8db-45af-99c0-a673c3c6d8e1", - "hostname": "docker-fleet-agent", - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "ephemeral_id": "88d27df6-beee-4299-bf35-56742db35e98", + "id": "f6f3ddbc-7ab7-4a74-aeeb-152405dea56f", "name": "docker-fleet-agent", "type": "filebeat", - "version": "7.16.2" + "version": "8.1.2" + }, + "data_stream": { + "dataset": "zscaler_zia.dns", + "namespace": "ep", + "type": "logs" }, - "client": { + "destination": { + "as": { + "number": 29518, + "organization": { + "name": "Bredband2 AB" + } + }, "geo": { - "city_name": "London", + "city_name": "Linköping", "continent_name": "Europe", - "country_iso_code": "GB", - "country_name": "United Kingdom", + "country_iso_code": "SE", + "country_name": "Sweden", "location": { - "lat": 51.5142, - "lon": -0.0931 + "lat": 58.4167, + "lon": 15.6167 }, - "region_iso_code": "GB-ENG", - "region_name": "England" + "region_iso_code": "SE-E", + "region_name": "Östergötland County" }, - "ip": "81.2.69.193" - }, - "data_stream": { - "dataset": "zscaler_zia.dns", - "namespace": "ep", - "type": "logs" + "ip": "89.160.20.156", + "port": 8080 }, "dns": { "answers": { @@ -391,15 +426,18 @@ An example event for `dns` looks as following: "version": "8.2.0" }, "elastic_agent": { - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "id": "f6f3ddbc-7ab7-4a74-aeeb-152405dea56f", "snapshot": false, - "version": "7.16.2" + "version": "8.1.2" }, "event": { "agent_id_status": "verified", - "category": "network", + "category": [ + "network" + ], "dataset": "zscaler_zia.dns", - "ingested": "2022-02-04T06:32:56Z", + "duration": 123456000000, + "ingested": "2022-04-20T06:45:24Z", "kind": "event", "type": [ "info" @@ -410,33 +448,41 @@ An example event for `dns` looks as following: }, "log": { "source": { - "address": "172.21.0.7:54202" + "address": "1.128.3.4:32902" } }, + "network": { + "protocol": "dns" + }, "related": { "hosts": [ "Machine9000" ], "ip": [ - "81.2.69.193", - "81.2.69.144" + "89.160.20.112", + "89.160.20.156" ] }, - "server": { + "source": { + "as": { + "number": 29518, + "organization": { + "name": "Bredband2 AB" + } + }, "geo": { - "city_name": "London", + "city_name": "Linköping", "continent_name": "Europe", - "country_iso_code": "GB", - "country_name": "United Kingdom", + "country_iso_code": "SE", + "country_name": "Sweden", "location": { - "lat": 51.5142, - "lon": -0.0931 + "lat": 58.4167, + "lon": 15.6167 }, - "region_iso_code": "GB-ENG", - "region_name": "England" + "region_iso_code": "SE-E", + "region_name": "Östergötland County" }, - "ip": "81.2.69.144", - "port": 8080 + "ip": "89.160.20.112" }, "tags": [ "forwarded", @@ -481,7 +527,6 @@ An example event for `dns` looks as following: | Field | Description | Type | |---|---|---| | @timestamp | Event timestamp. | date | -| client.bytes | Bytes sent from the client to the server. | long | | cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | cloud.availability_zone | Availability zone in which this host is running. | keyword | | cloud.image.id | Image ID for the cloud instance. | keyword | @@ -498,9 +543,14 @@ An example event for `dns` looks as following: | data_stream.dataset | Data stream dataset. | constant_keyword | | data_stream.namespace | Data stream namespace. | constant_keyword | | data_stream.type | Data stream type. | constant_keyword | +| destination.bytes | Bytes sent from the destination to the source. | long | +| destination.geo.country_name | Country name. | keyword | +| destination.ip | IP address of the destination (IPv4 or IPv6). | ip | +| destination.port | Port of the destination. | long | | ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword | | event.action | The action captured by the event. This describes the information in the event. It is more specific than `event.category`. Examples are `group-add`, `process-started`, `file-created`. The value is normally defined by the implementer. | keyword | | event.dataset | Event dataset | constant_keyword | +| event.duration | Duration of the event in nanoseconds. If event.start and event.end are known this value should be the difference between the end and start time. | long | | event.module | Event module | constant_keyword | | host.architecture | Operating system architecture. | keyword | | host.containerized | If the host is a container. | boolean | @@ -523,38 +573,36 @@ An example event for `dns` looks as following: | log.offset | Log offset | long | | log.source.address | Source address from which the log event was read / sent from. | keyword | | network.application | When a specific application or service is identified from network connection details (source/dest IPs, ports, certificates, or wire format), this field captures the application's or service's name. For example, the original event identifies the network connection being from a specific web service in a `https` network connection, like `facebook` or `twitter`. The field value must be normalized to lowercase for querying. | keyword | +| network.community_id | A hash of source and destination IPs and ports, as well as the protocol used in a communication. This is a tool-agnostic standard to identify flows. Learn more at https://github.com/corelight/community-id-spec. | keyword | | network.protocol | In the OSI Model this would be the Application Layer protocol. For example, `http`, `dns`, or `ssh`. The field value must be normalized to lowercase for querying. | keyword | | network.transport | Same as network.iana_number, but instead using the Keyword name of the transport layer (udp, tcp, ipv6-icmp, etc.) The field value must be normalized to lowercase for querying. | keyword | | related.ip | All of the IPs seen on your event. | ip | | rule.name | The name of the rule or signature generating the event. | keyword | -| server.bytes | Bytes sent from the server to the client. | long | -| server.geo.country_name | Country name. | keyword | +| source.bytes | Bytes sent from the source to the destination. | long | +| source.ip | IP address of the source (IPv4 or IPv6). | ip | +| source.port | Port of the source. | long | | tags | List of keywords used to tag each event. | keyword | | user.email | User email address. | keyword | | user.name | Short name or login of the user. | keyword | | user.name.text | Multi-field of `user.name`. | match_only_text | | zscaler_zia.firewall.aggregate | | keyword | | zscaler_zia.firewall.client.destination.ip | Client destination IP address. For aggregated sessions, this is the client destination IP address of the last session in the aggregate. | keyword | -| zscaler_zia.firewall.client.destination.port | Client destination port. For aggregated sessions, this is the client destination port of the last session in the aggregate. | double | -| zscaler_zia.firewall.client.source.ip | Client source IP address. For aggregated sessions, this is the client source IP address of the last session in the aggregate. | keyword | -| zscaler_zia.firewall.client.source.port | Client source port. For aggregated sessions, this is the client source port of the last session in the aggregate. | double | +| zscaler_zia.firewall.client.destination.port | Client destination port. For aggregated sessions, this is the client destination port of the last session in the aggregate. | long | | zscaler_zia.firewall.department | Department of the user. | keyword | -| zscaler_zia.firewall.duration.avg | Average session duration, in milliseconds, if the sessions were aggregated. | double | -| zscaler_zia.firewall.duration.milliseconds | Session or request duration in milliseconds. | double | -| zscaler_zia.firewall.duration.seconds | Session or request duration in seconds. | double | +| zscaler_zia.firewall.duration.avg | Average session duration, in milliseconds, if the sessions were aggregated. | long | +| zscaler_zia.firewall.duration.milliseconds | Session or request duration in milliseconds. | long | +| zscaler_zia.firewall.duration.seconds | Average session duration, in milliseconds, if the sessions were aggregated. | long | | zscaler_zia.firewall.ip_category | URL category that corresponds to the server IP address. | keyword | | zscaler_zia.firewall.location.name | Name of the location from which the session was initiated. | keyword | | zscaler_zia.firewall.nat | Indicates if the destination NAT policy was applied. | keyword | -| zscaler_zia.firewall.server.destination.ip | Server Destination IP address. For aggregated sessions, this is the server destination IP address of the last session in the aggregate. | keyword | -| zscaler_zia.firewall.server.destination.port | Server destination port. For aggregated sessions, this is the server destination port of the last session in the aggregate. | double | | zscaler_zia.firewall.server.source.ip | Server source IP address. For aggregated sessions, this is the server source IP address of the last session in the aggregate. | keyword | -| zscaler_zia.firewall.server.source.port | Server source port. For aggregated sessions, this is the server source port of the last session in the aggregate. | double | +| zscaler_zia.firewall.server.source.port | Server source port. For aggregated sessions, this is the server source port of the last session in the aggregate. | long | | zscaler_zia.firewall.session.count | Number of sessions that were aggregated. | double | | zscaler_zia.firewall.stateful | | keyword | | zscaler_zia.firewall.threat.category | Category of the threat in the Firewall session by the IPS engine. | keyword | | zscaler_zia.firewall.threat.name | Name of the threat detected in the Firewall session by the IPS engine. | keyword | | zscaler_zia.firewall.tunnel.ip | Tunnel IP address of the client (source). For aggregated sessions, this is the client's tunnel IP address corresponding to the last session in the aggregate. | keyword | -| zscaler_zia.firewall.tunnel.port | Tunnel port on the client side. For aggregated sessions, this is the client's tunnel port corresponding to the last session in the aggregate. | double | +| zscaler_zia.firewall.tunnel.port | Tunnel port on the client side. For aggregated sessions, this is the client's tunnel port corresponding to the last session in the aggregate. | long | | zscaler_zia.firewall.tunnel.type | Traffic forwarding method used to send the traffic to the firewall. | keyword | @@ -562,40 +610,48 @@ An example event for `firewall` looks as following: ```json { - "@timestamp": "2021-12-17T07:27:54.000Z", + "@timestamp": "2021-12-31T07:08:09.000Z", "agent": { - "ephemeral_id": "41987f90-74dc-4b4b-9936-4347028cf558", - "hostname": "docker-fleet-agent", - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "ephemeral_id": "2c292e52-b6ea-4ca0-bfc7-692dadde1a7d", + "id": "f6f3ddbc-7ab7-4a74-aeeb-152405dea56f", "name": "docker-fleet-agent", "type": "filebeat", - "version": "7.16.2" - }, - "client": { - "bytes": 1734 + "version": "8.1.2" }, "data_stream": { "dataset": "zscaler_zia.firewall", "namespace": "ep", "type": "logs" }, + "destination": { + "bytes": 19052, + "geo": { + "country_name": "Ireland" + }, + "ip": "0.0.0.0", + "port": 443 + }, "ecs": { "version": "8.2.0" }, "elastic_agent": { - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "id": "f6f3ddbc-7ab7-4a74-aeeb-152405dea56f", "snapshot": false, - "version": "7.16.2" + "version": "8.1.2" }, "event": { "action": "drop", "agent_id_status": "verified", - "category": "network", + "category": [ + "network" + ], "dataset": "zscaler_zia.firewall", - "ingested": "2022-02-04T06:34:17Z", + "duration": 486000000, + "ingested": "2021-12-31T05:06:07Z", "kind": "event", - "original": "{ \"sourcetype\" : \"zscalernss-fw\", \"event\" :{\"datetime\":\"Fri Dec 17 07:27:54 2021\",\"user\":\"some_user@example.com\",\"department\":\"Unknown\",\"locationname\":\"TestLoc%20DB\",\"cdport\":443,\"csport\":55018,\"sdport\":443,\"ssport\":0,\"csip\":\"0.0.0.0\",\"cdip\":\"0.0.0.0\",\"ssip\":\"0.0.0.0\",\"sdip\":\"0.0.0.0\",\"tsip\":\"0.0.0.0\",\"tunsport\":0,\"tuntype\":\"ZscalerClientConnector\",\"action\":\"Drop\",\"dnat\":\"No\",\"stateful\":\"Yes\",\"aggregate\":\"No\",\"nwsvc\":\"HTTPS\",\"nwapp\":\"http\",\"proto\":\"TCP\",\"ipcat\":\"Test Name\",\"destcountry\":\"Ireland\",\"avgduration\":486,\"rulelabel\":\"Access%20Blocked\",\"inbytes\":19052,\"outbytes\":1734,\"duration\":0,\"durationms\":486,\"numsessions\":1,\"ipsrulelabel\":\"None\",\"threatcat\":\"None\",\"threatname\":\"None\",\"deviceowner\":\"admin77\",\"devicehostname\":\"Machine9000\"}}", - "type": "info" + "type": [ + "info" + ] }, "host": { "hostname": "Machine9000" @@ -605,11 +661,12 @@ An example event for `firewall` looks as following: }, "log": { "source": { - "address": "172.21.0.7:58194" + "address": "1.128.3.4:43634" } }, "network": { "application": "http", + "community_id": "1:hQwW1HWTOUYlk7y4+T2D+UPDU1c=", "protocol": "https", "transport": "tcp" }, @@ -624,11 +681,10 @@ An example event for `firewall` looks as following: "None" ] }, - "server": { - "bytes": 19052, - "geo": { - "country_name": "Ireland" - } + "source": { + "bytes": 1734, + "ip": "0.0.0.0", + "port": 55018 }, "tags": [ "forwarded", @@ -645,17 +701,12 @@ An example event for `firewall` looks as following: "destination": { "ip": "0.0.0.0", "port": 443 - }, - "source": { - "ip": "0.0.0.0", - "port": 55018 } }, "department": "Unknown", "duration": { "avg": 486, - "milliseconds": 486, - "seconds": 0 + "milliseconds": 486 }, "ip_category": "Test Name", "location": { @@ -663,10 +714,6 @@ An example event for `firewall` looks as following: }, "nat": "No", "server": { - "destination": { - "ip": "0.0.0.0", - "port": 443 - }, "source": { "ip": "0.0.0.0", "port": 0 @@ -743,6 +790,9 @@ An example event for `firewall` looks as following: | input.type | Input type | keyword | | log.offset | Log offset | long | | log.source.address | Source address from which the log event was read / sent from. | keyword | +| network.community_id | A hash of source and destination IPs and ports, as well as the protocol used in a communication. This is a tool-agnostic standard to identify flows. Learn more at https://github.com/corelight/community-id-spec. | keyword | +| network.iana_number | IANA Protocol Number (https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml). Standardized list of protocols. This aligns well with NetFlow and sFlow related logs which use the IANA Protocol Number. | keyword | +| network.transport | Same as network.iana_number, but instead using the Keyword name of the transport layer (udp, tcp, ipv6-icmp, etc.) The field value must be normalized to lowercase for querying. | keyword | | related.ip | All of the IPs seen on your event. | ip | | related.user | All the user names or other user identifiers seen on the event. | keyword | | source.bytes | Bytes sent from the source to the destination. | long | @@ -781,14 +831,13 @@ An example event for `tunnel` looks as following: ```json { - "@timestamp": "2021-12-30T11:20:12.000Z", + "@timestamp": "2021-12-31T11:12:13.000Z", "agent": { - "ephemeral_id": "63ac98b6-0ff6-4943-820e-8505eff15937", - "hostname": "docker-fleet-agent", - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "ephemeral_id": "b187ac54-dab8-4e34-b72d-36772d818767", + "id": "f6f3ddbc-7ab7-4a74-aeeb-152405dea56f", "name": "docker-fleet-agent", "type": "filebeat", - "version": "7.16.2" + "version": "8.1.2" }, "data_stream": { "dataset": "zscaler_zia.tunnel", @@ -802,16 +851,18 @@ An example event for `tunnel` looks as following: "version": "8.2.0" }, "elastic_agent": { - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "id": "f6f3ddbc-7ab7-4a74-aeeb-152405dea56f", "snapshot": false, - "version": "7.16.2" + "version": "8.1.2" }, "event": { "agent_id_status": "verified", - "category": "network", + "category": [ + "network" + ], "dataset": "zscaler_zia.tunnel", "id": "1111111111111111111", - "ingested": "2022-02-04T06:36:16Z", + "ingested": "2021-12-31T05:06:07Z", "kind": "event", "type": [ "info" @@ -822,9 +873,12 @@ An example event for `tunnel` looks as following: }, "log": { "source": { - "address": "172.21.0.7:44374" + "address": "1.128.3.4:58370" } }, + "network": { + "transport": "ipsec ikev 1" + }, "related": { "ip": [ "81.2.69.143", @@ -904,10 +958,6 @@ An example event for `tunnel` looks as following: | Field | Description | Type | |---|---|---| | @timestamp | Event timestamp. | date | -| client.ip | IP address of the client (IPv4 or IPv6). | ip | -| client.nat.ip | Translated IP of source based NAT sessions (e.g. internal client to internet). Typically connections traversing load balancers, firewalls, or routers. | ip | -| client.user.name | Short name or login of the user. | keyword | -| client.user.name.text | Multi-field of `client.user.name`. | match_only_text | | cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | cloud.availability_zone | Availability zone in which this host is running. | keyword | | cloud.image.id | Image ID for the cloud instance. | keyword | @@ -924,6 +974,7 @@ An example event for `tunnel` looks as following: | data_stream.dataset | Data stream dataset. | constant_keyword | | data_stream.namespace | Data stream namespace. | constant_keyword | | data_stream.type | Data stream type. | constant_keyword | +| destination.ip | IP address of the destination (IPv4 or IPv6). | ip | | ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword | | event.action | The action captured by the event. This describes the information in the event. It is more specific than `event.category`. Examples are `group-add`, `process-started`, `file-created`. The value is normally defined by the implementer. | keyword | | event.dataset | Event dataset | constant_keyword | @@ -960,6 +1011,10 @@ An example event for `tunnel` looks as following: | related.ip | All of the IPs seen on your event. | ip | | rule.name | The name of the rule or signature generating the event. | keyword | | rule.ruleset | Name of the ruleset, policy, group, or parent category in which the rule used to generate this event is a member. | keyword | +| source.ip | IP address of the source (IPv4 or IPv6). | ip | +| source.nat.ip | Translated ip of source based NAT sessions (e.g. internal client to internet) Typically connections traversing load balancers, firewalls, or routers. | ip | +| source.user.name | Short name or login of the user. | keyword | +| source.user.name.text | Multi-field of `source.user.name`. | match_only_text | | tags | List of keywords used to tag each event. | keyword | | url.domain | Domain of the url, such as "www.elastic.co". In some cases a URL may refer to an IP and/or port directly, without a domain name. In this case, the IP address would go to the `domain` field. If the URL contains a literal IPv6 address enclosed by `[` and `]` (IETF RFC 2732), the `[` and `]` characters should also be captured in the `domain` field. | keyword | | url.extension | The field contains the file extension from the original request url, excluding the leading dot. The file extension is only set if it exists, as not every url has a file extension. The leading period must not be included. For example, the value must be "png", not ".png". Note that when the file name has multiple extensions (example.tar.gz), only the last one should be captured ("gz", not "tar.gz"). | keyword | @@ -986,7 +1041,6 @@ An example event for `tunnel` looks as following: | zscaler_zia.web.app.class | The web application class of the application that was accessed. Equivalent to module. | keyword | | zscaler_zia.web.app.name | Cloud application name. | keyword | | zscaler_zia.web.bandwidth_throttle | Indicates whether the transaction was throttled due to a configured bandwidth policy. | keyword | -| zscaler_zia.web.client.internet.ip | The client Internet (NATted Public) IP address. This is different from the cip value if the internal IP address is visible. Otherwise, same as cip. | keyword | | zscaler_zia.web.ctime | The time from when the first byte of the request hits the ZEN to the time in which the last byte of the response is sent from the ZEN back to the browser. | long | | zscaler_zia.web.department | Department of the user. | keyword | | zscaler_zia.web.device.hostname | The obfuscated version of the device owner. This field must be changed manually. | keyword | @@ -1014,41 +1068,36 @@ An example event for `web` looks as following: { "@timestamp": "2021-12-17T07:04:57.000Z", "agent": { - "ephemeral_id": "ced1fd2e-2f17-4f67-b8b1-d38a1920abbb", - "hostname": "docker-fleet-agent", - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "ephemeral_id": "6f164483-9eb8-4219-bb09-cd2ff3532390", + "id": "f6f3ddbc-7ab7-4a74-aeeb-152405dea56f", "name": "docker-fleet-agent", "type": "filebeat", - "version": "7.16.2" - }, - "client": { - "ip": "81.2.69.193", - "nat": { - "ip": "81.2.69.145" - }, - "user": { - "name": "administrator1" - } + "version": "8.1.2" }, "data_stream": { "dataset": "zscaler_zia.web", "namespace": "ep", "type": "logs" }, + "destination": { + "ip": "81.2.69.145" + }, "ecs": { "version": "8.2.0" }, "elastic_agent": { - "id": "d03794ae-c5b7-46b2-8a63-42f00010ac23", + "id": "f6f3ddbc-7ab7-4a74-aeeb-152405dea56f", "snapshot": false, - "version": "7.16.2" + "version": "8.1.2" }, "event": { "action": "blocked", "agent_id_status": "verified", - "category": "web", + "category": [ + "web" + ], "dataset": "zscaler_zia.web", - "ingested": "2022-02-04T08:48:57Z", + "ingested": "2021-12-31T05:06:07Z", "kind": "event", "risk_score": 0, "type": [ @@ -1072,7 +1121,7 @@ An example event for `web` looks as following: }, "log": { "source": { - "address": "172.21.0.7:48722" + "address": "1.128.3.4:37608" } }, "network": { @@ -1091,6 +1140,14 @@ An example event for `web` looks as following: "name": "Zscaler Proxy Traffic", "ruleset": "FwFilter" }, + "source": { + "nat": { + "ip": "81.2.69.193" + }, + "user": { + "name": "administrator1" + } + }, "tags": [ "forwarded", "zscaler_zia-web" diff --git a/packages/zscaler_zia/img/cloud_nss_feeds.png b/packages/zscaler_zia/img/cloud_nss_feeds.png new file mode 100644 index 00000000000..14a098f2c92 Binary files /dev/null and b/packages/zscaler_zia/img/cloud_nss_feeds.png differ diff --git a/packages/zscaler_zia/kibana/dashboard/zscaler_zia-66597790-4ded-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/dashboard/zscaler_zia-66597790-4ded-11ec-ad09-d9f49962d407.json index 7148c61b14a..6bec1ec3877 100644 --- a/packages/zscaler_zia/kibana/dashboard/zscaler_zia-66597790-4ded-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/dashboard/zscaler_zia-66597790-4ded-11ec-ad09-d9f49962d407.json @@ -227,10 +227,10 @@ "title": "[Zscaler] [ZIA] Firewall Logs", "version": 1 }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-66597790-4ded-11ec-ad09-d9f49962d407", "migrationVersion": { - "dashboard": "7.16.0" + "dashboard": "8.1.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/dashboard/zscaler_zia-85380a00-4de3-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/dashboard/zscaler_zia-85380a00-4de3-11ec-ad09-d9f49962d407.json index 60e83c67697..081b00ab99f 100644 --- a/packages/zscaler_zia/kibana/dashboard/zscaler_zia-85380a00-4de3-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/dashboard/zscaler_zia-85380a00-4de3-11ec-ad09-d9f49962d407.json @@ -182,10 +182,10 @@ "title": "[Zscaler] [ZIA] Web Logs", "version": 1 }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-85380a00-4de3-11ec-ad09-d9f49962d407", "migrationVersion": { - "dashboard": "7.16.0" + "dashboard": "8.1.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/dashboard/zscaler_zia-9447f5b0-4eaf-11ec-9527-b704eaaa5c53.json b/packages/zscaler_zia/kibana/dashboard/zscaler_zia-9447f5b0-4eaf-11ec-9527-b704eaaa5c53.json index 0210e194eee..35b0cd5de83 100644 --- a/packages/zscaler_zia/kibana/dashboard/zscaler_zia-9447f5b0-4eaf-11ec-9527-b704eaaa5c53.json +++ b/packages/zscaler_zia/kibana/dashboard/zscaler_zia-9447f5b0-4eaf-11ec-9527-b704eaaa5c53.json @@ -134,10 +134,10 @@ "title": "[Zscaler] [ZIA] Tunnel Logs", "version": 1 }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-9447f5b0-4eaf-11ec-9527-b704eaaa5c53", "migrationVersion": { - "dashboard": "7.16.0" + "dashboard": "8.1.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/dashboard/zscaler_zia-d4977590-4de8-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/dashboard/zscaler_zia-d4977590-4de8-11ec-ad09-d9f49962d407.json index 333284bf917..cc87d6b271c 100644 --- a/packages/zscaler_zia/kibana/dashboard/zscaler_zia-d4977590-4de8-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/dashboard/zscaler_zia-d4977590-4de8-11ec-ad09-d9f49962d407.json @@ -100,10 +100,10 @@ "title": "[Zscaler] [ZIA] DNS Logs", "version": 1 }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-d4977590-4de8-11ec-ad09-d9f49962d407", "migrationVersion": { - "dashboard": "7.16.0" + "dashboard": "8.1.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/map/zscaler_zia-48a188a0-4de8-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/map/zscaler_zia-48a188a0-4de8-11ec-ad09-d9f49962d407.json index f40e79e9ebf..3a0528acbbe 100644 --- a/packages/zscaler_zia/kibana/map/zscaler_zia-48a188a0-4de8-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/map/zscaler_zia-48a188a0-4de8-11ec-ad09-d9f49962d407.json @@ -11,12 +11,13 @@ "minZoom": 0, "sourceDescriptor": { "isAutoSelect": true, + "lightModeDefault": "road_map", "type": "EMS_TMS" }, "style": { "type": "TILE" }, - "type": "VECTOR_TILE", + "type": "EMS_VECTOR_TILE", "visible": true }, { @@ -34,7 +35,7 @@ "applyForceRefresh": true, "applyGlobalQuery": true, "applyGlobalTime": true, - "geoField": "client.geo.location", + "geoField": "source.geo.location", "id": "636e2366-af59-41da-a0af-83b10b7a1b47", "indexPatternRefName": "layer_1_source_index_pattern", "metrics": [ @@ -106,10 +107,10 @@ "openTOCDetails": [] } }, - "coreMigrationVersion": "7.16.2", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-48a188a0-4de8-11ec-ad09-d9f49962d407", "migrationVersion": { - "map": "7.14.0" + "map": "8.1.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-0334d8c0-4de4-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-0334d8c0-4de4-11ec-ad09-d9f49962d407.json index e5b35d65cc1..5ad790e4f13 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-0334d8c0-4de4-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-0334d8c0-4de4-11ec-ad09-d9f49962d407.json @@ -153,10 +153,10 @@ "type": "horizontal_bar" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-0334d8c0-4de4-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-05cc16a0-4eae-11ec-9527-b704eaaa5c53.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-05cc16a0-4eae-11ec-9527-b704eaaa5c53.json index 906f9839d74..e863df872a5 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-05cc16a0-4eae-11ec-9527-b704eaaa5c53.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-05cc16a0-4eae-11ec-9527-b704eaaa5c53.json @@ -55,10 +55,10 @@ "type": "table" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-05cc16a0-4eae-11ec-9527-b704eaaa5c53", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-2958ae90-4de5-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-2958ae90-4de5-11ec-ad09-d9f49962d407.json index 6d78fd2c782..7ff761b44fc 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-2958ae90-4de5-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-2958ae90-4de5-11ec-ad09-d9f49962d407.json @@ -130,10 +130,10 @@ "type": "horizontal_bar" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-2958ae90-4de5-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-2c8eb9f0-4eae-11ec-9527-b704eaaa5c53.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-2c8eb9f0-4eae-11ec-9527-b704eaaa5c53.json index a66f7bb47b5..26a843495a8 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-2c8eb9f0-4eae-11ec-9527-b704eaaa5c53.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-2c8eb9f0-4eae-11ec-9527-b704eaaa5c53.json @@ -73,10 +73,10 @@ "type": "pie" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-2c8eb9f0-4eae-11ec-9527-b704eaaa5c53", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-35612ae0-4de6-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-35612ae0-4de6-11ec-ad09-d9f49962d407.json index 3a0c98abeed..3f69687d7e6 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-35612ae0-4de6-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-35612ae0-4de6-11ec-ad09-d9f49962d407.json @@ -12,22 +12,7 @@ } }, "title": "[Zscaler] [ZIA] Distribution of Web Events by Action, Malware Category, App Class, Response Code, Department, Username, URL", - "uiStateJSON": { - "vis": { - "params": { - "colWidth": [ - { - "colIndex": 5, - "width": 137.71428571428572 - }, - { - "colIndex": 6, - "width": 194.0408163265306 - } - ] - } - } - }, + "uiStateJSON": {}, "version": 1, "visState": { "aggs": [ @@ -43,7 +28,7 @@ "id": "7", "params": { "customLabel": "Username", - "field": "client.user.name", + "field": "source.user.name", "missingBucket": false, "missingBucketLabel": "Missing", "order": "desc", @@ -172,10 +157,10 @@ "type": "table" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-35612ae0-4de6-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-3faec910-4ded-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-3faec910-4ded-11ec-ad09-d9f49962d407.json index f179b75d26b..b48ec098080 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-3faec910-4ded-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-3faec910-4ded-11ec-ad09-d9f49962d407.json @@ -191,10 +191,10 @@ "type": "table" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-3faec910-4ded-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-4d4b4fa0-4eae-11ec-9527-b704eaaa5c53.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-4d4b4fa0-4eae-11ec-9527-b704eaaa5c53.json index ff3e5c593e6..7f5b0ebfc06 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-4d4b4fa0-4eae-11ec-9527-b704eaaa5c53.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-4d4b4fa0-4eae-11ec-9527-b704eaaa5c53.json @@ -73,10 +73,10 @@ "type": "pie" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-4d4b4fa0-4eae-11ec-9527-b704eaaa5c53", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-4e583660-4deb-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-4e583660-4deb-11ec-ad09-d9f49962d407.json index 007cd4468ff..7049307a2c9 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-4e583660-4deb-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-4e583660-4deb-11ec-ad09-d9f49962d407.json @@ -55,10 +55,10 @@ "type": "table" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-4e583660-4deb-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-5b68c940-4eaf-11ec-9527-b704eaaa5c53.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-5b68c940-4eaf-11ec-9527-b704eaaa5c53.json index 5a2d639ce6f..16121ffdc94 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-5b68c940-4eaf-11ec-9527-b704eaaa5c53.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-5b68c940-4eaf-11ec-9527-b704eaaa5c53.json @@ -157,10 +157,10 @@ "type": "table" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-5b68c940-4eaf-11ec-9527-b704eaaa5c53", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-5ebff250-4de5-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-5ebff250-4de5-11ec-ad09-d9f49962d407.json index 08b89ee2974..892d02814ae 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-5ebff250-4de5-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-5ebff250-4de5-11ec-ad09-d9f49962d407.json @@ -151,10 +151,10 @@ "type": "line" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-5ebff250-4de5-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-63155460-4e82-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-63155460-4e82-11ec-ad09-d9f49962d407.json index cb1618b5e46..5433d10be85 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-63155460-4e82-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-63155460-4e82-11ec-ad09-d9f49962d407.json @@ -134,10 +134,10 @@ "type": "histogram" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-63155460-4e82-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-652829d0-4eb9-11ec-9527-b704eaaa5c53.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-652829d0-4eb9-11ec-9527-b704eaaa5c53.json index 91f7e48e029..14200e1db93 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-652829d0-4eb9-11ec-9527-b704eaaa5c53.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-652829d0-4eb9-11ec-9527-b704eaaa5c53.json @@ -55,10 +55,10 @@ "type": "table" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-652829d0-4eb9-11ec-9527-b704eaaa5c53", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-68d16b80-4de4-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-68d16b80-4de4-11ec-ad09-d9f49962d407.json index 24060d75d86..546771149eb 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-68d16b80-4de4-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-68d16b80-4de4-11ec-ad09-d9f49962d407.json @@ -153,10 +153,10 @@ "type": "horizontal_bar" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-68d16b80-4de4-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-6d29cc50-4de8-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-6d29cc50-4de8-11ec-ad09-d9f49962d407.json index 3d02ae30426..66719ab2f17 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-6d29cc50-4de8-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-6d29cc50-4de8-11ec-ad09-d9f49962d407.json @@ -73,10 +73,10 @@ "type": "pie" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-6d29cc50-4de8-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-72169a60-4deb-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-72169a60-4deb-11ec-ad09-d9f49962d407.json index c0d4bf656b6..9bd5c1fa51f 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-72169a60-4deb-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-72169a60-4deb-11ec-ad09-d9f49962d407.json @@ -55,10 +55,10 @@ "type": "table" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-72169a60-4deb-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-7a0a40d0-4de3-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-7a0a40d0-4de3-11ec-ad09-d9f49962d407.json index bd2e6dab644..08b9cd4fba0 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-7a0a40d0-4de3-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-7a0a40d0-4de3-11ec-ad09-d9f49962d407.json @@ -81,10 +81,10 @@ "type": "metric" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-7a0a40d0-4de3-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-8058c4e0-4eae-11ec-9527-b704eaaa5c53.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-8058c4e0-4eae-11ec-9527-b704eaaa5c53.json index 313c4077ad7..b988b8b2883 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-8058c4e0-4eae-11ec-9527-b704eaaa5c53.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-8058c4e0-4eae-11ec-9527-b704eaaa5c53.json @@ -73,10 +73,10 @@ "type": "pie" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-8058c4e0-4eae-11ec-9527-b704eaaa5c53", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-91813c00-4de8-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-91813c00-4de8-11ec-ad09-d9f49962d407.json index 563accac21d..92ded91328f 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-91813c00-4de8-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-91813c00-4de8-11ec-ad09-d9f49962d407.json @@ -73,10 +73,10 @@ "type": "pie" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-91813c00-4de8-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-9e6d2890-4deb-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-9e6d2890-4deb-11ec-ad09-d9f49962d407.json index 78319bdfc54..2bde940e35e 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-9e6d2890-4deb-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-9e6d2890-4deb-11ec-ad09-d9f49962d407.json @@ -28,7 +28,7 @@ "id": "2", "params": { "customLabel": "Server Destination IP", - "field": "zscaler_zia.firewall.server.destination.ip", + "field": "destination.ip", "missingBucket": false, "missingBucketLabel": "Missing", "order": "desc", @@ -55,10 +55,10 @@ "type": "table" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-9e6d2890-4deb-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-a536b890-4e80-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-a536b890-4e80-11ec-ad09-d9f49962d407.json index e6d2153278a..d9f80f29584 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-a536b890-4e80-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-a536b890-4e80-11ec-ad09-d9f49962d407.json @@ -153,10 +153,10 @@ "type": "horizontal_bar" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-a536b890-4e80-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-a9ac0260-4de3-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-a9ac0260-4de3-11ec-ad09-d9f49962d407.json index 4ccd7d951af..d6c82373827 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-a9ac0260-4de3-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-a9ac0260-4de3-11ec-ad09-d9f49962d407.json @@ -77,10 +77,10 @@ "type": "table" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-a9ac0260-4de3-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-bcddbd40-4ead-11ec-9527-b704eaaa5c53.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-bcddbd40-4ead-11ec-9527-b704eaaa5c53.json index b226e833e29..df07c7ab593 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-bcddbd40-4ead-11ec-9527-b704eaaa5c53.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-bcddbd40-4ead-11ec-9527-b704eaaa5c53.json @@ -55,10 +55,10 @@ "type": "table" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-bcddbd40-4ead-11ec-9527-b704eaaa5c53", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-bd00f230-4de8-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-bd00f230-4de8-11ec-ad09-d9f49962d407.json index a8dfa30d257..77b55dc97ec 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-bd00f230-4de8-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-bd00f230-4de8-11ec-ad09-d9f49962d407.json @@ -131,10 +131,10 @@ "type": "horizontal_bar" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-bd00f230-4de8-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-c8b23580-4de3-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-c8b23580-4de3-11ec-ad09-d9f49962d407.json index 8b2e5a40ae1..6119cb80edf 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-c8b23580-4de3-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-c8b23580-4de3-11ec-ad09-d9f49962d407.json @@ -55,10 +55,10 @@ "type": "table" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-c8b23580-4de3-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-da1734d0-4deb-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-da1734d0-4deb-11ec-ad09-d9f49962d407.json index 97d6454cb18..7dee1854458 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-da1734d0-4deb-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-da1734d0-4deb-11ec-ad09-d9f49962d407.json @@ -28,7 +28,7 @@ "id": "2", "params": { "customLabel": "Client Source IP", - "field": "zscaler_zia.firewall.client.source.ip", + "field": "source.ip", "missingBucket": false, "missingBucketLabel": "Missing", "order": "desc", @@ -55,10 +55,10 @@ "type": "table" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-da1734d0-4deb-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-db1241f0-4e80-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-db1241f0-4e80-11ec-ad09-d9f49962d407.json index 18ee199633f..1aadd82cd7e 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-db1241f0-4e80-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-db1241f0-4e80-11ec-ad09-d9f49962d407.json @@ -28,7 +28,7 @@ "id": "2", "params": { "customLabel": "Destination Country", - "field": "server.geo.country_name", + "field": "destination.geo.country_name", "missingBucket": false, "missingBucketLabel": "Missing", "order": "desc", @@ -102,7 +102,7 @@ }, "times": [], "truncateLegend": true, - "type": "histogram", + "type": "horizontal_bar", "valueAxes": [ { "id": "ValueAxis-1", @@ -131,10 +131,10 @@ "type": "horizontal_bar" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-db1241f0-4e80-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-dff0d0b0-4dea-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-dff0d0b0-4dea-11ec-ad09-d9f49962d407.json index d2526e4cd62..1b3b9419c4f 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-dff0d0b0-4dea-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-dff0d0b0-4dea-11ec-ad09-d9f49962d407.json @@ -81,10 +81,10 @@ "type": "metric" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-dff0d0b0-4dea-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-e4f2aa20-4ead-11ec-9527-b704eaaa5c53.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-e4f2aa20-4ead-11ec-9527-b704eaaa5c53.json index bcc6c6fb628..628743893bc 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-e4f2aa20-4ead-11ec-9527-b704eaaa5c53.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-e4f2aa20-4ead-11ec-9527-b704eaaa5c53.json @@ -55,10 +55,10 @@ "type": "table" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-e4f2aa20-4ead-11ec-9527-b704eaaa5c53", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-e54e9f20-4de4-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-e54e9f20-4de4-11ec-ad09-d9f49962d407.json index c2d451fd945..1ff2792c917 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-e54e9f20-4de4-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-e54e9f20-4de4-11ec-ad09-d9f49962d407.json @@ -130,10 +130,10 @@ "type": "horizontal_bar" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-e54e9f20-4de4-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/kibana/visualization/zscaler_zia-f5a2e730-4deb-11ec-ad09-d9f49962d407.json b/packages/zscaler_zia/kibana/visualization/zscaler_zia-f5a2e730-4deb-11ec-ad09-d9f49962d407.json index 75be53e9698..9055494a87e 100644 --- a/packages/zscaler_zia/kibana/visualization/zscaler_zia-f5a2e730-4deb-11ec-ad09-d9f49962d407.json +++ b/packages/zscaler_zia/kibana/visualization/zscaler_zia-f5a2e730-4deb-11ec-ad09-d9f49962d407.json @@ -55,10 +55,10 @@ "type": "table" } }, - "coreMigrationVersion": "7.16.0", + "coreMigrationVersion": "8.1.2", "id": "zscaler_zia-f5a2e730-4deb-11ec-ad09-d9f49962d407", "migrationVersion": { - "visualization": "7.14.0" + "visualization": "8.0.0" }, "references": [ { diff --git a/packages/zscaler_zia/manifest.yml b/packages/zscaler_zia/manifest.yml index 512bcc59559..513de6f32b1 100644 --- a/packages/zscaler_zia/manifest.yml +++ b/packages/zscaler_zia/manifest.yml @@ -1,7 +1,7 @@ format_version: 1.0.0 name: zscaler_zia -title: "Zscaler Internet Access" -version: 0.2.0 +title: Zscaler Internet Access +version: 2.0.0 license: basic description: Collect logs from Zscaler Internet Access (ZIA) with Elastic Agent. type: integration @@ -9,7 +9,7 @@ categories: - security release: beta conditions: - kibana.version: ^7.16.2 || ^8.0.0 + kibana.version: ^8.3.0 screenshots: - src: /img/zscaler-zia-screenshot.png title: Zscaler ZIA web log dashboard screenshot @@ -66,5 +66,46 @@ policy_templates: # -----END CERTIFICATE----- title: Collect Zscaler Internet Access logs via TCP input description: Collecting Zscaler Internet Access logs via TCP input + - type: http_endpoint + title: Collect Zscaler Internet Access logs via HTTP Endpoint + description: Collecting Zscaler Internet Access logs via HTTP Endpoint + vars: + - name: listen_address + type: text + title: Listen Address + description: The bind address to listen for http endpoint connections. Set to `0.0.0.0` to bind to all available interfaces. + multi: false + required: true + show_user: true + default: localhost + - name: ssl + type: yaml + title: SSL Configuration + description: i.e. certificate_authorities, supported_protocols, verification_mode etc. + multi: false + required: false + show_user: false + default: | + #certificate_authorities: + # - | + # -----BEGIN CERTIFICATE----- + # MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF + # ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2 + # MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB + # BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n + # fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl + # 94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t + # /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP + # PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41 + # CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O + # BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux + # 8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D + # 874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw + # 3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA + # H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu + # 8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0 + # yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk + # sxSmbIUfc2SGJGCJD4I= + # -----END CERTIFICATE----- owner: github: elastic/security-external-integrations