[zscaler_zia] Add ZScaler ZIA package. by vinit-chauhan · Pull Request #2459 · elastic/integrations

vinit-chauhan · 2022-01-03T10:43:35Z

What does this PR do?

Generated the skeleton of the ZScaler ZIA integration package.
Added 5 data streams (Alert, DNS, Firewall, Tunnel, and Web Logs).
Added data collection logic for all 5 data streams.
Added the ingest pipeline for all the data streams.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files
Added dashboards and visualizations.
Added test for pipeline for all the data streams.
Added system test cases for all the data streams.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.16.2).

How to test this PR locally

Clone integrations repo.
Install elastic-package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/zscaler_zia directory.
Run the following command to run tests.

elastic-package test

Screenshots

elasticmachine · 2022-01-03T10:47:36Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-02-16T17:39:44.695+0000
Duration: 25 min 46 sec

Test stats 🧪

Test	Results
Failed	0
Passed	60
Skipped	0
Total	60

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-01-06T13:58:19Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

efd6

Query: update ECS version to 8.0.0 here?

packages/zscaler_zia/_dev/build/docs/README.md

efd6 · 2022-01-09T21:26:43Z

packages/zscaler_zia/_dev/build/docs/README.md

+
+ZScaler response format:  
+```json
+\{ "sourcetype" : "zscalernss-dns", "event" :\{"datetime":"%s{time}","user":"%s{elogin}","department":"%s{edepartment}","location":"%s{elocation}","reqaction":"%s{reqaction}","resaction":"%s{resaction}","reqrulelabel":"%s{reqrulelabel}","resrulelabel":"%s{resrulelabel}","dns_reqtype":"%s{reqtype}","dns_req":"%s{req}","dns_resp":"%s{res}","srv_dport":"%d{sport}","durationms":"%d{durationms}","clt_sip":"%s{cip}","srv_dip":"%s{sip}","category":"%s{domcat}","deviceowner":"%s{deviceowner}","devicehostname":"%s{devicehostname}"\}\}


Query why the outer braces are escaped here and below.

Basically, it is the default syntax that the user has to put in the response format configuration section in the ZScaler.

OK so they are not really JSON, but rather opaque strings that just happen to resolve to JSON when fed to the configuration. I'd suggest that ~~the json syntax marker be removed and~~ some additional text be added explaining how the string should be used.

Note that these look different to the ZPA response format strings that lack the \-escaped braces (see here). Is this expected; are they actually different or is it just an accident of documentation editing?

Firstly, We have removed the JSON marker from the section as it is not a valid JSON. Additionally, The SOP for the end-user is mentioned clearly in the README.md file.

Secondly, to answer your second comment, yes, the response formats for both ZIA and ZPA are different. For both ZIA and ZPA it is required that the exact string, which we have given in the user guide is to be put in the Response Format section.

Thank you for clarifying. I think that the user instructions for use of the response format strings could be expanded a little. I doesn't need a lot, just where they should be pasted.

Sure, I will update the steps to be simpler, and I will also add a step where it describes where to put the given response format.

efd6 · 2022-01-09T21:47:45Z

/test

- Corrected the case in vendor name

vinit-chauhan · 2022-01-21T13:45:48Z

/test

marc-gr · 2022-01-25T08:46:36Z

/test

packages/zscaler_zia/_dev/build/docs/README.md

packages/zscaler_zia/data_stream/alerts/_dev/test/pipeline/test-alerts.log-expected.json

packages/zscaler_zia/data_stream/alerts/_dev/test/pipeline/test-common-config.yml

packages/zscaler_zia/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml

packages/zscaler_zia/data_stream/firewall/_dev/test/pipeline/test-firewall.log-expected.json

packages/zscaler_zia/data_stream/web/_dev/test/pipeline/test-web.log-expected.json

packages/zscaler_zia/_dev/deploy/docker/sample_logs/alerts.log

packages/zscaler_zia/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml

packages/zscaler_zia/data_stream/dns/_dev/test/pipeline/test-common-config.yml

andrewkroh · 2022-02-01T16:40:43Z

As a follow up to merging this the zscaler package needs to be marked as deprecated.

- Removed dynamic_fields section from pipeline tests - Updated ecs version to 8.0.0 - Converted values of network.protocol to lowercase. - Updated mappings - Added newline in logfiles - Re-generated pipeline test expacted files.

andrewkroh · 2022-02-09T23:43:50Z

/test

andrewkroh · 2022-02-10T06:17:57Z

/test

packages/zscaler_zia/_dev/build/docs/README.md

packages/zscaler_zia/manifest.yml

packages/zscaler_zia/data_stream/alerts/manifest.yml

andrewkroh · 2022-02-16T17:38:59Z

/test

There are now Zscaler ZIA and Zscaler ZPA integrations that are specific Zscaler products. This package is being marked as deprecated. The new integrations are not generated parsers based on rsa2elk. Relates #2459

Initial commit for zscaler zia package

4acb04e

vinit-chauhan force-pushed the package_zscaler_zia branch from d827d56 to 4acb04e Compare January 3, 2022 10:44

andrewkroh added Team:Security-External Integrations Integration:zscaler_zia Zscaler Internet Access labels Jan 6, 2022

vinit-chauhan marked this pull request as ready for review January 6, 2022 13:58

efd6 reviewed Jan 9, 2022

View reviewed changes

- Updated documentation.

9ec2a26

- Corrected the case in vendor name

andrewkroh reviewed Jan 27, 2022

View reviewed changes

efd6 reviewed Feb 1, 2022

View reviewed changes

- Removed extra newline ( README.md: line 123 )

edc02f8

- Removed dynamic_fields section from pipeline tests - Updated ecs version to 8.0.0 - Converted values of network.protocol to lowercase. - Updated mappings - Added newline in logfiles - Re-generated pipeline test expacted files.

vinit-chauhan force-pushed the package_zscaler_zia branch from cf91193 to edc02f8 Compare February 7, 2022 06:30

Added newline at the end of files.

1a4d693

vinit-chauhan requested a review from efd6 February 10, 2022 12:29

andrewkroh reviewed Feb 10, 2022

View reviewed changes

packages/zscaler_zia/_dev/build/docs/README.md Outdated Show resolved Hide resolved

packages/zscaler_zia/manifest.yml Outdated Show resolved Hide resolved

packages/zscaler_zia/data_stream/alerts/manifest.yml Outdated Show resolved Hide resolved

Updated ports and configuration parameter name.

2bcc0d1

vinit-chauhan requested a review from andrewkroh February 14, 2022 15:35

jamiehynds linked an issue Feb 16, 2022 that may be closed by this pull request

Zscaler Internet Access #1134

Closed

15 tasks

andrewkroh approved these changes Feb 16, 2022

View reviewed changes

andrewkroh merged commit cde118b into elastic:main Feb 16, 2022

This was referenced Feb 16, 2022

[zscaler] Mark package as deprecated #2708

Merged

[staging] Promote packages from snapshot (zscaler_zia-0.1.0) elastic/package-storage#3636

Merged

andrewkroh mentioned this pull request Feb 22, 2022

[production] Promote packages from staging (gcp-1.4.1, juniper-1.1.0, o365-1.4.1, zscaler-0.5.1, zscaler_zia-0.1.0, zscaler_zpa-0.1.0) elastic/package-storage#3664

Merged

andrewkroh added the New Integration Issue or pull request for creating a new integration package. label Aug 13, 2024

Conversation

vinit-chauhan commented Jan 3, 2022

What does this PR do?

Checklist

How to test this PR locally

Screenshots

Uh oh!

elasticmachine commented Jan 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

elasticmachine commented Jan 6, 2022

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

efd6 Jan 9, 2022

Choose a reason for hiding this comment

Uh oh!

vinit-chauhan Jan 10, 2022

Choose a reason for hiding this comment

Uh oh!

efd6 Jan 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vinit-chauhan Feb 7, 2022

Choose a reason for hiding this comment

Uh oh!

efd6 Feb 14, 2022

Choose a reason for hiding this comment

Uh oh!

vinit-chauhan Feb 15, 2022

Choose a reason for hiding this comment

Uh oh!

efd6 commented Jan 9, 2022

Uh oh!

vinit-chauhan commented Jan 21, 2022

Uh oh!

marc-gr commented Jan 25, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewkroh commented Feb 1, 2022

Uh oh!

andrewkroh commented Feb 9, 2022

Uh oh!

andrewkroh commented Feb 10, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewkroh commented Feb 16, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

elasticmachine commented Jan 3, 2022 •

edited

Loading

efd6 Jan 31, 2022 •

edited

Loading