Skip to content

zscaler_zpa: fix handling of multiple remote IPs, and event categorisation #13755

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
efd6 merged 1 commit into from
May 23, 2025
Merged

zscaler_zpa: fix handling of multiple remote IPs, and event categorisation #13755

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions packages/zscaler_zpa/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,12 @@
# newer versions go on top
- version: "1.22.2"
changes:
- description: Fix handling of remote IP lists in audit data stream.
type: bugfix
link: https://github.com/elastic/integrations/pull/13755
- description: Fix ECS event type, category and outcome mapping of audit events.
type: bugfix
link: https://github.com/elastic/integrations/pull/13755
- version: "1.22.1"
changes:
- description: Do not set `error.message` for expected behavior related to Zscaler `Host` field.
Expand Down
1 change: 1 addition & 0 deletions packages/zscaler_zpa/data_stream/audit/_dev/test/pipeline/test-audit.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
{"ModifiedTime":"2021-11-17T04:29:38.000Z","CreationTime":"2021-11-17T04:29:38.000Z","ModifiedBy":12345678901234567,"RequestID":"11111111-1111-1111-1111-111111111111","SessionID":"1idn23nlfm2q1txa5h3r4mep6","AuditOldValue":"","AuditNewValue":"{\"id\":\"72058340288495701\",\"name\":\"Some-Name\",\"domainOrIpAddress\":\"81.2.69.144\",\"description\":\"This is a description field\",\"enabled\":\"true\"}","AuditOperationType":"Create","ObjectType":"Server","ObjectName":"Some-Name","ObjectID":12345678901234567,"CustomerID":98765432109876543,"User":"zpaadmin@xxxxxxxxxxxxxxxxx.zpa-customer.com","ClientAuditUpdate":0}
{"ModifiedTime":"","CreationTime":"2021-11-17T04:29:38.000Z","ModifiedBy":12345678901234567,"RequestID":"11111111-1111-1111-1111-111111111111","SessionID":"1idn23nlfm2q1txa5h3r4mep6","AuditOldValue":"","AuditNewValue":"{\"id\":\"72058340288495701\",\"name\":\"Some-Name\",\"domainOrIpAddress\":\"example.com\",\"description\":\"This is a description field\",\"enabled\":\"true\"}","AuditOperationType":"Create","ObjectType":"Server","ObjectName":"Some-Name","ObjectID":12345678901234567,"CustomerID":98765432109876543,"User":"zpaadmin@xxxxxxxxxxxxxxxxx.zpa-customer.com","ClientAuditUpdate":0}
{"ModifiedTime":"","CreationTime":"2025-04-30T16:23:40.000Z","ModifiedBy":288263728720249833,"RequestID":"12d6eccc-718c-4657-b267-83cc1c3f35f6","SessionID":"1samau4fwi7xbsf3317mkd5vz","AuditOldValue":"","AuditNewValue":"{\"loginAttempt\":\"2025-04-30 16:23:40 UTC\",\"remoteIP\":\"81.2.69.142, 81.2.69.144\"}","AuditOperationType":"Sign In","ObjectType":"Authentication","ObjectName":"xxxx","ObjectID":"xxxxx","CustomerID":"xxxxx","User":"xxxx","ClientAuditUpdate":1}
68 changes: 68 additions & 0 deletions packages/zscaler_zpa/data_stream/audit/_dev/test/pipeline/test-audit.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
"id": "11111111-1111-1111-1111-111111111111",
"kind": "event",
"original": "{\"ModifiedTime\":\"2021-11-17T04:29:38.000Z\",\"CreationTime\":\"2021-11-17T04:29:38.000Z\",\"ModifiedBy\":12345678901234567,\"RequestID\":\"11111111-1111-1111-1111-111111111111\",\"SessionID\":\"1idn23nlfm2q1txa5h3r4mep6\",\"AuditOldValue\":\"\",\"AuditNewValue\":\"{\\\"id\\\":\\\"72058340288495701\\\",\\\"name\\\":\\\"Some-Name\\\",\\\"domainOrIpAddress\\\":\\\"81.2.69.144\\\",\\\"description\\\":\\\"This is a description field\\\",\\\"enabled\\\":\\\"true\\\"}\",\"AuditOperationType\":\"Create\",\"ObjectType\":\"Server\",\"ObjectName\":\"Some-Name\",\"ObjectID\":12345678901234567,\"CustomerID\":98765432109876543,\"User\":\"zpaadmin@xxxxxxxxxxxxxxxxx.zpa-customer.com\",\"ClientAuditUpdate\":0}",
"outcome": "success",
"type": [
"creation"
]
Expand Down Expand Up @@ -77,6 +78,7 @@
"id": "11111111-1111-1111-1111-111111111111",
"kind": "event",
"original": "{\"ModifiedTime\":\"\",\"CreationTime\":\"2021-11-17T04:29:38.000Z\",\"ModifiedBy\":12345678901234567,\"RequestID\":\"11111111-1111-1111-1111-111111111111\",\"SessionID\":\"1idn23nlfm2q1txa5h3r4mep6\",\"AuditOldValue\":\"\",\"AuditNewValue\":\"{\\\"id\\\":\\\"72058340288495701\\\",\\\"name\\\":\\\"Some-Name\\\",\\\"domainOrIpAddress\\\":\\\"example.com\\\",\\\"description\\\":\\\"This is a description field\\\",\\\"enabled\\\":\\\"true\\\"}\",\"AuditOperationType\":\"Create\",\"ObjectType\":\"Server\",\"ObjectName\":\"Some-Name\",\"ObjectID\":12345678901234567,\"CustomerID\":98765432109876543,\"User\":\"zpaadmin@xxxxxxxxxxxxxxxxx.zpa-customer.com\",\"ClientAuditUpdate\":0}",
"outcome": "success",
"type": [
"creation"
]
Expand Down Expand Up @@ -123,6 +125,72 @@
}
}
}
},
{
"@timestamp": "2025-04-30T16:23:40.000Z",
"client": {
"ip": [
"81.2.69.142",
"81.2.69.144"
]
},
"ecs": {
"version": "8.11.0"
},
"event": {
"category": [
"authentication",
"session"
],
"created": "2025-04-30T16:23:40.000Z",
"id": "12d6eccc-718c-4657-b267-83cc1c3f35f6",
"kind": "event",
"original": "{\"ModifiedTime\":\"\",\"CreationTime\":\"2025-04-30T16:23:40.000Z\",\"ModifiedBy\":288263728720249833,\"RequestID\":\"12d6eccc-718c-4657-b267-83cc1c3f35f6\",\"SessionID\":\"1samau4fwi7xbsf3317mkd5vz\",\"AuditOldValue\":\"\",\"AuditNewValue\":\"{\\\"loginAttempt\\\":\\\"2025-04-30 16:23:40 UTC\\\",\\\"remoteIP\\\":\\\"81.2.69.142, 81.2.69.144\\\"}\",\"AuditOperationType\":\"Sign In\",\"ObjectType\":\"Authentication\",\"ObjectName\":\"xxxx\",\"ObjectID\":\"xxxxx\",\"CustomerID\":\"xxxxx\",\"User\":\"xxxx\",\"ClientAuditUpdate\":1}",
"outcome": "success",
"type": [
"start"
]
},
"organization": {
"id": "xxxxx"
},
"related": {
"ip": [
"81.2.69.142",
"81.2.69.144"
],
"user": [
"288263728720249833",
"xxxx"
]
},
"tags": [
"preserve_original_event"
],
"user": {
"id": "288263728720249833",
"name": "xxxx"
},
"zscaler_zpa": {
"audit": {
"client_audit_update": 1,
"object": {
"id": "xxxxx",
"name": "xxxx",
"type": "Authentication"
},
"operation_type": "Sign In",
"session": {
"id": "1samau4fwi7xbsf3317mkd5vz"
},
"value": {
"new": {
"loginAttempt": "2025-04-30 16:23:40 UTC",
"remoteIP": "81.2.69.142, 81.2.69.144"
}
}
}
}
}
]
}
93 changes: 75 additions & 18 deletions packages/zscaler_zpa/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,30 +39,74 @@ processors:
field: event.created
copy_from: '@timestamp'
ignore_failure: true
- append:
field: event.category
value: iam
- set:
field: event.kind
value: event
- script:
if: ctx.json?.AuditOperationType != null && ctx.json.AuditOperationType != ''
lang: painless
params:
event_classification:
'create':
type:
- creation
category:
- iam
outcome: success
'delete':
type:
- deletion
category:
- iam
outcome: success
'update':
type:
- change
category:
- iam
outcome: success
'sign in':
type:
- start
category:
- authentication
- session
outcome: success
'sign in failure':
type:
- start
- error
category: failure
- session
outcome:
'download':
type:
- info
- access
outcome: success
category:
- file
'sign out':
type:
- end
category:
- session
outcome: success
'client session revoked':
type:
- change
- deletion
category:
- iam
outcome: success
source: |
def eventType = ctx.json.AuditOperationType?.toLowerCase();
ctx.event.type = new ArrayList();
Map referenceTable = [
'create': ['creation'],
'delete': ['deletion'],
'update': ['change'],
'sign in': ['access', 'allowed'],
'sign in failure': ['access', 'error'],
'download': ['info'],
'sign out': ['access'],
'client session revoked': ['end']
];

ctx.event.type = referenceTable[eventType];
def class = params.event_classification[ctx.json.AuditOperationType?.toLowerCase()];
if (class == null) {
return;
}
ctx.event.type = class.type;
ctx.event.category = class.category;
ctx.event.outcome = class.outcome;
- rename:
field: json.RequestID
target_field: event.id
Expand Down Expand Up @@ -216,12 +260,25 @@ processors:
ctx.server = new HashMap();
ctx.server.address = valuesMap?.domainOrIpAddress;
}
- split:
field: client.ip
separator: ', *'
if: ctx.client?.ip instanceof String && ctx.client.ip.contains(',')
Comment on lines +263 to +266

@kcreddy kcreddy May 5, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think having a list of client.ip/ source.ip will prevent us from using geoip processor on them. Maybe we should find the public IP among the 2 IPs and populate just with it?

@efd6 efd6 May 7, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm following this up to see if there is a known behaviour we can rely on to do this.

@efd6 efd6 May 15, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@StellaFreyju can you provide some insight here? I understand that the first element of the array is the public IP address, but that the second may be somewhat public. I do not think that it is a good idea to do geoip look-ups on both, but doing it on the first element seems fine; users interested in the second (and subsequent) addresses can do look-ups in an @custom pipeline. Does this sound reasonable to you?

- append:
field: related.ip
value: '{{{client.ip}}}'
if: ctx.client?.ip != null
if: ctx.client?.ip instanceof String
allow_duplicates: false
ignore_failure: true
- foreach:
field: client.ip
if: ctx.client?.ip instanceof List
processor:
append:
field: related.ip
value: '{{{_ingest._value}}}'
allow_duplicates: false
ignore_failure: true
- convert:
field: server.address
target_field: server.ip
Expand Down
2 changes: 1 addition & 1 deletion packages/zscaler_zpa/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: "3.0.3"
name: zscaler_zpa
title: Zscaler Private Access
version: "1.22.1"
version: "1.22.2"
source:
license: Elastic-2.0
description: Collect logs from Zscaler Private Access (ZPA) with Elastic Agent.
Expand Down