Skip to content

[santa] update ingest pipeline to match new log format #3347

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
efd6 merged 6 commits into from
May 18, 2022
Merged

[santa] update ingest pipeline to match new log format #3347

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
fab226c
update log samples for Santa based on version 2022.4
cl-damiapoquet May 12, 2022
dda224f
modify ingest pipeline and fields to match with Santa logs
cl-damiapoquet May 12, 2022
d7850eb
update README.md
cl-damiapoquet May 12, 2022
4ca682b
update documentation
cl-damiapoquet May 16, 2022
575dd1d
mark release as breaking-change
cl-damiapoquet May 16, 2022
f4f44c6
change pidversion field and package version description
cl-damiapoquet May 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions packages/santa/_dev/build/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@ binaries.

## Compatibility

The Google Santa integration was tested with logs from Santa 0.9.14.
The Google Santa integration was tested with logs from Santa 2022.4.

**Google Santa is available for MacOS only.**

The integration is by default configured to read logs from `/var/log/santa.log`.
The integration is by default configured to read logs from `/var/db/santa/santa.log`.

## Logs

Expand Down
21 changes: 11 additions & 10 deletions packages/santa/_dev/deploy/docker/sample_logs/santa.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
[2018-12-10T06:45:16.802Z] I santad: action=EXEC|decision=ALLOW|reason=CERT|sha256=c4bc09fd2f248534552f517acf3edb9a635aba2b02e46f49df683ea9b778e5b4|path=/usr/libexec/xpcproxy|args=/usr/sbin/newsyslog|cert_sha256=2aa4b9973b7ba07add447ee4da8b5337c3ee2c3a991911e80e7282e8a751fc32|cert_cn=Software Signing|pid=29678|ppid=1|uid=0|user=root|gid=0|group=wheel|mode=M
[2018-12-10T06:45:16.802Z] I santad: action=EXEC|decision=ALLOW|reason=CERT|sha256=c4bc09fd2f248534552f517acf3edb9a635aba2b02e46f49df683ea9b778e5b4|path=/usr/libexec/xpcproxy|args=xpcproxy com.apple.systemstats.daily|cert_sha256=2aa4b9973b7ba07add447ee4da8b5337c3ee2c3a991911e80e7282e8a751fc32|cert_cn=Software Signing|pid=29679|ppid=1|uid=0|user=root|gid=0|group=wheel|mode=M
[2018-12-10T06:45:16.851Z] I santad: action=EXEC|decision=ALLOW|reason=CERT|sha256=746f0dbafb7e675d5ce67131e5544772ee612b894e8ab51d3ce2d21f7cb7332d|path=/usr/sbin/newsyslog|args=/usr/sbin/newsyslog|cert_sha256=2aa4b9973b7ba07add447ee4da8b5337c3ee2c3a991911e80e7282e8a751fc32|cert_cn=Software Signing|pid=29678|ppid=1|uid=0|user=root|gid=0|group=wheel|mode=M
[2018-12-10T06:45:16.859Z] I santad: action=EXEC|decision=ALLOW|reason=CERT|sha256=d6be9bfbd777ac5dcd30488014acc787a2df5ce840f1fe4d5742d323ee00392f|path=/usr/sbin/systemstats|args=/usr/sbin/systemstats --daily|cert_sha256=2aa4b9973b7ba07add447ee4da8b5337c3ee2c3a991911e80e7282e8a751fc32|cert_cn=Software Signing|pid=29679|ppid=1|uid=0|user=root|gid=0|group=wheel|mode=M
[2018-12-10T08:45:27.810Z] I santad: action=EXEC|decision=ALLOW|reason=CERT|sha256=c4bc09fd2f248534552f517acf3edb9a635aba2b02e46f49df683ea9b778e5b4|path=/usr/libexec/xpcproxy|args=/usr/sbin/newsyslog|cert_sha256=2aa4b9973b7ba07add447ee4da8b5337c3ee2c3a991911e80e7282e8a751fc32|cert_cn=Software Signing|pid=29681|ppid=1|uid=0|user=root|gid=0|group=wheel|mode=M
[2018-12-10T08:45:27.810Z] I santad: action=EXEC|decision=ALLOW|reason=CERT|sha256=c4bc09fd2f248534552f517acf3edb9a635aba2b02e46f49df683ea9b778e5b4|path=/usr/libexec/xpcproxy|args=xpcproxy com.adobe.AAM.Scheduler-1.0|cert_sha256=2aa4b9973b7ba07add447ee4da8b5337c3ee2c3a991911e80e7282e8a751fc32|cert_cn=Software Signing|pid=29680|ppid=1|uid=0|user=root|gid=0|group=wheel|mode=M
[2018-12-10T21:37:27.247Z] I santad: action=EXEC|decision=ALLOW|reason=UNKNOWN|sha256=08bd61582657cd6d78c9e071d34d79a32bb59e7210077a44919d2c5477e988a1|path=/usr/local/Cellar/osquery/3.3.0_1/bin/osqueryd|args=/usr/local/bin/osqueryd --flagfile=/private/var/osquery/osquery.flags --logger_min_stderr=1|pid=45084|ppid=1|uid=0|user=root|gid=0|group=wheel|mode=M
[2018-12-10T16:24:43.992Z] I santad: action=EXEC|decision=ALLOW|reason=CERT|sha256=63b6a54848d7b4adf726d68f11409a4ac05b43926cb0f2792f7d41dc0221c106|path=/usr/bin/basename|cert_sha256=2aa4b9973b7ba07add447ee4da8b5337c3ee2c3a991911e80e7282e8a751fc32|cert_cn=Software Signing|pid=40757|ppid=40756|uid=501|user=akroh|gid=20|group=staff|mode=M
[2018-12-14T05:35:38.313Z] I santad: action=EXEC|decision=ALLOW|reason=UNKNOWN|sha256=a8defc1b24c45f6dabeb8298af5f8e1daf39e1504e16f878345f15ac94ae96d7|path=/Applications/Google Chrome.app/Contents/Versions/70.0.3538.110/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper|args=/Applications/Google Chrome.app/Contents/Versions/70.0.3538.110/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper --type=utility --field-trial-handle=120122713615061869,9401617251746517350,131072 --lang=en-US --service-sandbox-type=utility --service-request-channel-token=10458143409865682077 --seatbelt-client=262|cert_sha256=345a8e098bd04794aaeefda8c9ef56a0bf3d3706d67d35bc0e23f11bb3bffce5|cert_cn=Developer ID Application: Google, Inc. (EQHXZ8M8AV)|pid=89238|ppid=704|uid=501|user=akroh|gid=20|group=staff|mode=M
[2018-12-17T03:03:52.337Z] I santad: action=DISKAPPEAR|mount=/Volumes/Recovery|volume=Recovery|bsdname=disk1s3|fs=apfs|model=APPLE SSD SM0512L|serial=C026495006UHCHH1Q|bus=PCI-Express|dmgpath=
[2022-05-12T11:38:03.923Z] I santad: action=EXEC|decision=ALLOW|reason=BINARY|explain=critical system binary|sha256=43158bf397bf52001067319c591249307e2862daf8690828c15cdcc1ddf6166d|cert_sha256=d84db96af8c2e60ac4c851a21ec460f6f84e0235beb17d24a78712b9b021ed57|cert_cn=Software Signing|pid=71993|pidversion=1097732|ppid=1|uid=0|user=root|gid=0|group=wheel|mode=M|path=/usr/libexec/xpcproxy|args=xpcproxy com.apple.CoreAuthentication.agent
[2022-05-12T11:38:42.781Z] I santad: action=EXEC|decision=ALLOW|reason=CERT|sha256=7207307ca09d2707368ec394e67c6ccff6e48a2d1d86225a3115fe3535a8237c|cert_sha256=d84db96af8c2e60ac4c851a21ec460f6f84e0235beb17d24a78712b9b021ed57|cert_cn=Software Signing|pid=72012|pidversion=1097765|ppid=1|uid=208|user=_trustevaluationagent|gid=208|group=_trustevaluationagent|mode=M|path=/System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/Resources/trustevaluationagent|args=trustevaluationagent
[2022-05-12T11:33:56.696Z] I santad: action=DELETE|path=/private/var/db/SystemPolicy-journal|pid=377|pidversion=833|ppid=1|process=syspolicyd|processpath=/usr/libexec/syspolicyd|uid=0|user=root|gid=0|group=wheel
[2022-05-12T11:30:05.248Z] I santad: action=LINK|path=/private/var/db/santa/santa.log|newpath=/private/var/db/santa/santa.log.0|pid=71559|pidversion=1096716|ppid=1|process=newsyslog|processpath=/usr/sbin/newsyslog|uid=0|user=root|gid=0|group=wheel
[2022-05-12T11:30:16.125Z] I santad: action=RENAME|path=/System/Volumes/Data/.Spotlight-V100/Store-V2/DA884A3F-2513-426E-95F0-59244D1625ED/Cache/0000/0000/00e1/14761134.tmp|newpath=/System/Volumes/Data/.Spotlight-V100/Store-V2/DA884A3F-2513-426E-95F0-59244D1625ED/Cache/0000/0000/00e1/14761134.txt|pid=546|pidversion=1285|ppid=1|process=mds_stores|processpath=/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Support/mds_stores|uid=0|user=root|gid=0|group=wheel
[2022-05-12T11:38:05.278Z] I santad: action=WRITE|path=/private/var/log/com.apple.xpc.launchd/launchd.log|pid=1|pidversion=521|ppid=0|process=launchd|processpath=/sbin/launchd|uid=0|user=root|gid=0|group=wheel
[2022-05-12T11:32:33.718Z] I santad: action=DISKDISAPPEAR|mount=/Volumes/GoogleDrive|volume=Google Drive|bsdname=(null)
[2022-05-12T11:32:44.184Z] I santad: action=DISKAPPEAR|mount=/Volumes/GoogleDrive|volume=Google Drive|bsdname=|fs=smbfs|model=|serial=(null)|bus=|dmgpath=|appearance=2001-01-01T00:00:00.000Z
[2022-05-12T11:33:57.166Z] I santad: action=DISKAPPEAR|mount=|volume=Install Google Drive|bsdname=disk4s2|fs=hfs|model=Apple Disk Image|serial=|bus=Virtual Interface|dmgpath=|appearance=2022-05-12T11:33:57.043Z
[2022-05-12T11:33:57.235Z] I santad: action=DISKAPPEAR|mount=/Volumes/Install Google Drive|volume=Install Google Drive|bsdname=disk4s2|fs=hfs|model=Apple Disk Image|serial=|bus=Virtual Interface|dmgpath=|appearance=2022-05-12T11:33:57.043Z
[2022-05-12T11:35:31.436Z] I santad: action=DISKDISAPPEAR|mount=|volume=Install Google Drive|bsdname=disk4s2
5 changes: 5 additions & 0 deletions packages/santa/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "2.2.0"
Comment thread
DamiaPoquet marked this conversation as resolved.
Outdated
changes:
- description: Update log format
Comment thread
DamiaPoquet marked this conversation as resolved.
Outdated
type: breaking-change
link: https://github.com/elastic/integrations/pull/3347
- version: "2.1.0"
changes:
- description: Update to ECS 8.2
Expand Down
21 changes: 11 additions & 10 deletions packages/santa/data_stream/log/_dev/test/pipeline/test-santa-raw.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
[2018-12-10T06:45:16.802Z] I santad: action=EXEC|decision=ALLOW|reason=CERT|sha256=c4bc09fd2f248534552f517acf3edb9a635aba2b02e46f49df683ea9b778e5b4|path=/usr/libexec/xpcproxy|args=/usr/sbin/newsyslog|cert_sha256=2aa4b9973b7ba07add447ee4da8b5337c3ee2c3a991911e80e7282e8a751fc32|cert_cn=Software Signing|pid=29678|ppid=1|uid=0|user=root|gid=0|group=wheel|mode=M
[2018-12-10T06:45:16.802Z] I santad: action=EXEC|decision=ALLOW|reason=CERT|sha256=c4bc09fd2f248534552f517acf3edb9a635aba2b02e46f49df683ea9b778e5b4|path=/usr/libexec/xpcproxy|args=xpcproxy com.apple.systemstats.daily|cert_sha256=2aa4b9973b7ba07add447ee4da8b5337c3ee2c3a991911e80e7282e8a751fc32|cert_cn=Software Signing|pid=29679|ppid=1|uid=0|user=root|gid=0|group=wheel|mode=M
[2018-12-10T06:45:16.851Z] I santad: action=EXEC|decision=ALLOW|reason=CERT|sha256=746f0dbafb7e675d5ce67131e5544772ee612b894e8ab51d3ce2d21f7cb7332d|path=/usr/sbin/newsyslog|args=/usr/sbin/newsyslog|cert_sha256=2aa4b9973b7ba07add447ee4da8b5337c3ee2c3a991911e80e7282e8a751fc32|cert_cn=Software Signing|pid=29678|ppid=1|uid=0|user=root|gid=0|group=wheel|mode=M
[2018-12-10T06:45:16.859Z] I santad: action=EXEC|decision=ALLOW|reason=CERT|sha256=d6be9bfbd777ac5dcd30488014acc787a2df5ce840f1fe4d5742d323ee00392f|path=/usr/sbin/systemstats|args=/usr/sbin/systemstats --daily|cert_sha256=2aa4b9973b7ba07add447ee4da8b5337c3ee2c3a991911e80e7282e8a751fc32|cert_cn=Software Signing|pid=29679|ppid=1|uid=0|user=root|gid=0|group=wheel|mode=M
[2018-12-10T08:45:27.810Z] I santad: action=EXEC|decision=ALLOW|reason=CERT|sha256=c4bc09fd2f248534552f517acf3edb9a635aba2b02e46f49df683ea9b778e5b4|path=/usr/libexec/xpcproxy|args=/usr/sbin/newsyslog|cert_sha256=2aa4b9973b7ba07add447ee4da8b5337c3ee2c3a991911e80e7282e8a751fc32|cert_cn=Software Signing|pid=29681|ppid=1|uid=0|user=root|gid=0|group=wheel|mode=M
[2018-12-10T08:45:27.810Z] I santad: action=EXEC|decision=ALLOW|reason=CERT|sha256=c4bc09fd2f248534552f517acf3edb9a635aba2b02e46f49df683ea9b778e5b4|path=/usr/libexec/xpcproxy|args=xpcproxy com.adobe.AAM.Scheduler-1.0|cert_sha256=2aa4b9973b7ba07add447ee4da8b5337c3ee2c3a991911e80e7282e8a751fc32|cert_cn=Software Signing|pid=29680|ppid=1|uid=0|user=root|gid=0|group=wheel|mode=M
[2018-12-10T21:37:27.247Z] I santad: action=EXEC|decision=ALLOW|reason=UNKNOWN|sha256=08bd61582657cd6d78c9e071d34d79a32bb59e7210077a44919d2c5477e988a1|path=/usr/local/Cellar/osquery/3.3.0_1/bin/osqueryd|args=/usr/local/bin/osqueryd --flagfile=/private/var/osquery/osquery.flags --logger_min_stderr=1|pid=45084|ppid=1|uid=0|user=root|gid=0|group=wheel|mode=M
[2018-12-10T16:24:43.992Z] I santad: action=EXEC|decision=ALLOW|reason=CERT|sha256=63b6a54848d7b4adf726d68f11409a4ac05b43926cb0f2792f7d41dc0221c106|path=/usr/bin/basename|cert_sha256=2aa4b9973b7ba07add447ee4da8b5337c3ee2c3a991911e80e7282e8a751fc32|cert_cn=Software Signing|pid=40757|ppid=40756|uid=501|user=akroh|gid=20|group=staff|mode=M
[2018-12-14T05:35:38.313Z] I santad: action=EXEC|decision=ALLOW|reason=UNKNOWN|sha256=a8defc1b24c45f6dabeb8298af5f8e1daf39e1504e16f878345f15ac94ae96d7|path=/Applications/Google Chrome.app/Contents/Versions/70.0.3538.110/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper|args=/Applications/Google Chrome.app/Contents/Versions/70.0.3538.110/Google Chrome Helper.app/Contents/MacOS/Google Chrome Helper --type=utility --field-trial-handle=120122713615061869,9401617251746517350,131072 --lang=en-US --service-sandbox-type=utility --service-request-channel-token=10458143409865682077 --seatbelt-client=262|cert_sha256=345a8e098bd04794aaeefda8c9ef56a0bf3d3706d67d35bc0e23f11bb3bffce5|cert_cn=Developer ID Application: Google, Inc. (EQHXZ8M8AV)|pid=89238|ppid=704|uid=501|user=akroh|gid=20|group=staff|mode=M
[2018-12-17T03:03:52.337Z] I santad: action=DISKAPPEAR|mount=/Volumes/Recovery|volume=Recovery|bsdname=disk1s3|fs=apfs|model=APPLE SSD SM0512L|serial=C026495006UHCHH1Q|bus=PCI-Express|dmgpath=
[2022-05-12T11:38:03.923Z] I santad: action=EXEC|decision=ALLOW|reason=BINARY|explain=critical system binary|sha256=43158bf397bf52001067319c591249307e2862daf8690828c15cdcc1ddf6166d|cert_sha256=d84db96af8c2e60ac4c851a21ec460f6f84e0235beb17d24a78712b9b021ed57|cert_cn=Software Signing|pid=71993|pidversion=1097732|ppid=1|uid=0|user=root|gid=0|group=wheel|mode=M|path=/usr/libexec/xpcproxy|args=xpcproxy com.apple.CoreAuthentication.agent
[2022-05-12T11:38:42.781Z] I santad: action=EXEC|decision=ALLOW|reason=CERT|sha256=7207307ca09d2707368ec394e67c6ccff6e48a2d1d86225a3115fe3535a8237c|cert_sha256=d84db96af8c2e60ac4c851a21ec460f6f84e0235beb17d24a78712b9b021ed57|cert_cn=Software Signing|pid=72012|pidversion=1097765|ppid=1|uid=208|user=_trustevaluationagent|gid=208|group=_trustevaluationagent|mode=M|path=/System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/Resources/trustevaluationagent|args=trustevaluationagent
[2022-05-12T11:33:56.696Z] I santad: action=DELETE|path=/private/var/db/SystemPolicy-journal|pid=377|pidversion=833|ppid=1|process=syspolicyd|processpath=/usr/libexec/syspolicyd|uid=0|user=root|gid=0|group=wheel
[2022-05-12T11:30:05.248Z] I santad: action=LINK|path=/private/var/db/santa/santa.log|newpath=/private/var/db/santa/santa.log.0|pid=71559|pidversion=1096716|ppid=1|process=newsyslog|processpath=/usr/sbin/newsyslog|uid=0|user=root|gid=0|group=wheel
[2022-05-12T11:30:16.125Z] I santad: action=RENAME|path=/System/Volumes/Data/.Spotlight-V100/Store-V2/DA884A3F-2513-426E-95F0-59244D1625ED/Cache/0000/0000/00e1/14761134.tmp|newpath=/System/Volumes/Data/.Spotlight-V100/Store-V2/DA884A3F-2513-426E-95F0-59244D1625ED/Cache/0000/0000/00e1/14761134.txt|pid=546|pidversion=1285|ppid=1|process=mds_stores|processpath=/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Support/mds_stores|uid=0|user=root|gid=0|group=wheel
[2022-05-12T11:38:05.278Z] I santad: action=WRITE|path=/private/var/log/com.apple.xpc.launchd/launchd.log|pid=1|pidversion=521|ppid=0|process=launchd|processpath=/sbin/launchd|uid=0|user=root|gid=0|group=wheel
[2022-05-12T11:32:33.718Z] I santad: action=DISKDISAPPEAR|mount=/Volumes/GoogleDrive|volume=Google Drive|bsdname=(null)
[2022-05-12T11:32:44.184Z] I santad: action=DISKAPPEAR|mount=/Volumes/GoogleDrive|volume=Google Drive|bsdname=|fs=smbfs|model=|serial=(null)|bus=|dmgpath=|appearance=2001-01-01T00:00:00.000Z
[2022-05-12T11:33:57.166Z] I santad: action=DISKAPPEAR|mount=|volume=Install Google Drive|bsdname=disk4s2|fs=hfs|model=Apple Disk Image|serial=|bus=Virtual Interface|dmgpath=|appearance=2022-05-12T11:33:57.043Z
[2022-05-12T11:33:57.235Z] I santad: action=DISKAPPEAR|mount=/Volumes/Install Google Drive|volume=Install Google Drive|bsdname=disk4s2|fs=hfs|model=Apple Disk Image|serial=|bus=Virtual Interface|dmgpath=|appearance=2022-05-12T11:33:57.043Z
[2022-05-12T11:35:31.436Z] I santad: action=DISKDISAPPEAR|mount=|volume=Install Google Drive|bsdname=disk4s2
Loading