[santa] update ingest pipeline to match new log format

[DamiaPoquet]

What does this PR do?

The log output for Santa changed on versions 0.9.19, v0.9.20, and 0.9.21, making this integration fundamentally broken since then. This PR addresses the new log format parsing.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• Check log parsing
• ECS 8.2 doesn't contains a field for process pidversion

How to test this PR locally

Related issues

I didn't find any related issue, although it seems to be broken since 2017.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-05-17T08:33:25.434+0000

• Duration: 15 min 32 sec

Test stats 🧪

Test	Results
Failed	0
Passed	13
Skipped	0
Total	13

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

Can we leave the original tests in place while adding the new ones.

[efd6]

/test

[DamiaPoquet]

Can we leave the original tests in place while adding the new ones.

Is there any use case for preserving those old log samples? If I leave the original ones, the tests will fail.

[efd6]

Do we know that these are no users still on the old version? If we are going to drop support for the older log format then this should really be a breaking change and marked as such.

The preferred option though would be to retain support for the old format while adding the new.

[DamiaPoquet]

In my opinion, it doesn't make sense to support a pre-release output format for Santa. All the GA versions are outputting logs in the new format.

I'll mark this release as a breaking change.

[andrewkroh]

/test

[andrewkroh]

Thanks for the fix!

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (1/1)	💚
Files	100.0% (1/1)	💚 3.589
Classes	100.0% (1/1)	💚 3.589
Methods	100.0% (12/12)	💚 11.714
Lines	91.0% (91/100)	👍 1.968
Conditionals	100.0% (0/0)	💚

[efd6]

process.pidversion is not an allowed ECS field, so it should be placed in santa, but we could construct a process.entity_id with it and process.pid, something like "{{{process.pid}}}-{{[process.pidversion}}}". @andrewkroh WDYT? We don't have any examples of it being used anywhere else AFAICS.

[andrewkroh]

That sounds like a good idea. I am looking for some documentation on what pidversion in the Santa logs means just to confirm that it makes sense to use it as part of process.entity_id.

[efd6]

From what I could find it's just another pid, but from the kernel to prevent pid collision on pid rollover, e.g. https://www.exploit-db.com/exploits/46648. The Apple documentation is also not wildly helpful https://developer.apple.com/documentation/kernel/proc_persona_info/1560796-pidversion, the source helps https://github.com/apple/darwin-xnu/blob/2ff845c2e033bd0ff64b5b6aa6063a1f8f65aa32/bsd/kern/kern_exec.c#L6215-L6216.

This in the context of this gives the best answer I could find.

[DamiaPoquet]

I have changed the field name to santa.pidversion while the ECS is not ready yet to support it.

[efd6]

/test

[andrewkroh]

LGTM. We could add a process.entity_id independently.

I was curious how Elastic Endpoint generates its process.entity_id to see if we could replicate that here. The code is

    (void)StringLib::FormatDuplicate(str, "%s-%llu-%llu.%llu", Cache::GetEndpointID().c_str(),
        (unsigned long long)pid, (unsigned long long)ts.tv_sec, (unsigned long long)ts.tv_nsec);
    (void)Titan_Encoding_toBase64(str.c_str(), str.size(), (void **)&(pBase64), &(base64Size));

But given that the start time we have is coming from the logger timestamp and lacks nanosecond precision, I don't think we can. So the only thing I would say is that if we do add an entity_id we should consider incorporating the agent ID.

[efd6]

But given that the start time we have is coming from the logger timestamp and lacks nanosecond precision, I don't think we can. So the only thing I would say is that if we do add an entity_id we should consider incorporating the agent ID.

SGTM