Include examples of a detected log entry in the rule

[kelnage]

When developing integration for a new backend, it would be useful if a rule was able/encouraged to include one (or more!) examples of the entries in a sample log files that it is trying to detect. This could allow the backend engine to test whether or not it correctly identifies the log entry - helping ensure that the generated rules are operating correctly. My suggestion would be to add an optional section to the schema, something along the lines of this:

    examples:
        type: //arr
        contents: //str

Obviously some backends/types of logs may not be able to be tested in this way, but for those that could, it could allow greater confidence in the Sigma rule itself and generated rules for the backend.

One potential risk in doing this is that it might mean the rule files themselves start being picked up by other detection systems. The risk/impact of this seems low to me - especially as the rules already likely contain keywords that would trigger detection anyway - but I thought it would be worth highlighting.

[frack113]

For me rule should not have an example in this way but a link to a dataset.
This method gives the impression that sigma is only made for 1 line log = 1 detection.

I'm more for a section like the references

datasets:
  - https://raw.githubusercontent.com/OTRF/Security-Datasets/master/datasets/atomic/windows/discovery/host/empire_shell_net_local_users.zip

[kelnage]

I understand where you're coming from - but I do want to make some counter-points:

1. YAML strings can be multi-line - so whilst an entry in the example list might be a single line from a log file (and in my experience, many Sigma rules will work for such data), it certainly can contain multiple log entries (assuming entries are separated with new lines - for XML-style logs I suppose they would be separated with open/close tags), and hence we could certainly assume 1 entry = 1 detection (though I do agree that should probably be formalised in the spec)
2. My primary goal here is to automate this process to some extent. Using hyperlinks significantly complicates that process - it would mean fetching/caching those links which would slow down the process, determining what (if any) pre-processing needs to happen to it (for example, presumably your example dataset would need decompression before being used for validation - which compression methods would be supported?) and introduces additional questions that I'm not sure we can answer (i.e., what should be done if a link is not reachable at validation time?)
3. Sigma rules already contains a references section that such a link to a sample dataset could be included in

Personally I don't think the cons of including the examples in the rules would outweigh the benefits of hosting them externally.

[Neo23x0]

some backends/types of logs may not be able to be tested in this way

Most backends/types of logs could not be tested that way. E.g. all Windows logs are actually XML structures.
Currently, I would only see a 1:1 match for all kinds of web logs, proxy logs, linux logs and possibly PowerShell script block logs.

Personally I don't think the cons of including the examples in the rules would outweigh the benefits of hosting them externally.

Which cons did you identify?
For me, the costs outweigh the benefits.

Costs:

• rules get bigger and more complex
• the rules would suddenly contain full strings of possibly malicious contents (E.g. log4shell payloads) that could trigger an AV detection or YARA sigs
• the authors would provide the source logs in different formats - e.g. someone working with Sentinel would provide a log line that looks very different from the log line someone exports in his CrowdStrike platform, or a Sysmon event
• the authors would provide the wrong raw data - e.g. the rendered log event from Windows Event Viewer and not the XML source

Benefit:

• you could (try to!) automate tests to verify a rule works as expected (flawed - only works for a few formats)
• refactoring of the rule and changes to the rule (e.g. false positive filtering) could be made while looking at the example log line - this would help during the refactoring because the reviewer, which is often not the original author, would be able to factor in all contents and not just the extract that the rule author provided

[Neo23x0]

Do you have a specific example of a log entry that would be detected as malicious whilst the Sigma signature wouldn't?

This happened multiple times so far. The last time I remember was with the Log4Shell detection rules

Rule
https://valhalla.nextron-systems.com/info/sigma-rule/5ea8faa8-db8b-45be-89b0-151b84c82702

I had to shorten them to get them through and users could download and use the sigma rule again.

SigmaHQ/sigma@e2aa3665a

[Neo23x0]

Yes, we've discussed something very similar before, with @WojciechLesicki - I mean the part about the validations.

That's why we started the repo "sigma-validation-guides". We wanted to keep that information out of the rules. But we never filled the guides with content, because it is so much work to do.

It was a PR followed by a discussion: SigmaHQ/sigma#2207

[kelnage]

Thank you for the context @Neo23x0 - it's been really helpful! I've looked and I can't find SigmaHQ/validation-guides nor SigmaHQ/sigma-validation-guides - are they possibly private?

I think, from reading the discussion around SigmaHQ/sigma#2207, the key underlying question is whether or not Sigma rules should contain information that is more relevant for the rule/backend developers, or whether it should be primarily focused on the analysts who are monitoring them. Personally, I would say that providing sample log events that provide examples of detection events could be relevant for either party - but I will concede that they would probably have less relevance for analysts on average.

I've been working on a Python script to automate the testing of a pySigma backend I am developing (coming soon!), and for the time being I will take the route of maintaining my own repository of sample logs that map to the rules in the SigmaHQ/sigma repository (e.g., in order to test rules/web/web_cve_2021_43798_grafana.yml there will be a file samples/web/web_cve_2021_43798_grafana.log). This will at least minimise the risk around detection events on the main sigma repository.

If we do decide to include external links to datasets (which I'd still like to do!), I can then incorporate fetching those files into my testing script.

[Neo23x0]

I'm not opposed to an external reference and a new structure that holds all kinds of additional information, e.g., like a validation guide to reproduce the logs or example logs.

Let me see if I can publish the validation guides repo.

The thing is: Currently, it takes me and others working on the rules between 10 and 30min to write rule. I already spend half of that time finding the right MITRE ATTACK technique IDs.
Adding an example log line in a separate file would be another overhead for the main contributors, but yes, it would be optional and others could provide these example log lines for the existing rules.
I'm not sure if we can get a critical mass of example lines so that people see the value and start using it. It's a hen/egg problem.

[Neo23x0]

Here it is : https://github.com/SigmaHQ/validation-guides

But the only thing we (@frack113) did so far, was to specify a template and generate some blank guides for log sources.

You see, what we wanted to do is to provide a complete new structure for each rule that contains a guide on how to reproduce an event, event examples and possibly more.