Add OpenSearch PPL support

[Adrian256256]

Add OpenSearch PPL support

This PR adds support for the OpenSearch PPL query language.
Since it is independent from any Elasticsearch query language and not based on the existing backend code, the implementation resides in its own file (sigma/backends/opensearch/opensearch_ppl.py).
The backend extends pySigma's TextQueryBackend class.

What it does

Converts Sigma rules into OpenSearch PPL queries:

• Regular detection rules
• Correlation rules

Target Versions

Note that this targets OpenSearch 3.3+ for regular detection rules. Correlation rules require OpenSearch 3.4+ since they depend on the multisearch command.

PPL Documentation: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/index.md

Features

Correlation Support

Support for Sigma correlation types with automatic conversion to PPL aggregation queries:

• event_count - Frequency-based detection (e.g., >10 failed logins in 5 minutes)

  • Uses PPL's stats count() with time-based grouping with span() function

• value_count - Cardinality-based detection (e.g., same password tried on more than 5 accounts)

  • Uses PPL's dc(field) (distinct count) function

• temporal / temporal_ordered - Multi-stage attack detection (e.g., failed login followed by successful login)

  • Always uses the multisearch command to combine detection rules from multiple indices
  • Uses span() for time-based bucketing and dc(EventID) to verify all event types occurred

About multisearch: When you have 2 or more detection rules in a correlation, the backend uses the multisearch command. This means each detection rule runs as its own separate search and the results get combined. This approach works the same way whether your rules search the same index or different indices.

Custom Attributes

You can configure specific rules by adding custom attributes directly in the Sigma YAML file. This lets you override the backend's default behavior for individual rules:

• opensearch_ppl_index - Override index pattern (useful for non-standard naming)
• opensearch_ppl_min_time - Add minimum time filter (e.g., "-30d" for last 30 days)
• opensearch_ppl_max_time - Add maximum time filter (e.g., "now" for up to current time)

Example:

custom:
  opensearch_ppl_index: "my-custom-logs-*"
  opensearch_ppl_min_time: "-7d"
  opensearch_ppl_max_time: "now"

Backend Options

Default values when instantiating the backend (similar to Splunk's -O flag) - apply to all rules unless overridden by custom attributes:

• custom_logsource - Override auto-generated index patterns globally
• min_time / max_time - Add time filters to all queries for performance optimization

Example:

backend = OpenSearchPPLBackend(
    custom_logsource="security-logs-*",
    min_time="-30d",
    max_time="now"
)

Priority system: YAML custom attributes > Backend options > Default values

---

Acknowledgments

Special thanks to @VladNastase for guidance.

[VladNastase]

I'll add that while OpenSearch PPL added a lot of features to get close to Splunk's SPL, some parts are still a bit unstable.

While working on the backend we found a few bugs (opensearch-project/sql#5145, opensearch-project/sql#5146, opensearch-project/sql#5147) that make some correlation rules unusable. There is a fix in opensearch-project/sql#5154, but we have no idea when it will be merged and/or if it will be backported.

There might still be some bugs in the newer PPL commands / features, but the translation from Sigma to PPL should be ok.

[thomaspatzke]

Looks great to me! Could you please add tests as done here for Lucene query generation?

[Adrian256256]

I added the tests for the OpenSearch PPL backend.