[ti_anomali] Add Benchmark and Policy Test

[mohitjha-elastic]

Proposed commit message

ti_anomali: Add benchmark and policy test.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

• Clone integrations repo.
• Install the elastic package locally.
• Start the elastic stack using the elastic package.
• Move to integrations/packages/ti_anomali directory.
• Run the following command to run tests.

elastic-package test -v

Related Issues

• Closes ti_anomali: Add policy tests for integration quality checks #17069
• Closes ti_anomali: Add benchmarks for integration quality checks #17071

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[chrisberkhout]

Looks good. One minor suggestion in a policy test.

It looks like this has rally benchmarks, system benchmarks and pipeline benchmarks. It would be great if you could remind me of how those benchmark types differ in function and value. Are all three worthwhile here?

[mohitjha-elastic]

Looks good. One minor suggestion in a policy test.

It looks like this has rally benchmarks, system benchmarks and pipeline benchmarks. It would be great if you could remind me of how those benchmark types differ in function and value. Are all three worthwhile here?

Pipeline benchmarking focuses only on the ingest pipeline layer and measures how efficiently events are processed by processors such as grok, dissect, script, and convert. It helps identify event processing latency, and throughput in docs/sec. Its main value is optimizing parsing and enrichment logic without involving indexing or cluster performance.

Rally benchmarking evaluates core Elasticsearch indexing and query performance at scale. It measures bulk indexing rate, search latency, disk I/O, and the impact of mappings and shard configurations. This is primarily used for cluster sizing, capacity planning, and understanding how many events per second a given infrastructure can handle.

System benchmarking tests the full end-to-end flow—Elastic Agent to ingest pipelines to Elasticsearch indexing—under realistic conditions. It captures overall throughput, resource usage, and bottlenecks across the stack, making it the closest representation of production behavior. Its value lies in validating real integration performance and identifying whether limitations come from the agent, pipelines, or the cluster.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 4043e44

History

• 💚 Build #38290 succeeded ddbadd5

cc @mohitjha-elastic

[chrisberkhout]

Looks good. One minor suggestion in a policy test.
It looks like this has rally benchmarks, system benchmarks and pipeline benchmarks. It would be great if you could remind me of how those benchmark types differ in function and value. Are all three worthwhile here?

Pipeline benchmarking focuses only on the ingest pipeline layer and measures how efficiently events are processed by processors such as grok, dissect, script, and convert. It helps identify event processing latency, and throughput in docs/sec. Its main value is optimizing parsing and enrichment logic without involving indexing or cluster performance.

Rally benchmarking evaluates core Elasticsearch indexing and query performance at scale. It measures bulk indexing rate, search latency, disk I/O, and the impact of mappings and shard configurations. This is primarily used for cluster sizing, capacity planning, and understanding how many events per second a given infrastructure can handle.

System benchmarking tests the full end-to-end flow—Elastic Agent to ingest pipelines to Elasticsearch indexing—under realistic conditions. It captures overall throughput, resource usage, and bottlenecks across the stack, making it the closest representation of production behavior. Its value lies in validating real integration performance and identifying whether limitations come from the agent, pipelines, or the cluster.

Thanks. So let's merge this.

I think pipeline benchmarking is definitely useful for integrations. A lot of our logic is there. The system benchmarking is probably also useful because sometimes the agent may be the bottleneck and that's definitely something that the integration is in charge of. The rally benchmarking seems a bit less relevant because things like sharding, etc. are not really integration concerns. These seem like they would be useful less for monitoring integration quality and more for informing cluster resourcing in actual setups. Where we get value from these might be a topic for future discussion in the team, but for now let's have them all.

Thanks for the good work!