Measure current batch byte size and event count

[andsel]

Release notes

Implements current batch event count and byte size metrics.

What does this PR do?

Introduce a new gauge metric to collect list of values, used a a couple, the first element count of events in the batch and the second one is the estimated memory occupation of the batch.
This list is later grabbed in the API layer to populate the two current values for event_count and batch_size.

Why is it important/What is the impact to the user?

Exposes the current batch size in terms of events and estimated memory consumption.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files (and/or docker env variables)
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• [ ]

How to test this PR locally

Start Logstash with pipeline.batch.metrics.sampling_mode set to full running an HTTP input pipeline, load it with wrk and verify that the metric API pipelines.main.batch.event_count.current and pipelines.main.batch.byte_size.current goes down to 0 when wrk is interrupted and gets a value other than 0 when the load wrk resumes.

1. Create a file named input_sample.txt with some content
2. Create a script send_file.lua as

wrk.method = "POST"
local f = io.open("input_sample.txt", "r")
wrk.body   = f:read("*all")%

3. enable the metric in config/logstash.yml:

pipeline.batch.metrics.sampling_mode: full

4. run Logstash with pipeline:

input {
  http {
    response_headers => {"Content-Type" => "application/json"}
    ecs_compatibility => disabled
  }
}
output {
  sink {}
}

5. open a shell to verify the metrics on Logstash:

while true; do curl http://localhost:9600/_node/stats | jq .pipelines.main.batch; sleep 1; clear; done

6. start and stop wrk, checking the metrics:

wrk --threads 4 --connections 12 -d10m -s send_file.lua --latency http://localhost:8080

Related issues

• Closes Implements current value for batch's byte size and event count average metric #17997
• Relates Measure average batch byte size and event count #18000

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label. Could you fix it @andsel? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• If no backport is necessary, please add the backport-skip label

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: dc5fe28

Failed CI Steps

• 🛂 License check - Fedramp High Mode
• :java: Java unit tests - FIPS mode
• Observability SRE container smoke test
• 🥼 Integration Tests - FIPS mode / part 5-of-6

History

• 💚 Build #3570 succeeded ec0891b
• 💛 Build #3565 was flaky cc22aa2
• 💛 Build #3558 was flaky 00a395b
• 💚 Build #3490 succeeded 2e3ea64
• 💚 Build #3489 succeeded b4733b7

cc @andsel

[jsvd]

LGTM

When testing I noticed that the branch this is built on has a custom config/logstash.yml with an uncommented sample_mode setting https://github.com/andsel/logstash/blob/feature/current_metric_byte_size_event_count/config/logstash.yml#L57, so be careful in merging to ensure this change doesn't carry to main branch.

[andsel]

That flew into the #18000, was fixed in #18204. Thanks for noting :-)