Pq compression user metrics #18227

yaauie · 2025-09-26T22:42:58Z

Release notes

[rn: skip]

What does this PR do?

Adds metrics to the pipeline's queue namespace to reflect the cost and benefit of compression (see: #18107):

name definition expected value range

encode.spend.lifetime (encode_time / uptime) [0,(N_CPUS)]

encode.ratio.lifetime (compressed_bytes / decompresssed_bytes) [0,1]

decode.spend.lifetime (decode_time / uptime) [0,(N_CPUS)]

decode.ratio.lifetime (decompressed_bytes / compresssed_bytes) [1,)

Why is it important/What is the impact to the user?

It gives a user insight into the cost and benefit of compression options, so that they can make informed decisions about expending resources.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~[ ] I have made corresponding change to the default configuration files (and/or docker env variables)~~
I have added tests that prove my fix is effective or that my feature works

Author's Checklist

[ ]

How to test this PR locally

create an ndjson file example-input.ndjson:

{"this":"that","banana":["orange",{"elephant":1}]}
{"event":{"original":"this is an original event"}}

add my feed utility to your path

Invoke Logstash with options that activate the PQ and compression, feeding your input to stdin:

bin/logstash --log.level=info \
-Squeue.type=persisted \
-Squeue.drain=true \
-Squeue.compression=balanced \
--config.string 'input { stdin { codec => json_lines } } output { sink {} }' \
< <(feed example-input.ndjson)

while that runs, query the HTTP API:

curl --silent 'localhost:9600/_node/stats?pretty=true' | jq .pipelines.main.queue.compression

github-actions · 2025-09-26T22:43:09Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2025-09-26T22:43:35Z

This pull request does not have a backport label. Could you fix it @yaauie? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
If no backport is necessary, please add the backport-skip label

github-actions · 2025-09-26T22:43:51Z

🔍 Preview links for changed docs

yaauie · 2025-09-29T17:34:33Z

logstash-core/src/main/java/org/logstash/ackedqueue/AtomicIORatioMetric.java

+                return new ImmutableRatio(combinedBytesIn, combinedBytesOut);
+            } catch (ArithmeticException e) {
+                // don't crash, just start over.
+                return new ImmutableRatio(bytesIn, bytesOut);


An alternative that would be less lossy would be to shift the stored number (dividing it by 2 or 4).

But at 100GiB/sec this counter would take ~272 years to overflow so it's not really a concern in practice.

robbavey · 2025-09-29T18:27:22Z

logstash-core/src/main/java/org/logstash/ackedqueue/AtomicIORatioMetric.java

+                return new ImmutableRatio(combinedBytesIn, combinedBytesOut);
+            } catch (ArithmeticException e) {
+                // don't crash, just start over.
+                return new ImmutableRatio(bytesIn, bytesOut);


Can we log here, just in case there is some misplaced calculation that triggers this

I've addressed this in 7ae2395 by logging and halving the existing long value, and then logging. Combined with changing the interface of IORatioMetric#incrementBy to take int, int this should make those invalid calculations even harder to come by.

robbavey · 2025-09-29T19:01:21Z

docs/static/spec/openapi/logstash-api.yaml

+                compression:
+                  type: object
+                  properties:
+                    encode:


Would including the goal (speed, balanced, etc) in the payload provide value for monitoring, or checking diags without needing to refer back to logstash.yml?

(This is ok to add as a follow-on PR if we think it is worthwhile)

This would be trivial.

But the queue.compression.encode namespace is only present when encoding operations are possible (e.g., speed, balanced, size) and missing when configured with none. Do you have a preference for where we would put it?

I think queue.compression.encode.goal would work ok if that namespace is present. If it is absent, then none is implied, which makes sense, I think

robbavey · 2025-09-29T19:21:45Z

logstash-core/src/main/java/org/logstash/ackedqueue/AtomicIORatioMetric.java

+
+        if (bytesIn.signum() == 0) {
+            return switch(bytesOut.signum()) {
+                case -1 -> Double.NEGATIVE_INFINITY;


Is this possible? Or should incrementBy only accept positive values?

I've added safeguards to what is acceptable, so this should never get hit, but I would rather leave it in place instead of crashing if the impossible occurs.

robbavey · 2025-09-29T19:23:05Z

logstash-core/src/test/java/org/logstash/ackedqueue/AtomicIORatioMetricTest.java

+    public void testZeroBytesInNegativeBytesOut() {
+        final AtomicIORatioMetric ioRatioMetric = new AtomicIORatioMetric("name");
+
+        ioRatioMetric.incrementBy(0, -768);


Is this a use case that the ratio should support? Or should this raise an IllegalArgumentException?

I've addressed in 7ae2395 by making it log and ignore attempts to decrement. This is only ever called with the result of byte[]#length (which is never negative), but I would very much not like this to be a crashing problem that puts in-flight events at risk.

By using `int` type in `IORatioMetric#incrementBy(int,int)`, we simplify the failure scenarios while still allowing the desired behaviour, since this is always called in practice with `byte[]#length`. We ensure that attempts to decrement the value are ignored, and result in a log message, and that overflows reduce precision and are also logged. Together, these ensure that long overflows won't ever result in pipeline crashes.

elastic-sonarqube · 2025-09-29T21:33:16Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

robbavey

Changes made LGTM

Premature full LGTM

elasticmachine · 2025-09-29T21:42:27Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 7ae2395

Failed CI Steps

🥼 Integration Tests / part 3-of-6

History

💛 Build #3606 was flaky 6dc75f1
💚 Build #3605 succeeded e0f1215
💔 Build #3601 failed 91da26c

robbavey

LGTM

* pq-compression: wire through custom user metrics for ratio/spend * add test for queue compression metrics * pq metrics: IORatioMetric edge-case logging, use int at interface By using `int` type in `IORatioMetric#incrementBy(int,int)`, we simplify the failure scenarios while still allowing the desired behaviour, since this is always called in practice with `byte[]#length`. We ensure that attempts to decrement the value are ignored, and result in a log message, and that overflows reduce precision and are also logged. Together, these ensure that long overflows won't ever result in pipeline crashes.

yaauie force-pushed the pq-compression-user-metrics branch from 91da26c to 4d9a9dc Compare September 29, 2025 15:58

pq-compression: wire through custom user metrics for ratio/spend

e0f1215

yaauie force-pushed the pq-compression-user-metrics branch from 4d9a9dc to e0f1215 Compare September 29, 2025 16:12

add test for queue compression metrics

6dc75f1

yaauie marked this pull request as ready for review September 29, 2025 17:30

yaauie added the backport-skip Skip automated backport with mergify label Sep 29, 2025

yaauie commented Sep 29, 2025

View reviewed changes

robbavey reviewed Sep 29, 2025

View reviewed changes

yaauie requested a review from robbavey September 29, 2025 21:30

This comment was marked as outdated.

Sign in to view

robbavey reviewed Sep 29, 2025

View reviewed changes

robbavey approved these changes Sep 29, 2025

View reviewed changes

yaauie merged commit 1b3b3ee into elastic:main Sep 29, 2025
12 checks passed

yaauie mentioned this pull request Sep 29, 2025

metrics: add gauge with compression goal if enabled #18230

Merged

4 tasks

robbavey mentioned this pull request Sep 30, 2025

PQ compression: metrics #18107

Closed

donoghuc mentioned this pull request Oct 15, 2025

Release notes for 9.2.0 #18313

Merged

name	definition	expected value range
`encode.spend.lifetime`	(`encode_time` / `uptime`)	[`0`,(`N_CPUS`)]
`encode.ratio.lifetime`	(compressed_bytes / decompresssed_bytes)	[`0`,`1`]
`decode.spend.lifetime`	(`decode_time` / `uptime`)	[`0`,(`N_CPUS`)]
`decode.ratio.lifetime`	(decompressed_bytes / compresssed_bytes)	[`1`,)

Pq compression user metrics #18227

Pq compression user metrics #18227

Uh oh!

Conversation

yaauie commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release notes

What does this PR do?

Why is it important/What is the impact to the user?

Checklist

Author's Checklist

How to test this PR locally

Uh oh!

github-actions bot commented Sep 26, 2025

🤖 GitHub comments

Uh oh!

mergify bot commented Sep 26, 2025

Uh oh!

github-actions bot commented Sep 26, 2025

🔍 Preview links for changed docs

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaauie Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elastic-sonarqube bot commented Sep 29, 2025

Quality Gate passed

Uh oh!

This comment was marked as outdated.

Uh oh!

robbavey left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Sep 29, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

History

Uh oh!

robbavey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yaauie commented Sep 26, 2025 •

edited

Loading

yaauie Sep 29, 2025 •

edited

Loading