draft: experiment with compressing the persistent queue data

[jsvd]

since compression can be quite cheap compared to i/o, maybe we could default to compressing the data being written to the PQ.

this is a draft and shows a significant performance in performance, moving the bottleneck closer to the cbor mapper deserialization instead of PQ writing in my local laptop.

such a change must be however evaluated as to not introduce breaking changes, potentially with a V3 PQ format.

code-wise Logstash::Event could move to a extension of the Queueable interface that allows compression.

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 8449368

Failed CI Steps

• :java: Java unit tests
• :java: Java unit tests
• :java: Java unit tests
• :java: Java unit tests
• 🥼 Integration Tests / part 1

History

[mergify]

This pull request does not have a backport label. Could you fix it @jsvd? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• backport-8.x is the label to automatically backport to the 8.x branch.

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label.