PQ settings refactor: propagate builder upward #18180
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The `ackedqueue.SettingsImpl` uses an _immutable_ builder, which makes adding options cumbersome; each additional property added needs to modify code from all existing options. By introducing an api-internal temporary mutable builder, we can simplify the process of creating an immutable copy that has a single component modified.
Contributor
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
Contributor
|
This pull request does not have a backport label. Could you fix it @yaauie? 🙏
|
yaauie
changed the title
5 tasks
andsel
requested changes
Sep 18, 2025
Contributor
andsel
left a comment
andsel
left a comment
There was a problem hiding this comment.
The changes seems fine, I left a question regarding the need to use a MutableBuilder bridge class to collect field values.
LMK your thoughts about that.
logstash-core/src/main/java/org/logstash/ackedqueue/SettingsImpl.java
Outdated
Show resolved
Hide resolved
logstash-core/src/main/java/org/logstash/ackedqueue/ext/JRubyWrappedAckedQueueExt.java
Outdated
Show resolved
Hide resolved
Co-authored-by: Andrea Selva <[email protected]>
|
Collaborator
💚 Build Succeeded
History
|
jsvd
added a commit
that referenced
this pull request
Oct 27, 2025
…8366) The queue factory setting "queueMaxBytes" used to be a long [link](https://github.com/elastic/logstash/pull/18180/files#diff-a377851670e06cd113751aa73b726b446dfb721d03d6979b9195afff7facd9f5L60): ```ruby long queueMaxBytes = RubyFixnum.num2long(args[6]); ``` But with the refactoring of #18180, it [became an Integer by mistake](https://github.com/elastic/logstash/pull/18180/files#diff-01eec670d2beabcd43041ff7fca8bc907eb227471dce886bb757b6414a655429R125): ```java .queueMaxBytes(getSetting(context, settings, QUEUE_MAX_BYTES).toJava(Integer.class)) ``` Which prevents queues being bigger than 2^31-1, causing crashes to anyone setting their PQ to 2GB or more: ```yaml queue.max_bytes: 2147483647 # 2GB-1 OK queue.max_bytes: 2147483648 # 2GB FAIL ``` This change makes queueMaxBytes a Long again, with a test confirming that queues larger than 2ˆ31-1 (max int) can be created. Without this change the new test fails: ``` ❯ bin/rspec logstash-core/spec/logstash/queue_factory_spec.rb ..F. Failures: 1) LogStash::QueueFactory when `queue.type` is `persisted` when queue.max_bytes is larger than Java int does not raise error Failure/Error: expect { queue = subject.create(settings) }.to_not raise_error expected no Exception, got #<RangeError: too big for int: 2147483648> with backtrace: # org/logstash/ackedqueue/QueueFactoryExt.java:97:in `create' # org/logstash/ackedqueue/QueueFactoryExt.java:88:in `create' # ./logstash-core/spec/logstash/queue_factory_spec.rb:85:in `block in <main>' # ./logstash-core/spec/logstash/queue_factory_spec.rb:85:in `block in <main>' # ./lib/bootstrap/rspec.rb:36:in `<main>' # ./logstash-core/spec/logstash/queue_factory_spec.rb:85:in `block in <main>' # ./lib/bootstrap/rspec.rb:36:in `<main>' Finished in 0.03879 seconds (files took 0.03787 seconds to load) 4 examples, 1 failure Failed examples: rspec ./logstash-core/spec/logstash/queue_factory_spec.rb:84 # LogStash::QueueFactory when `queue.type` is `persisted` when queue.max_bytes is larger than Java int does not raise error ``` But with the change it passes: ``` ❯ bin/rspec logstash-core/spec/logstash/queue_factory_spec.rb -fd LogStash::QueueFactory when `queue.type` is `persisted` returns a `WrappedAckedQueue` per pipeline id subdirectory creation creates a queue directory based on the pipeline id when queue.max_bytes is larger than Java int does not raise error when `queue.type` is `memory` returns a `WrappedSynchronousQueue` Finished in 0.03292 seconds (files took 0.04052 seconds to load) 4 examples, 0 failures ```
mergify bot
pushed a commit
that referenced
this pull request
Oct 27, 2025
…8366) The queue factory setting "queueMaxBytes" used to be a long [link](https://github.com/elastic/logstash/pull/18180/files#diff-a377851670e06cd113751aa73b726b446dfb721d03d6979b9195afff7facd9f5L60): ```ruby long queueMaxBytes = RubyFixnum.num2long(args[6]); ``` But with the refactoring of #18180, it [became an Integer by mistake](https://github.com/elastic/logstash/pull/18180/files#diff-01eec670d2beabcd43041ff7fca8bc907eb227471dce886bb757b6414a655429R125): ```java .queueMaxBytes(getSetting(context, settings, QUEUE_MAX_BYTES).toJava(Integer.class)) ``` Which prevents queues being bigger than 2^31-1, causing crashes to anyone setting their PQ to 2GB or more: ```yaml queue.max_bytes: 2147483647 # 2GB-1 OK queue.max_bytes: 2147483648 # 2GB FAIL ``` This change makes queueMaxBytes a Long again, with a test confirming that queues larger than 2ˆ31-1 (max int) can be created. Without this change the new test fails: ``` ❯ bin/rspec logstash-core/spec/logstash/queue_factory_spec.rb ..F. Failures: 1) LogStash::QueueFactory when `queue.type` is `persisted` when queue.max_bytes is larger than Java int does not raise error Failure/Error: expect { queue = subject.create(settings) }.to_not raise_error expected no Exception, got #<RangeError: too big for int: 2147483648> with backtrace: # org/logstash/ackedqueue/QueueFactoryExt.java:97:in `create' # org/logstash/ackedqueue/QueueFactoryExt.java:88:in `create' # ./logstash-core/spec/logstash/queue_factory_spec.rb:85:in `block in <main>' # ./logstash-core/spec/logstash/queue_factory_spec.rb:85:in `block in <main>' # ./lib/bootstrap/rspec.rb:36:in `<main>' # ./logstash-core/spec/logstash/queue_factory_spec.rb:85:in `block in <main>' # ./lib/bootstrap/rspec.rb:36:in `<main>' Finished in 0.03879 seconds (files took 0.03787 seconds to load) 4 examples, 1 failure Failed examples: rspec ./logstash-core/spec/logstash/queue_factory_spec.rb:84 # LogStash::QueueFactory when `queue.type` is `persisted` when queue.max_bytes is larger than Java int does not raise error ``` But with the change it passes: ``` ❯ bin/rspec logstash-core/spec/logstash/queue_factory_spec.rb -fd LogStash::QueueFactory when `queue.type` is `persisted` returns a `WrappedAckedQueue` per pipeline id subdirectory creation creates a queue directory based on the pipeline id when queue.max_bytes is larger than Java int does not raise error when `queue.type` is `memory` returns a `WrappedSynchronousQueue` Finished in 0.03292 seconds (files took 0.04052 seconds to load) 4 examples, 0 failures ``` (cherry picked from commit 6c48e51)
jsvd
added a commit
that referenced
this pull request
Oct 27, 2025
…8366) (#18367) The queue factory setting "queueMaxBytes" used to be a long [link](https://github.com/elastic/logstash/pull/18180/files#diff-a377851670e06cd113751aa73b726b446dfb721d03d6979b9195afff7facd9f5L60): ```ruby long queueMaxBytes = RubyFixnum.num2long(args[6]); ``` But with the refactoring of #18180, it [became an Integer by mistake](https://github.com/elastic/logstash/pull/18180/files#diff-01eec670d2beabcd43041ff7fca8bc907eb227471dce886bb757b6414a655429R125): ```java .queueMaxBytes(getSetting(context, settings, QUEUE_MAX_BYTES).toJava(Integer.class)) ``` Which prevents queues being bigger than 2^31-1, causing crashes to anyone setting their PQ to 2GB or more: ```yaml queue.max_bytes: 2147483647 # 2GB-1 OK queue.max_bytes: 2147483648 # 2GB FAIL ``` This change makes queueMaxBytes a Long again, with a test confirming that queues larger than 2ˆ31-1 (max int) can be created. Without this change the new test fails: ``` ❯ bin/rspec logstash-core/spec/logstash/queue_factory_spec.rb ..F. Failures: 1) LogStash::QueueFactory when `queue.type` is `persisted` when queue.max_bytes is larger than Java int does not raise error Failure/Error: expect { queue = subject.create(settings) }.to_not raise_error expected no Exception, got #<RangeError: too big for int: 2147483648> with backtrace: # org/logstash/ackedqueue/QueueFactoryExt.java:97:in `create' # org/logstash/ackedqueue/QueueFactoryExt.java:88:in `create' # ./logstash-core/spec/logstash/queue_factory_spec.rb:85:in `block in <main>' # ./logstash-core/spec/logstash/queue_factory_spec.rb:85:in `block in <main>' # ./lib/bootstrap/rspec.rb:36:in `<main>' # ./logstash-core/spec/logstash/queue_factory_spec.rb:85:in `block in <main>' # ./lib/bootstrap/rspec.rb:36:in `<main>' Finished in 0.03879 seconds (files took 0.03787 seconds to load) 4 examples, 1 failure Failed examples: rspec ./logstash-core/spec/logstash/queue_factory_spec.rb:84 # LogStash::QueueFactory when `queue.type` is `persisted` when queue.max_bytes is larger than Java int does not raise error ``` But with the change it passes: ``` ❯ bin/rspec logstash-core/spec/logstash/queue_factory_spec.rb -fd LogStash::QueueFactory when `queue.type` is `persisted` returns a `WrappedAckedQueue` per pipeline id subdirectory creation creates a queue directory based on the pipeline id when queue.max_bytes is larger than Java int does not raise error when `queue.type` is `memory` returns a `WrappedSynchronousQueue` Finished in 0.03292 seconds (files took 0.04052 seconds to load) 4 examples, 0 failures ``` (cherry picked from commit 6c48e51) Co-authored-by: João Duarte <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Release notes
[rn: skip]
What does this PR do?
This is an internal no-net-change refactor to pull upward the use of the
org.logstash.ackedqueue.Settingsinto the JRuby queue extension classes, pared off of the existing PR #18121.Why is it important/What is the impact to the user?
There is no net change to the user.
The changes here underly code that is also going to need changes in @andsel 's current in-flight work. Paring it off as a separate PR unblocks his work.
Checklist
[ ] I have made corresponding changes to the documentation[ ] I have made corresponding change to the default configuration files (and/or docker env variables)[ ] I have added tests that prove my fix is effective or that my feature worksHow to test this PR locally
No net change. Rely on existing test coverage and usage.
Related issues
Use cases
Screenshots
Logs