NoMethodError: undefined method `deep_replace' for LogStash::Outputs::ElasticSearch:Class running local integration tests

[dblock]

JVM

openjdk 14.0.2 2020-07-14
OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)

JRuby

$ ruby --version
jruby 9.2.9.0 (2.5.7) 2019-10-30 458ad3e OpenJDK 64-Bit Server VM 14.0.2+12-Ubuntu-120.04 on 14.0.2+12-Ubuntu-120.04 +jit [linux-x86_64]

(also tried with 9.2.13)

OS version (uname -a if on a Unix-like system):

$ uname -a
Linux ip-172-31-41-248 5.4.0-1049-aws #51-Ubuntu SMP Wed May 12 21:13:52 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

Can't run integration tests locally, these work in CI but for the life of me I cannot figure out why.

Steps to reproduce:

1. Check out this repo.
2. Ensure ES_URL is correct in .ci/logstash-run.sh
3. Ensure ES is running/
4. Run INTEGRATION=true SECURE_INTEGRATION=false DISTRIBUTION=oss .ci/logstash-run.sh

Observe:

1.2) Failure/Error: else super

          NoMethodError:
            undefined method `deep_replace' for LogStash::Outputs::ElasticSearch:Class
          # /home/ubuntu/.rvm/gems/jruby-9.2.9.0/gems/logstash-mixin-ecs_compatibility_support-1.0.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support.rb:90:in `validate_
value'
          # ./lib/logstash/outputs/elasticsearch/data_stream_support.rb:193:in `validate_value'
          # /home/ubuntu/.rvm/gems/jruby-9.2.9.0/gems/logstash-core-5.4.0-java/lib/logstash/config/mixin.rb:369:in `process_parameter_value'
          # /home/ubuntu/.rvm/gems/jruby-9.2.9.0/gems/logstash-core-5.4.0-java/lib/logstash/config/mixin.rb:388:in `block in validate_check_parameter_values'
          # /home/ubuntu/.rvm/gems/jruby-9.2.9.0/gems/logstash-core-5.4.0-java/lib/logstash/config/mixin.rb:382:in `block in validate_check_parameter_values'
          # /home/ubuntu/.rvm/gems/jruby-9.2.9.0/gems/logstash-core-5.4.0-java/lib/logstash/config/mixin.rb:381:in `validate_check_parameter_values'
          # /home/ubuntu/.rvm/gems/jruby-9.2.9.0/gems/logstash-core-5.4.0-java/lib/logstash/config/mixin.rb:271:in `validate'
          # /home/ubuntu/.rvm/gems/jruby-9.2.9.0/gems/logstash-core-5.4.0-java/lib/logstash/config/mixin.rb:129:in `config_init'
          # ./lib/logstash/outputs/elasticsearch.rb:327:in `config_init'
          # /home/ubuntu/.rvm/gems/jruby-9.2.9.0/gems/logstash-core-5.4.0-java/lib/logstash/outputs/base.rb:63:in `initialize'
          # ./lib/logstash/outputs/elasticsearch.rb:260:in `initialize'
          # ./spec/integration/outputs/no_es_on_startup_spec.rb:9:in `block in subject'
          # ./spec/integration/outputs/no_es_on_startup_spec.rb:32:in `block in <main>'
          # /home/ubuntu/.rvm/gems/jruby-9.2.9.0/gems/rspec-wait-0.0.9/lib/rspec/wait.rb:46:in `block in <main>'

I tracked the deep_replace method to this mixin, but I don't understand how it's supposed to end up in LogStash::Outputs::ElasticSearch:Class.

[kares]

Thanks Daniel, we no longer test against old LS 5.x.

The part the LS team hasn't updated is advice on plugins to use a local LOGSTASH_HOME path instead of relying on the core gems to be released in RubyGems (these gems are available from any LS installation).

I would recommend you try export LOGSTASH_SOURCE=1 and use export LOGSTASH_PATH=../path/to/logstash/6.8/or/greater, if the directory exists it will pick up the logstash-core gem from there instead of RubyGems.
Than bundle install and re-run the specs.

[dblock]

Thanks, so it's a versioning problem where a minimum of 6.8 is required? should some/a dependency be updated to have a correct minimum in .gemspec?

[kares]

kind of yes but only for testing, the plugin is likely to work on old LS versions (the reason it hasn't been bumped).
we could bump I think having a spec.required_ruby_version = '>= 2.3' should effectively limit usage with old LS.
that would not solve your testing problem though - you would still need gem 'logstash-core' '>= 6.8' dependency.