Remove ruby execution engine

[kaisecheng]

Fixed: #11236

This PR removes the Ruby Execution Engine
The Java Execution Engine has been the default engine since Logstash 7.0, and is capable of using plugins written in either Ruby or Java; removing the Ruby Execution Engine will not affect our ability to run existing pipelines, but will greatly reduce the surface area of all features and fixes within the engine(s). We will no longer need to target two engines simultaneously.

Ruby execution engine uses Treetop to parse Logstash config files and convert them to executable code which is being referenced by Java pipeline, so Treetop stays in the project.

• remove Logstash::Pipeline
• rewrite test cases which depend on ruby pipeline
• remove logstash/event, Logstash::SignalEvent
• remove Engine.Ruby on java side
• remove config pipeline.java_execution

This PR will split into three smaller to feature branch for review
#12496 remove Logstash::Pipeline, Engine.Ruby, rewrite test cases
#12505 remove Logstash::SignalEvent
#12506 remove config pipeline.java_execution

A follow up will be removing logstash/pipeline in logstash-devutils

• clean up devutils remove logstash/pipeline logstash-devutils#89
• check tier 1 plugins reference
• clean up input-redis remove ruby pipeline dependency logstash-plugins/logstash-input-redis#84
• clean up integration-rabbitmq remove logstash/pipeline logstash-plugins/logstash-integration-rabbitmq#39
• clean up mixin-rabbitmq remove logstash/pipeline logstash-plugins/logstash-mixin-rabbitmq_connection#64
• check tier 2 plugins reference

Fix Logstash::SHUTDOWN event dependency

• kafka integration Avoid usage of SHUTDOWN event in Logstash 8.0.0 or greater logstash-plugins/logstash-integration-kafka#71
• cloudwatch Remove of unusefull SHUTDOWN event logstash-plugins/logstash-output-cloudwatch#18
• output-lumberjack Remove of unusefull SHUTDOWN event logstash-plugins/logstash-output-lumberjack#31

[andsel]

I think the commits should be squashed, LGTM

[kares]

NOTE: any plugin doing require 'logstash/pipeline' (e.g. in specs) will be broken by this.
Wonder if it was considered to have this file around as:

require 'logstash/java_pipeline'
module LogStash
  Pipeline = JavaPipeline
end

Seems an implementation detail whether require 'logstash/java_pipeline' gets required,
unless there's a strong need to break old users of require 'logstash/pipeline'

[kaisecheng]

This is a nice suggestion. I was thinking to remove everything completely including the plugins.

[yaauie]

Latergram:

I think we can do to make this transition easier for our users, especially for those who opt for explicit configuration or who control their configuration with environment variables.

• 7.x: the false value for --java-execution or java.execution is deprecated. We emit a deprecation log that REE is deprecated and will be removed in "a future major release of Logstash". To solve this deprecation warning, the user can either remove the flag or set its value to true.
• 8.x: the --java-execution or java.execution option itself is deprecated, and only accepts the true value. When this is specified, we emit a deprecation log about the option being removed in "a future major release of Logstash"

The primary goal that this path accomplishes is that explicit configuration from Logstash 7.$last that contains no deprecations works without modification in 8.0.0. It allows our users to be explicit in their configuration of 7.x and does not require an intermediate step where they are required to remove settings from their config that are meaningful in Logstash 7 prior to upgrading to Logstash 8.

The other transition option is to fast-track the removal of the setting, but I think that may be less clear to the user.

• 7.x: the --java-execution or java.execution option itself is deprecated.
  • When this option is specified as true, we emit a deprecation log indicating that it is not needed and the option will be removed in "a future major release of Logstash".
  • When the option is specified as false, we emit a different deprecation log indicating that both REE and the option will be removed in "a future major release of Logstash".

Note that the fast-track will make environment variable control of the config file not possible.