Backport PR #13015 to 7.15: Bundler: freeze lockfile on run, and "normalize" platform on plugin changes

Backport PR #13015 to 7.15 branch. Original Message:

This PR enables the upgrade of bundler to the latest version.

Prior to this PR, the ability to do so was blocked by bundler.setup in versions of bundler > `2.23` making runtime changes to `Gemfile.lock` (unless the lock file was `frozen`) based on the specific platform the application was being run on, overriding any platforms (including generic `java` platform) set during build time. This was in conflict with changes made in #12782, which prevented the logstash user writing to files in `/usr/share/logstash`.

This PR will freeze the lockfile when logstash is run, and unfreeze it when manipulating plugins (install, update, remove, install from offline pack) to allow new plugins to be added. While unfrozen, changes are also made to ensure that the platform list remains as the generic `java` platform, and not changed to the specific platform for the runtime JVM.

This PR also introduces a new runtime flag, `--enable-local-plugin-development`. This flag is intended for use by Logstash developers only, and enables a mode of operation where a Gemfile can be manipulated, eg

```
gem "logstash-integration-kafka", :path => '/users/developer/code/plugins/logstash-integration-kafka'
```

to facilitate quick and simple plugin testing.

This PR also sets the `silence_root_warning` flag to avoid bundler printing out alarming looking warning messages when `sudo` is used. This warning message was concerning for users - it would be printed out during normal operation of `bin/logstash-plugin install/update/remove` when run under `sudo`, which is the expected mode of operation when logstash is installed to run as a service via rpm/deb packages.

This PR also updates the vagrant based integration tests to ensure that Logstash still runs after plugin update/install/remove operations, fixes up some regular expressions that would cause test failures, and removes some dead code from tests.

* Updated Bundler to latest version
* Ensured that `Gemfile.lock` are appropriately frozen
* Added new developer-only flag to facilitate local plugin development to allow unfrozen lockfile in a development environment

(cherry picked from commit 4707cb)

Author: robbavey

ci/acceptance_tests.sh

@@ -27,6 +27,7 @@ QA_DIR="$PWD/qa"
 
 # Always run the halt, even if the test times out or an exit is sent
 cleanup() {
+
   cd $QA_DIR
   bundle check || bundle install
   bundle exec rake qa:vm:halt
@@ -39,6 +40,7 @@ cleanup
 if [[ $SELECTED_TEST_SUITE == $"redhat" ]]; then
   echo "Generating the RPM, make sure you start with a clean environment before generating other packages."
   cd $LS_HOME
+  ./gradlew clean bootstrap
   rake artifact:rpm
   echo "Acceptance: Installing dependencies"
   cd $QA_DIR
@@ -52,6 +54,7 @@ if [[ $SELECTED_TEST_SUITE == $"redhat" ]]; then
 elif [[ $SELECTED_TEST_SUITE == $"debian" ]]; then
   echo "Generating the DEB, make sure you start with a clean environment before generating other packages."
   cd $LS_HOME
+  ./gradlew clean bootstrap
   rake artifact:deb
   echo "Acceptance: Installing dependencies"
   cd $QA_DIR
@@ -65,6 +68,7 @@ elif [[ $SELECTED_TEST_SUITE == $"debian" ]]; then
 elif [[ $SELECTED_TEST_SUITE == $"all" ]]; then
   echo "Building Logstash artifacts"
   cd $LS_HOME
+  ./gradlew clean bootstrap
   rake artifact:all
 
   echo "Acceptance: Installing dependencies"

docs/static/running-logstash-command-line.asciidoc

@@ -227,5 +227,14 @@ With this command, Logstash concatenates three config files, `/tmp/one`, `/tmp/t
   as the log4j logging configuration. This can also be set through the LS_SETTINGS_DIR environment variable.
   The default is the `config` directory under Logstash home.
 
+*`--enable-local-plugin-development`*::
+  This flag enables developers to update their local Gemfile without running into issues caused by a frozen lockfile.
+  This flag can be helpful when you are developing/testing plugins locally.
+
+NOTE: This flag is for Logstash developers only. End users should not need it.
+
+
+
+
 *`-h, --help`*::
   Print help

lib/bootstrap/bundler.rb

@@ -73,7 +73,7 @@ def setup!(options = {})
       # in the context of Bundler.setup it looks like this is useless here because Gemfile path can only be specified using
       # the ENV, see https://github.com/bundler/bundler/blob/v1.8.3/lib/bundler/shared_helpers.rb#L103
       ::Bundler.settings.set_local(:gemfile, Environment::GEMFILE_PATH)
-
+      ::Bundler.settings.set_local(:frozen, true) unless options[:allow_gemfile_changes]
       ::Bundler.reset!
       ::Bundler.setup
     end
@@ -95,7 +95,6 @@ def invoke!(options = {})
                  :jobs => 12, :all => false, :package => false, :without => [:development]}.merge(options)
       options[:without] = Array(options[:without])
       options[:update] = Array(options[:update]) if options[:update]
-
       ::Gem.clear_paths
       ENV['GEM_HOME'] = ENV['GEM_PATH'] = LogStash::Environment.logstash_gem_home
       ::Gem.paths = ENV
@@ -128,13 +127,18 @@ def invoke!(options = {})
       ::Bundler.settings.set_local(:gemfile, LogStash::Environment::GEMFILE_PATH)
       ::Bundler.settings.set_local(:without, options[:without])
       ::Bundler.settings.set_local(:force, options[:force])
-
-      if !debug?
-        # Will deal with transient network errors
-        execute_bundler_with_retry(options)
-      else
-        options[:verbose] = true
-        execute_bundler(options)
+      # This env setting avoids the warning given when bundler is run as root, as is required
+      # to update plugins when logstash is run as a service
+      # Note: Using an `ENV` here because ::Bundler.settings.set_local(:silence_root_warning, true)
+      # does not work (set_global *does*, but that seems too drastic a change)
+      with_env("BUNDLE_SILENCE_ROOT_WARNING" => "true") do
+        if !debug?
+          # Will deal with transient network errors
+          execute_bundler_with_retry(options)
+        else
+          options[:verbose] = true
+          execute_bundler(options)
+        end
       end
     end
 
@@ -176,6 +180,18 @@ def execute_bundler(options)
       ::Bundler::CLI.start(bundler_arguments(options))
     end
 
+    def specific_platforms(platforms=::Gem.platforms)
+      platforms.find_all {|plat| plat.is_a?(::Gem::Platform) && plat.os=='java' && !plat.cpu.nil?}
+    end
+
+    def genericize_platform
+      output = LogStash::Bundler.invoke!({:add_platform => 'java'})
+      specific_platforms.each do |platform|
+        output << LogStash::Bundler.invoke!({:remove_platform => platform})
+      end
+      output
+    end
+
     def debug?
       ENV["DEBUG"]
     end
@@ -185,7 +201,6 @@ def debug?
     # @return [Array<String>] Bundler::CLI.start string arguments array
     def bundler_arguments(options = {})
       arguments = []
-
       if options[:install]
         arguments << "install"
         arguments << "--clean" if options[:clean]
@@ -202,14 +217,30 @@ def bundler_arguments(options = {})
       elsif options[:package]
         arguments << "package"
         arguments << "--all" if options[:all]
+      elsif options[:add_platform]
+        arguments << "lock"
+        arguments << "--add_platform"
+        arguments << options[:add_platform]
+      elsif options[:remove_platform]
+        arguments << "lock"
+        arguments << "--remove_platform"
+        arguments << options[:remove_platform]
       end
 
       arguments << "--verbose" if options[:verbose]
-
       arguments.flatten
     end
 
-   # capture any $stdout from the passed block. also trap any exception in that block, in which case the trapped exception will be returned
+    def with_env(modifications)
+      backup_env = ENV.to_hash
+      ENV.replace(backup_env.merge(modifications))
+
+      yield
+    ensure
+      ENV.replace(backup_env)
+    end
+
+    # capture any $stdout from the passed block. also trap any exception in that block, in which case the trapped exception will be returned
     # @param [Proc] the code block to execute
     # @return [String, Exception] the captured $stdout string and any trapped exception or nil if none
     def capture_stdout(&block)

lib/bootstrap/environment.rb

@@ -81,7 +81,12 @@ def pattern_path(path)
 # defined and exposing the LogStash::Runner#main instance method which will be called with the current ARGV
 # currently lib/logstash/runner.rb and lib/pluginmanager/main.rb are called using this.
 if $0 == __FILE__
-  LogStash::Bundler.setup!({:without => [:build, :development]})
+  bundler_options = {:without => [:build, :development]}
+  ## Check for dev flags - this needs to be done before the runner is invoked to set bundler options
+  if ARGV.include?("--enable-local-plugin-development")
+    bundler_options[:allow_gemfile_changes] = true
+  end
+  LogStash::Bundler.setup!(bundler_options)
   require_relative "patches/jar_dependencies"
 
   require ARGV.shift

lib/pluginmanager/bundler/logstash_injector.rb

@@ -93,6 +93,10 @@ def inject(gemfile_path, lockfile_path, dependencies)
 
           builder.eval_gemfile("bundler file", gemfile.generate())
           definition = builder.to_definition(lockfile_path, {})
+          LogStash::Bundler.specific_platforms(definition.platforms).each do |specific_platform|
+            definition.remove_platform(specific_platform)
+          end
+          definition.add_platform(Gem::Platform.new('java'))
           definition.lock(lockfile_path)
           gemfile.save
         rescue => e

lib/pluginmanager/install.rb

@@ -202,9 +202,12 @@ def install_gems_list!(install_list)
     bundler_options[:without] = [] if development?
     bundler_options[:rubygems_source] = gemfile.gemset.sources
     bundler_options[:local] = true if local?
-
-    output = LogStash::Bundler.invoke!(bundler_options)
-
+    output = nil
+    # Unfreeze the bundle when installing gems
+    Bundler.settings.temporary({:frozen => false}) do
+      output = LogStash::Bundler.invoke!(bundler_options)
+      output << LogStash::Bundler.genericize_platform.to_s
+    end
     puts("Installation successful")
   rescue => exception
     gemfile.restore!

lib/pluginmanager/remove.rb

@@ -54,7 +54,7 @@ def execute
     signal_error("This plugin has not been previously installed") unless LogStash::PluginManager.installed_plugin?(plugin, gemfile)
 
     exit(1) unless ::Bundler::LogstashUninstall.uninstall!(plugin)
-
+    LogStash::Bundler.genericize_platform
     remove_unused_locally_installed_gems!
   rescue => exception
     report_exception("Operation aborted, cannot remove plugin", exception)

lib/pluginmanager/update.rb

@@ -80,7 +80,13 @@ def update_gems!
     # Bundler cannot update and clean gems in one operation so we have to call the CLI twice.
     options = {:update => plugins, :rubygems_source => gemfile.gemset.sources}
     options[:local] = true if local?
-    output = LogStash::Bundler.invoke!(options)
+    output=nil
+    # Unfreeze the bundle when updating gems
+    Bundler.settings.temporary({:frozen => false}) do
+      output = LogStash::Bundler.invoke!(options)
+      output << LogStash::Bundler.genericize_platform unless output.nil?
+    end
+
     # We currently dont removed unused gems from the logstash installation
     # see: https://github.com/elastic/logstash/issues/6339
     # output = LogStash::Bundler.invoke!(:clean => true)

logstash-core/lib/logstash/environment.rb

@@ -69,6 +69,7 @@ module Environment
             Setting::String.new("log.level", "info", true, ["fatal", "error", "warn", "debug", "info", "trace"]),
            Setting::Boolean.new("version", false),
            Setting::Boolean.new("help", false),
+            Setting::Boolean.new("enable-local-plugin-development", false),
             Setting::String.new("log.format", "plain", true, ["json", "plain"]),
            Setting::Boolean.new("http.enabled", true),
             Setting::String.new("http.host", "127.0.0.1"),

logstash-core/lib/logstash/runner.rb

@@ -73,6 +73,11 @@ class LogStash::Runner < Clamp::StrictCommand
     :attribute_name => "node.name",
     :default => LogStash::SETTINGS.get_default("node.name")
 
+  option ["--enable-local-plugin-development"], :flag,
+         I18n.t("logstash.runner.flag.enable-local-plugin-development"),
+         :attribute_name => "enable-local-plugin-development",
+         :default => LogStash::SETTINGS.get_default("enable-local-plugin-development")
+
   # Config Settings
   option ["-f", "--path.config"], "CONFIG_PATH",
     I18n.t("logstash.runner.flag.config"),