From 414ce4ea90ed5765dac5793a6dd0e89d1b7fb858 Mon Sep 17 00:00:00 2001 From: Andy Bristol Date: Wed, 1 Nov 2017 15:08:21 -0700 Subject: [PATCH] [TEST] portable packaging test project + windows This creates projects for new packaging tests written in groovy that can run on linux and windows, and adds awareness of windows boxes to the gradle build and Vagrantfile. No tests have been ported here, the groovy tests just exit immediately. Also creates the project :qa:packaging-common for code that should be shared between tests, e.g. the kind of content that lives in the utils/ portion of the bats tests. This is a separate project so that extra plugins' projects may depend on it without conflicting with the tests' project. The default is to not test on any windows boxes unless explicitly specified (we can't provide windows boxes) to gradle via project properties or to vagrant via environment variables. Once the bats tests in :qa:vagrant have all been migrated, that project can be deleted and the vagrant test plugin applied to :qa:packaging directly, making it serve as both the groovy build for the tests, and the tasks that setup the VMs the tests run on. For #26741 --- TESTING.asciidoc | 147 +++-- Vagrantfile | 520 ++++++++++++------ Vagrantfile.alternate.rb | 462 ++++++++++++++++ .../vagrant/VagrantPropertiesExtension.groovy | 14 +- .../gradle/vagrant/VagrantTestPlugin.groovy | 318 ++++++++--- qa/packaging-common/build.gradle | 22 + qa/packaging/build.gradle | 30 + .../packaging/PackagingMain.groovy | 27 + qa/vagrant/build.gradle | 8 +- settings.gradle | 2 + 10 files changed, 1268 insertions(+), 282 deletions(-) create mode 100644 Vagrantfile.alternate.rb create mode 100644 qa/packaging-common/build.gradle create mode 100644 qa/packaging/build.gradle create mode 100644 qa/packaging/src/main/groovy/org/elasticsearch/packaging/PackagingMain.groovy diff --git a/TESTING.asciidoc b/TESTING.asciidoc index 1cb05792b8298..54a447316f6f9 100644 --- a/TESTING.asciidoc +++ b/TESTING.asciidoc @@ -302,15 +302,22 @@ comma separated list of nodes to connect to (e.g. localhost:9300). A transport c be created based on that and used for all the before|after test operations, and to extract the http addresses of the nodes so that REST requests can be sent to them. -== Testing scripts +== Testing distributions and packaging -The simplest way to test scripts and the packaged distributions is to use -Vagrant. You can get started by following there five easy steps: +The packaging tests use Vagrant virtual machines to verify that installing +and running elasticsearch distributions works correctly on supported operating systems. +These tests should really only be run in vagrant vms because they're destructive. + +The existing packaging tests are written with https://github.com/sstephenson/bats[Bats] +and run on Linux boxes. These tests are being ported to a Groovy project that will +run on all boxes. + +To set up your environment and run the packaging tests, follow these steps: . Install Virtual Box and Vagrant. -. (Optional) Install vagrant-cachier to squeeze a bit more performance out of -the process: +. (Optional) Install https://github.com/fgrehm/vagrant-cachier[vagrant-cachier] to squeeze +a bit more performance out of the process: -------------------------------------- vagrant plugin install vagrant-cachier @@ -322,10 +329,9 @@ vagrant plugin install vagrant-cachier gradle :qa:vagrant:vagrantCheckVersion ------------------------------------- -. Download and smoke test the VMs with `gradle vagrantSmokeTest` or -`gradle -Pvagrant.boxes=all vagrantSmokeTest`. The first time you run this it will -download the base images and provision the boxes and immediately quit. If you -you this again it'll skip the download step. +. Download and smoke test the VMs with `gradle vagrantSmokeTest`. The first time +you run this it will download the base images and provision the boxes and immediately +quit. If run again it will use the boxes already downloaded. . Run the tests with `gradle packagingTest`. This will cause gradle to build the tar, zip, and deb packages and all the plugins. It will then run the tests @@ -343,7 +349,7 @@ All the regular vagrant commands should just work so you can get a shell in a VM running trusty by running `vagrant up ubuntu-1404 --provider virtualbox && vagrant ssh ubuntu-1404`. -These are the linux flavors the Vagrantfile currently supports: +These are the linux flavors supported, all of which have boxes provided * ubuntu-1404 aka trusty * ubuntu-1604 aka xenial @@ -363,12 +369,42 @@ quality boxes available in vagrant atlas: * sles-11 -We're missing the following because our tests are very linux/bash centric: +=== Testing packaging on Windows + +The packaging tests also support Windows Server 2012R2 and Windows Server 2016. +Unfortunately we're not able to provide boxes for them in open source use +because of licensing issues. Any Virtualbox image that has WinRM enabled +for remote management and supports Powershell for remote users should work. + +Testing on Windows requires the https://github.com/criteo/vagrant-winrm[vagrant-winrm] plugin. + +------------------------------------ +vagrant plugin install vagrant-winrm +------------------------------------ + +To specify the Windows boxes to use, pass the project properties to gradle: + +* `-Pvagrant.windows.2012r2.box` +* `-Pvagrant.windows.2016.box` + +These properties are required for Windows support in all gradle tasks that +handle packaging tests. Either or both may be specified. Remember that to run tests +on these boxes, they still need to be included in the value of `-Pvagrant.boxes`. +When these properties are present, passing `-Pvagrant.boxes=all` will include the +Windows boxes. You can also use `-Pvagrant.boxes=windows-all` to only test on Windows boxes. + +If running vagrant directly, pass the box names in the environment variables: -* Windows Server 2012 +* `VAGRANT_WINDOWS_2012R2_BOX` +* `VAGRANT_WINDOWS_2016_BOX` -It's important to think of VMs like cattle. If they become lame you just shoot -them and let vagrant reprovision them. Say you've hosed your precise VM: +=== Testing VMs are disposable + +If your testing VM gets into a bad state, just destroy it and let Vagrant +create a new one. It only takes a minute or two and Vagrant handles all +the provisioning for you. + +Let's say you've hosed your Ubuntu Precise VM: ---------------------------------------------------- vagrant ssh ubuntu-1404 -c 'sudo rm -rf /bin'; echo oops @@ -380,9 +416,6 @@ All you've got to do to get another one is vagrant destroy -f ubuntu-1404 && vagrant up ubuntu-1404 --provider virtualbox ---------------------------------------------- -The whole process takes a minute and a half on a modern laptop, two and a half -without vagrant-cachier. - Its possible that some downloads will fail and it'll be impossible to restart them. This is a bug in vagrant. See the instructions here for how to work around it: @@ -398,58 +431,72 @@ vagrant destroy -f `vagrant up` would normally start all the VMs but we've prevented that because that'd consume a ton of ram. -== Testing scripts more directly +=== Running packaging tests more directly -In general its best to stick to testing in vagrant because the bats scripts are -destructive. When working with a single package it's generally faster to run its -tests in a tighter loop than gradle provides. In one window: +Gradle's full packaging test build takes a while, but you can iterate faster +by managing the VM and running the tests yourself. + +To get the build ready for the packaging tests, run (on the host) -------------------------------- -gradle :distribution:rpm:assemble +gradle :qa:vagrant:setupPackaging :qa:vagrant:prepareGradleBuild -------------------------------- -and in another window: +and in another window, run the gradle task to bring up the VM. In this +example let's test on Centos 7 + +------------------------------------------------- +gradle :qa:vagrant:vagrantCentos7#up +------------------------------------------------- + +This task is available for all boxes specified with `-Pvagrant.boxes` - +to see what's available, run `gradle :qa:vagrant:tasks --all`. + +Then in another terminal, ssh into the VM + +---------------------------------------------------- +vagrant ssh centos-7 +---------------------------------------------------- + +To run the Bats tests for just the RPM package (on the guest) ---------------------------------------------------- -vagrant up centos-7 --provider virtualbox && vagrant ssh centos-7 cd $BATS_ARCHIVES sudo -E bats $BATS_TESTS/*rpm*.bats ---------------------------------------------------- -If you wanted to retest all the release artifacts on a single VM you could: +Or to retest all the release artifacts on a single VM (on the guest) ------------------------------------------------- -gradle setupBats -cd qa/vagrant; vagrant up ubuntu-1404 --provider virtualbox && vagrant ssh ubuntu-1404 cd $BATS_ARCHIVES sudo -E bats $BATS_TESTS/*.bats ------------------------------------------------- -You can also use Gradle to prepare the test environment and then starts a single VM: - -------------------------------------------------- -gradle vagrantFedora27#up -------------------------------------------------- - -Or any of vagrantCentos6#up, vagrantCentos7#up, vagrantDebian8#up, -vagrantDebian9#up, vagrantFedora26#up, vagrantFedora27#up, vagrantOel6#up, vagrantOel7#up, -vagrantOpensuse42#up,vagrantSles12#up, vagrantUbuntu1404#up, vagrantUbuntu1604#up. - -Once up, you can then connect to the VM using SSH from the elasticsearch directory: +To run the Groovy packaging tests, run (on the guest) -------------------------------------------------- -vagrant ssh fedora-27 -------------------------------------------------- - -Or from another directory: - -------------------------------------------------- -VAGRANT_CWD=/path/to/elasticsearch vagrant ssh fedora-27 -------------------------------------------------- +------------------------------------------------ +cd ~/.elasticsearch && gradle :qa:packaging:run +------------------------------------------------ -Note: Starting vagrant VM outside of the elasticsearch folder requires to -indicates the folder that contains the Vagrantfile using the VAGRANT_CWD -environment variable. +=== Testing plugins in packaging tests + +If you're building plugins in an `elasticsearch-extra` directory, the Vagrantfile +will mount that directory to the VM so it's included in the VM's gradle build. If +your plugin's build needs some additional setup steps before it runs, you can add +task dependencies to the `#prepareGradleBuild` tasks. For example + +[source, groovy] +---------------------------------------- +project(':qa:vagrant).tasks.findAll { t -> t.name.endsWith('prepareGradleBuild') }.each { task -> + Task myPluginSetupTask = project.tasks.create(task.replace('prepareGradleBuild, 'mySetup')) { + doLast { + String vagrantBox = task.ext.box + // vagrant command to setup box + } + } + task.dependsOn(myPluginSetupTask) +} +---------------------------------------- == Testing backwards compatibility diff --git a/Vagrantfile b/Vagrantfile index 021b4d630a1e6..531a376960914 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -21,167 +21,207 @@ # specific language governing permissions and limitations # under the License. +GRADLE_VERSION = '3.3' + +define_opts = { + autostart: false +}.freeze + Vagrant.configure(2) do |config| - config.vm.define "ubuntu-1404" do |config| - config.vm.box = "elastic/ubuntu-14.04-x86_64" - ubuntu_common config + + config.vm.provider 'virtualbox' do |vbox| + # Give the box more memory and cpu because our tests are beasts! + vbox.memory = Integer(ENV['VAGRANT_MEMORY'] || 8192) + vbox.cpus = Integer(ENV['VAGRANT_CPUS'] || 4) end - config.vm.define "ubuntu-1604" do |config| - config.vm.box = "elastic/ubuntu-16.04-x86_64" - ubuntu_common config, extra: <<-SHELL - # Install Jayatana so we can work around it being present. - [ -f /usr/share/java/jayatanaag.jar ] || install jayatana - SHELL + + # Vagrant and ruby stdlib expand '.' differently, so look for the build + # in the directory containing the Vagrantfile + vagrantfile_dir = File.expand_path('..', __FILE__) + + # Switch the default share for the project root from /vagrant to + # /elasticsearch because /vagrant is confusing when there is a project inside + # the elasticsearch project called vagrant.... + config.vm.synced_folder vagrantfile_dir, '/vagrant', disabled: true + config.vm.synced_folder vagrantfile_dir, '/elasticsearch' + + # If building with extra plugins, mount that too + extra_projects = "#{vagrantfile_dir}-extra" + if Dir.exists?(extra_projects) + config.vm.synced_folder extra_projects, '/elasticsearch-extra' + end + + # Expose project directory. Note that VAGRANT_CWD may not be the same as Dir.pwd + PROJECT_DIR = ENV['VAGRANT_PROJECT_DIR'] || Dir.pwd + config.vm.synced_folder PROJECT_DIR, '/project' + + 'ubuntu-1404'.tap do |box| + config.vm.define box, define_opts do |config| + config.vm.box = 'elastic/ubuntu-14.04-x86_64' + deb_common config, box + end + end + 'ubuntu-1604'.tap do |box| + config.vm.define box, define_opts do |config| + config.vm.box = 'elastic/ubuntu-16.04-x86_64' + deb_common config, box, extra: <<-SHELL + # Install Jayatana so we can work around it being present. + [ -f /usr/share/java/jayatanaag.jar ] || install jayatana + SHELL + end end # Wheezy's backports don't contain Openjdk 8 and the backflips # required to get the sun jdk on there just aren't worth it. We have # jessie and stretch for testing debian and it works fine. - config.vm.define "debian-8" do |config| - config.vm.box = "elastic/debian-8-x86_64" - deb_common config - end - config.vm.define "debian-9" do |config| - config.vm.box = "elastic/debian-9-x86_64" - deb_common config - end - config.vm.define "centos-6" do |config| - config.vm.box = "elastic/centos-6-x86_64" - rpm_common config + 'debian-8'.tap do |box| + config.vm.define box, define_opts do |config| + config.vm.box = 'elastic/debian-8-x86_64' + deb_common config, box + end end - config.vm.define "centos-7" do |config| - config.vm.box = "elastic/centos-7-x86_64" - rpm_common config + 'debian-9'.tap do |box| + config.vm.define box, define_opts do |config| + config.vm.box = 'elastic/debian-9-x86_64' + deb_common config, box + end end - config.vm.define "oel-6" do |config| - config.vm.box = "elastic/oraclelinux-6-x86_64" - rpm_common config + 'centos-6'.tap do |box| + config.vm.define box, define_opts do |config| + config.vm.box = 'elastic/centos-6-x86_64' + rpm_common config, box + end end - config.vm.define "oel-7" do |config| - config.vm.box = "elastic/oraclelinux-7-x86_64" - rpm_common config + 'centos-7'.tap do |box| + config.vm.define box, define_opts do |config| + config.vm.box = 'elastic/centos-7-x86_64' + rpm_common config, box + end end - config.vm.define "fedora-26" do |config| - config.vm.box = "elastic/fedora-26-x86_64" - dnf_common config + 'oel-6'.tap do |box| + config.vm.define box, define_opts do |config| + config.vm.box = 'elastic/oraclelinux-6-x86_64' + rpm_common config, box + end end - config.vm.define "fedora-27" do |config| - config.vm.box = "elastic/fedora-27-x86_64" - dnf_common config + 'oel-7'.tap do |box| + config.vm.define box, define_opts do |config| + config.vm.box = 'elastic/oraclelinux-7-x86_64' + rpm_common config, box + end end - config.vm.define "opensuse-42" do |config| - config.vm.box = "elastic/opensuse-42-x86_64" - opensuse_common config + 'fedora-26'.tap do |box| + config.vm.define box, define_opts do |config| + config.vm.box = 'elastic/fedora-26-x86_64' + dnf_common config, box + end end - config.vm.define "sles-12" do |config| - config.vm.box = "elastic/sles-12-x86_64" - sles_common config + 'fedora-27'.tap do |box| + config.vm.define box, define_opts do |config| + config.vm.box = 'elastic/fedora-27-x86_64' + dnf_common config, box + end end - # Switch the default share for the project root from /vagrant to - # /elasticsearch because /vagrant is confusing when there is a project inside - # the elasticsearch project called vagrant.... - config.vm.synced_folder ".", "/vagrant", disabled: true - config.vm.synced_folder ".", "/elasticsearch" - # Expose project directory - PROJECT_DIR = ENV['VAGRANT_PROJECT_DIR'] || Dir.pwd - config.vm.synced_folder PROJECT_DIR, "/project" - config.vm.provider "virtualbox" do |v| - # Give the boxes 3GB because Elasticsearch defaults to using 2GB - v.memory = 3072 + 'opensuse-42'.tap do |box| + config.vm.define box, define_opts do |config| + config.vm.box = 'elastic/opensuse-42-x86_64' + suse_common config, box + end end - if Vagrant.has_plugin?("vagrant-cachier") - config.cache.scope = :box + 'sles-12'.tap do |box| + config.vm.define box, define_opts do |config| + config.vm.box = 'elastic/sles-12-x86_64' + sles_common config, box + end end - config.vm.defined_vms.each do |name, config| - config.options[:autostart] = false - set_prompt = lambda do |config| - # Sets up a consistent prompt for all users. Or tries to. The VM might - # contain overrides for root and vagrant but this attempts to work around - # them by re-source-ing the standard prompt file. - config.vm.provision "prompt", type: "shell", inline: <<-SHELL - cat \<\ /etc/profile.d/elasticsearch_prompt.sh -export PS1='#{name}:\\w$ ' -PROMPT - grep 'source /etc/profile.d/elasticsearch_prompt.sh' ~/.bashrc | - cat \<\> ~/.bashrc -# Replace the standard prompt with a consistent one -source /etc/profile.d/elasticsearch_prompt.sh -SOURCE_PROMPT - grep 'source /etc/profile.d/elasticsearch_prompt.sh' ~vagrant/.bashrc | - cat \<\> ~vagrant/.bashrc -# Replace the standard prompt with a consistent one -source /etc/profile.d/elasticsearch_prompt.sh -SOURCE_PROMPT - SHELL - # Creates a file to mark the machine as created by vagrant. Tests check - # for this file and refuse to run if it is not present so that they can't - # be run unexpectedly. - config.vm.provision "markerfile", type: "shell", inline: <<-SHELL - touch /etc/is_vagrant_vm - SHELL + + windows_2012r2_box = ENV['VAGRANT_WINDOWS_2012R2_BOX'] + if windows_2012r2_box && windows_2012r2_box.empty? == false + 'windows-2012r2'.tap do |box| + config.vm.define box, define_opts do |config| + config.vm.box = windows_2012r2_box + windows_common config, box + end end - config.config_procs.push ['2', set_prompt] end -end -def ubuntu_common(config, extra: '') - deb_common config, extra: extra + windows_2016_box = ENV['VAGRANT_WINDOWS_2016_BOX'] + if windows_2016_box && windows_2016_box.empty? == false + 'windows-2016'.tap do |box| + config.vm.define box, define_opts do |config| + config.vm.box = windows_2016_box + config.vm.provision 'enable long paths', type: 'shell', inline: <<-SHELL + Set-ItemProperty -Path "HKLM:/SYSTEM/CurrentControlSet/Control/Filesystem/" -Name "LongPathsEnabled" -Value 1 + SHELL + windows_common config, box + end + end + end end -def deb_common(config, extra: '') +def deb_common(config, name, extra: '') # http://foo-o-rama.com/vagrant--stdin-is-not-a-tty--fix.html - config.vm.provision "fix-no-tty", type: "shell" do |s| + config.vm.provision 'fix-no-tty', type: 'shell' do |s| s.privileged = false s.inline = "sudo sed -i '/tty/!s/mesg n/tty -s \\&\\& mesg n/' /root/.profile" end - provision(config, - update_command: "apt-get update", - update_tracking_file: "/var/cache/apt/archives/last_update", - install_command: "apt-get install -y", - extra: extra) + linux_common( + config, + name, + update_command: 'apt-get update', + update_tracking_file: '/var/cache/apt/archives/last_update', + install_command: 'apt-get install -y', + extra: extra + ) end -def rpm_common(config) - provision(config, - update_command: "yum check-update", - update_tracking_file: "/var/cache/yum/last_update", - install_command: "yum install -y") +def rpm_common(config, name) + linux_common( + config, + name, + update_command: 'yum check-update', + update_tracking_file: '/var/cache/yum/last_update', + install_command: 'yum install -y' + ) end -def dnf_common(config) - provision(config, - update_command: "dnf check-update", - update_tracking_file: "/var/cache/dnf/last_update", - install_command: "dnf install -y", - install_command_retries: 5) - if Vagrant.has_plugin?("vagrant-cachier") - # Autodetect doesn't work.... +def dnf_common(config, name) + # Autodetect doesn't work.... + if Vagrant.has_plugin?('vagrant-cachier') config.cache.auto_detect = false - config.cache.enable :generic, { :cache_dir => "/var/cache/dnf" } + config.cache.enable :generic, { :cache_dir => '/var/cache/dnf' } end + linux_common( + config, + name, + update_command: 'dnf check-update', + update_tracking_file: '/var/cache/dnf/last_update', + install_command: 'dnf install -y', + install_command_retries: 5 + ) end -def opensuse_common(config) - suse_common config, '' -end - -def suse_common(config, extra) - provision(config, - update_command: "zypper --non-interactive list-updates", - update_tracking_file: "/var/cache/zypp/packages/last_update", - install_command: "zypper --non-interactive --quiet install --no-recommends", - extra: extra) +def suse_common(config, name, extra: '') + linux_common( + config, + name, + update_command: 'zypper --non-interactive list-updates', + update_tracking_file: '/var/cache/zypp/packages/last_update', + install_command: 'zypper --non-interactive --quiet install --no-recommends', + extra: extra + ) end -def sles_common(config) +def sles_common(config, name) extra = <<-SHELL zypper rr systemsmanagement_puppet puppetlabs-pc1 zypper --non-interactive install git-core -SHELL - suse_common config, extra + SHELL + suse_common config, name, extra: extra end -# Register the main box provisioning script. +# Configuration needed for all linux boxes # @param config Vagrant's config object. Required. +# @param name [String] The box name. Required. # @param update_command [String] The command used to update the package # manager. Required. Think `apt-get update`. # @param update_tracking_file [String] The location of the file tracking the @@ -189,27 +229,77 @@ end # is cached by vagrant-cachier. # @param install_command [String] The command used to install a package. # Required. Think `apt-get install #{package}`. -# @param extra [String] Extra provisioning commands run before anything else. -# Optional. Used for things like setting up the ppa for Java 8. -def provision(config, - update_command: 'required', - update_tracking_file: 'required', - install_command: 'required', - install_command_retries: 0, - extra: '') - # Vagrant run ruby 2.0.0 which doesn't have required named parameters.... - raise ArgumentError.new('update_command is required') if update_command == 'required' - raise ArgumentError.new('update_tracking_file is required') if update_tracking_file == 'required' - raise ArgumentError.new('install_command is required') if install_command == 'required' - config.vm.provider "virtualbox" do |v| - # Give the box more memory and cpu because our tests are beasts! - v.memory = Integer(ENV['VAGRANT_MEMORY'] || 8192) - v.cpus = Integer(ENV['VAGRANT_CPUS'] || 4) +# @param install_command_retries [Integer] Number of times to retry +# a failed install command +# @param extra [String] Additional script to run before installing +# dependencies +# +def linux_common(config, + name, + update_command: 'required', + update_tracking_file: 'required', + install_command: 'required', + install_command_retries: 0, + extra: '') + + raise ArgumentError, 'update_command is required' if update_command == 'required' + raise ArgumentError, 'update_tracking_file is required' if update_tracking_file == 'required' + raise ArgumentError, 'install_command is required' if install_command == 'required' + + if Vagrant.has_plugin?('vagrant-cachier') + config.cache.scope = :box end - config.vm.synced_folder "#{Dir.home}/.gradle/caches", "/home/vagrant/.gradle/caches", + + gradle_cache config, '/home/vagrant/.gradle/caches' + + config.vm.provision 'markerfile', type: 'shell', inline: <<-SHELL + touch /etc/is_vagrant_vm + SHELL + sh_set_prompt config, name + sh_install_deps( + config, + update_command, + update_tracking_file, + install_command, + install_command_retries, + extra + ) +end + +def gradle_cache(config, guest_path) + config.vm.synced_folder "#{Dir.home}/.gradle/caches", guest_path, create: true, - owner: "vagrant" - config.vm.provision "dependencies", type: "shell", inline: <<-SHELL + owner: 'vagrant' +end + +# Sets up a consistent prompt for all users. Or tries to. The VM might +# contain overrides for root and vagrant but this attempts to work around +# them by re-source-ing the standard prompt file. +def sh_set_prompt(config, name) + config.vm.provision 'set prompt', type: 'shell', inline: <<-SHELL + cat \<\ /etc/profile.d/elasticsearch_prompt.sh +export PS1='#{name}:\\w$ ' +PROMPT + grep 'source /etc/profile.d/elasticsearch_prompt.sh' ~/.bashrc | + cat \<\> ~/.bashrc +# Replace the standard prompt with a consistent one +source /etc/profile.d/elasticsearch_prompt.sh +SOURCE_PROMPT + grep 'source /etc/profile.d/elasticsearch_prompt.sh' ~vagrant/.bashrc | + cat \<\> ~vagrant/.bashrc +# Replace the standard prompt with a consistent one +source /etc/profile.d/elasticsearch_prompt.sh +SOURCE_PROMPT + SHELL +end + +def sh_install_deps(config, + update_command, + update_tracking_file, + install_command, + install_command_retries, + extra) + config.vm.provision 'install dependencies', type: 'shell', inline: <<-SHELL set -e set -o pipefail @@ -243,9 +333,9 @@ def provision(config, echo "==> Installing $1" if [ #{install_command_retries} -eq 0 ] then - #{install_command} $1 + #{install_command} $1 else - retry_installcommand $1 #{install_command_retries} + retry_installcommand $1 #{install_command_retries} fi } @@ -256,12 +346,13 @@ def provision(config, #{extra} installed java || { - echo "==> Java is not installed on vagrant box ${config.vm.box}" + echo "==> Java is not installed" return 1 } ensure tar ensure curl ensure unzip + ensure rsync installed bats || { # Bats lives in a git repository.... @@ -275,31 +366,22 @@ def provision(config, installed gradle || { echo "==> Installing Gradle" - curl -sS -o /tmp/gradle.zip -L https://services.gradle.org/distributions/gradle-3.3-bin.zip + curl -sS -o /tmp/gradle.zip -L https://services.gradle.org/distributions/gradle-#{GRADLE_VERSION}-bin.zip unzip -q /tmp/gradle.zip -d /opt rm -rf /tmp/gradle.zip - ln -s /opt/gradle-3.3/bin/gradle /usr/bin/gradle + ln -s /opt/gradle-#{GRADLE_VERSION}/bin/gradle /usr/bin/gradle # make nfs mounted gradle home dir writeable chown vagrant:vagrant /home/vagrant/.gradle } - cat \<\ /etc/profile.d/elasticsearch_vars.sh -export ZIP=/elasticsearch/distribution/zip/build/distributions -export TAR=/elasticsearch/distribution/tar/build/distributions -export RPM=/elasticsearch/distribution/rpm/build/distributions -export DEB=/elasticsearch/distribution/deb/build/distributions -export BATS=/project/build/bats -export BATS_UTILS=/project/build/bats/utils -export BATS_TESTS=/project/build/bats/tests -export BATS_ARCHIVES=/project/build/bats/archives -export GRADLE_HOME=/opt/gradle-3.3 +export BATS=/project/build/packaging/bats +export BATS_UTILS=/project/build/packaging/bats/utils +export BATS_TESTS=/project/build/packaging/bats/tests +export BATS_ARCHIVES=/project/build/packaging/archives +export GRADLE_HOME=/opt/gradle-#{GRADLE_VERSION} VARS cat \<\ /etc/sudoers.d/elasticsearch_vars -Defaults env_keep += "ZIP" -Defaults env_keep += "TAR" -Defaults env_keep += "RPM" -Defaults env_keep += "DEB" Defaults env_keep += "BATS" Defaults env_keep += "BATS_UTILS" Defaults env_keep += "BATS_TESTS" @@ -308,3 +390,125 @@ SUDOERS_VARS chmod 0440 /etc/sudoers.d/elasticsearch_vars SHELL end + +def windows_common(config, name) + gradle_cache config, '/Users/vagrant/.gradle/caches' + + config.vm.provision 'markerfile', type: 'shell', inline: <<-SHELL + $ErrorActionPreference = "Stop" + New-Item C:/is_vagrant_vm -ItemType file -Force | Out-Null + SHELL + + config.vm.provision 'set prompt', type: 'shell', inline: <<-SHELL + $ErrorActionPreference = "Stop" + $ps_prompt = 'function Prompt { "#{name}:$($ExecutionContext.SessionState.Path.CurrentLocation)>" }' + $ps_prompt | Out-File $PsHome/Microsoft.PowerShell_profile.ps1 + SHELL + + # Windows' system APIs limit paths to 260 characters. In server 2016 we can raise this limit, + # (see LongPathsEnabled) but not in server 2012r2. This adds a powershell module that wraps + # robocopy, which can handle paths longer than the system limit. The wrapper converts + # robocopy's unusual exit code and error handling to the behavior we'd normally expect + # + # https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx + # https://ss64.com/nt/robocopy-exit.html + config.vm.provision 'long path shim module', type: 'shell' do |s| + s.privileged = false + s.inline = <<-SHELL + $ErrorActionPreference = "Stop" + $longPathScript = @' +#{powershell_long_path_module} +'@ + $ModuleDir = "C:/Users/vagrant/Documents/WindowsPowerShell/Modules/LongPathShims" + $ModuleFile = "$ModuleDir/LongPathShims.psm1" + if (-Not (Test-Path "$ModuleDir")) { + New-Item "$ModuleDir" -ItemType Directory | Out-Null + } + $longPathScript | Out-File "$ModuleFile" + SHELL + end + + powershell_install_deps config +end + +def powershell_long_path_module + <<-SHELL + function Copy-Long-Path { + param( + [string]$Source, + [string]$Destination + ) + + robocopy "$Source" "$Destination" /E /COPY:DT /DCOPY:T /NFL /NDL /NJH /NJS /NC /NS /NP + Handle-Robocopy-Exit-Code $LASTEXITCODE + } + + function Remove-Long-Path { + param( + [string]$Target + ) + + $EmptyDir = "C:\\intentionally_empty" + + try { + New-Item "$EmptyDir" -ItemType Directory | Out-Null + # There doesn't appear to be a way to silence output about files that are purged, + # so just sent it to null as writing it to the terminal is very slow + robocopy "$EmptyDir" "$Target" /PURGE /NFL /NDL /NJH /NJS /NC /NS /NP | Out-Null + $RobocopyExitCode = $LASTEXITCODE + Remove-Item "$Target" -Recurse + Handle-Robocopy-Exit-Code $RobocopyExitCode + } finally { + Remove-Item "$EmptyDir" -Recurse + } + } + + function Handle-Robocopy-Exit-Code { + param( + [int]$ExitCode + ) + + Write-Verbose "robocopy returned exit code $ExitCode" + if ($ExitCode -ge 7) { + Write-Error "robocopy encountered an error and returned exit code $ExitCode" + } + } + SHELL +end + +def powershell_install_deps(config) + config.vm.provision 'install deps', type: 'shell', inline: <<-SHELL + $ErrorActionPreference = "Stop" + + function Installed { + Param( + [string]$command + ) + + try { + Get-Command $command + return $true + } catch { + return $false + } + } + + if (-Not (Installed java)) { + Write-Error "java is not installed" + } + + if (-Not (Installed gradle)) { + Write-Host "==> Installing gradle" + $Source="https://services.gradle.org/distributions/gradle-#{GRADLE_VERSION}-bin.zip" + $Zip="C:\\tmp\\gradle.zip" + $Destination="C:\\gradle" + New-Item (Split-Path $Zip) -ItemType Directory -ErrorAction Ignore | Out-Null + (New-Object Net.WebClient).DownloadFile($Source, $Zip) + Add-Type -assembly "System.IO.Compression.Filesystem" + [IO.Compression.ZipFile]::ExtractToDirectory($Zip, $Destination) + Remove-Item $Zip + [Environment]::SetEnvironmentVariable("Path", $env:Path + ";$Destination\\gradle-#{GRADLE_VERSION}\\bin", "Machine") + [Environment]::SetEnvironmentVariable("GRADLE_HOME", "$Destination\\gradle-#{GRADLE_VERSION}", "Machine") + } + SHELL +end diff --git a/Vagrantfile.alternate.rb b/Vagrantfile.alternate.rb new file mode 100644 index 0000000000000..3723b4505907e --- /dev/null +++ b/Vagrantfile.alternate.rb @@ -0,0 +1,462 @@ +# -*- mode: ruby -*- +# vi: set ft=ruby : + +# This Vagrantfile exists to test packaging. Read more about its use in the +# vagrant section in TESTING.asciidoc. + +# Licensed to Elasticsearch under one or more contributor +# license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright +# ownership. Elasticsearch licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +GRADLE_VERSION = '3.3' + +boxes = {} +linux_boxes = {} + +ubuntu_boxes = { + 'ubuntu-1404' => 'elastic/ubuntu-14.04-x86_64', + 'ubuntu-1604' => 'elastic/ubuntu-16.04-x86_64' +}.freeze +linux_boxes.merge!(ubuntu_boxes) + +# Wheezy's backports don't contain Openjdk 8 and the backflips +# required to get the sun jdk on there just aren't worth it. We have +# jessie and stretch for testing debian and it works fine. +debian_boxes = { + 'debian-8' => 'elastic/debian-8-x86_64', + 'debian-9' => 'elastic/debian-9-x86_64' +}.freeze +linux_boxes.merge!(debian_boxes) + +centos_boxes = { + 'centos-6' => 'elastic/centos-6-x86_64', + 'centos-7' => 'elastic/centos-7-x86_64' +}.freeze +linux_boxes.merge!(centos_boxes) + +oel_boxes = { + 'oel-6' => 'elastic/oraclelinux-6-x86_64', + 'oel-7' => 'elastic/oraclelinux-7-x86_64' +}.freeze +linux_boxes.merge!(oel_boxes) + +fedora_boxes = { + 'fedora-25' => 'elastic/fedora-25-x86_64', + 'fedora-26' => 'elastic/fedora-26-x86_64' +}.freeze +linux_boxes.merge!(fedora_boxes) + +opensuse_boxes = { + 'opensuse-42' => 'elastic/opensuse-42-x86_64' +}.freeze +linux_boxes.merge!(opensuse_boxes) + +sles_boxes = { + 'sles-12' => 'elastic/sles-12-x86_64' +}.freeze +linux_boxes.merge!(sles_boxes) + +linux_boxes.freeze +boxes.merge!(linux_boxes) + +windows_boxes = {} + +windows_2012r2_box = ENV['VAGRANT_WINDOWS_2012R2_BOX'] +if windows_2012r2_box && windows_2012r2_box.empty? == false + windows_boxes['windows-2012r2'] = windows_2012r2_box +end + +windows_2016_box = ENV['VAGRANT_WINDOWS_2016_BOX'] +if windows_2016_box && windows_2016_box.empty? == false + windows_boxes['windows-2016'] = windows_2016_box +end + +windows_boxes.freeze + +boxes = linux_boxes.merge(windows_boxes).freeze + +def gradle_cache(guest_path) + lambda do |config| + config.vm.synced_folder "#{Dir.home}/.gradle/caches", guest_path, + create: true, + owner: 'vagrant' + end +end + +def sh_set_prompt(name) + # Sets up a consistent prompt for all users. Or tries to. The VM might + # contain overrides for root and vagrant but this attempts to work around + # them by re-source-ing the standard prompt file. + lambda do |config| + config.vm.provision 'prompt sh', type: 'shell', inline: <<-SHELL + cat \<\ /etc/profile.d/elasticsearch_prompt.sh +export PS1='#{name}:\\w$ ' +PROMPT + grep 'source /etc/profile.d/elasticsearch_prompt.sh' ~/.bashrc | + cat \<\> ~/.bashrc +# Replace the standard prompt with a consistent one +source /etc/profile.d/elasticsearch_prompt.sh +SOURCE_PROMPT + grep 'source /etc/profile.d/elasticsearch_prompt.sh' ~vagrant/.bashrc | + cat \<\> ~vagrant/.bashrc +# Replace the standard prompt with a consistent one +source /etc/profile.d/elasticsearch_prompt.sh +SOURCE_PROMPT + SHELL + end +end + +configuration_procs = Hash.new { |hash, key| hash[key] = Array.new } + +# Share gradle cache +linux_boxes.keys.each do |name| + configuration_procs[name] << gradle_cache('/home/vagrant/.gradle/caches') +end + +windows_boxes.keys.each do |name| + configuration_procs[name] << gradle_cache('/Users/vagrant/.gradle/caches') +end + +# Workaround provisoning bug +# http://foo-o-rama.com/vagrant--stdin-is-not-a-tty--fix.html +(debian_boxes.keys + ubuntu_boxes.keys).each do |name| + configuration_procs[name] << lambda do |config| + config.vm.provision 'fix-no-tty', type: 'shell' do |s| + s.privileged = false + s.inline = "sudo sed -i '/tty/!s/mesg n/tty -s \\&\\& mesg n/' /root/.profile" + end + end +end + +# Create vagrant vm markerfile +linux_boxes.keys.each do |name| + configuration_procs[name] << lambda do |config| + config.vm.provision 'markerfile', type: 'shell', inline: <<-SHELL + touch /etc/is_vagrant_vm + SHELL + end +end + +windows_boxes.keys.each do |name| + configuration_procs[name] << lambda do |config| + config.vm.provision 'markerfile', type: 'shell', inline: <<-SHELL + New-Item \\is_vagrant_vm -ItemType file -Force | Out-Null + SHELL + end +end + +# Set prompt to be machine name +linux_boxes.keys.each do |name| + configuration_procs[name] << sh_set_prompt(name) + + configuration_procs[name] << lambda do |config| + if Vagrant.has_plugin?('vagrant-cachier') + config.cache.scope = :box + end + end +end + +windows_boxes.keys.each do |name| + configuration_procs[name] << lambda do |config| + config.vm.provision 'prompt powershell', type: 'shell', inline: <<-SHELL + $ErrorActionPreference = "Stop" + # set powershell prompt for all users + $ps_prompt = 'function Prompt { "#{name}:$($ExecutionContext.SessionState.Path.CurrentLocation)>" }' + $ps_prompt | Out-File $PsHome\\Microsoft.PowerShell_profile.ps1 + SHELL + end +end + + +# Install Jayatana so we can work around it being present. +configuration_procs['ubuntu-1604'] << lambda do |config| + config.vm.provision 'install jayatana', type: 'shell', inline: sh_install_apt( + '[ -f /usr/share/java/jayatanaag.jar ] || install jayatana' + ) +end +(debian_boxes.keys + ubuntu_boxes.keys).each do |name| + configuration_procs[name] << lambda do |config| + config.vm.provision 'install common deps (apt)', type: 'shell', inline: sh_install_apt(linux_deps) + end +end + +(centos_boxes.keys + oel_boxes.keys).each do |name| + configuration_procs[name] << lambda do |config| + config.vm.provision 'install common deps (rpm)', type: 'shell', inline: sh_install_rpm(linux_deps) + end +end + +fedora_boxes.keys.each do |name| + configuration_procs[name] << lambda do |config| + config.vm.provision 'install common deps (dnf)', type: 'shell', inline: sh_install_dnf(linux_deps) + if Vagrant.has_plugin?("vagrant-cachier") + # Autodetect doesn't work.... + config.cache.auto_detect = false + config.cache.enable :generic, { :cache_dir => "/var/cache/dnf" } + end + end +end + +sles_boxes.keys.each do |name| + configuration_procs[name] << lambda do |config| + config.vm.provision 'install puppet and git for sles', type: 'shell', inline: <<-SHELL + zypper rr systemsmanagement_puppet puppetlabs-pc1 + zypper --non-interactive install git-core + SHELL + end +end + +(sles_boxes.keys + opensuse_boxes.keys).each do |name| + configuration_procs[name] << lambda do |config| + config.vm.provision 'install common deps (zypper)', type: 'shell', inline: sh_install_zypper(linux_deps) + end +end + + +windows_boxes.keys.each do |name| + configuration_procs[name] << lambda do |config| + config.vm.provision 'install common deps', type: 'shell', inline: windows_deps + end +end + +Vagrant.configure(2) do |config| + + config.vm.provider 'virtualbox' do |vbox| + # Give the box more memory and cpu because our tests are beasts! + vbox.memory = Integer(ENV['VAGRANT_MEMORY'] || 8192) + vbox.cpus = Integer(ENV['VAGRANT_CPUS'] || 4) + end + + # Vagrant and ruby stdlib expand '.' differently, so look for the build + # in the directory containing the Vagrantfile + vagrantfile_dir = File.expand_path('..', __FILE__) + + # Switch the h share for the project root from /vagrant to + # /elasticsearch because /vagrant is confusing when there is a project inside + # the elasticsearch project called vagrant.... + config.vm.synced_folder vagrantfile_dir, '/vagrant', disabled: true + config.vm.synced_folder vagrantfile_dir, '/elasticsearch' + + extra_projects = "#{vagrantfile_dir}-extra" + if Dir.exists?(extra_projects) + config.vm.synced_folder extra_projects, '/elasticsearch-extra' + end + + # Expose project directory + PROJECT_DIR = ENV['VAGRANT_PROJECT_DIR'] || Dir.pwd + config.vm.synced_folder PROJECT_DIR, '/project' + + boxes.each do |name, image| + config.vm.define name, autostart: false do |config| + config.vm.box = image + configuration_procs[name].each do |configuration_proc| + configuration_proc.call(config) + end + end + end +end + + +def sh_install_apt(script) + sh_install( + script, + update_command: 'apt-get update', + update_tracking_file: '/var/cache/apt/archives/last_update', + install_command: 'apt-get install -y' + ) +end + +def sh_install_rpm(script) + sh_install( + script, + update_command: 'yum check-update', + update_tracking_file: '/var/cache/yum/last_update', + install_command: 'yum install -y' + ) +end + +def sh_install_dnf(script) + sh_install( + script, + update_command: 'dnf check-update', + update_tracking_file: '/var/cache/dnf/last_update', + install_command: 'dnf install -y', + install_command_retries: 5 + ) +end + +def sh_install_zypper(script) + sh_install( + script, + update_command: 'zypper --non-interactive list-updates', + update_tracking_file: '/var/cache/zypp/packages/last_update', + install_command: 'zypper --non-interactive --quiet install --no-recommends' + ) +end + +def linux_deps + <<-SHELL + installed java || { + echo "==> Java is not installed" + return 1 + } + ensure tar + ensure curl + ensure unzip + ensure rsync + + installed bats || { + # Bats lives in a git repository.... + ensure git + echo "==> Installing bats" + git clone https://github.com/sstephenson/bats /tmp/bats + # Centos doesn't add /usr/local/bin to the path.... + /tmp/bats/install.sh /usr + rm -rf /tmp/bats + } + + installed gradle || { + echo "==> Installing Gradle" + curl -sS -o /tmp/gradle.zip -L https://services.gradle.org/distributions/gradle-#{GRADLE_VERSION}-bin.zip + unzip -q /tmp/gradle.zip -d /opt + rm -rf /tmp/gradle.zip + ln -s /opt/gradle-#{GRADLE_VERSION}/bin/gradle /usr/bin/gradle + # make nfs mounted gradle home dir writeable + chown vagrant:vagrant /home/vagrant/.gradle + } + + cat \<\ /etc/profile.d/elasticsearch_vars.sh +export BATS=/project/build/packaging/bats +export BATS_UTILS=/project/build/packaging/bats/utils +export BATS_TESTS=/project/build/packaging/bats/tests +export BATS_ARCHIVES=/project/build/packaging/archives +export GRADLE_HOME=/opt/gradle-#{GRADLE_VERSION} +VARS + cat \<\ /etc/sudoers.d/elasticsearch_vars +Defaults env_keep += "BATS" +Defaults env_keep += "BATS_UTILS" +Defaults env_keep += "BATS_TESTS" +Defaults env_keep += "BATS_ARCHIVES" +SUDOERS_VARS + chmod 0440 /etc/sudoers.d/elasticsearch_vars + SHELL +end + +def windows_deps + <<-SHELL + $ErrorActionPreference = "Stop" + + function Installed { + Param( + [string]$command + ) + + try { + Get-Command $command + return $true + } catch { + return $false + } + } + + if (-Not (Installed java)) { + Write-Error "java is not installed" + } + + if (-Not (Installed gradle)) { + Write-Host "==> Installing gradle" + $Source="https://services.gradle.org/distributions/gradle-#{GRADLE_VERSION}-bin.zip" + $Zip="\\tmp\\gradle.zip" + $Destination="\\gradle" + New-Item (Split-Path $Zip) -Type Directory -ErrorAction Ignore | Out-Null + (New-Object Net.WebClient).DownloadFile($Source, $Zip) + Add-Type -assembly "System.IO.Compression.Filesystem" + [IO.Compression.ZipFile]::ExtractToDirectory($Zip, $Destination) + Remove-Item $Zip + [Environment]::SetEnvironmentVariable("Path", $env:Path + ";$Destination\\gradle-#{GRADLE_VERSION}\\bin", "Machine") + [Environment]::SetEnvironmentVariable("GRADLE_HOME", "$Destination\\gradle-#{GRADLE_VERSION}", "Machine") + } + SHELL +end + +# Build the script for installing build dependencies +# @param script [String] the script to run with install statements +# @param update_command [String] The command used to update the package +# manager. Required. Think `apt-get update`. +# @param update_tracking_file [String] The location of the file tracking the +# last time the update command was run. Required. Should be in a place that +# is cached by vagrant-cachier. +# @param install_command [String] The command used to install a package. +# Required. Think `apt-get install #{package}`. +# +def sh_install(script, + update_command: 'required', + update_tracking_file: 'required', + install_command: 'required', + install_command_retries: 0) + + raise ArgumentError, 'update_command is required' if update_command == 'required' + raise ArgumentError, 'update_tracking_file is required' if update_tracking_file == 'required' + raise ArgumentError, 'install_command is required' if install_command == 'required' + + command = '' + command << <<-SHELL + set -e + set -o pipefail + + # Retry install command up to $2 times, if failed + retry_installcommand() { + n=0 + while true; do + #{install_command} $1 && break + let n=n+1 + if [ $n -ge $2 ]; then + echo "==> Exhausted retries to install $1" + return 1 + fi + echo "==> Retrying installing $1, attempt $((n+1))" + # Add a small delay to increase chance of metalink providing updated list of mirrors + sleep 5 + done + } + + installed() { + command -v $1 2>&1 >/dev/null + } + + install() { + # Only apt-get update if we haven't in the last day + if [ ! -f #{update_tracking_file} ] || [ "x$(find #{update_tracking_file} -mtime +0)" == "x#{update_tracking_file}" ]; then + echo "==> Updating repository" + #{update_command} || true + touch #{update_tracking_file} + fi + echo "==> Installing $1" + if [ #{install_command_retries} -eq 0 ] + then + #{install_command} $1 + else + retry_installcommand $1 #{install_command_retries} + fi + } + + ensure() { + installed $1 || install $1 + } + SHELL + command << script + command +end diff --git a/buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantPropertiesExtension.groovy b/buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantPropertiesExtension.groovy index e6e7fca62f97e..4c7fab93b5d86 100644 --- a/buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantPropertiesExtension.groovy +++ b/buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantPropertiesExtension.groovy @@ -22,9 +22,16 @@ import org.gradle.api.tasks.Input class VagrantPropertiesExtension { + /** The boxes that we will actually run the tests on - not all boxes **/ @Input List boxes + @Input + Map vagrantEnvVars + + @Input + String testTask + @Input String upgradeFromVersion @@ -43,8 +50,7 @@ class VagrantPropertiesExtension { @Input Boolean inheritTestUtils - VagrantPropertiesExtension(List availableBoxes) { - this.boxes = availableBoxes + VagrantPropertiesExtension() { this.batsDir = 'src/test/resources/packaging' } @@ -52,6 +58,10 @@ class VagrantPropertiesExtension { this.boxes = Arrays.asList(boxes) } + void setTestTask(String testTask) { + this.testTask = testTask + } + void setBatsDir(String batsDir) { this.batsDir = batsDir } diff --git a/buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantTestPlugin.groovy b/buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantTestPlugin.groovy index 2510620ac712b..663752688b724 100644 --- a/buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantTestPlugin.groovy +++ b/buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantTestPlugin.groovy @@ -1,6 +1,5 @@ package org.elasticsearch.gradle.vagrant -import com.carrotsearch.gradle.junit4.RandomizedTestingPlugin import org.elasticsearch.gradle.FileContentsTask import org.gradle.api.* import org.gradle.api.artifacts.dsl.RepositoryHandler @@ -13,8 +12,8 @@ import org.gradle.api.tasks.TaskState class VagrantTestPlugin implements Plugin { - /** All available boxes **/ - static List BOXES = [ + /** All Linux boxes **/ + static List LINUX_BOXES = [ 'centos-6', 'centos-7', 'debian-8', @@ -29,6 +28,12 @@ class VagrantTestPlugin implements Plugin { 'ubuntu-1604' ] + /** Windows boxes we have images for and can build with **/ + static Map AVAILABLE_WINDOWS_BOXES = [:] + + /** All available boxes **/ + static List BOXES + /** Boxes used when sampling the tests **/ static List SAMPLE = [ 'centos-7', @@ -41,21 +46,29 @@ class VagrantTestPlugin implements Plugin { /** Packages onboarded for upgrade tests **/ static List UPGRADE_FROM_ARCHIVES = ['rpm', 'deb'] + static final PACKAGING_CONFIGURATION = 'packaging' + private static final BATS = 'bats' private static final String BATS_TEST_COMMAND ="cd \$BATS_ARCHIVES && sudo bats --tap \$BATS_TESTS/*.$BATS" - private static final String PLATFORM_TEST_COMMAND ="rm -rf ~/elasticsearch && rsync -r /elasticsearch/ ~/elasticsearch && cd ~/elasticsearch && \$GRADLE_HOME/bin/gradle test integTest" @Override void apply(Project project) { + collectAvailableBoxes(project) + // Creates the Vagrant extension for the project - project.extensions.create('esvagrant', VagrantPropertiesExtension, listVagrantBoxes(project)) + project.extensions.create('esvagrant', VagrantPropertiesExtension) + List boxesToRun = listSelectedVagrantBoxes(project) + assert BOXES.containsAll(boxesToRun) + project.extensions.esvagrant.boxes = boxesToRun + + project.extensions.esvagrant.vagrantEnvVars = collectVagrantEnvVars(project) // Add required repositories for Bats tests configureBatsRepositories(project) // Creates custom configurations for Bats testing files (and associated scripts and archives) - createBatsConfiguration(project) + createPackagingConfiguration(project) // Creates all the main Vagrant tasks createVagrantTasks(project) @@ -74,14 +87,54 @@ class VagrantTestPlugin implements Plugin { createVagrantBoxesTasks(project) } - private List listVagrantBoxes(Project project) { + private static void collectAvailableBoxes(Project project) { + String windows_2012r2_box = project.properties.get('vagrant.windows.2012r2.box', null) + if (windows_2012r2_box != null && windows_2012r2_box.isEmpty() == false) { + AVAILABLE_WINDOWS_BOXES['windows-2012r2'] = windows_2012r2_box + } + + String windows_2016_box = project.properties.get('vagrant.windows.2016.box', null) + if (windows_2016_box != null && windows_2016_box.isEmpty() == false) { + AVAILABLE_WINDOWS_BOXES['windows-2016'] = windows_2016_box + } + + BOXES = LINUX_BOXES + AVAILABLE_WINDOWS_BOXES.keySet() + project.ext.boxes = BOXES + } + + private static Map collectVagrantEnvVars(Project project) { + /* + * We always use the main project.rootDir as Vagrant's current working directory (VAGRANT_CWD) + * so that boxes are not duplicated for every Gradle project that use this VagrantTestPlugin. + */ + def vagrantEnvVars = [ + 'VAGRANT_CWD' : "${project.rootDir.absolutePath}", + 'VAGRANT_VAGRANTFILE' : 'Vagrantfile', + 'VAGRANT_PROJECT_DIR' : "${project.projectDir.absolutePath}" + ] + if ('windows-2012r2' in AVAILABLE_WINDOWS_BOXES) { + vagrantEnvVars['VAGRANT_WINDOWS_2012R2_BOX'] = AVAILABLE_WINDOWS_BOXES['windows-2012r2'] + } + if ('windows-2016' in AVAILABLE_WINDOWS_BOXES) { + vagrantEnvVars['VAGRANT_WINDOWS_2016_BOX'] = AVAILABLE_WINDOWS_BOXES['windows-2016'] + } + + return vagrantEnvVars + } + + private static List listSelectedVagrantBoxes(Project project) { String vagrantBoxes = project.getProperties().get('vagrant.boxes', 'sample') - if (vagrantBoxes == 'sample') { - return SAMPLE - } else if (vagrantBoxes == 'all') { - return BOXES - } else { - return vagrantBoxes.split(',') + switch (vagrantBoxes) { + case 'sample': + return SAMPLE + case 'linux-all': + return LINUX_BOXES + case 'windows-all': + return AVAILABLE_WINDOWS_BOXES.keySet().toList() + case 'all': + return BOXES + default: + return vagrantBoxes.split(',') } } @@ -100,10 +153,10 @@ class VagrantTestPlugin implements Plugin { } } - private static void createBatsConfiguration(Project project) { - project.configurations.create(BATS) + private static void createPackagingConfiguration(Project project) { + project.configurations.create(PACKAGING_CONFIGURATION) - String upgradeFromVersion = System.getProperty("tests.packaging.upgradeVersion"); + String upgradeFromVersion = System.getProperty("tests.packaging.upgradeVersion") if (upgradeFromVersion == null) { String firstPartOfSeed = project.rootProject.testSeed.tokenize(':').get(0) final long seed = Long.parseUnsignedLong(firstPartOfSeed, 16) @@ -113,12 +166,14 @@ class VagrantTestPlugin implements Plugin { DISTRIBUTION_ARCHIVES.each { // Adds a dependency for the current version - project.dependencies.add(BATS, project.dependencies.project(path: ":distribution:${it}", configuration: 'archives')) + project.dependencies.add(PACKAGING_CONFIGURATION, + project.dependencies.project(path: ":distribution:${it}", configuration: 'archives')) } UPGRADE_FROM_ARCHIVES.each { // The version of elasticsearch that we upgrade *from* - project.dependencies.add(BATS, "org.elasticsearch.distribution.${it}:elasticsearch:${upgradeFromVersion}@${it}") + project.dependencies.add(PACKAGING_CONFIGURATION, + "org.elasticsearch.distribution.${it}:elasticsearch:${upgradeFromVersion}@${it}") } project.extensions.esvagrant.upgradeFromVersion = upgradeFromVersion @@ -147,22 +202,23 @@ class VagrantTestPlugin implements Plugin { } private static void createPrepareVagrantTestEnvTask(Project project) { - File batsDir = new File("${project.buildDir}/${BATS}") + File packagingDir = new File("${project.buildDir}/${PACKAGING_CONFIGURATION}") + File batsDir = new File(packagingDir, BATS) - Task createBatsDirsTask = project.tasks.create('createBatsDirs') - createBatsDirsTask.outputs.dir batsDir - createBatsDirsTask.doLast { + Task createPackagingDirsTask = project.tasks.create('createBatsDirs') + createPackagingDirsTask.outputs.dir batsDir + createPackagingDirsTask.doLast { batsDir.mkdirs() } - Copy copyBatsArchives = project.tasks.create('copyBatsArchives', Copy) { - dependsOn createBatsDirsTask - into "${batsDir}/archives" - from project.configurations[BATS] + Copy copyArchives = project.tasks.create('copyPackagingArchives', Copy) { + dependsOn createPackagingDirsTask + into "${packagingDir}/archives" + from project.configurations[PACKAGING_CONFIGURATION] } Copy copyBatsTests = project.tasks.create('copyBatsTests', Copy) { - dependsOn createBatsDirsTask + dependsOn createPackagingDirsTask into "${batsDir}/tests" from { "${project.extensions.esvagrant.batsDir}/tests" @@ -170,7 +226,7 @@ class VagrantTestPlugin implements Plugin { } Copy copyBatsUtils = project.tasks.create('copyBatsUtils', Copy) { - dependsOn createBatsDirsTask + dependsOn createPackagingDirsTask into "${batsDir}/utils" from { "${project.extensions.esvagrant.batsDir}/utils" @@ -180,7 +236,7 @@ class VagrantTestPlugin implements Plugin { // Now we iterate over dependencies of the bats configuration. When a project dependency is found, // we bring back its own archives, test files or test utils. project.afterEvaluate { - project.configurations.bats.dependencies.findAll {it.targetConfiguration == BATS }.each { d -> + project.configurations.packaging.dependencies.findAll {it.targetConfiguration == PACKAGING_CONFIGURATION }.each { d -> if (d instanceof DefaultProjectDependency) { DefaultProjectDependency externalBatsDependency = (DefaultProjectDependency) d Project externalBatsProject = externalBatsDependency.dependencyProject @@ -189,9 +245,6 @@ class VagrantTestPlugin implements Plugin { if (project.extensions.esvagrant.inheritTests) { copyBatsTests.from(externalBatsProject.files("${externalBatsDir}/tests")) } - if (project.extensions.esvagrant.inheritTestArchives) { - copyBatsArchives.from(externalBatsDependency.projectConfiguration.files) - } if (project.extensions.esvagrant.inheritTestUtils) { copyBatsUtils.from(externalBatsProject.files("${externalBatsDir}/utils")) } @@ -200,20 +253,20 @@ class VagrantTestPlugin implements Plugin { } Task createVersionFile = project.tasks.create('createVersionFile', FileContentsTask) { - dependsOn createBatsDirsTask - file "${batsDir}/archives/version" + dependsOn createPackagingDirsTask + file "${packagingDir}/archives/version" contents project.version } Task createUpgradeFromFile = project.tasks.create('createUpgradeFromFile', FileContentsTask) { - dependsOn createBatsDirsTask - file "${batsDir}/archives/upgrade_from_version" + dependsOn createPackagingDirsTask + file "${packagingDir}/archives/upgrade_from_version" contents project.extensions.esvagrant.upgradeFromVersion } - Task vagrantSetUpTask = project.tasks.create('setupBats') + Task vagrantSetUpTask = project.tasks.create('setupPackaging') vagrantSetUpTask.dependsOn 'vagrantCheckVersion' - vagrantSetUpTask.dependsOn copyBatsTests, copyBatsUtils, copyBatsArchives, createVersionFile, createUpgradeFromFile + vagrantSetUpTask.dependsOn copyBatsTests, copyBatsUtils, copyArchives, createVersionFile, createUpgradeFromFile } private static void createPackagingTestTask(Project project) { @@ -251,6 +304,9 @@ class VagrantTestPlugin implements Plugin { private static void createVagrantBoxesTasks(Project project) { assert project.extensions.esvagrant.boxes != null + assert project.extensions.esvagrant.vagrantEnvVars != null + Map vagrantEnvVars = project.extensions.esvagrant.vagrantEnvVars + assert project.tasks.stop != null Task stop = project.tasks.stop @@ -263,8 +319,8 @@ class VagrantTestPlugin implements Plugin { assert project.tasks.virtualboxCheckVersion != null Task virtualboxCheckVersion = project.tasks.virtualboxCheckVersion - assert project.tasks.setupBats != null - Task setupBats = project.tasks.setupBats + assert project.tasks.setupPackaging != null + Task setupPackaging = project.tasks.setupPackaging assert project.tasks.packagingTest != null Task packagingTest = project.tasks.packagingTest @@ -272,25 +328,16 @@ class VagrantTestPlugin implements Plugin { assert project.tasks.platformTest != null Task platformTest = project.tasks.platformTest - /* - * We always use the main project.rootDir as Vagrant's current working directory (VAGRANT_CWD) - * so that boxes are not duplicated for every Gradle project that use this VagrantTestPlugin. - */ - def vagrantEnvVars = [ - 'VAGRANT_CWD' : "${project.rootDir.absolutePath}", - 'VAGRANT_VAGRANTFILE' : 'Vagrantfile', - 'VAGRANT_PROJECT_DIR' : "${project.projectDir.absolutePath}" - ] - // Each box gets it own set of tasks for (String box : BOXES) { - String boxTask = box.capitalize().replace('-', '') + String boxTask = box.capitalize().replace('-', '') // always add a halt task for all boxes, so clean makes sure they are all shutdown Task halt = project.tasks.create("vagrant${boxTask}#halt", VagrantCommandTask) { command 'halt' boxName box environmentVars vagrantEnvVars + description "Runs 'vagrant halt' for box ${box}" } stop.dependsOn(halt) @@ -300,8 +347,9 @@ class VagrantTestPlugin implements Plugin { boxName box environmentVars vagrantEnvVars dependsOn vagrantCheckVersion, virtualboxCheckVersion + description "Runs 'vagrant update' for box ${box}" } - update.mustRunAfter(setupBats) + update.mustRunAfter(setupPackaging) Task up = project.tasks.create("vagrant${boxTask}#up", VagrantCommandTask) { command 'up' @@ -321,58 +369,169 @@ class VagrantTestPlugin implements Plugin { /* It'd be possible to check if the box is already up here and output SKIPPED but that would require running vagrant status which is slow! */ dependsOn update + description "Runs 'vagrant up' for box ${box} (with provisioning)" } + // We use an Exec task here (instead of VagrantCommandTask) because we want the full output displayed Task smoke = project.tasks.create("vagrant${boxTask}#smoketest", Exec) { environment vagrantEnvVars dependsOn up finalizedBy halt - commandLine 'vagrant', 'ssh', box, '--command', - "set -o pipefail && echo 'Hello from ${project.path}' | sed -ue 's/^/ ${box}: /'" + description "Tries to run a 'hello world' shell command on box ${box}" + } + if (box in LINUX_BOXES) { + smoke.commandLine = ['vagrant', 'ssh', box, '--command', """ + set -euo pipefail + echo 'Hello from ${project.path}' | sed -ue 's/^/ ${box}: /' + """] + } else { + smoke.commandLine = ['vagrant', 'winrm', box, '--command', wrapPowershell(""" + Write-Host ' ${box}: Hello from ${project.path}' + """)] } vagrantSmokeTest.dependsOn(smoke) - Task packaging = project.tasks.create("vagrant${boxTask}#packagingTest", BatsOverVagrantTask) { - remoteCommand BATS_TEST_COMMAND + Task relocateProject = project.tasks.create("vagrant${boxTask}#relocateProject", VagrantCommandTask) { boxName box environmentVars vagrantEnvVars - dependsOn up, setupBats + dependsOn up finalizedBy halt + description "Run a command on box ${box} to move the project to vagrant's home directory" } + if (box in LINUX_BOXES) { + relocateProject.command = 'ssh' + relocateProject.args = ['--command', """ + set -euo pipefail + rm -rf ~/elasticsearch ~/elasticsearch-extra + rsync -rlt /elasticsearch/ ~/elasticsearch + if [ -d /elasticsearch-extra ]; then + rsync -rlt /elasticsearch-extra/ ~/elasticsearch-extra + fi + """] + } else { + relocateProject.command = 'winrm' + relocateProject.args = ['--command', wrapPowershell(""" + if (Test-Path "~/elasticsearch") { + Remove-Long-Path C:\\Users\\vagrant\\elasticsearch + } + if (Test-Path "~/elasticsearch-extra") { + Remove-Long-Path C:\\Users\\vagrant\\elasticsearch-extra + } - TaskExecutionAdapter packagingReproListener = new TaskExecutionAdapter() { + Copy-Long-Path C:\\elasticsearch C:\\Users\\vagrant\\elasticsearch + + if (Test-Path "C:/elasticsearch-extra") { + Copy-Long-Path C:\\elasticsearch-extra C:\\Users\\vagrant\\elasticsearch-extra + } + """)] + } + + /* + * Other projects can hook dependencies onto this task to get the vm ready to run the gradle build + */ + Task prepareGradle = project.tasks.create("vagrant${boxTask}#prepareGradleBuild") { + dependsOn relocateProject + } + prepareGradle.ext.box = box + + if (box in LINUX_BOXES) { + Task batsPackagingTest = project.tasks.create("vagrant${boxTask}#batsPackagingTest", BatsOverVagrantTask) { + remoteCommand BATS_TEST_COMMAND + boxName box + environmentVars vagrantEnvVars + dependsOn up, setupPackaging + finalizedBy halt + description "Runs the BATS packaging test scripts on box ${box}" + } + + TaskExecutionAdapter batsPackagingReproListener = new TaskExecutionAdapter() { + @Override + void afterExecute(Task task, TaskState state) { + if (state.failure != null) { + println "REPRODUCE WITH: gradle ${batsPackagingTest.path} " + + "-Dtests.seed=${project.testSeed} " + } + } + } + batsPackagingTest.doFirst { + project.gradle.addListener(batsPackagingReproListener) + } + batsPackagingTest.doLast { + project.gradle.removeListener(batsPackagingReproListener) + } + if (project.extensions.esvagrant.boxes.contains(box)) { + packagingTest.dependsOn(batsPackagingTest) + } + } + + Task groovyPackagingTest = project.tasks.create("vagrant${boxTask}#groovyPackagingTest", VagrantCommandTask) { + boxName box + environmentVars vagrantEnvVars + dependsOn up, setupPackaging, prepareGradle + finalizedBy halt + description "Runs the Groovy portable packaging tests on box ${box}" + } + if (box in LINUX_BOXES) { + groovyPackagingTest.command = 'ssh' + groovyPackagingTest.args = ['--command', """ + set -euo pipefail + cd ~/elasticsearch + \$GRADLE_HOME/bin/gradle ${-> project.extensions.esvagrant.testTask} -Dtests.seed=${-> project.testSeed} + """] + } else { + groovyPackagingTest.command = 'winrm' + groovyPackagingTest.args = ['--command', wrapPowershell(""" + cd ~/elasticsearch + & "\$Env:GRADLE_HOME/bin/gradle.bat" "${-> project.extensions.esvagrant.testTask}" "-Dtests.seed=${-> project.testSeed}" + """)] + } + + TaskExecutionAdapter groovyPackagingReproListener = new TaskExecutionAdapter() { @Override void afterExecute(Task task, TaskState state) { if (state.failure != null) { - println "REPRODUCE WITH: gradle ${packaging.path} " + - "-Dtests.seed=${project.testSeed} " + println "REPRODUCE WITH: gradle ${groovyPackagingTest.path} -Dtests.seed=${project.testSeed}" } } } - packaging.doFirst { - project.gradle.addListener(packagingReproListener) + + groovyPackagingTest.doFirst { + project.gradle.addListener(groovyPackagingReproListener) } - packaging.doLast { - project.gradle.removeListener(packagingReproListener) + groovyPackagingTest.doLast { + project.gradle.removeListener(groovyPackagingReproListener) } if (project.extensions.esvagrant.boxes.contains(box)) { - packagingTest.dependsOn(packaging) + packagingTest.dependsOn(groovyPackagingTest) } + Task platform = project.tasks.create("vagrant${boxTask}#platformTest", VagrantCommandTask) { - command 'ssh' boxName box environmentVars vagrantEnvVars - dependsOn up + dependsOn up, prepareGradle finalizedBy halt - args '--command', PLATFORM_TEST_COMMAND + " -Dtests.seed=${-> project.testSeed}" } + if (box in LINUX_BOXES) { + platform.command = 'ssh' + platform.args = ['--command', """ + set -euo pipefail + cd ~/elasticsearch + \$GRADLE_HOME/bin/gradle test integTest -Dtests.seed=${-> project.testSeed} + """] + } else { + platform.command = 'winrm' + platform.args = ['--command', wrapPowershell(""" + cd ~/elasticsearch + & "\$Env:GRADLE_HOME/bin/gradle.bat" "test" "integTest" "-Dtests.seed=${-> project.testSeed}" + """)] + } + TaskExecutionAdapter platformReproListener = new TaskExecutionAdapter() { @Override void afterExecute(Task task, TaskState state) { if (state.failure != null) { - println "REPRODUCE WITH: gradle ${platform.path} " + - "-Dtests.seed=${project.testSeed} " + println "REPRODUCE WITH: gradle ${platform.path} -Dtests.seed=${project.testSeed}" } } } @@ -387,4 +546,23 @@ class VagrantTestPlugin implements Plugin { } } } + + /* + * The library that vagrant uses to talk to WinRM [1] executes commands in such a way that causes commands to return a success exit + * code when $ErrorActionPreference = "Stop" is set and the script is terminated by an error. If we wrap our command inside a single + * call to the powershell executable, then failures cause that call to return a failure exit code, which is picked up by WinRB's + * wrapper script + * + * This returns GString and not String because several of the scripts that use closures that are lazily evaluated + * (of the form "foo ${-> bar}") to pull in configuration from the plugin's extension after the plugin has already been applied + * + * [1] https://github.com/WinRb/WinRM/blob/52918d73590449466332aaf06f69b0cf77d91dc7/lib/winrm/shells/power_shell.rb#L99-L115 + */ + static GString wrapPowershell(GString script) { + "powershell -Command { \$ErrorActionPreference = 'Stop'; ${-> script} }" + } + + static GString wrapPowershell(String script) { + wrapPowershell("${script}") + } } diff --git a/qa/packaging-common/build.gradle b/qa/packaging-common/build.gradle new file mode 100644 index 0000000000000..104a0741c6a7e --- /dev/null +++ b/qa/packaging-common/build.gradle @@ -0,0 +1,22 @@ +/* + * Licensed to Elasticsearch under one or more contributor + * license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright + * ownership. Elasticsearch licenses this file to you under + * the Apache License, Version 2.0 (the "License"); you may + * not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +apply plugin: 'groovy' + +description = 'Shared utilities for packaging tests' diff --git a/qa/packaging/build.gradle b/qa/packaging/build.gradle new file mode 100644 index 0000000000000..3e0e3d402c582 --- /dev/null +++ b/qa/packaging/build.gradle @@ -0,0 +1,30 @@ +/* + * Licensed to Elasticsearch under one or more contributor + * license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright + * ownership. Elasticsearch licenses this file to you under + * the Apache License, Version 2.0 (the "License"); you may + * not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +apply plugin: 'groovy' +apply plugin: 'application' + +description = 'Tests that packaging (distributions) and plugins can be installed and run' + +mainClassName = 'org.elasticsearch.packaging.PackagingMain' + +dependencies { + compile localGroovy() + compile project(':qa:packaging-common') +} diff --git a/qa/packaging/src/main/groovy/org/elasticsearch/packaging/PackagingMain.groovy b/qa/packaging/src/main/groovy/org/elasticsearch/packaging/PackagingMain.groovy new file mode 100644 index 0000000000000..4c3b39fdafb1f --- /dev/null +++ b/qa/packaging/src/main/groovy/org/elasticsearch/packaging/PackagingMain.groovy @@ -0,0 +1,27 @@ +/* + * Licensed to Elasticsearch under one or more contributor + * license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright + * ownership. Elasticsearch licenses this file to you under + * the Apache License, Version 2.0 (the "License"); you may + * not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.elasticsearch.packaging + +class PackagingMain { + + static void main(String[] args) { + println("hello world groovy packaging test") + } +} diff --git a/qa/vagrant/build.gradle b/qa/vagrant/build.gradle index 40fc3998185c8..abd19e2293325 100644 --- a/qa/vagrant/build.gradle +++ b/qa/vagrant/build.gradle @@ -20,19 +20,23 @@ apply plugin: 'elasticsearch.vagrantsupport' apply plugin: 'elasticsearch.vagrant' +esvagrant { + testTask ':qa:packaging:run' +} + List plugins = [] for (Project subproj : project.rootProject.subprojects) { if (subproj.path.startsWith(':plugins:')) { // add plugin as a dep dependencies { - bats project(path: "${subproj.path}", configuration: 'zip') + packaging project(path: "${subproj.path}", configuration: 'zip') } plugins.add(subproj.name) } } plugins = plugins.toSorted() -setupBats { +setupPackaging { doFirst { File expectedPlugins = file('build/plugins/expected') expectedPlugins.parentFile.mkdirs() diff --git a/settings.gradle b/settings.gradle index cdac374198b5e..863b3f9ce5f62 100644 --- a/settings.gradle +++ b/settings.gradle @@ -70,6 +70,8 @@ List projects = [ 'qa:mixed-cluster', 'qa:multi-cluster-search', 'qa:no-bootstrap-tests', + 'qa:packaging', + 'qa:packaging-common', 'qa:reindex-from-old', 'qa:rolling-upgrade', 'qa:smoke-test-client',