Upgrade from 6.2.4 to 6.4.2 - unable to use elasticsearch-plugin command to remove x-pack plugin after upgrade

[danleerunk]

Elasticsearch version (bin/elasticsearch --version): 6.2.4 upgrading to 6.4.2

Plugins installed: [xpack]

JVM version (java -version): openjdk version "1.8.0_171"

OS version (uname -a if on a Unix-like system): CentOS 3.10.0-693.11.6.el7.x86_64

Description of the problem including expected versus actual behavior:
When upgrading from versions of Elasticsearch prior to 6.3 to 6.3 or higher, the x-pack plugin must be removed. We advise running bin/elasticsearch-plugin remove x-pack after the upgrade, but before restarting the node, which worked in 6.3 to remove the plugin. However, when upgrading to 6.4, attempting this command results in the error mentioned at the bottom of this issue.

Expected: bin/elasticsearch-plugin remove x-pack removes the x-pack plugin after upgrade from 6.2.4 to 6.4.2
Actual: the command fails with the error I've included below

Steps to reproduce:

1. Install ES 6.2.4, then install the x-pack plugin
2. Stop ES and perform an upgrade to ES 6.4.2 (rpm -Uvh)
3. Attempt to run bin/elasticsearch-plugin remove x-pack from the ES_HOME directory

Provide logs (if relevant):

[root@localhost elasticsearch]# bin/elasticsearch-plugin remove x-pack
Exception in thread "main" java.lang.IllegalStateException: Could not load plugin descriptor for plugin directory [x-pack]
	at org.elasticsearch.plugins.PluginsService.readPluginBundle(PluginsService.java:394)
	at org.elasticsearch.plugins.PluginsService.findBundles(PluginsService.java:379)
	at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:372)
	at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:89)
	at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:67)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
	at org.elasticsearch.cli.Command.main(Command.java:90)
	at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/plugins/x-pack/plugin-descriptor.properties
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
	at java.nio.file.Files.newByteChannel(Files.java:361)
	at java.nio.file.Files.newByteChannel(Files.java:407)
	at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
	at java.nio.file.Files.newInputStream(Files.java:152)
	at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:162)
	at org.elasticsearch.plugins.PluginsService.readPluginBundle(PluginsService.java:391)
	... 10 more

Notes:
I've marked as a bug, but upon further thought - we might just change the doc to advise removing the plugin prior to actually upgrading ES? I don't know if this has potential risk should the upgrade fail/etc. Easiest example of the doc is the upgrade guide

[elasticmachine]

Pinging @elastic/es-core-infra

[jasontedor]

This is a result of the fact that we structured the x-pack plugin in a certain way (as a meta-plugin) in 6.2.0 and with the migration of X-Pack to the default distribution in 6.3.0, we took advantage of the opportunity to simplify the code base by removing the notion of meta-plugins in 6.4.0. This means that the plugin tool does not understand how to read a meta-plugin packaged for 6.2.0, an oversight. We are not going to add the meta-plugin infrastructure back solely so that the plugin tool can understand how to remove such a plugin.

Instead, I think that we should indeed document this as a known issue and advise removing the plugin before performing the upgrade. Since Elasticsearch will refuse to start here (because the on-disk x-pack plugin will be in a format that Elasticsearch does not understand) there is no risk of performing a downgrade to use the 6.2.4 plugin tool to remove x-pack, and then proceeding with the upgrade again.

[jasontedor]

@nik9000 Would you address this as a documentation issue, adding this as a known issue to the 6.4.x breaking changes and release notes?

[nik9000]

I backported #32016 to 6.x and 6.4. I rewrote it somewhat and moved it to right before you upgrade the node (master, 6.x, and 6.4). Then I added breaking changes and release notes (6.4).