Switch to gradle build system

[rjernst]

We currently have a very large maven build system, comprising of 6k lines of xml. While maven parent POMs have been very useful in de-duplicating build logic across elasticsearch and plugins, the rigidity of maven has required us to begin implementing most of our test logic through maven-antrun-plugin. Another major problem with multi-module builds is trap: building a module requires explicitly listing an upstream module in order to get a local build of the dependency, or building from a higher level module (which implicitly builds all sub-modules). And finally, the maven model is incapable of decoupling test dependencies from main dependencies, in that tests for module A, cannot depend on module B, which in turn depends on module A (this is not a cycle).

I have a branch which contains elasticsearch switched to using gradle:
https://github.com/elastic/elasticsearch/tree/burn_maven_with_fire

This is a tracking issue for completing the migration to gradle. The tasks are separated into two groups: those remaining required before pushing to master (as to not completely break development), and those required for the migration to be complete, which must happen before the next release (tentatively 2.2).

Required for master push:

• Maven publishing
• Windows integ test support
• Setup publishing of buildSrc (ie ES gradle build plugins)
• Publish "attach project plugin" (allows dynamically including a multi-project build inside another build)

Required for release:

• Copyright header check
• Vagrant qa tests
• Multi node qa tests (Get multi node smoke tests working #14921)
• Rpm/deb building complete
• Rpm signing
• Distribution tests
• Update plugins not in elasticsearch repo to use ES gradle build plugins (mapper attachment: Build: Switch mapper attachments to gradle elasticsearch-mapper-attachments#184)
• Document new ES gradle build plugins (for ES plugin authors)
• Document building/testing in README/TESTING (this is done except for jacoco reports)
• integ tests parity: Maven to gradle Integ test parity #14464

Addtional nice to haves:

• Make qa plugins work in eclipse (need to configure custom classpath)
• Add run task for elasticsearch and plugins to replace run.sh/run.bat: fixing run.bat and run.sh #14451

[nik9000]

Document building/testing in README/TESTING

You should probably add some of this to the pre-merge checklist to. Maybe just the some basic build instructions and a big note saying "![under construction][underconstruction.gif] All references to maven in this file are obsolete and will be replaced with gradle soon. We're very sorry for the inconvenience. ![under construction][underconstruction.gif]"

Document building/testing in README/TESTING
Rpm/deb building complete
Distribution tests
Vagrant qa tests
Rpm signing

I'm cool with grabbing these later this week. If anyone that wants them before I get to them then please take them with my best wishes.

Document new ES gradle build plugins (for ES plugin authors)

I'm cool with grabbing this later, maybe porting github.com/wikimedia/search-highlighter or something to prove it out.

[rjernst]

I published a snapshot of the project-attachment-plugin. See the repo here:
https://github.com/elastic/gradle-project-attachment-plugin
And the snapshot is here:
https://oss.sonatype.org/content/repositories/snapshots/org/elasticsearch/gradle/project-attachment-plugin/1.0.0-SNAPSHOT/

[dadoonet]

I was wondering if we can contribute this plugin to Gradle as it sounds like to me a useful one for Gradle community. Is it doable at some point (when stable enough)?

[rjernst]

The gradle folks are working on a longer term solution for combining multiple multi-project builds. This is just a semi-temporary solution.

[dadoonet]

The gradle folks are working on a longer term solution for combining multiple multi-project builds.

Great news!

[nik9000]

I'm cool with grabbing these later this week. If anyone that wants them before I get to them then please take them with my best wishes.

And now that I'm finally done with @Test I can think of helping again! Hurray!

[mrsolo]

I would add 'ability to do third party license analysis' as part of release requirement.
currently 3rd party uses license maven plugin
with 'mvn -P license site' call

gradle seems to have one

[rjernst]

@mrsolo afaik, we only use the maven license plugin to check files have a license header?

[mrsolo]

@rjernst we use http://code.mycila.com/license-maven-plugin/ to check the license header which is a different license plug in than http://www.mojohaus.org/license-maven-plugin/

we use depends on maven license plugin to do third party library license auditing

[mrsolo]

Hmm .. since 2.x, the licenses are managed in https://github.com/elastic/elasticsearch/tree/2.1/distribution/licenses and some if not all libraries are not shaded, this pretty much invalidate pre 2.x auditing workflo, gradle 3rd party dependencies therefore is nice to have but not necessary.

Auditing will be

1. Cross check content of distribution *.zip against information under distribution/licenses
2. Generate information for external consumption

[rjernst]

#1 is already done in the port of the license checker (perl script on master). Not sure what you mean by #2

[mrsolo]

#2 basically mean generation of google spreadsheet, currently not automated

[nik9000]

Ok - I'm going to look at the vagrant tests.

[nik9000]

Ok - I'm going to look at the vagrant tests.

I've got the vagrant tests working as well as I can without rpm and deb packages so I'm going to start work on fixing the deb package.

[nik9000]

except for instructions for vagrant tests and jacoco reports

Ok - I'm going to see about these.

[clintongormley]

This is done, bar the docs for plugin authors (#15280). Closing

[agentgt]

@rjernst I'm curious did you not know about <relativePath> in Maven. I looked at the previous versions of the project you don't include <relativePath> which makes me think your entire hate for Maven could have stemmed from not knowing this feature.

From maven: https://maven.apache.org/pom.html#Inheritance

"Notice the relativePath element. It is not required, but may be used as a signifier to Maven to first search the path given for this project's parent, before searching the local and then remote repositories."

Thus if you had put the relative path to your parent pom in your submodules maven would not have to go "download" the internet.

I only bring this up because I have people point to your gradle video about how the maven reactor won't let them build a part of the project without downloading other modules and thus fundamentally flawed.

EDIT: Oh I thought your parent was in a non standard directory. I guess you mean it goes and downloads snapshot pom files. There are ways around that but yes that is the reactor behavior.