Use Nebula version plugins

[rdblue]

This replaces gradle-consistent-versions with nebula.dependency-recommender and nebula.dependency-lock. The purpose of this change is to enable having separate Spark 2.x and Spark 3.x modules in the build. An empty spark3 project is included.

The dependency recommender plugin is used to get versions from versions.props or the per-project dependency lock files.

The dependency lock plugin is used to lock versions. Locks are now generated in the build/ folders using ./gradlew generateLock, and used in the build after running ./gradlew saveLock. The JSON lock files are added in this commit. These are large because they include transitive information and a set of dependencies per configuration.

This also fixes a problem where JMH dependencies needed to be declared in the compile configuration. Now JMH dependencies can use the jmh configuration.

[rdblue]

@jerryshao, you may be interested in this change because it allows multiple Spark versions in the build. @massdosage, this should help fix your dependency issues as well.

@mccheah, you may be interested in reviewing this because it updates how dependencies are locked.

[massdosage]

OK, this does indeed look interesting. So we should be able to use this to specify a different version of Guava for a certain subproject? We'll give it a go on a branch we're working on to demonstrate the Guava version issue for Hive tests. Thanks for pointing this out.

[rdblue]

Looks like Jackson 2.10.2 breaks Spark 2.4.4:

org.apache.iceberg.spark.TestSparkDataFile > testValueConversion FAILED
    java.lang.ExceptionInInitializerError
        at org.apache.spark.SparkContext.withScope(SparkContext.scala:699)
        at org.apache.spark.SparkContext.parallelize(SparkContext.scala:716)
        at org.apache.spark.api.java.JavaSparkContext.parallelize(JavaSparkContext.scala:134)
        at org.apache.spark.api.java.JavaSparkContext.parallelize(JavaSparkContext.scala:146)
        at org.apache.iceberg.spark.TestSparkDataFile.checkSparkDataFile(TestSparkDataFile.java:149)
        at org.apache.iceberg.spark.TestSparkDataFile.testValueConversion(TestSparkDataFile.java:130)
        Caused by:
        com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.10.2
            at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)
            at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
            at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:808)
            at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
            at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
            ... 6 more

I don't think that this plugin locks transitive dependencies by default, so we will probably need to turn that on to fix this.

[jzhuge]

+1 LGTM
We have used Nebula in our internal fork at Netflix for more than 6 months.

[massdosage]

I tried this out in a branch we have which demonstrates the version conflicts we were getting with Hive and Guava (https://github.com/ExpediaGroup/iceberg/tree/add-hiverunner-test) and it works!

[aokolnychyi]

+1

[rdblue]

Thanks for reviewing, everyone! Good to have this in to unblock the Hive and Spark 3 work!