Enable building Apache Beam using Gradle #4096

lukecwik · 2017-11-08T01:24:33Z

Follow this checklist to help us incorporate your contribution quickly and easily:

Make sure there is a JIRA issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
Each commit in the pull request should have a meaningful subject line and body.
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
Run mvn clean verify to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

Things that work:

compiling Java/Python/Go code
generating source from protos
generating source from avro
running rat, checkstyle, findbugs, jacoco
building docker containers
running Python tests
shading Java

Partially working:

running Java tests (~99% pass, hcatalog, hbase, hadoop input format tests are having dependency problems because of shaded/unshaded dependencies)
generating maven pom (albeit with wrong dependencies for some subprojects)

Things that aren't done:

running @NeedsRunner/@ValidatesRunner/integration tests

kennknowles

Awesome. I looked most closely at the global bits and then skimmed about 30 of the individual bits. Lots of questions, and mostly I think it deserves a ton of commenting so everyone can see exactly what is going on.

Even though I have lots of suggestions here, our maven configs are 10x the size and super crufty also, so as far as the choice between them, this is already way more clear and maintainable.

At sdks/java/core/src/test/avro/user.avsc:2:

  "namespace": "org.apache.beam.sdk.io",

I see how src/test/avro is better for this than src/test/java but should keep the namespace directories IMO.

kennknowles · 2017-11-08T04:29:52Z

build_rules.gradle

+ext.applyGrpcNature = {
+  println "applyGrpcNature with " + (it ? "$it" : "default configuration") + " for project $project.name"
+  apply plugin: "com.google.protobuf"
+  protobuf {


What do these methods do, exactly?

kennknowles · 2017-11-08T04:29:52Z

build_rules.gradle

+    }
+
+    // Create a new configuration 'shadowTest' like 'shadow' for the test scope
+    configurations {


This configuration stuff does make me think it would be ideal to have a single shaded dep in its own module, so there's none of this spooky configuration at a distance of all projects in really nontrivial ways.

kennknowles · 2017-11-08T04:29:52Z

build_rules.gradle

+    }
+  }
+
+  if (configuration.enableFindbugs) {


Why if? Can it be always on and be a separate target? It would be preferable to manage all aspects of what runs by specifying targets. I know in Gradle these are "tasks" but aren't those about the same?

Applying a plugin typically adds configuration sets, enables tasks to execute. I have an if here to prevent the plugin from applying for a few modules (google-cloud-dataflow-java, google-cloud-platform, kafka, model/*) mirroring the Maven configuration.

Gradle doesn't follow the Maven plugin style where there is a skip flag so either you enable the plugin or you don't. Some plugins have the ability to not fail the build if they fail.

kennknowles · 2017-11-08T04:29:52Z

build_rules.gradle

+    maxErrors = 0
+  }
+
+  apply plugin: 'com.diffplug.gradle.spotless'


Here, too, comments about what plugin is being loaded, why, how it is configured.

kennknowles · 2017-11-08T04:29:52Z

build_rules.gradle

+    useJUnit { }
+  }
+
+  apply plugin: "net.ltgt.apt"


These two lines are close to each other - are they linked by groovy syntax or is it just coincidence?

Not certain what your asking, please clarify

kennknowles · 2017-11-08T04:29:52Z

runners/spark/build.gradle

+  // Only one SparkContext may be running in a JVM (SPARK-2243)
+  forkEvery 1
+  useJUnit {
+    excludeCategories "org.apache.beam.runners.spark.UsesCheckpointRecovery"


Where's the bit where it scans tests from sdks/java/core?

There are no validates runner tests configured and wanted to make all validates runner tests their own test sets independent of the normal unit tests. This will allow us to have different dependency configurations for the different test sets.

kennknowles · 2017-11-08T04:29:52Z

build_rules.gradle

+  jcenter()
+
+  // Release staging repository
+  maven { url "https://oss.sonatype.org/content/repositories/staging/" }


Should these have names? Or how does the build distinguish which ones to use for which purpose? Etc.

No, I believe they are harmless and will only become useful once we start staging artifacts.

kennknowles · 2017-11-08T04:29:53Z

build_rules.gradle

+repositories {
+  mavenLocal()
+  mavenCentral()
+  jcenter()


What is jcenter?

kennknowles · 2017-11-08T04:29:53Z

build_rules.gradle

+
+// Define the default set of properties
+repositories {
+  mavenLocal()


Ah, so here is where the actual dep repos are configured, and where I commented in the other place was just where to find Gradle plugin deps?

kennknowles · 2017-11-08T04:29:53Z

build_rules.gradle

+ * limitations under the License.
+ */
+
+println "Applying build_rules.gradle to $project.name"


So this runs in multiple projects and by this line you can tell when it hits each of them?

This is to be able to follow during evaluation/configuration what is being applied to what project and when.

This specifically says when a project "imports" build_rules.gradle".
There are others in the applyYYYNature() methods which state when a large configuration is being applied with what settings.

kennknowles · 2017-11-08T04:29:54Z

runners/google-cloud-dataflow-java/build.gradle

+  shadow library.java.google_auth_library_oauth2_http
+  shadow library.java.bigdataoss_util
+  shadow library.java.avro
+  shadow library.java.joda_time


Is this shading way more than we do today? Or is this not shaded and shadow means something else?

Its the opposite of what you would expect since the shadow plugin conveniently by default decides to bundle everything instead of bundling nothing. Marking something with shadow or the equivalent test one means you want to keep it. Anything in the runtime/testRuntime configurations is meant to be shaded and removed from the final artifact.

kennknowles · 2017-11-08T04:29:55Z

examples/java/build.gradle

+  shadow library.java.google_http_client
+  shadow library.java.bigdataoss_util
+  shadow library.java.google_auth_library_oauth2_http
+  shadow library.java.google_auth_library_credentials


On the surface, this shadow configuration seems way way better than maven-shade-plugin, where we have to configure things twice.

kennknowles · 2017-11-08T04:29:55Z

model/fn-execution/build.gradle

+ * limitations under the License.
+ */
+
+apply from: project(":").file("build_rules.gradle")


It seems like we are not required to blanket apply the root build_rules.gradle. It might be cleaner to break it up into things that really are across all modules and separately things that are not. For example, checkstyle, findbugs, etc should not even by loaded for building Python, Go, etc. So those configurations would go in a root level java_base.gradle or some such.

That is what build_rules.gradle does already.
It contains configuration that applies to everything and then has a few closures which apply additional configuration like applyJavaNature/applyGrpcNature which projects can optionally call.

kennknowles · 2017-11-08T04:29:55Z

model/pipeline/build.gradle

+
+apply from: project(":").file("build_rules.gradle")
+applyJavaNature(enableFindbugs: false)
+applyGrpcNature()


Where are the source files specified? Are they implied? How do we configure the output? Currently, our protos are actually put in the wrong place in the jar, really, and we should namespace them. How can we do that? And can we pull in protos from one module to another without having to pull in a jar? We should be able to build three dependent proto modules all to Python without touching any Java technology.

All great questions, I'm not familiar enough with the proto plugin to know how it does everything but the default configuration + gRPC did what we wanted it to do.

kennknowles · 2017-11-08T04:29:56Z

build.gradle

+    classpath "com.diffplug.spotless:spotless-plugin-gradle:3.6.0"
+    classpath "gradle.plugin.com.github.blindpirate:gogradle:0.7.0"
+    classpath "gradle.plugin.com.palantir.gradle.docker:gradle-docker:0.13.0"
+    classpath "cz.malohlava:visteg:1.0.3"


I think a few of them are not self-documenting, like this one. It wouldn't hurt to comment each one separately, even obvious ones like protobuf-gradle-plugin.

kennknowles · 2017-11-08T04:29:56Z

build.gradle

+    maven { url "https://plugins.gradle.org/m2/" }
+    maven { url "http://repo.spring.io/plugins-release" }
+  }
+  dependencies {


These are just dependencies for running the build, right?

kennknowles · 2017-11-08T04:29:56Z

build.gradle

+  ],
+]
+
+buildscript {


Since many people will not have used Gradle, and even those that have will edit builds less often than their main code, can you signpost the significance of sections and commands? Especially commenting on any that affect whole-project things, like this maven config looks like it does.

I commented build_rules.gradle heavily for parts that I had a lot of knowledge in and gave some thought to what made sense.

Also some parts of build.gradle.

kennknowles · 2017-11-08T04:29:56Z

build.gradle

+
+apply from: project(":").file("build_rules.gradle")
+
+def google_cloud_bigdataoss_version = "1.4.5"


Could use some documentation about why some are pulled out and some are inlined.

kennknowles · 2017-11-08T04:29:56Z

build.gradle

+    "**/build/**",
+    "**/vendor/**",
+
+    // .gitignore: Ignore files generated by the Maven build process


Incidentally, the plugin might already check .gitignore. It was my mistake thinking that the Maven plugin didn't.

I believe improving this minor point can happen in the future.

kennknowles · 2017-11-08T04:29:56Z

build.gradle

+    classpath "com.google.protobuf:protobuf-gradle-plugin:0.8.1"
+    classpath "io.spring.gradle:propdeps-plugin:0.0.9.RELEASE"
+    classpath "gradle.plugin.org.nosphere.apache:creadur-rat-gradle:0.3.1"
+    classpath "com.commercehub.gradle.plugin:gradle-avro-plugin:0.11.0"


Can we isolate plugin dependencies to just the modules that use them?

That would lead to an inconsistent experience because the dependencies declared with a project’s buildscript() method are available to the build scripts of all its sub-projects and we have a couple of places where go/python/docker intermix (beam-sdks-python-container is a subproject of beam-sdks-python). So I feel that we should load all the plugins everytime so that every project has access to the same set irrespective of where they are in the three.

kennknowles · 2017-11-08T04:29:57Z

settings.gradle

+project(':beam-runners-parent:beam-runners-reference-parent:beam-runners-reference-job-orchestrator').projectDir = "$rootDir/runners/reference/job-server" as File
+project(':beam-runners-parent:beam-runners-reference-parent').projectDir = "$rootDir/runners/reference" as File
+project(':beam-runners-parent:beam-runners-direct-java').projectDir = "$rootDir/runners/direct-java" as File
+project(':beam-runners-parent:beam-runners-flink_2.10').projectDir = "$rootDir/runners/flink" as File


The fact that we have to set this looks pretty suspicious. Probably it means we are just reaching into other project directories when we should depend on declared outputs?

To my knowledge setting the projectDir is how you tell gradle where subprojects are located.

kennknowles · 2017-11-08T04:29:58Z

sdks/java/core/src/main/resources/org/apache/beam/sdk/sdk.properties

@@ -16,7 +16,7 @@
 #
 # SDK source version

-version=${pom.version}
+version=@pom.version@


Ditto here that we might not make this change unless maven and gradle both support the same syntax. I'm sure we could find a way. But maybe this just works for both?

Most of our other filter plugin changes that we have done have migrated to using @ as the separator instead of $, see the archetypes.

kennknowles · 2017-11-08T04:29:58Z

...-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties

-legacy.environment.major.version=${dataflow.legacy_environment_major_version}
-fnapi.environment.major.version=${dataflow.fnapi_environment_major_version}
-container.version=${dataflow.container_version}
+[email protected]_environment_major_version@


This change will make it so mvn and gradle cannot coexist?

Maven supports $ and @ syntax. Most of our project actually uses @, see the archetypes.

This brings these parts inline and enables them to be used without additional configuration of the gradle filter plugin.

kennknowles · 2017-11-08T04:29:59Z

gradlew

+
+##############################################################################
+##
+##  Gradle start up script for UN*X


Why do we have this or need it?

Its the recommended way for users to use the build without needing to install the version of gradle that this build expects (4.2.1). Its also the recommended approach when using Jenkins. Gradle will automatically download its own version and execute itself via the gradlew script. Ditto for the .bat file for windows users.

For developers who only develop on this project, they can install Gradle themselves but having this script removes the need for them to do so.

kennknowles

At sdks/java/core/src/test/avro/user.avsc:2:

  "namespace": "org.apache.beam.sdk.io",

I see how src/test/avro is better for this than src/test/java but should keep the namespace directories IMO.

kennknowles · 2017-11-08T04:31:13Z

runners/spark/build.gradle

+  // Only one SparkContext may be running in a JVM (SPARK-2243)
+  forkEvery 1
+  useJUnit {
+    excludeCategories "org.apache.beam.runners.spark.UsesCheckpointRecovery"


Where's the bit where it scans tests from sdks/java/core?

kennknowles · 2017-11-08T04:31:13Z

build_rules.gradle

+
+  apply plugin: "net.ltgt.apt"
+  dependencies {
+    def auto_value = "com.google.auto.value:auto-value:1.5.1"


These don't go in the other toplevel file?

kennknowles · 2017-11-08T04:31:13Z

build_rules.gradle

+    useJUnit { }
+  }
+
+  apply plugin: "net.ltgt.apt"


These two lines are close to each other - are they linked by groovy syntax or is it just coincidence?

kennknowles · 2017-11-08T04:31:13Z

build_rules.gradle

+    maxErrors = 0
+  }
+
+  apply plugin: 'com.diffplug.gradle.spotless'


Here, too, comments about what plugin is being loaded, why, how it is configured.

kennknowles · 2017-11-08T04:31:14Z

build_rules.gradle

+    }
+  }
+
+  if (configuration.enableFindbugs) {


Why if? Can it be always on and be a separate target? It would be preferable to manage all aspects of what runs by specifying targets. I know in Gradle these are "tasks" but aren't those about the same?

kennknowles · 2017-11-08T04:31:14Z

build_rules.gradle

+ * limitations under the License.
+ */
+
+println "Applying build_rules.gradle to $project.name"


So this runs in multiple projects and by this line you can tell when it hits each of them?

kennknowles · 2017-11-08T04:31:14Z

build_rules.gradle

+  jcenter()
+
+  // Release staging repository
+  maven { url "https://oss.sonatype.org/content/repositories/staging/" }


Should these have names? Or how does the build distinguish which ones to use for which purpose? Etc.

kennknowles · 2017-11-08T04:31:14Z

build_rules.gradle

+repositories {
+  mavenLocal()
+  mavenCentral()
+  jcenter()


What is jcenter?

kennknowles · 2017-11-08T04:31:14Z

build_rules.gradle

+  apply plugin: "java"
+
+  sourceCompatibility = configuration.javaVersion
+  targetCompatibility = configuration.javaVersion


I know that there exist third party tools to compile lambdas to Java 7 bytecode. Do they perhaps have easy gradle plugins so you can just say sourceCompatibility=1.8 targetCompatibility=1.7? Might be a sneaky way to get started.

kennknowles · 2017-11-08T04:31:14Z

build_rules.gradle

+/*************************************************************************************************/
+
+/** Used as configuration within the applyJavaNature closure. */
+class JavaNatureConfiguration {


To be clear, this is our own POJO and the fields are just for us to go ahead and use? Or are these magic fields that some plugin looks at?

kennknowles · 2017-11-08T04:31:15Z

runners/google-cloud-dataflow-java/build.gradle

+  shadow library.java.google_auth_library_oauth2_http
+  shadow library.java.bigdataoss_util
+  shadow library.java.avro
+  shadow library.java.joda_time


Is this shading way more than we do today? Or is this not shaded and shadow means something else?

kennknowles · 2017-11-08T04:31:16Z

examples/java/build.gradle

+  shadow library.java.google_http_client
+  shadow library.java.bigdataoss_util
+  shadow library.java.google_auth_library_oauth2_http
+  shadow library.java.google_auth_library_credentials


On the surface, this shadow configuration seems way way better than maven-shade-plugin, where we have to configure things twice.

kennknowles · 2017-11-08T04:31:16Z

model/fn-execution/build.gradle

+ * limitations under the License.
+ */
+
+apply from: project(":").file("build_rules.gradle")


It seems like we are not required to blanket apply the root build_rules.gradle. It might be cleaner to break it up into things that really are across all modules and separately things that are not. For example, checkstyle, findbugs, etc should not even by loaded for building Python, Go, etc. So those configurations would go in a root level java_base.gradle or some such.

kennknowles · 2017-11-08T04:31:16Z

model/pipeline/build.gradle

+
+apply from: project(":").file("build_rules.gradle")
+applyJavaNature(enableFindbugs: false)
+applyGrpcNature()


Where are the source files specified? Are they implied? How do we configure the output? Currently, our protos are actually put in the wrong place in the jar, really, and we should namespace them. How can we do that? And can we pull in protos from one module to another without having to pull in a jar? We should be able to build three dependent proto modules all to Python without touching any Java technology.

kennknowles · 2017-11-08T04:31:17Z

build.gradle

+
+apply from: project(":").file("build_rules.gradle")
+
+def google_cloud_bigdataoss_version = "1.4.5"


Could use some documentation about why some are pulled out and some are inlined.

kennknowles · 2017-11-08T04:31:17Z

build.gradle

+  ],
+]
+
+buildscript {


Since many people will not have used Gradle, and even those that have will edit builds less often than their main code, can you signpost the significance of sections and commands? Especially commenting on any that affect whole-project things, like this maven config looks like it does.

kennknowles · 2017-11-08T04:31:17Z

build.gradle

+    classpath "com.diffplug.spotless:spotless-plugin-gradle:3.6.0"
+    classpath "gradle.plugin.com.github.blindpirate:gogradle:0.7.0"
+    classpath "gradle.plugin.com.palantir.gradle.docker:gradle-docker:0.13.0"
+    classpath "cz.malohlava:visteg:1.0.3"


I think a few of them are not self-documenting, like this one. It wouldn't hurt to comment each one separately, even obvious ones like protobuf-gradle-plugin.

kennknowles · 2017-11-08T04:31:17Z

build.gradle

+    maven { url "https://plugins.gradle.org/m2/" }
+    maven { url "http://repo.spring.io/plugins-release" }
+  }
+  dependencies {


These are just dependencies for running the build, right?

kennknowles · 2017-11-08T04:31:17Z

build.gradle

+    "**/build/**",
+    "**/vendor/**",
+
+    // .gitignore: Ignore files generated by the Maven build process


Incidentally, the plugin might already check .gitignore. It was my mistake thinking that the Maven plugin didn't.

kennknowles · 2017-11-08T04:31:17Z

build.gradle

+    classpath "com.google.protobuf:protobuf-gradle-plugin:0.8.1"
+    classpath "io.spring.gradle:propdeps-plugin:0.0.9.RELEASE"
+    classpath "gradle.plugin.org.nosphere.apache:creadur-rat-gradle:0.3.1"
+    classpath "com.commercehub.gradle.plugin:gradle-avro-plugin:0.11.0"


Can we isolate plugin dependencies to just the modules that use them?

kennknowles · 2017-11-08T04:31:18Z

settings.gradle

+project(':beam-runners-parent:beam-runners-reference-parent:beam-runners-reference-job-orchestrator').projectDir = "$rootDir/runners/reference/job-server" as File
+project(':beam-runners-parent:beam-runners-reference-parent').projectDir = "$rootDir/runners/reference" as File
+project(':beam-runners-parent:beam-runners-direct-java').projectDir = "$rootDir/runners/direct-java" as File
+project(':beam-runners-parent:beam-runners-flink_2.10').projectDir = "$rootDir/runners/flink" as File


The fact that we have to set this looks pretty suspicious. Probably it means we are just reaching into other project directories when we should depend on declared outputs?

kennknowles · 2017-11-08T04:31:18Z

sdks/java/core/src/main/resources/org/apache/beam/sdk/sdk.properties

@@ -16,7 +16,7 @@
 #
 # SDK source version

-version=${pom.version}
+version=@pom.version@


Ditto here that we might not make this change unless maven and gradle both support the same syntax. I'm sure we could find a way. But maybe this just works for both?

Are there tests of what this is expected to be?

kennknowles · 2017-11-08T04:31:19Z

...-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties

-legacy.environment.major.version=${dataflow.legacy_environment_major_version}
-fnapi.environment.major.version=${dataflow.fnapi_environment_major_version}
-container.version=${dataflow.container_version}
+[email protected]_environment_major_version@


This change will make it so mvn and gradle cannot coexist?

kennknowles · 2017-11-08T04:31:19Z

gradlew

+
+##############################################################################
+##
+##  Gradle start up script for UN*X


Why do we have this or need it?

jbonofre · 2017-11-08T08:30:58Z

Do we have an equivalent to the enforcer plugin ? By the way, I'm testing the build right now.

jbonofre · 2017-11-08T08:38:17Z

FYI, I have a build failure with Gradle 4.3:

FAILURE: Build failed with an exception.

* Where:
Script '/home/jbonofre/Workspace/beam/build_rules.gradle' line: 87

due to:

Caused by: java.lang.NoClassDefFoundError: Could not initialize class net.ltgt.gradle.apt.CompatibilityUtils
        at net.ltgt.gradle.apt.AptPlugin$5.execute(AptPlugin.java:177)
        at net.ltgt.gradle.apt.AptPlugin$5.execute(AptPlugin.java:173)
        at org.gradle.internal.Actions$FilteredAction.execute(Actions.java:205)
       ....

jbonofre · 2017-11-08T08:44:24Z

Gradle 4.2.x fails on rat and python:

FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':rat'.
> Found 905 files with unapproved/unknown licenses. See file:/home/jbonofre/Workspace/beam/build/reports/rat/rat-report.txt

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/home/jbonofre/Workspace/beam/sdks/python/build.gradle' line: 64

* What went wrong:
Execution failed for task ':beam-sdks-parent:beam-sdks-python:lint'.
> Process 'command 'tox'' finished with non-zero exit value 1

On the other hand, I will take a look for the Gradle 4.3 support.

iemejia · 2017-11-08T14:12:57Z

build.gradle

+    maven { url "http://repo.spring.io/plugins-release" }
+  }
+  dependencies {
+    classpath "net.ltgt.gradle:gradle-apt-plugin:0.12"


This one need to be version 0.13 or it breaks with the latest version of gradle (4.3)

lukecwik · 2017-11-08T17:23:42Z

@iemejia @jbonofre
I believe users/developers should either install the specific version of gradle (4.2.1) that this build uses or should rely on using the gradle wrapper script. I would like to defer upgrading to gradle 4.3 to another PR.

@jbonofre
For the part of maven enforcer which prevents certain dependencies, I just ban them globally here:
https://github.com/lukecwik/incubator-beam/blob/8ca3fdca276a74973cd428d7321b995f7b7ebccf/build_rules.gradle#L172
I found http://www.mojohaus.org/extra-enforcer-rules/dependency-info.html but have yet to configure it.

lukecwik · 2017-11-08T19:26:48Z

All builds passed:
https://builds.apache.org/job/beam_PreCommit_Go_MavenInstall/631/
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15565/
https://builds.apache.org/job/beam_PreCommit_Python_MavenInstall/717/

Following changes are only going to be for Gradle build files so we know that the current set of changes don't break the Maven based build.

kennknowles · 2017-11-08T20:48:42Z

OK. The only changes I could imagine messing with maven would be the move from ${property} to @property@. I think we should have some more discussion and review but I'm pretty excited to try it out. Once it is in we can try setting up some additional presubmits that use it, hopefully seeing how much the caching can help for modules that are not touched.

kennknowles · 2017-11-08T20:49:27Z

run dataflow validatesrunner

lukecwik · 2017-11-09T00:09:18Z

We have some basic unit tests for the property rules in: https://github.com/apache/beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ReleaseInfoTest.java

lukecwik · 2017-11-09T00:09:46Z

Dataflow validates runner tests also passed:
https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/4310/

lukecwik · 2017-11-09T22:49:02Z

PTAL

lukecwik · 2017-11-13T17:08:26Z

R: @kennknowles

jbonofre · 2017-11-14T09:47:12Z

Testing the latest changes. Thanks !

kennknowles · 2017-11-17T17:18:06Z

LGTM but I think there have been a couple things that changed since the tests last ran.

Resolve a spurious error that gogradle was not adding a task dependency between resolving build dependencies and building the dependency

…ependencies that changed

…cts.

… that Dockerfile expects

…ts to use it. Add shadowTest configuration for sdks/java/core updating all other projects to use it.

…way guava

Update repackaged locations to always use org.apache.beam.${project.name}.repackaged as prefix

Fix whitespace formatting.

…ally.

…th any other go project Fix up evaluation message.

Fix Spark runners tests by excluding slf4j-jdk14 from test runtime classpath

…c/test/avro

Conflicts: build_rules.gradle

This was working in the past because http://creadur.apache.org/rat/apache-rat-plugin/check-mojo.html#useIdeaDefaultExcludes is true.

…aster branch up to f10399d

…extensions/sql

This closes #4096

lukecwik · 2017-11-17T22:30:04Z

I got this merged and updated the build changes so the gradle build is upto sync with all pom changes up to current master.

It currently passes on my desktop.

asfgit · 2017-11-17T23:34:15Z

SUCCESS

--none--

kennknowles reviewed Nov 8, 2017

View reviewed changes

iemejia reviewed Nov 8, 2017

View reviewed changes

lukecwik added 23 commits November 17, 2017 14:27

Add a task dependency visualization plugin.

cd713b1

Resolve a spurious error that gogradle was not adding a task dependency between resolving build dependencies and building the dependency

Fix classpath bundling issue caused by Hive and its dependencies

6bd364c

Define Tika dependencies version as variable

d83520b

Rebase on apache/master and add new subprojects, update versions of d…

a5d3407

…ependencies that changed

Skip test failures in hadoop related modules due to dependency confli…

54c57a1

…cts.

Add Jacoco plugin

83cfc07

Add Gradle builds for gcemd and gcsproxy

9c9d39b

Add Java SDK Container Gradle build file

c03d0c6

Create uber jar of the Java SDK harness and copy it into the location…

7fe8943

… that Dockerfile expects

Make docker tasks correctly depend on required build tasks

8578628

Fix runners/gcp/* go package paths

45c4db6

Add shadow configuration for sdks/java/core updating all other projec…

e4951a2

…ts to use it. Add shadowTest configuration for sdks/java/core updating all other projects to use it.

Configure default shading profile on all java sub-projects to shade a…

ac9bf03

…way guava

Shade direct-java and sql projects.

4fcee85

Update repackaged locations to always use org.apache.beam.${project.name}.repackaged as prefix

Fix gcemd and gcsproxy builds.

17da0e0

Fix whitespace formatting.

Enforce that gogradle dependency resolution/installation is done seri…

3e567c2

…ally.

Also ensure that resolving test dependencies are done sequentially wi…

2cce2dd

…th any other go project Fix up evaluation message.

Use valid repackage path on shadow relocations

89dc9b5

Fix Spark runners tests by excluding slf4j-jdk14 from test runtime classpath

Move user.avsc to directory which corresponds with namespace under sr…

fd1f5db

…c/test/avro

Add comments to various parts of build_rules.gradle

6067fc9

Conflicts: build_rules.gradle

Fix excludes to correctly match the .idea/ directory.

788d958

This was working in the past because http://creadur.apache.org/rat/apache-rat-plugin/check-mojo.html#useIdeaDefaultExcludes is true.

Update build.gradle files to correspond with pom.xml changes within m…

aff736f

…aster branch up to f10399d

Enable FFMP -> JavaCC path to generate Beam SQL parser for sdks/java/…

6f55ccd

…extensions/sql

lukecwik force-pushed the gradle branch from a53e074 to 6f55ccd Compare November 17, 2017 22:28

asfgit merged commit 6f55ccd into apache:master Nov 17, 2017

asfgit pushed a commit that referenced this pull request Nov 17, 2017

Enable building Apache Beam using Gradle

50e58f0

This closes #4096

swegner mentioned this pull request Oct 22, 2018

[BEAM-5804] remove javacc bad option warning #6768

Merged

2 tasks

kennknowles mentioned this pull request Nov 14, 2022

[Feature Request]: fastjson & fastjson2 dependencies #24154

Closed


		apply from: project(":").file("build_rules.gradle")

		def google_cloud_bigdataoss_version = "1.4.5"

Enable building Apache Beam using Gradle #4096

Enable building Apache Beam using Gradle #4096

Conversation

lukecwik commented Nov 8, 2017 • edited Loading

kennknowles left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kennknowles left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukecwik commented Nov 8, 2017 •

edited

Loading