Set gradle build heap size to avoid build failure

[jun-he]

Set gradle build heap size to avoid build failure because JVM heap space is exhausted.
For #1846.

[jun-he]

This build failure is similar to #1817 (cc @pvary) with the error:

2020-11-29T08:18:49.5344738Z org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithCustomCatalog > testCreatePartitionedTable[fileFormat=ORC, engine=tez] FAILED
2020-11-29T08:18:49.5349046Z     java.lang.IllegalArgumentException: Failed to execute Hive query 'SELECT * FROM default.customers ORDER BY customer_id DESC': Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
2020-11-29T08:18:49.5350923Z 
2020-11-29T08:18:49.5351370Z         Caused by:
2020-11-29T08:18:49.5353221Z         org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask

and

2020-11-29T08:28:35.2356807Z org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithHiveCatalog > testScanTable[fileFormat=ORC, engine=tez] FAILED
2020-11-29T08:28:35.2361330Z     java.lang.IllegalArgumentException: Failed to execute Hive query 'SELECT first_name, customer_id FROM default.customers ORDER BY customer_id DESC': Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
2020-11-29T08:28:35.2363464Z 
2020-11-29T08:28:35.2363962Z         Caused by:
2020-11-29T08:28:35.2365843Z         org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask

[pvary]

@jun-he: How sure are we that this is the Heap size? Do we have a log for it (See: #1789 how to get the logs <ICEBERG_ROOT>/build/testlogs/)? The #1817 is about creating the Metrics RPC, which is fixed there.

Thanks,
Peter

[jun-he]

@pvary Thanks for the info to get the test log.
The error seems related to Metrics source already exists. I will rebase the PR. thanks.

- Test log for: Test testCreatePartitionedTable[fileFormat=ORC, engine=tez](org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithCustomCatalog)
--------
StdOut 2020-11-29T08:18:38,090 WARN  [pool-3263-thread-5] metastore.ObjectStore (ObjectStore.java:correctAutoStartMechanism(638)) - datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored
StdErr OKStdErr 
StdErr Query ID = runner_20201129081838_936fe330-7888-41a9-9219-c7fa5ef3a5b8StdErr 
StdErr Total jobs = 1StdErr 
StdErr Launching Job 1 out of 1StdErr 
StdOut 2020-11-29T08:18:38,604 INFO  [337376db-6c07-463a-a7cf-213c9d8ec809 Test worker] exec.Task (TezTask.java:execute(190)) - Subscribed to counters: [] for queryId: runner_20201129081838_936fe330-7888-41a9-9219-c7fa5ef3a5b8
StdOut 2020-11-29T08:18:38,605 INFO  [337376db-6c07-463a-a7cf-213c9d8ec809 Test worker] exec.Task (TezTask.java:ensureSessionHasResources(367)) - Tez session hasn't been created yet. Opening session
StdOut 2020-11-29T08:18:38,777 ERROR [ServiceThread:DAGClientRPCServer] client.DAGClientServer (DAGClientServer.java:serviceStart(99)) - Failed to start DAGClientServer: 
StdOut org.apache.hadoop.metrics2.MetricsException: Metrics source RpcActivityForPort41173 already exists!

- Test log for: Test testScanTable[fileFormat=ORC, engine=tez](org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithHiveCatalog)
--------
StdOut 2020-11-29T08:28:18,310 WARN  [pool-6496-thread-4] metastore.ObjectStore (ObjectStore.java:correctAutoStartMechanism(638)) - datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored
StdErr OKStdErr 
StdErr Query ID = runner_20201129082818_d6cdb374-f476-4a20-8593-48bbd4c859d8StdErr 
StdErr Total jobs = 1StdErr 
StdErr Launching Job 1 out of 1StdErr 
StdOut 2020-11-29T08:28:18,476 INFO  [4be58fef-bdce-46ea-9f60-91894aae50a7 Test worker] exec.Task (TezTask.java:execute(190)) - Subscribed to counters: [] for queryId: runner_20201129082818_d6cdb374-f476-4a20-8593-48bbd4c859d8
StdOut 2020-11-29T08:28:18,477 INFO  [4be58fef-bdce-46ea-9f60-91894aae50a7 Test worker] exec.Task (TezTask.java:ensureSessionHasResources(367)) - Tez session hasn't been created yet. Opening session
StdOut 2020-11-29T08:28:18,691 ERROR [ServiceThread:DAGClientRPCServer] client.DAGClientServer (DAGClientServer.java:serviceStart(99)) - Failed to start DAGClientServer: 
StdOut org.apache.hadoop.metrics2.MetricsException: Metrics source RpcActivityForPort37535 already exists!

[pvary]

@jun-he: the fix for the Metrics error is merged. You might want to rebase and run the tests again.
Thanks, Peter

[jun-he]

@rdblue @aokolnychyi Can you please review it for the issue #1846 (i.e. ./gradlew clean build fails due to Expiring Daemon because JVM heap space is exhausted)? Thanks.

[rdblue]

@jun-he, is this still happening after the other fixes?

I'd like to avoid increasing the heap size if we don't need to. Sometimes failures like this can help us catch runtime issues.

[jun-he]

@rdblue Yes, it is still happening with the latest master (commit dcc539aaf671b4642a8fda95ab299ac98cfc89c9).
./gradlew clean build fails with errors Expiring Daemon because JVM heap space is exhausted if I set org.gradle.jvmargs=-Xmx512m in the gradle.properties.

[rdblue]

Sounds good to me then. I don't think this would hurt anything.