Use CUDF's "UNBOUNDED" window boundaries for time-range queries.

[mythrocks]

Fixes #1039. Depends on rapidsai/cudf#6811.

This change aims to allow for unbounded time-range window function queries to return the right results. It depends on the explicit support to specify unbounded window boundaries in CUDF. This cannot be checked in without rapidsai/cudf#6811.

[revans2]

@mythrocks you need to update the commit with a signoff

[mythrocks]

Depends on rapidsai/cudf#6811.

This was merged a couple of hours ago. I have rebased the change and signed the commit.

[mythrocks]

mythrocks dismissed revans2’s stale review via 1d6d130 26 minutes ago

With apologies to @revans2, this is a little harsh; I did no such thing. Signing the commit invalidated the previous review. The new commit has no material difference from the old one.

[revans2]

build

[mythrocks]

An odd test failure:

11:59:31  �[32m- a block fits entirely, but a subsequent block doesn't�[0m
11:59:31  �[32mHashAggregatesSuite:�[0m
11:59:46  �[31m*** RUN ABORTED ***�[0m
11:59:46  �[31m  java.lang.NoSuchMethodError: org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.canChangeNumPartitions()Z�[0m
11:59:46  �[31m  at org.apache.spark.sql.rapids.execution.GpuShuffleMeta.convertToGpu(GpuShuffleExchangeExec.scala:68)�[0m
11:59:46  �[31m  at org.apache.spark.sql.rapids.execution.GpuShuffleMeta.convertToGpu(GpuShuffleExchangeExec.scala:42)�[0m
11:59:46  �[31m  at com.nvidia.spark.rapids.SparkPlanMeta.convertIfNeeded(RapidsMeta.scala:615)�[0m
11:59:46  �[31m  at com.nvidia.spark.rapids.GpuOverrides$$anon$138.convertToGpu(GpuOverrides.scala:2046)�[0m
11:59:46  �[31m  at com.nvidia.spark.rapids.GpuOverrides$$anon$138.convertToGpu(GpuOverrides.scala:2040)�[0m
11:59:46  �[31m  at com.nvidia.spark.rapids.SparkPlanMeta.convertIfNeeded(RapidsMeta.scala:615)�[0m
11:59:46  �[31m  at com.nvidia.spark.rapids.GpuSortMeta.convertToGpu(GpuSortExec.scala:42)�[0m
11:59:46  �[31m  at com.nvidia.spark.rapids.GpuSortMeta.convertToGpu(GpuSortExec.scala:33)�[0m
11:59:46  �[31m  at com.nvidia.spark.rapids.SparkPlanMeta.convertIfNeeded(RapidsMeta.scala:615)�[0m
11:59:46  �[31m  at com.nvidia.spark.rapids.GpuSortAggregateMeta.convertToGpu(aggregate.scala:285)�[0m
11:59:46  �[31m  ...�[0m
11:59:46  [INFO] ------------------------------------------------------------------------
11:59:46  [INFO] Reactor Summary:
11:59:46  [INFO] 
11:59:46  [INFO] RAPIDS Accelerator for Apache Spark Root Project ... SUCCESS [  1.368 s]
11:59:46  [INFO] RAPIDS Accelerator for Apache Spark SQL Plugin ..... SUCCESS [  7.770 s]
11:59:46  [INFO] RAPIDS Accelerator for Apache Spark Shuffle Plugin . SUCCESS [  0.460 s]
11:59:46  [INFO] RAPIDS Accelerator for Apache Spark SQL Plugin Shims SUCCESS [  0.068 s]
11:59:46  [INFO] RAPIDS Accelerator for Apache Spark SQL Plugin Spark 3.0.0 Shim SUCCESS [  0.592 s]
11:59:46  [INFO] RAPIDS Accelerator for Apache Spark SQL Plugin Spark 3.0.0 EMR Shim SUCCESS [  0.432 s]
11:59:46  [INFO] RAPIDS Accelerator for Apache Spark SQL Plugin Spark 3.0.1 Shim SUCCESS [  0.528 s]
11:59:46  [INFO] RAPIDS Accelerator for Apache Spark SQL Plugin Spark 3.0.1 EMR Shim SUCCESS [  0.445 s]
11:59:46  [INFO] RAPIDS Accelerator for Apache Spark SQL Plugin Spark 3.1.0 Shim SUCCESS [ 11.990 s]
11:59:46  [INFO] RAPIDS Accelerator for Apache Spark SQL Plugin Spark 3.0.2 Shim SUCCESS [ 11.075 s]
11:59:46  [INFO] RAPIDS Accelerator for Apache Spark SQL Plugin Shim Aggregator SUCCESS [  0.069 s]
11:59:46  [INFO] RAPIDS Accelerator for Apache Spark Scala UDF Plugin SUCCESS [ 35.900 s]
11:59:46  [INFO] RAPIDS Accelerator for Apache Spark Distribution ... SUCCESS [  0.124 s]
11:59:46  [INFO] RAPIDS Accelerator for Apache Spark Tests .......... FAILURE [ 44.202 s]
11:59:46  [INFO] rapids-4-spark-integration-tests_2.12 .............. SKIPPED
11:59:46  [INFO] rapids-4-spark-api-validation ...................... SKIPPED
11:59:46  [INFO] ------------------------------------------------------------------------
11:59:46  [INFO] BUILD FAILURE
11:59:46  [INFO] ------------------------------------------------------------------------
11:59:46  [INFO] Total time: 01:55 min
11:59:46  [INFO] Finished at: 2020-11-25T19:59:48+00:00
11:59:46  [INFO] Final Memory: 116M/1315M
11:59:46  [INFO] ------------------------------------------------------------------------

[abellina]

Looks related to: apache/spark#30432. So ShuffleExchangeLike changed and canChangeNumPartitions became shuffleOrigin

[mythrocks]

Looks related to: apache/spark#30432. So ShuffleExchangeLike changed and canChangeNumPartitions became shuffleOrigin

#1206 has been committed. (Thanks, @jlowe.) I'll try a rebuild.

[mythrocks]

build

[jlowe]

#1206 has been committed. (Thanks, @jlowe.) I'll try a rebuild.\

Note that you'll need to upmerge to branch-0.3 to pick up the fix in CI.

[mythrocks]

build

[sameerz]

build

[mythrocks]

build

Thanks, @sameerz. The builds launched when I rebased seemed to hang as well. It seems to have worked that time. w007!