[auto-merge] release/26.08 to main [skip ci] [bot]

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,6 +1,6 @@
 ---
 name: Bug report
-about: Create a bug report to help us improve RAPIDS Accelerator for Apache Spark
+about: Create a bug report to help us improve the NVIDIA cuDF plugin for Apache Spark
 title: "[BUG]"
 labels: "? - Needs Triage, bug"
 assignees: ''

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,6 +1,6 @@
 ---
 name: Feature request
-about: Suggest an idea for RAPIDS Accelerator for Apache Spark
+about: Suggest an idea for the NVIDIA cuDF plugin for Apache Spark
 title: "[FEA]"
 labels: "? - Needs Triage, feature request"
 assignees: ''

.github/ISSUE_TEMPLATE/submit-question.md

@@ -1,6 +1,6 @@
 ---
 name: Submit question
-about: Ask a general question about RAPIDS Accelerator for Apache Spark here, or open a thread in the Discussions tab
+about: Ask a general question about the NVIDIA cuDF plugin for Apache Spark here, or open a thread in the Discussions tab
 title: "[QST]"
 labels: "? - Needs Triage, question"
 assignees: ''

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,8 +1,8 @@
 <!--
 
-Thank you for contributing to RAPIDS Accelerator for Apache Spark!
+Thank you for contributing to the NVIDIA cuDF plugin for Apache Spark!
 
-Please read https://github.com/NVIDIA/spark-rapids/blob/HEAD/CONTRIBUTING.md#creating-a-pull-request before making this PR.
+Please read https://github.com/NVIDIA/cudf-spark/blob/HEAD/CONTRIBUTING.md#creating-a-pull-request before making this PR.
 
 The following are the guidelines to help the review process go smoothly. Please read them carefully and fill out relevant information as much as possible.
 

.github/copilot-instructions.md

@@ -1,6 +1,6 @@
 # GitHub Copilot Review Instructions for spark-rapids
 
-spark-rapids is a GPU acceleration plugin for Apache Spark.
+cudf-spark is a GPU acceleration plugin for Apache Spark.
 For full project context, coding conventions, and code examples,
 see [AGENTS.md](../AGENTS.md).
 
@@ -10,7 +10,7 @@ When reviewing shim code or GPU operator implementations, cross-reference
 these upstream repositories to verify correctness:
 - **apache/spark** — verify GPU behavior matches the CPU implementation in the target Spark version (expression evaluation semantics, null handling, type coercion, catalog/partition behavior)
 - **rapidsai/cudf** — verify cuDF API usage (column operations, memory allocation semantics, null handling)
-- **NVIDIA/spark-rapids-jni** — verify JNI binding signatures, memory ownership rules, error codes
+- **NVIDIA/cudf-spark-jni** — verify JNI binding signatures, memory ownership rules, error codes
 
 ## Review Checklist
 
@@ -26,7 +26,7 @@ these upstream repositories to verify correctness:
 - [ ] H5: Configuration — new RapidsConf without docs/defaults; should use .internal() if not user-visible; new features default off
 - [ ] H6: Magic numbers — unexplained numeric literals without named constants or comments
 - [ ] H7: Pre-merge CI gaps — only selected shims run unit tests; [databricks] needed for DB CI; feature-gated tests need explicit enable; limited Scala 2.13 coverage
-- [ ] H8: Upstream dependencies — SNAPSHOT changes from spark-rapids-jni/cudf may break; verify API usage against upstream repos
+- [ ] H8: Upstream dependencies — SNAPSHOT changes from cudf-spark-jni/cudf may break; verify API usage against upstream repos
 
 ## Do NOT Comment On
 

CODE_OF_CONDUCT.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-Define the code of conduct followed and enforced by the RAPIDS Accelerator for Apache Spark project
+Define the code of conduct followed and enforced by the NVIDIA cuDF plugin for Apache Spark project
 
 ### Intended audience
 

CONTRIBUTING.md

@@ -1,15 +1,15 @@
-# Contributing to RAPIDS Accelerator for Apache Spark
+# Contributing to the NVIDIA cuDF plugin for Apache Spark
 
-Contributions to RAPIDS Accelerator for Apache Spark fall into the following three categories.
+Contributions to the NVIDIA cuDF plugin for Apache Spark fall into the following three categories.
 
 1. To report a bug, request a new feature, or report a problem with
-    documentation, please file an [issue](https://github.com/NVIDIA/spark-rapids/issues/new/choose)
+    documentation, please file an [issue](https://github.com/NVIDIA/cudf-spark/issues/new/choose)
     describing in detail the problem or new feature. The project team evaluates
     and triages issues, and schedules them for a release. If you believe the
     issue needs priority attention, please comment on the issue to notify the
     team.
 2. To propose and implement a new Feature, please file a new feature request
-    [issue](https://github.com/NVIDIA/spark-rapids/issues/new/choose). Describe the
+    [issue](https://github.com/NVIDIA/cudf-spark/issues/new/choose). Describe the
     intended feature and discuss the design and implementation with the team and
     community. Once the team agrees that the plan looks good, go ahead and
     implement it using the [code contributions](#code-contributions) guide below.
@@ -48,7 +48,7 @@ building at least running to the `verify` phase, e.g.:
 mvn verify
 ```
 
-After a successful build, the RAPIDS Accelerator jar will be in the `dist/target/` directory.
+After a successful build, the cuDF plugin jar will be in the `dist/target/` directory.
 This will build the plugin for a single version of Spark.  By default, this is Apache Spark
 3.3.0. To build against other versions of Spark you use the `-Dbuildver=XXX` command line option
 to Maven. For instance to build Spark 3.3.0 you would use:
@@ -83,7 +83,7 @@ build versions. See the next section for more details.
 
 You might see a warning during scala-maven-plugin compile goal invocation.
 ```
-[INFO] Compiling 94 Scala sources and 1 Java source to /home/user/gits/NVIDIA/spark-rapids/tests/target/spark3XY/test-classes ...
+[INFO] Compiling 94 Scala sources and 1 Java source to /home/user/gits/NVIDIA/cudf-spark/tests/target/spark3XY/test-classes ...
 OpenJDK 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
 OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
 CodeCache: size=245760Kb used=236139Kb max_used=243799Kb free=9620Kb
@@ -167,7 +167,7 @@ In many situations the user knows that the Plugin jar will be deployed for a sin
 release. It is most commonly the case when a container image for a cloud or local deployment includes
 Spark binaries as well. In such a case it is advantageous to create a jar with
 a conventional class directory structure avoiding complications such as
-[#3704](https://github.com/NVIDIA/spark-rapids/issues/3704). To this end add
+[#3704](https://github.com/NVIDIA/cudf-spark/issues/3704). To this end add
 `-DallowConventionalDistJar=true` when invoking Maven.
 
 ```bash
@@ -214,15 +214,15 @@ for a single Spark version Shim alone.
 To this end in a pre-production build you can set the Boolean property
 `dist.jar.compress` to `false`, its default value is `true`.
 
-Furthermore, after the first build execution on the clean repository the spark-rapids-jni
+Furthermore, after the first build execution on the clean repository the cudf-spark-jni
 SNAPSHOT dependency typically does not change until the next nightly CI build, or the next install
 to the local Maven repo if you are working on a change to the native code. So you can save
 significant time spent on repeated unpacking these dependencies by adding `-Drapids.jni.unpack.skip`
 to the `dist` build command.
 
 The time saved is more significant if you are merely changing
 the `aggregator` module, or the `dist` module, or just incorporating changes from
-[spark-rapids-jni](https://github.com/NVIDIA/spark-rapids-jni/blob/branch-23.04/CONTRIBUTING.md#local-testing-of-cross-repo-contributions-cudf-spark-rapids-jni-and-spark-rapids)
+[cudf-spark-jni](https://github.com/NVIDIA/cudf-spark-jni/blob/branch-23.04/CONTRIBUTING.md#local-testing-of-cross-repo-contributions-cudf-spark-rapids-jni-and-spark-rapids)
 
 For example, to quickly repackage `rapids-4-spark` after the
 initial `./build/buildall` you can iterate by invoking
@@ -280,7 +280,7 @@ Last tested with IntelliJ IDEA 2023.1.2 (Community Edition)
 
 ##### Manual Maven Install for a target Spark build
 
-Before proceeding with importing spark-rapids into IDEA or switching to a different Spark release
+Before proceeding with importing cudf-spark into IDEA or switching to a different Spark release
 profile, execute the install phase with the corresponding `buildver`, e.g. for Spark 3.4.0:
 
 ```bash
@@ -378,7 +378,7 @@ rm -vf .bloop
 ln -s .bloop-spark330 .bloop
 ```
 
-You can now open the spark-rapids as a
+You can now open the cudf-spark as a
 [BSP project in IDEA](https://www.jetbrains.com/help/idea/bsp-support.html)
 
 Read on for VS Code Scala Metals instructions.
@@ -446,15 +446,15 @@ We welcome pull requests with tips on how to setup your favorite IDE!
 
 ### Your first issue
 
-1. Read the [Developer Overview](docs/dev/README.md) to understand how the RAPIDS Accelerator
+1. Read the [Developer Overview](docs/dev/README.md) to understand how the cuDF
     plugin works.
 2. Find an issue to work on. The best way is to look for the
-    [good first issue](https://github.com/NVIDIA/spark-rapids/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)
-    or [help wanted](https://github.com/NVIDIA/spark-rapids/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22)
+    [good first issue](https://github.com/NVIDIA/cudf-spark/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)
+    or [help wanted](https://github.com/NVIDIA/cudf-spark/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22)
     labels.
 3. Comment on the issue stating that you are going to work on it.
 4. Code! Make sure to update unit tests and integration tests if needed! [refer to test section](#testing-your-code)
-5. When done, [create your pull request](https://github.com/NVIDIA/spark-rapids/compare).
+5. When done, [create your pull request](https://github.com/NVIDIA/cudf-spark/compare).
 6. Verify that CI passes all [status checks](https://help.github.com/articles/about-status-checks/).
     Fix if needed.
 7. Wait for other developers to review your code and update code as needed.
@@ -464,7 +464,7 @@ Remember, if you are unsure about anything, don't hesitate to comment on issues
 and ask for clarifications!
 
 ### Code Formatting
-RAPIDS Accelerator for Apache Spark follows the same coding style guidelines as the Apache Spark
+The NVIDIA cuDF plugin for Apache Spark follows the same coding style guidelines as the Apache Spark
 project.  For IntelliJ IDEA users, an
 [example code style settings file](docs/dev/idea-code-style-settings.xml) is available in the
 `docs/dev/` directory.

DF_UDF_README.md

@@ -7,21 +7,21 @@ are also used as a way to standardize processing logic across an organization or
 But UDFs come with some downsides. The biggest one is visibility into the processing being done. SQL is a language that
 can be highly optimized. But a UDF in most cases is a black box, that the SQL optimizer cannot do anything about.
 This can result in less than ideal query planning. Additionally, accelerated execution environments, like the
-RAPIDS Accelerator for Apache Spark have no easy way to replace UDFs with accelerated versions, which can result in
+NVIDIA cuDF plugin for Apache Spark have no easy way to replace UDFs with accelerated versions, which can result in
 slow performance.
 
 This attempts to add visibility to the code reuse use case by providing a way to implement a UDF in terms of dataframe
 commands. 
 
 ## Setup
 
-The dataframe UDF plugin is packaged in the same jar as the RAPIDS Accelerator for Apache Spark. This jar will need to
+The dataframe UDF plugin is packaged in the same jar as the cuDF plugin. This jar will need to
 be added as a compile time dependency for code that wants to use this feature as well as adding the jar to your Spark
 classpath just like you would do for GPU acceleration.
 
 If you plan to not use the GPU accelerated processing, but still want dataframe UDF support on CPU applications then
 add `com.nvidia.spark.DFUDFPlugin` to the `spark.sql.extensions` config. If you do use GPU accelerated processing
-the RAPIDS Plugin will enable this automatically. You don't need to set the `spark.sql.extensions` config, but it
+the cuDF plugin will enable this automatically. You don't need to set the `spark.sql.extensions` config, but it
 won't hurt anything if you do add it. Now you can implement a UDF in terms of Dataframe operations.
 
 ## Usage

README.md

@@ -1,15 +1,19 @@
-# RAPIDS Accelerator For Apache Spark
-NOTE: For the latest stable [README.md](https://github.com/nvidia/spark-rapids/blob/main/README.md) ensure you are on the main branch.
+# NVIDIA cuDF plugin for Apache Spark
+NOTE: For the latest stable [README.md](https://github.com/nvidia/cudf-spark/blob/main/README.md) ensure you are on the main branch.
 
-The RAPIDS Accelerator for Apache Spark provides a set of plugins for
-[Apache Spark](https://spark.apache.org) that leverage GPUs to accelerate processing 
-via the [RAPIDS](https://rapids.ai) libraries.
+The NVIDIA cuDF plugin for [Apache Spark](https://spark.apache.org) provides a plugin library that
+leverages GPUs to accelerate processing via the [cuDF](https://github.com/rapidsai/cudf) (CUDA
+DataFrame) libraries.
 
 Documentation on the current release can be found [here](https://nvidia.github.io/cudf-spark/).
 
 To get started and try the plugin out use the [getting started guide](https://docs.nvidia.com/spark-rapids/user-guide/latest/getting-started/overview.html).
 
-[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/NVIDIA/spark-rapids)
+[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/NVIDIA/cudf-spark)
+
+Note: The NVIDIA cuDF plugin for Apache Spark was formerly known as the RAPIDS Accelerator for
+Apache Spark.  The RAPIDS name will be sunset over time.  Github links from `spark-rapids` will
+redirect to `cudf-spark`.  Artifact names will remain the same for now.
 
 ## Compatibility
 
@@ -29,8 +33,8 @@ The plugin has a set of Spark configs that control its behavior and are document
 ## Issues & Questions
 
 We use github to track bugs, feature requests, and answer questions. File an
-[issue](https://github.com/NVIDIA/spark-rapids/issues/new/choose) for a bug or feature request. Ask
-or answer a question on the [discussion board](https://github.com/NVIDIA/spark-rapids/discussions).
+[issue](https://github.com/NVIDIA/cudf-spark/issues/new/choose) for a bug or feature request. Ask
+or answer a question on the [discussion board](https://github.com/NVIDIA/cudf-spark/discussions).
 
 ## Download
 
@@ -46,8 +50,8 @@ See the [build instructions in the contributing guide](CONTRIBUTING.md#building-
 Tests are described [here](tests/README.md).
 
 ## Integration
-The RAPIDS Accelerator For Apache Spark does provide some APIs for doing zero copy data
-transfer into other GPU enabled applications.  It is described
+The cuDF plugin provides some APIs for doing zero copy data transfer into other GPU enabled
+applications.  It is described
 [here](https://docs.nvidia.com/spark-rapids/user-guide/latest/additional-functionality/ml-integration.html).
 
 Currently, we are working with XGBoost to try to provide this integration out of the box.
@@ -67,9 +71,8 @@ for more details on how to use the tools.
 
 ## Dependency for External Projects
 
-If you need to develop some functionality on top of RAPIDS Accelerator For Apache Spark (we currently
-limit support to GPU-accelerated UDFs) we recommend you declare our distribution artifact
-as a `provided` dependency.
+If you develop functionality on top of the cuDF plugin (we currently limit support to
+GPU-accelerated UDFs) we recommend you declare the distribution artifact as a `provided` dependency.
 
 ```xml
 <dependency>

datagen/README.md

@@ -8,12 +8,11 @@ data between tables for joins. To accomplish this we wrote
 
 ## Setup Environment
 
-To get started with big data generation the first thing you need to do is
-to include the appropriate jar on the classpath for your version of Apache Spark.
-Note that this does not run on the GPU, but it does use parts of the shim framework
-that the RAPIDS Accelerator does. The jar is specific to the version of Spark you
-are using and is not pushed to Maven Central. Because of this you will have to
-build it from source yourself.
+To get started with big data generation the first thing you need to do is to include the appropriate
+jar on the classpath for your version of Apache Spark.  Note that this does not run on the GPU, but
+it does use parts of the shim framework that the NVIDIA cuDF plugin for Apache Spark does. The jar
+is specific to the version of Spark you are using and is not pushed to Maven Central. Because of
+this you will have to build it from source yourself.
 
 ```shell
 cd datagen
@@ -344,7 +343,7 @@ spark.time(fdf.join(ddf).groupBy("agg_key").agg(min("value"),
 Time taken: 890163 ms
 ```
 
-Or you could run it with the RAPIDS Accelerator where the data generations is still
+Or you could run it with the cuDF plugin where the data generations is still
 done on the CPU and cut the time down to just 75,696 ms
 
 But what if we wanted to join on multiple columns, or do an aggregation with

delta-lake/README.md

@@ -1,4 +1,4 @@
-# RAPIDS Accelerator for Apache Spark Delta Lake Support
+# NVIDIA cuDF plugin for Apache Spark Delta Lake Support
 
 The Delta Lake support is organized into multiple Maven projects, one per Delta Lake minor
 version that is supported. This allows each submodule to build against the Delta Lake minor

delta-lake/common/src/main/delta-33x-41x/scala/com/nvidia/spark/rapids/delta/GpuDeltaCatalogBase.scala

@@ -33,7 +33,7 @@ import org.apache.spark.sql.catalyst.catalog.{CatalogTable, CatalogTableType, Ca
 import org.apache.spark.sql.connector.catalog.{Identifier, StagedTable, StagingTableCatalog, SupportsWrite, Table, TableCapability, TableCatalog, TableChange}
 import org.apache.spark.sql.connector.catalog.TableCapability._
 import org.apache.spark.sql.connector.expressions.Transform
-import org.apache.spark.sql.connector.write.{LogicalWriteInfo, V1Write, WriteBuilder}
+import org.apache.spark.sql.connector.write.{LogicalWriteInfo, SupportsTruncate, V1Write, WriteBuilder}
 import org.apache.spark.sql.delta.{ColumnWithDefaultExprUtils, DeltaConfigs, DeltaErrors, DeltaLog, DeltaOptions}
 import org.apache.spark.sql.delta.catalog.DeltaCatalog
 import org.apache.spark.sql.delta.commands.{TableCreationModes, WriteIntoDelta}
@@ -471,7 +471,7 @@ abstract class GpuDeltaCatalogBase(
     override def abortStagedChanges(): Unit = {}
 
     override def capabilities(): util.Set[TableCapability] = {
-      Set(V1_BATCH_WRITE).asJava
+      Set(V1_BATCH_WRITE, TRUNCATE).asJava
     }
 
     override def newWriteBuilder(info: LogicalWriteInfo): WriteBuilder = {
@@ -482,7 +482,9 @@ abstract class GpuDeltaCatalogBase(
     /*
      * WriteBuilder for creating a Delta table.
      */
-    private class DeltaV1WriteBuilder extends WriteBuilder {
+    private class DeltaV1WriteBuilder extends WriteBuilder with SupportsTruncate {
+      override def truncate(): this.type = this
+
       override def build(): V1Write = new V1Write {
         override def toInsertableRelation(): InsertableRelation = {
           new InsertableRelation {