Removed spark.rapids.sql.castDecimalToString.enabled and enabled GPU decimal to string by default [databricks] #8481
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…or GPU decimal to string Signed-off-by: Mike Wilson <[email protected]>
Signed-off-by: Mike Wilson <[email protected]>
Signed-off-by: Mike Wilson <[email protected]>
…/spark-rapids into fix_decimal_to_string
Signed-off-by: Robert (Bobby) Evans <[email protected]>
|
build |
jlowe
previously approved these changes
Jun 2, 2023
|
looks like databricks 3.3.0 pulled in the Spark 3.4.0 cast change. Need to work on this a little more... |
|
build |
jlowe
approved these changes
Jun 5, 2023
This was referenced Jun 5, 2023
|
🎉 |
thirtiseven
added a commit
to thirtiseven/spark-rapids
that referenced
this pull request
Jun 7, 2023
* Removed spark.rapids.sql.castDecimalToString.enabled and enabled GPU decimal to string by default [databricks] (NVIDIA#8481) * Adding shim for decimal to string and removing configuration option for GPU decimal to string Signed-off-by: Mike Wilson <[email protected]> * adding stub comparison function back in for now Signed-off-by: Mike Wilson <[email protected]> * Updating from review comments Signed-off-by: Mike Wilson <[email protected]> * Fix some corner cases with decimal cast to string Signed-off-by: Robert (Bobby) Evans <[email protected]> * Move include cast change in databricks 330 --------- Signed-off-by: Mike Wilson <[email protected]> Signed-off-by: Robert (Bobby) Evans <[email protected]> Co-authored-by: Mike Wilson <[email protected]> * Move task completion listener registration to after variables are initialized (NVIDIA#8486) Signed-off-by: Jason Lowe <[email protected]> * [Doc] address Spark RAPIDS NVAIE VDR issues [skip ci] (NVIDIA#8427) * address Spark RAPIDS NVAIE VDR Signed-off-by: liyuan <[email protected]> * reword the getting started guide on gcp secure boot part Signed-off-by: liyuan <[email protected]> * address the nit rewording Signed-off-by: liyuan <[email protected]> --------- Signed-off-by: liyuan <[email protected]> * Fix warning about deprecated parquet config (NVIDIA#8515) Signed-off-by: Robert (Bobby) Evans <[email protected]> * Avoid calling Table.merge with BinaryType columns (NVIDIA#8516) Signed-off-by: Jason Lowe <[email protected]> --------- Signed-off-by: Mike Wilson <[email protected]> Signed-off-by: Robert (Bobby) Evans <[email protected]> Signed-off-by: Jason Lowe <[email protected]> Signed-off-by: liyuan <[email protected]> Co-authored-by: Robert (Bobby) Evans <[email protected]> Co-authored-by: Mike Wilson <[email protected]> Co-authored-by: Jason Lowe <[email protected]> Co-authored-by: liyuan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This includes everything in #8439, but adds some fixes in the final commit. The heavy lifting was done by @hyperbolic2346
My part just adds in a few new integration tests so that the we are comparing to Spark instead of what we think Spark would produce in the current unit tests. It also ads in a slow path for negative scale decimals in ANSI mode, which both of them are not common, so I think we might be okay with the slow path.