Skip to content

Add casting of count to UInt64 in array_repeat function to ensure consistent integer type handling #14236

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
korowa merged 3 commits into from
Jan 24, 2025
Merged

Add casting of count to UInt64 in array_repeat function to ensure consistent integer type handling #14236

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1d41184
Add casting of `count` to `Int64` in `array_repeat` function to ensur…
jatin510 Jan 22, 2025
869ef99
updated the "array_repeat" function signature to automatically cast u…
jatin510 Jan 23, 2025
2cafea2
fixed clippy issue
jatin510 Jan 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion datafusion/functions-nested/src/repeat.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
use crate::utils::make_scalar_function;
use arrow::array::{Capacities, MutableArrayData};
use arrow::compute;
use arrow::compute::cast;
use arrow_array::{
new_null_array, Array, ArrayRef, GenericListArray, Int64Array, ListArray,
OffsetSizeTrait,
Expand Down Expand Up @@ -136,7 +137,17 @@ pub fn array_repeat_inner(args: &[ArrayRef]) -> Result<ArrayRef> {
}

let element = &args[0];
let count_array = as_int64_array(&args[1])?;
let count_array = &args[1];

let count_array = match count_array.data_type() {
DataType::Int8 | DataType::Int16 | DataType::Int32 => {

@korowa korowa Jan 22, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likely, UInt should also be allowed here

&cast(count_array, &DataType::Int64)?
}
DataType::Int64 => count_array,
_ => return exec_err!("count must be an integer type"),
};

let count_array = as_int64_array(&count_array)?;

@korowa korowa Jan 22, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe uint should be used as a target type here? Right now DF panics when negative value passed as an argument, which also could be fixed along with type casting

@jatin510 jatin510 Jan 23, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we want the target type of array_repeat count function to be UInt64, we have to change the behaviour of array_repeat. As of now, by default count is of type Int64.
It will be a breaking change, are we okay with that ?

@korowa

with current implementation of array_repeat and default data type of Int64.

It is behaving like this , if we are converting all Signed Integer type to Unsigned Integer, but letting the input of count to be of Int64

> select array_repeat(1,-20000);
+--------------------------------------+
| array_repeat(Int64(1),Int64(-20000)) |
+--------------------------------------+
| []                                   |
+--------------------------------------+

In Spark, it gives empty arrow for negative count argument

@korowa korowa Jan 23, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By target type I've meant conversion to UInt internallly instead of Int, so it wouldn't affect any input arguments, only prevent this function from panicking on negative values (but I thought it would return cast error 🤔 ), so I don't think it's a breaking change.


match element.data_type() {
List(_) => {
Expand Down
9 changes: 9 additions & 0 deletions datafusion/sqllogictest/test_files/array.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2750,6 +2750,15 @@ select
----
[[1], [1], [1], [1], [1]] [[1.1, 2.2, 3.3], [1.1, 2.2, 3.3], [1.1, 2.2, 3.3]] [[NULL, NULL], [NULL, NULL], [NULL, NULL]] [[[1, 2], [3, 4]], [[1, 2], [3, 4]]]

# array_repeat scalar function with count of different integer types
query ???
Select
array_repeat(1, arrow_cast(2,'Int8')),
array_repeat(2, arrow_cast(2,'Int16')),
array_repeat(3, arrow_cast(2,'Int32'));
----
[1, 1] [2, 2] [3, 3]

# array_repeat with columns #1

statement ok
Expand Down