Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move Covariance (Population) covar_pop to be a User Defined Aggregate Function #10418

Merged
merged 2 commits into from
May 10, 2024

Conversation

yyy1000
Copy link
Contributor

@yyy1000 yyy1000 commented May 7, 2024

Which issue does this PR close?

Closes #10389 .

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions bot added logical-expr Logical plan and expressions physical-expr Physical Expressions labels May 7, 2024
@@ -319,281 +225,3 @@ impl Accumulator for CovarianceAccumulator {
std::mem::size_of_val(self)
}
}

#[cfg(test)]
mod tests {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking whether we should move these tests into the new files rather than deleting them. 🤔

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can move it to aggregate.slt like what in #10384 does

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can create another PR for moving test

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it.

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @yyy1000 and @jayzhan211 -- I agree once we port over the tests this PR should be good to go 🙏

@yyy1000
Copy link
Contributor Author

yyy1000 commented May 8, 2024

I agree once we port over the tests this PR should be good to go 🙏

Would you prefer moving test in a separate PR or in this PR? I'm good either way. :)

@jayzhan211
Copy link
Contributor

I agree once we port over the tests this PR should be good to go 🙏

Would you prefer moving test in a separate PR or in this PR? I'm good either way. :)

If the test can be verified one by one less than 5min, than it is fine for me to see them in one PR.

@github-actions github-actions bot added the sqllogictest SQL Logic Tests (.slt) label May 10, 2024
Comment on lines -595 to -597
let state2 = get_accum_scalar_values_as_arrays(accum2.as_mut())?;
accum1.merge_batch(&state2)?;
accum1.evaluate()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added all unit tests to sqllogictest excluding covariance_f64_merge_1 and covariance_f64_merge_2
Don't know how to mock these merge operations. 🥲

));

let actual = merge(&batch1, &batch2, agg1, agg2)?;
assert!(actual == ScalarValue::from(0.6666666666666666));
Copy link
Contributor

@jayzhan211 jayzhan211 May 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we just need to test with multi column

covariance(a, b), where a is [1, 2,3]. b is [4,5,6]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, thanks!
I checked these two test case has been covered in covariance_f64_1 and covariance_f64_6 so all the tests are ported.

Copy link
Contributor

@jayzhan211 jayzhan211 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@alamb
Copy link
Contributor

alamb commented May 10, 2024

Thanks @yyy1000 and @jayzhan211

@alamb alamb merged commit 9f0e016 into apache:main May 10, 2024
23 checks passed
@yyy1000 yyy1000 deleted the covar_pop branch May 14, 2024 02:26
findepi pushed a commit to findepi/datafusion that referenced this pull request Jul 16, 2024
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
logical-expr Logical plan and expressions physical-expr Physical Expressions sqllogictest SQL Logic Tests (.slt)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Move Covariance (Population) covar_pop to be a UDAF
3 participants