Support User Defined Table Function #8306

Veeupup · 2023-11-22T14:10:30Z

Which issue does this PR close?

Closes #7926

Rationale for this change

~~This is a draft PR to show how to implement User-Defined-Table Function.~~

It remains a lot of things to do, but maybe it's a good start.

Such as:

Internal Table Defined Function
More input functions types
More friendly register
...

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Veeupup · 2023-11-22T14:12:20Z

@alamb PTAL, a draft implementation of user-defined-table-function : )

alamb · 2023-11-22T20:41:32Z

Thank you @Veeupup -- I am very excited to check this out. I plan to do so tomorrow if possible (though it is a holiday in the US so it may not be until Friday or the weekend)

alamb

Ok, I couldn't help myself and took a quick look now @Veeupup -- this is really cool, thank you. Also thank you for putting up a draft PR for early feedback.

I left some comments, let me know what you think.

Thanks again! 🏆

alamb · 2023-11-22T20:44:50Z

datafusion/core/src/datasource/function.rs

+    }
+
+    /// Get the function implementation and generate a table
+    pub fn create_table_provider(


I was thinking we would allow users to provide their own TableProvider directly (rather than a PartitionStream) as this permits them to implement predicate / projection / limit pushdown, among other things

I like how you have made a simple version in the example -- maybe you can just move the creation of StreamingTable::try_new() into datafusion-examples/examples/simple_udtf.rs 🤔

I am also thinking about how to support TableFunctions that take streams as input: #7926 (comment). Any ideas on this?

I was thinking we would allow users to provide their own TableProvider directly (rather than a PartitionStream) as this permits them to implement predicate / projection / limit pushdown, among other things

sure! Agree with you, just implementing their TableProvider seems more reasonable. I have change the func signature.

I am also thinking about how to support TableFunctions that take streams as input:

maybe users can implement PartitionStream and use inner StreamTable to take streams as input? see it in example

alamb · 2023-11-22T20:45:57Z

datafusion/core/src/datasource/function.rs

+    /// Get the function implementation and generate a table
+    pub fn create_table_provider(
+        &self,
+        args: &[String],


Instead of taking strings what would you think about taking Expr instead (which would allow for more sophisticated things like passing in now() as sugested by @yukkit in #7926 (comment))?

Love this idea too! I'll try to figure out if we can find a way to evaluate Expr correctly!

@alamb PTAL I have madeExprs and provider inputs and it looks just fine!

but we still have some problems to solve, you can check TODO in the latest commit.

how to evaluate expr simpler?

for now, I left it in fn scan to construct Expr -> LogicalPlan -> PhysicalPlan and evaluate it then, but I think we can make it simpler or elsewhere to evaluate Exprs.

how to evaluate expr simpler?

you can use ExprSimplifier::simplify to do this.

Here is an example https://github.com/apache/arrow-datafusion/blob/06bbe1298fa8aa042b6a6462e55b2890969d884a/datafusion-examples/examples/expr_api.rs#L56

I added a commit to Veeupup#1 showing how to make it work: Veeupup@2df01db

Love this idea too! I'll try to figure out if we can find a way to evaluate Expr correctly!

BTW since I agree evaluating expressions is non trivial, I added some additional documentation and examples of how to do so here: #8377

Veeupup · 2023-11-23T02:34:10Z

Happy Thanksgiving!! Enjoy your holiday! 😺

I'll checkout your constructive comments later today when I'm free : )

Signed-off-by: veeupup <[email protected]>

Veeupup · 2023-11-27T13:38:30Z

cc @yukkit hi, this PR is almost ready for review, comments are welcome!

alamb

Thank you @Veeupup -- I think this is looking very close. (Another) really nice PR 🦾

What I think is needed prior to merging this PR:

Gather any more feedback on the API
Add tests (in addition to the example) in https://github.com/apache/arrow-datafusion/tree/main/datafusion/core/tests/user_defined (the example outputs are not validated)

I also added some suggestions on how to make the code simpler / better but I don't think they are strictly necessary

alamb · 2023-11-28T11:45:15Z

datafusion/core/src/datasource/function.rs

+    }
+
+    /// Get the function implementation and generate a table
+    pub fn create_table_provider(&self, args: &[Expr]) -> Result<Arc<dyn TableProvider>> {


alamb · 2023-11-28T11:55:05Z

datafusion/core/src/datasource/function.rs

+/// A trait for table function implementations
+pub trait TableFunctionImpl: Sync + Send {
+    /// Create a table provider
+    fn call(&self, args: &[Expr]) -> Result<Arc<dyn TableProvider>>;


This API is nice and is as specified in #7926. I think it will work for using a table function as a relation in the query (aka like a table with parameters)

The one thing I don't think this API supports is TableFunctions that take other arguments (aka that are fed the result of a table / can use the value of correlated subqueries as mentioned by @yukkit and @Jesse-Bakker #7926 (comment).

I can think of two options:

Leave this API as is , and add a follow on / new API somehow to support that usecase

Try to extend this API somehow to support table inputs

I personally prefer 1 as I think it offers several additional use cases, even though it doesn't cover "take a table input".

Any other thoughts?

One specific use case for table-valued arguments to table-valued functions is, for example windowing tvf's like in apache flink.

Example which cannot be expressed by taking Expr arguments (maybe if Expr::Row() is added?):

SELECT window_start, window_end, SUM(price) FROM TABLE( TUMBLE(TABLE Bid, DESCRIPTOR(bidtime), INTERVAL '10' MINUTES)) GROUP BY window_start, window_end;

That can also be emulated, however, using something like:

SELECT window_start, window_end, SUM(price) FROM Bid, TUMBLE(Bid.bidtime, INTERVAL '10' MINUTES)) GROUP BY window_start, window_end;

which doesn't need table-valued arguments (but does need to resolve Expr::Column(name=bidtime). I'm not sure if the current API can do that?).

Anyway, the current API is nice, and definitely very useful 👍

datafusion/core/src/datasource/function.rs

datafusion/sql/src/planner.rs

alamb · 2023-11-28T12:00:29Z

datafusion/sql/src/relation/mod.rs

+                            FunctionArg::Unnamed(FunctionArgExpr::Expr(expr)) => {
+                                let expr = self.sql_expr_to_logical_expr(
+                                    expr,
+                                    // TODO(veeupup): for now, maybe it's little diffcult to resolve tables' schema before create provider


When the function is used like a base relation (aka a TableFactor / item in the FROM clause) I don't think there is any schema to use to resolve the input arguments (aka this code is correct and probably shouldn't be a todo).

I don't really know how we would represent the correlated subquery case in #7926 (comment) 🤔 I think it would need some sort of analysis pass after the initial plan is built

datafusion/sql/src/relation/mod.rs

datafusion-examples/examples/simple_udtf.rs

alamb · 2023-11-28T12:12:57Z

datafusion-examples/examples/simple_udtf.rs

+    }
+}
+
+// Option2: (use StreamingTable to make it simpler)


I actually think adding a second option is more confusing in this example, so I would recommend removing this part (perhaps you can just add a note that it is posisble)

Signed-off-by: veeupup <[email protected]>

alamb

Thank you @Veeupup -- I think this can be merged as is. Really nice

I have a suggestion about trying to simplify the code a bit (we can do it as a follow on PR as well): Veeupup#1

datafusion/core/tests/user_defined/user_defined_table_functions.rs

Simply table function example, add some comments

Veeupup · 2023-11-30T12:27:53Z

we should support table function inputs with Expr::Column and more complicated Expr.

Where should we track it? Still track it in #7926 ? or start a new issue

I can make some trials later.

…s.rs

alamb · 2023-11-30T22:16:24Z

I took the liberty of merging this branch up from main. If the tests pass I'll plan to merge it. While waiting I will likely file follow on tickets

alamb · 2023-11-30T22:54:10Z

I filed #8383 to track follow on work. Is there any other specific improvement that we discussed that isn't covered?

alamb · 2023-11-30T22:55:59Z

🚀

* Support User Defined Table Function Signed-off-by: veeupup <[email protected]> * fix comments Signed-off-by: veeupup <[email protected]> * add udtf test Signed-off-by: veeupup <[email protected]> * add file header * Simply table function example, add some comments * Simplfy exprs * make clippy happy * Update datafusion/core/tests/user_defined/user_defined_table_functions.rs --------- Signed-off-by: veeupup <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>

github-actions bot added sql SQL Planner core Core DataFusion crate labels Nov 22, 2023

alamb mentioned this pull request Nov 22, 2023

DataFusion weekly project plan (Andrew Lamb) - Nov 20, 2023 #8274

Closed

7 tasks

alamb reviewed Nov 22, 2023

View reviewed changes

Veeupup force-pushed the table_function branch 2 times, most recently from 26ddf6a to 050ab9d Compare November 25, 2023 07:47

Support User Defined Table Function

32ecce4

Signed-off-by: veeupup <[email protected]>

Veeupup force-pushed the table_function branch from 050ab9d to 32ecce4 Compare November 25, 2023 07:53

Veeupup requested a review from alamb November 25, 2023 07:55

Veeupup marked this pull request as ready for review November 25, 2023 08:10

This was referenced Nov 27, 2023

DataFusion weekly project plan (Andrew Lamb) - Nov 27, 2023 #8329

Closed

Support User Defined Table Functions / Table Value Functions #7926

Closed

alamb reviewed Nov 28, 2023

View reviewed changes

Veeupup and others added 4 commits November 28, 2023 21:46

fix comments

2f4e089

Signed-off-by: veeupup <[email protected]>

add udtf test

7a85a43

Signed-off-by: veeupup <[email protected]>

add file header

1fe0c8b

Simply table function example, add some comments

169b13e

alamb mentioned this pull request Nov 29, 2023

Simply table function example, add some comments Veeupup/arrow-datafusion#1

Merged

alamb approved these changes Nov 29, 2023

View reviewed changes

datafusion/core/tests/user_defined/user_defined_table_functions.rs Outdated Show resolved Hide resolved

Simplfy exprs

2df01db

alamb mentioned this pull request Nov 29, 2023

Implement parquet_metadata function in datafusion-cli #8367

Closed

Veeupup added 2 commits November 30, 2023 08:53

Merge pull request #1 from alamb/alamb/simplified_table_function

180e02e

Simply table function example, add some comments

make clippy happy

c2e7cb0

Veeupup mentioned this pull request Nov 30, 2023

Support User-Defined Table Function inputs with Expr::Column and more complicated Expr #8375

Closed

Update datafusion/core/tests/user_defined/user_defined_table_function…

0e98c0b

…s.rs

Merge remote-tracking branch 'apache/main' into table_function

ccc2959

alamb mentioned this pull request Nov 30, 2023

Add evaluate_demo and range_analysis_demo to Expr examples #8377

Merged

alamb merged commit e19c669 into apache:main Nov 30, 2023
23 checks passed

tshauck mentioned this pull request Dec 14, 2023

Update UDF Library Docs with UDTFs #8545

Closed

matthewgapp mentioned this pull request Jan 11, 2024

matt/feat/recursive ctes/config flag matthewgapp/arrow-datafusion#3

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support User Defined Table Function #8306

Support User Defined Table Function #8306

Veeupup commented Nov 22, 2023 •

edited by alamb

Loading

Veeupup commented Nov 22, 2023

alamb commented Nov 22, 2023

alamb left a comment •

edited

Loading

alamb Nov 22, 2023

Veeupup Nov 23, 2023

alamb Nov 22, 2023

Veeupup Nov 23, 2023 •

edited

Loading

Veeupup Nov 25, 2023

alamb Nov 29, 2023

alamb Nov 30, 2023

Veeupup commented Nov 23, 2023 •

edited

Loading

Veeupup commented Nov 27, 2023

alamb left a comment

alamb Nov 28, 2023

alamb Nov 28, 2023

Jesse-Bakker Nov 28, 2023 •

edited

Loading

alamb Nov 28, 2023

alamb Nov 28, 2023

alamb left a comment

Veeupup commented Nov 30, 2023 •

edited

Loading

alamb commented Nov 30, 2023

alamb commented Nov 30, 2023

alamb commented Nov 30, 2023

Support User Defined Table Function #8306

Support User Defined Table Function #8306

Conversation

Veeupup commented Nov 22, 2023 • edited by alamb Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Veeupup commented Nov 22, 2023

alamb commented Nov 22, 2023

alamb left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Veeupup Nov 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Veeupup commented Nov 23, 2023 • edited Loading

Veeupup commented Nov 27, 2023

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jesse-Bakker Nov 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

Veeupup commented Nov 30, 2023 • edited Loading

alamb commented Nov 30, 2023

alamb commented Nov 30, 2023

alamb commented Nov 30, 2023

Veeupup commented Nov 22, 2023 •

edited by alamb

Loading

alamb left a comment •

edited

Loading

Veeupup Nov 23, 2023 •

edited

Loading

Veeupup commented Nov 23, 2023 •

edited

Loading

Jesse-Bakker Nov 28, 2023 •

edited

Loading

Veeupup commented Nov 30, 2023 •

edited

Loading