[draft] Add `LogicalType`, try to support user-defined types #11160

notfilippo · 2024-06-28T09:26:10Z

Proposal

The proposal now lives here: #11513, the original draft PR follows:

Which issue does this PR close?

Closes #7923
Follows up on #8143, which is stale.

In the current state the PR is a draft implementation to validate the idea based on the discussion from #7421.

New additions

LogicalType enum.
ExtensionType trait. Abstraction for extension types.
TypeSignature struct. Uniquely identifies a data type.
LogicalSchema & LogicalField, equivalent to arrow's Schema and Field
- note: this was mostly a shortcut to be able to refactor somewhat easily
  without changing much of the logic of of DFSchema. In next iterations DFSchema and LogicalSchema could potentially merge.

Major changes

DFSchema uses LogicalSchema & LogicalField.
ExprSchemable and ExprSchema now use LogicalType.
ast to logical plan conversion now uses LogicalType.

To be implemented

Registering extension types through a ContextProvider.

To be determined

Note

Most of these open questions remain similar to the initial PR.

Should ScalarValue use LogicalType or arrow DataType?
- LogicalType
- DataType
Should TableSource return LogicalSchema or arrow Schema?
- Schema
- LogicalSchema (for now)

wjones127

Lot's to absorb here, so I'll be looking closer soon. But left some initial observations.

wjones127 · 2024-06-28T18:59:47Z

datafusion/common/src/logical_type/mod.rs

+pub mod fields;
+
+#[derive(Clone, Debug)]
+pub enum LogicalType {


There's probably a discussion to be had about what level we want logical type to be here. We previously discussed this, but I can't find that discussion anymore. (@alamb might remember.)

Some notes:

IMO the Large list/binary/string variants should be considered encodings of their respective logical types. IMO considering these different types is one of the biggest sources of friction in the Arrow ecosystem and one I would hope would be solved by a

There's probably a good argument for signed and unsigned integers, as well as decimals, to be grouped together as more general types. That is, DataType::Int8 and DataType::Int64 could both be an encoded version of LogicalType::Int.

Date32 and Date64 might be consolidated. Date64 is a weird type that exists for Pandas compatibility, and doesn't actually provide more precision.

Types where the choice provides meaningful differences in precision should be preserved as logical types. For example, I think Float64 and Float16 shouldn't be considered the same logical type. Similarly for timestamp precisions.

I generally agree with all the points discussed and this matches my intuition.

The current state of the PR mostly reflects the state for the original proposal in #8143 as I didn't want to include any additional changes without discussing them first. I'm not too familiar on how these changes should be discussed and if these points need to be also circulated in the mailing list. Let me know what's best :)

There's probably a discussion to be had about what level we want logical type to be here. We previously discussed this, but I can't find that discussion anymore. (@alamb might remember.)

I also remember but can't find this discussion

As I recall, the conclusion was that types that can store different ranges of values shoudl be different logical types.

So that would mean Int32 should be different than UInt32

I don't recall consensus about String vs LargeString, but I agree that treating these as the same logical type with different phsical encodings would be the easiest to reason about (even though theoretically LargeString can represent strings larger than 2GB that can not be represented in String)

I've added support for treating large types as their non-large logical type. I agree that types that can store different ranges of values should be different logical types

wjones127 · 2024-06-28T19:10:31Z

datafusion/common/src/logical_type/extension.rs

+pub trait ExtensionType: std::fmt::Debug {
+    fn display_name(&self) -> &str;
+    fn type_signature(&self) -> TypeSignature;
+    fn physical_type(&self) -> DataType;


Would this mean we need to have a variant of each extension type for each physical storage type?

A good built-in example is string. Right now LogicalType::String always returns DataType::String, but arguably could be DataType::StringView or DataType::LargeString.

Yup! Any LogicalType which has different physical representations should probably be implemented through multiple ExtensionTypes.

I've refactored the code a bit to keep a reference to the physical type.

alamb · 2024-06-30T12:23:30Z

I plan to check this out shortly -- thanks @notfilippo

jayzhan211 · 2024-07-01T03:45:27Z

datafusion/common/src/logical_type/type_signature.rs

+use std::sync::Arc;
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub struct TypeSignature {


We have existing TypeSignature for function

datafusion/datafusion/expr/src/signature.rs

Line 88 in a64df83

pub enum TypeSignature {

Therefore, I think what we need is rewriting existing signature with LogicalType, instead of introducing another similar concept here. Or we create a similar one and gradually replace the existing signature.

alamb

Thank you @notfilippo for reviving this PR.

I think this PR does a pretty good job of sketching out the structures required for logical types.

One thing I think would help me understand how complete / not complete this proposal is would be to show how we would use this code to create a user defined type. Would it be possible to sketch out an example, perhaps in https://github.com/apache/datafusion/tree/main/datafusion-examples, showing how we would use LogicalType to create a User Defined Type?

Obviously if we pursue this approach, there will be substantial churn for all downstream crates.

cc @yukkit @lewiszlw @samuelcolvin and @jayzhan211

Also this could be related to StringViewArray #10918 / @XiangpengHao

alamb · 2024-07-01T12:09:49Z

datafusion/common/src/logical_type/mod.rs

+pub mod fields;
+
+#[derive(Clone, Debug)]
+pub enum LogicalType {


There's probably a discussion to be had about what level we want logical type to be here. We previously discussed this, but I can't find that discussion anymore. (@alamb might remember.)

I also remember but can't find this discussion

As I recall, the conclusion was that types that can store different ranges of values shoudl be different logical types.

So that would mean Int32 should be different than UInt32

I don't recall consensus about String vs LargeString, but I agree that treating these as the same logical type with different phsical encodings would be the easiest to reason about (even though theoretically LargeString can represent strings larger than 2GB that can not be represented in String)

alamb · 2024-07-01T12:12:45Z

datafusion/common/src/scalar/mod.rs

+    type Error = DataFusionError;
+
+    /// Create a Null instance of ScalarValue for this datatype
+    fn try_from(data_type: &LogicalType) -> Result<Self> {


I would tend to agree that ScalarValue should follow LogicalType -- in fact we have had several issues when scalar value types don't quite match (like a logical string encoded as ScalarValue::String was compared to a DictionaryArray and was treated as a different type

It makes sense. I'll add a note to refactor ScalarValues at some point

notfilippo · 2024-07-02T14:35:41Z

@alamb I've added an example with some comments and TODOs remarking my open questions.

alamb

This is very cool @notfilippo

In addition to supporting user defined types, I think adding the notion of a LogicalType would make it easier for DataFusion to take better advantage of different encodings (specifically RunArray, StringViewArray and DictionaryArray)

We already have ok support for DictionaryArray but it is somewhat unnatural and required adding special DataType::Dictionary support all over the codebase and had various bugs like #11145 and #11032 that resulted when we missed support in some non obvious place

As we begin adding support for StringViewArray in #10918 the same pattern is emerging

That is all to say, I think we should pursue this idea

The biggest challenge of making this kind of change I think will be to manage the rollout and migration with downstream crates / make the transition as smooth as possible

So my suggestions for next steps are:

Try to update some downstream dependency to see what the change might require. Perhaps we can try with https://github.com/apache/datafusion-comet as that is open source
Write a more detailed plan (the proposal seems similar to what is in [Proposal] Support User-Defined Types (UDT) #7923) for what we are going to change (with a specific section "What this would mean for downstream crates")

I think I can find time to help with the writeup and communication, and maybe the code. I want to finish up some other inflight work first (specifically finishing porting aggregate functions, cleaning up parquet statistics, and the user defined sql parser)

cc @yukkit @samuelcolvin

alamb · 2024-07-03T12:27:50Z

datafusion-examples/examples/logical_type.rs

+    let df = ctx.sql("SELECT * FROM example").await?;
+    let records = df.collect().await?;
+
+    println!("{}", pretty_format_batches(&records)?);


Is the primary usecase of the LogicalType so that we could write functions that take the logical type, or define custom behavior?

Like I wonder is the idea that we could now create a ScalarFunctionImpl whose signature refers to LogicalType rather than PhysicalType 🤔 Or somehow plan a binary operation on logical types?

BTW I think the work we are doing with @samuelcolvin @dharanad and @jayzhan211 in #11207 would make it straightforward to implement a custom comparsion (via a function) for this magical type

That might be a good thing to show too

Like I wonder is the idea that we could now create a ScalarFunctionImpl whose signature refers to LogicalType rather than PhysicalType 🤔 Or somehow plan a binary operation on logical types?

Potentially both. The motivation behind this change is to simplify the interaction between standard types represented by different encodings (like RunArray, the various Views and DictionaryArray but potentially also user defined ones via Arrow Extension Types).

notfilippo · 2024-07-03T14:58:38Z

The biggest challenge of making this kind of change I think will be to manage the rollout and migration with downstream crates / make the transition as smooth as possible.

Completely agree. I will try to experiment with the user facing APIs (e.g. what's returned by the schema() method of structs implementing TableSource / TableProvider). I also think that most of the usage of DataType in physical and logical plans should be replaced with TypeRelation. I'll try to open a draft of what that would look like in comet in the coming days.

alamb · 2024-07-05T15:57:02Z

Thanks @notfilippo -- I will try and get the other projects I have under way to a better state so I can more fully help plan / communicate / coordinate this one.

notfilippo · 2024-07-05T19:30:54Z

So I can more fully help plan / communicate / coordinate this one.

Sounds good! Feel free to follow up here / on slack / on discord 😄

I've also noticed that this potential change could greatly benefit the substrait encoding / decoding of the logical plan. Its current implementation has troubles dealing with dictionaries. I'll look into that as well while waiting for further instructions.

notfilippo · 2024-07-17T12:33:17Z

@alamb, @wjones127, @jayzhan211 -- I've found some time to finally draft a proposal: [Proposal] Decouple logical from physical types

(this draft PR was updated to DataFusion v40 in order to test datafusion-comet)

github-actions bot added sql SQL Planner logical-expr Logical plan and expressions physical-expr Physical Expressions optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) substrait labels Jun 28, 2024

notfilippo force-pushed the fr/logical-types branch from fa0ef9e to c30746c Compare June 28, 2024 09:55

wjones127 reviewed Jun 28, 2024

View reviewed changes

draft: add logical types and support extension types

5427f4b

alamb mentioned this pull request Jun 30, 2024

DataFusion weekly project plan (Andrew Lamb) - June 24, 2024 #11106

Closed

9 tasks

jayzhan211 reviewed Jul 1, 2024

View reviewed changes

alamb mentioned this pull request Jul 1, 2024

DataFusion weekly project plan (Andrew Lamb) - July 1, 2024 #11190

Closed

10 tasks

alamb reviewed Jul 1, 2024

View reviewed changes

notfilippo added 2 commits July 2, 2024 14:42

Add type relation

996bbdc

Merge remote-tracking branch 'origin/main' into fr/logical-types

f8facb1

notfilippo force-pushed the fr/logical-types branch from c30746c to f8facb1 Compare July 2, 2024 13:05

Add example with TODOs

d65e857

alamb reviewed Jul 3, 2024

View reviewed changes

notfilippo and others added 6 commits July 8, 2024 10:59

Format

ce68d1b

Merge branch 'main' into fr/logical-types

f5f79bb

Merge remote-tracking branch 'origin/main' into fr/logical-types

481f6ca

Format

9fd793a

Union mode

8b9715d

Prepare 40 release

981486a

andygrove added 2 commits July 8, 2024 19:48

regenerate changelog

43fb645

manually remove a reverted PR from the breaking change section

4cae813

alamb mentioned this pull request Jul 12, 2024

2024 Q3-Q4 Roadmap? #11442

Open

Merge remote-tracking branch 'origin/branch-40' into fr/logical-types

e1eca31

github-actions bot added documentation Improvements or additions to documentation development-process Related to development process of DataFusion labels Jul 17, 2024

Rename types to reflect proposal

5be9029

notfilippo mentioned this pull request Jul 17, 2024

[Proposal] Decouple logical from physical types #11513

Open

2 tasks

kylebarron mentioned this pull request Jul 22, 2024

Spatial data support #7859

Open

alamb mentioned this pull request Jul 22, 2024

[draft] Add LogicalType, try to support user-defined types #8143

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[draft] Add `LogicalType`, try to support user-defined types #11160

[draft] Add `LogicalType`, try to support user-defined types #11160

notfilippo commented Jun 28, 2024 •

edited

Loading

wjones127 left a comment

wjones127 Jun 28, 2024

notfilippo Jun 28, 2024 •

edited

Loading

alamb Jul 1, 2024

notfilippo Jul 2, 2024

wjones127 Jun 28, 2024

notfilippo Jun 28, 2024 •

edited

Loading

notfilippo Jul 2, 2024

alamb commented Jun 30, 2024

jayzhan211 Jul 1, 2024 •

edited

Loading

alamb left a comment

alamb Jul 1, 2024

alamb Jul 1, 2024

notfilippo Jul 2, 2024 •

edited

Loading

notfilippo commented Jul 2, 2024 •

edited

Loading

alamb left a comment

alamb Jul 3, 2024

notfilippo Jul 3, 2024

notfilippo commented Jul 3, 2024

alamb commented Jul 5, 2024

notfilippo commented Jul 5, 2024

notfilippo commented Jul 17, 2024 •

edited

Loading

[draft] Add LogicalType, try to support user-defined types #11160

Are you sure you want to change the base?

[draft] Add LogicalType, try to support user-defined types #11160

Conversation

notfilippo commented Jun 28, 2024 • edited Loading

Proposal

Which issue does this PR close?

New additions

Major changes

To be implemented

To be determined

wjones127 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

notfilippo Jun 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

notfilippo Jun 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb commented Jun 30, 2024

jayzhan211 Jul 1, 2024 • edited Loading

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

notfilippo Jul 2, 2024 • edited Loading

Choose a reason for hiding this comment

notfilippo commented Jul 2, 2024 • edited Loading

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

notfilippo commented Jul 3, 2024

alamb commented Jul 5, 2024

notfilippo commented Jul 5, 2024

notfilippo commented Jul 17, 2024 • edited Loading

[draft] Add `LogicalType`, try to support user-defined types #11160

[draft] Add `LogicalType`, try to support user-defined types #11160

notfilippo commented Jun 28, 2024 •

edited

Loading

notfilippo Jun 28, 2024 •

edited

Loading

notfilippo Jun 28, 2024 •

edited

Loading

jayzhan211 Jul 1, 2024 •

edited

Loading

notfilippo Jul 2, 2024 •

edited

Loading

notfilippo commented Jul 2, 2024 •

edited

Loading

notfilippo commented Jul 17, 2024 •

edited

Loading