Evaluate ValuesExec's exprs during execution

[lewiszlw]

Is your feature request related to a problem or challenge?

Now ValuesExec's exprs will be evaluated during planning. This might make planning longer and ValuesExec can't be serialized in protobuf.

#[derive(Debug)]
pub struct ValuesExec {
    /// The schema
    schema: SchemaRef,
    /// The data
    data: Vec<RecordBatch>,
    /// Cache holding plan properties like equivalences, output partitioning etc.
    cache: PlanProperties,
}

What about making ValuesExec contains physical exprs and evaluating during execution?

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[alamb]

Now ValuesExec's exprs will be evaluated during planning. This might make planning longer and ValuesExec can't be serialized in protobuf.

Could we serialize the record batch using arrow IPC?

[lewiszlw]

I think we could. But serializing exprs seems cheaper as the generated record batch might be large.

[alamb]

I think we could. But serializing exprs seems cheaper as the generated record batch might be large.

In the other hand, many Exprs may be expensive too -- I bet that for small/medium sized values Expr would be better as the overhead of a RecordBatch will be non trivial. But then as the number of rows increases at some point serializing the RecordBatch would be better

I bet it would be straightforward to test using https://docs.rs/arrow-ipc/52.2.0/arrow_ipc/writer/struct.StreamWriter.html to write to a vec (I was hoping for a nice example but seems like the docs aren't great yet)

[lewiszlw]

For example, explain select * from values (generate_series(1, 100000)); will take 8s to complete in my machine.

[alamb]

For example, explain select * from values (generate_series(1, 100000)); will take 8s to complete in my machine.

It would be fascinating to run a profile / flamegraph and see where the time was going...

We should probably also change the EXPLAIN code to truncate showing the contents of VALUES after a while...