Library Guide: Building LogicalPlans

[alamb]

Is your feature request related to a problem or challenge?

Part of #7014

If we want to have DataFusion used as the core of many new systems, we need it to be as easy as possible for someone to get their idea working on top of DataFusion.

Thanks to @tshauck we now have a basic Library Users Guide ❤️ and this ticket describes expanding it out

Describe the solution you'd like

Fill in the content of https://arrow.apache.org/datafusion/library-user-guide/building-logical-plans.html

Example Outline

1. Building LogicalPlans directly
2. Building LogicalPlans using LogicalPlanBuilder
3. Translating LogicalPlan --> ExecutionPlan via physical planner

The examples directory holds a bunch more of examples: https://github.com/apache/arrow-datafusion/tree/main/datafusion-examples

Describe alternatives you've considered

No response

Additional context

No response

[edmondop]

@alamb I checked the previous @andygrove PR and it seems like we have a good setup. I read the docs, and noticed they lack a paragraph on how to translate the LogicalPlan into Physical Plan .

What do you think we should do? I can add that paragraph, but maybe we also want to link the examples? What else do you think it's needed to close this

[alamb]

What do you think we should do? I can add that paragraph, but maybe we also want to link the examples? What else do you think it's needed to close this

Those both sound like excellent ideas to close off this issue. Thank you @edmondop

[edmondop]

What do you think we should do? I can add that paragraph, but maybe we also want to link the examples? What else do you think it's needed to close this

Those both sound like excellent ideas to close off this issue. Thank you @edmondop

I noticed that the original work from @andygrove referred examples in the docs and these seems to be pretty "unique" in the sense that no other doc page seems to take this approach , see https://github.com/apache/datafusion/tree/main/docs/src

What do you think about moving those into examples and linking them from the docs? Do we have a preprocessor macro to includes the Rust code in a markdown upon compilation, or maybe to ensure that the embedded snippets compile and execute correctly ?

[alamb]

I think @andygrove started on a pre-processor that extracted the markdown and tested it here #7956

Maybe we can revive that idea

[edmondop]

I have created #10768 which I think would be a replacement of the Python scripts with something maintained somewhere else

[jc4x4]

I've added a first draft to the doc. Let me know if you would look for a more sophisticated example.