StatisticsV2: statistics framework initial redesign for Datafusion

[Fly-Style]

Rationale for this change

https://synnada.notion.site/Redesigning-and-Enhancing-the-Statistics-Framework-in-Datafusion-16bf46d2dab180448272dbbd1d1f7cea

What changes are included in this PR?

This patch presents a Statistics v.2 framework with the following main points:

• introduction to enum-based struct to support multiple distribution types, which initially include:
  • Uniform distribution (range)
  • Gaussian distribution (also known as normal)
  • Exponential distribution
  • Bernoulli distribution (holds probability, is used as resulting distribution of comparison operators),
  • Unknown distribution, abstracts any non-represented distribution.
• tree-based stats calculation is divided into bottom-up evaluation and top-down propagation;

Tables of statistic execution and propagation rules for PhysicalExpr-s:

Definitions

Details

UF = Uniform UN = Unknown EXP = Exponential GSS = Gaussian BRN = Bernoulli

Binary arithmetical operators, evaluation.

Details

Input 1	Input 2	(+)	(-)	(*)	(/)
UF [a,b]	UF [c,d]	UN	UN	UN	UN
UF [a,b]	GSS N(μ,σ²)	UN	UN	UN	UN
UF [a,b]	EXP Exp(λ)	UN	UN	UN	UN
UF [a,b]	BRN	UN	UN	UN	UN
UF [a,b]	UN	UN	UN	UN	UN
GSS N(μ₁,σ₁²)	GSS N(μ₂,σ₂²)	GSS	GSS	GSS	UN
GSS N(μ,σ²)	EXP Exp(λ)	UN	UN	UN	UN
GSS N(μ,σ²)	BRN	UN	UN	UN	UN
GSS N(μ,σ²)	UN	UN	UN	UN	UN
EXP Exp(λ₁)	EXP Exp(λ₂)	UN (gamma dist)	UN	UN	UN
EXP Exp(λ)	BRN	UN	UN	UN	UN
EXP Exp(λ)	UN	UN	UN	UN	UN
BRN Exp(λ)	BRN	UN	UN	UN	UN
BRN Exp(λ)	UN	UN	UN	UN	UN
UN	UN	UN	UN	UN	UN

Comparison operators, evaluation.

Details

Input 1	Input 2	Comparison (=, ≠)	Inequality (>, >=)	Inequality (<, <=)
UF [a,b]	UF [c,d]	BRN	BRN	BRN
GSS N(μ₁,σ₁²)	GSS N(μ₂,σ₂²)	UN	UN(Ф-func)	UN(Ф-func)
EXP Exp(λ₁)	EXP Exp(λ₂)	UN	UN	UN
BRN	BRN	BRN	BRN	BRN
UF [a,b]	GSS N(μ,σ²)	UN	UN	UN
UF [a,b]	EXP Exp(λ)	UN	UN	UN
UF [a,b]	BRN	UN?	UN?	UN?
UF [a,b]	UN	BRN?	BRN?	BRN?
GSS N(μ,σ²)	EXP Exp(λ)	UN	UN	UN
GSS N(μ,σ²)	BRN	BRN?/UN	BRN?/UN	BRN? /UN
GSS N(μ,σ²)	UN	UN	UN	UN
EXP Exp(λ)	UN	UN, estimate	UN, estimate	UN, estimate
EXP Exp(λ)	BRN	UN, estimate	UN, estimate	UN, estimate
UN	BRN	BRN/UN	BRN/UN	BRN/UN
UN	UN	BRN/UN	BRN/UN	BRN/UN

Are these changes tested?

These changes were massively tested, you can find a lot of unit tests coupled to this feature.

[berkaysynnada]

I really like how the code is organized, and your clean coding style, even though it’s still draft.

While reviewing ExprStatisticGraph, I wondered if introducing another distribution, singleton, to represent constants and known values, might improve us. It seems like this could simplify the representation of literals significantly. However, I’m not sure if it would complicate the computation process. Let’s discuss this too.

I’ll also think more about the propagation of distributions. However, it seems unlikely that the current approach for range propagations can be improved further.

[berkaysynnada]

I've sent a commit that has mostly minor code improvements. If you see any arguable change, we can discuss it.

What I see as a further improvement is implementing CastExpr methods as it is used very widely. Another major part is for ScalarFunctionExpr's. If we can also practice one of them (we can select the most trivial one), that would be highly beneficial.

[berkaysynnada]

Once we resolve these last items, @ozankabak will take a final look, and we can upstream this PR.

[berkaysynnada]

apache#14699 is merged