[SPARK-38138][SQL] Materialize QueryPlan subqueries

[pan3793]

What changes were proposed in this pull request?

This PR propose to materialize QueryPlan#subqueries and pruned by PLAN_EXPRESSION on searching to improve the SQL compile performance.

Why are the changes needed?

We found a query in production that cost lots of time in optimize phase (also include AQE optimize phase) when enable DPP, the SQL pattern likes

select <cols...>
from a
left join b on a.<col> = b.<col>
left join c on b.<col> = c.<col>
left join d on c.<col> = d.<col>
left join e on d.<col> = e.<col>
left join f on e.<col> = f.<col>
left join g on f.<col> = g.<col>
left join h on g.<col> = h.<col>
...

SPARK-36444 significantly reduces the optimize time (exclude AQE phase), see detail at #35431, but there are still lots of time costs in InsertAdaptiveSparkPlan on AQE optimize phase.

Before this change, the query costs 658s, after this change only costs 65s.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing UTs.

[pan3793]

cc @wangyum @cloud-fan @yaooqinn

[HyukjinKwon]

cc @maryannxue @allisonwang-db @sigmod FYI

[ulysses-you]

please add @transient

[pan3793]

Thanks for tips, updated.

[amaliujia]

Just for education purpose: why @transient is useful here?

[pan3793]

SparkPlan is the subclass of QueryPlan, which need to be sent to executor, use @transient to reduce the memory usage of executor.

abstract class SparkPlan extends QueryPlan[SparkPlan] with Logging with Serializable

[amaliujia]

cc @amaliujia

[AmplabJenkins]

Can one of the admins verify this patch?

[pan3793]

@cloud-fan would you please take a look? thanks

[cloud-fan]

thanks, merging to master!