Skip to content

[Fusion] Fix the operation planner failing to plan large federated schemas and discarding cheaper plans - #10515

Open
michaelstaib wants to merge 13 commits into
mainfrom
mst/planner-followups
Open

michaelstaib wants to merge 13 commits into
mainfrom
mst/planner-followups

Conversation

@michaelstaib

@michaelstaib michaelstaib commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

Follow-up to #10485. On large federated schemas the operation planner could fail to find any plan, and on every schema it could discard cheaper plans because its pruning bound was not a lower bound. This PR fixes both and bounds the extra search the correct bound causes.

Fixes

  • Lookup candidates that resolve nothing beyond echoing their entry key are no longer generated, so two schemas cannot hand the same unresolvable selection back and forth. On an anonymized 745-subgraph schema (32,020 composed types) an operation that found no plan within 120 s now plans in about 0.1 s.
  • The greedy pass keeps each step's sibling candidates and backtracks to the next cheapest when the chosen candidate dead-ends, instead of returning no plan. Before, 55 of about 300 test plans started their search without a plan to prune against.
  • The pruning bound is now a lower bound on the remaining work. Verified by a suite-wide assertion (184,275 checks, no violations) and by confirming that no test plan is cheaper with pruning disabled. Every changed plan is equal or cheaper by the planner's objective; five are strictly cheaper.
  • Lookups into a schema that owns only the key fields of a type are skipped. No plan changes; the corpus operation above needs 20 percent fewer expansions.
  • Children that cannot beat the current best plan are skipped before their operation document is built. No plan changes.
  • The greedy pass also falls back to the next root candidate when the cheapest root candidate dead-ends.
  • A merged requirement field is selected once when both requirements provably read the same value, and a lookup merge is declined when a nested selection conflict is detected. Nested conflicts that still return wrong data on main are tracked separately.
  • Deferred requirement work that the enclosing scope can host is lifted there when doing so does not deepen the main plan, so the initial payload is never delayed; otherwise the deferred plan stays self-contained. The lift lives in OperationPlanner.DeferRequirementChain.cs and lifted steps keep their Op_defer_* names.

Public changes

  • OperationPlannerOptions.MaxPlanImprovementNodes (int?, default 4096). Once a complete plan exists, the search expands at most this many further nodes and then returns the best complete plan found. null disables the budget. The budget is shared by the root search and all deferred searches, and a PlanImprovementBudgetExhausted planner event is emitted when it ends a search. Guardrail semantics are unchanged.
  • OperationPlannerOptions rejects non-finite or negative cost weights.
  • OperationPlanner.Version moves from 2.1.0 to 2.2.0, since existing operations can plan differently.

Cost

The correct bound makes the search expand far more nodes before it can prove a plan optimal. Measured on a dedicated idle Linux box (16 cores, Release, full BenchmarkDotNet jobs and 50-run planning medians), main 8fbb242 against this branch d61d46f:

case main this branch expansions
ShopPlanningTests, per plan 1.45 ms, 1.15 MB 36.5 ms, 27.9 MB 17 to 1,154
K6PlanTests DeepNesting, per plan 0.81 ms 3.70 ms 10 to 119
OperationPlannerBenchmark Complex 3.48 ms 20.4 ms
OperationPlannerBenchmark ConditionalRedundancy 0.93 ms 5.98 ms
OperationPlannerBenchmark Simple 1.18x

The cost per expansion fell by 9 to 63 percent; the increase is entirely expansion count. Plans are cached per operation, so this is paid once per distinct operation and on every schema reload. On the private 745-subgraph corpus, expansion counts went from no plan within the time limit to 1,870 for the 664-line operation, and the 1,683-line operation stops at the 4,096-node budget with the same plan as before; its timing was not measured on the box.

Whether this cost is acceptable for the cheaper plans, or whether the bound should wait for the search-space work, is an open decision on this PR. A sweep of the improvement budget default is being measured to see whether a lower default recovers most of the time while keeping the plan gains.

Internal

  • An opt-in corpus harness, PlannerCorpusTests, plans every operation in a local schema corpus given by FUSION_PLANNER_CORPUS_DIR (with FUSION_PLANNER_CORPUS_OVERRIDE_DIR to replace source schema files by name). It is skipped when the variable is unset and commits no corpus content.
  • The Fusion planner, compiler, rewriter, variable-merging and query benchmarks run in process so they work under the benchmark project's net11.0 target.

Benchmarks

The numbers above come from a pass on a dedicated idle Linux box with the load average checked before and after every run. The two tests that flaked on the loaded laptop passed 20 of 20 there. The planning harness used for the pass is being committed as a benchmark class, together with a fix for OperationPlannerBenchmark reusing a planner whose pooled field map is never cleared.

Deferred to a separate proposal

Costing the executed plan shape in the search and a stronger admissible bound, which would recover the cold-planning cost above.

@github-actions github-actions Bot added 📚 documentation This issue is about working on our documentation. 🌶️ hot chocolate labels Oct 8, 2026
@michaelstaib
michaelstaib marked this pull request as ready for review October 9, 2026 01:43
Copilot AI balanced review requested due to automatic review settings October 9, 2026 01:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

… with the production field map pool and the default job

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

📚 documentation This issue is about working on our documentation. 🌶️ hot chocolate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants