Repository navigation
[Fusion] Fix the operation planner failing to plan large federated schemas and discarding cheaper plans - #10515
Open
michaelstaib wants to merge 13 commits into
Open
michaelstaib wants to merge 13 commits into
michaelstaib wants to merge 13 commits into
Conversation
…across two schemas
…ery benchmarks in process
…ith the best complete plan
…building their operation
…at owns only the key fields
…ld twice and merging lookups with nested field conflicts
…ot candidate when the cheapest one dead-ends
michaelstaib
marked this pull request as ready for review
October 9, 2026 01:43
… with the production field map pool and the default job
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #10485. On large federated schemas the operation planner could fail to find any plan, and on every schema it could discard cheaper plans because its pruning bound was not a lower bound. This PR fixes both and bounds the extra search the correct bound causes.
Fixes
mainare tracked separately.OperationPlanner.DeferRequirementChain.csand lifted steps keep theirOp_defer_*names.Public changes
OperationPlannerOptions.MaxPlanImprovementNodes(int?, default 4096). Once a complete plan exists, the search expands at most this many further nodes and then returns the best complete plan found.nulldisables the budget. The budget is shared by the root search and all deferred searches, and aPlanImprovementBudgetExhaustedplanner event is emitted when it ends a search. Guardrail semantics are unchanged.OperationPlannerOptionsrejects non-finite or negative cost weights.OperationPlanner.Versionmoves from 2.1.0 to 2.2.0, since existing operations can plan differently.Cost
The correct bound makes the search expand far more nodes before it can prove a plan optimal. Measured on a dedicated idle Linux box (16 cores, Release, full BenchmarkDotNet jobs and 50-run planning medians),
main8fbb242 against this branch d61d46f:The cost per expansion fell by 9 to 63 percent; the increase is entirely expansion count. Plans are cached per operation, so this is paid once per distinct operation and on every schema reload. On the private 745-subgraph corpus, expansion counts went from no plan within the time limit to 1,870 for the 664-line operation, and the 1,683-line operation stops at the 4,096-node budget with the same plan as before; its timing was not measured on the box.
Whether this cost is acceptable for the cheaper plans, or whether the bound should wait for the search-space work, is an open decision on this PR. A sweep of the improvement budget default is being measured to see whether a lower default recovers most of the time while keeping the plan gains.
Internal
PlannerCorpusTests, plans every operation in a local schema corpus given byFUSION_PLANNER_CORPUS_DIR(withFUSION_PLANNER_CORPUS_OVERRIDE_DIRto replace source schema files by name). It is skipped when the variable is unset and commits no corpus content.Benchmarks
The numbers above come from a pass on a dedicated idle Linux box with the load average checked before and after every run. The two tests that flaked on the loaded laptop passed 20 of 20 there. The planning harness used for the pass is being committed as a benchmark class, together with a fix for
OperationPlannerBenchmarkreusing a planner whose pooled field map is never cleared.Deferred to a separate proposal
Costing the executed plan shape in the search and a stronger admissible bound, which would recover the cold-planning cost above.