-
Notifications
You must be signed in to change notification settings - Fork 5.5k
Description
Currently, native sidecar is available to have coordinator read the function registry from the worker node. This would allow inline-sql-functions to be enabled, and it allows users to invoke native only functions. Some current gaps exists in that C++ constant folding is not yet available, and there exists a special class of functions that exist only on the coordinator (like current_timestamp). Both these two gaps have not yet been addressed, but ideally these will be fixed in the medium term.
The change outlined here #25826 is created to meet these requirements:
- Allow users to access worker only functions
- Maintain existing behavior for functions with Java implementations by evaluating in coordinator and keeping constant folding
- Override SQL functions in coordinator if a corresponding implementation exists in the worker to take advantage of better performance of native implementation
- Allow users to continue using coordinator-only functions without specifying fully qualified function name
Call this solution B. Solution A #25818 is the change to move built in SQL functions into a plugin so that functions can be selectively registered and to use native.default as the default function namespace prefix. Currently, solution B supports all requirements but solution A does not meet the 4th requirement.
Here is the migration plan to add solution B as an opt-in feature for the short term:
- Merge in the change Native built in namespace manager #25826.
- Collect a comprehensive list of the functions that are only available in coordinator. This is accomplished by by shadowing production traffic using solution A. This would also serve as production testing to identify any other potential gaps in solution A. Work to implement these in C++.
- Once the gaps are understood, revert the change of solution B, and make solution A to be the solution that will handle all the requirements outlined above.
Both solutions A and B use a hack to always force functions that have java implementation to be evaluated in the coordinator so that it can be constant folded. Once this gap is fixed, tweak solution A so that it no longer relies on this hack.
Metadata
Metadata
Labels
Type
Projects
Status