[native pos] Fail, not hang, broadcast queries#19676
[native pos] Fail, not hang, broadcast queries#19676mbasmanova merged 2 commits intoprestodb:masterfrom
Conversation
|
Isn't it better to implement this check before the query execution starts (AbstractPrestoSparkQueryExecution.java#L785 )?, than during task-execution ? |
This is a good question. It would be nice to avoid duplicating the logic of what's supported and what is not. One option is to add an API to Prestissimo to check if plan fragment is supported. Then use this API on the driver to check that all fragments are supported before starting query execution. |
|
That would be ideal that this information lies with prestissimo. |
I think we can make changes to start native process on the driver for the purpose of plan validation. (We can also use it for constant folding of expressions that use external functions.) |
|
I am split between just hardcoding the info (simple but hardcoded) of whats currently supported v/s starting a sidecar to evaluate this info (complex but clean)
This might be a strong reason to add sidecar. I don't understand this use-case fully. Is there no other way to accomplish this. |
|
Thanks @mbasmanova for explaining constant folding of expressions, for external functions. Looks like we will need Velox library to evaluate these during planning. So makes sense that we have to go with sidecar approach. Do you want to still merge this PR ? This one might become throwaway work once we implement the sidecar solution. |
I think we still need to have proper checks in the C++ process as it cannot rely on the client (Java process) to always behave well. |
shrinidhijoshi
left a comment
There was a problem hiding this comment.
LGTM! Thanks for working on this
cac4d4d to
80011e2
Compare
Broadcast queries used to hang forever. Now these queries fail quickly with a clear error message: > Broadcast shuffle is not supported
xiaoxmeng
left a comment
There was a problem hiding this comment.
@mbasmanova LGTM. thanks!
…e mode This mode is currently not supported. Ignoring this flag may cause hard-to-debug incorrect results.
Broadcast queries used to hang forever. Now these queries fail quickly with a clear error message:
Also, check for replicate-nulls-and-any shuffle mode and reject these queries as well.