Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

We can hack Join operator not just logical relation operator. #747

Open
zhexuany opened this issue May 21, 2019 · 0 comments
Open

We can hack Join operator not just logical relation operator. #747

zhexuany opened this issue May 21, 2019 · 0 comments

Comments

@zhexuany
Copy link
Contributor

zhexuany commented May 21, 2019

case class TiStrategy(getOrCreateTiContext: SparkSession => TiContext)(sparkSession: SparkSession)

Too many hacks here since we hack the planning but we don't have full control over the planning stage. We cannot pass context around during planning so a re-extract needed for push-down since a plan tree might contain Join which causes a single tree has multiple plans to push-down.

override def apply(plan: LogicalPlan): Seq[SparkPlan] = {

Currently, we only hack the logical relation. We can do the same thing on the Join operator.

@birdstorm birdstorm changed the title We can hijack Join operator not just logical relation operator. We can hack Join operator not just logical relation operator. May 21, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants