Skip to content

[GLUTEN-7031][VL] Minimized backend API#7218

Merged
zhztheplayer merged 7 commits intoapache:mainfrom
zhztheplayer:wip-mini-be-api
Sep 13, 2024
Merged

[GLUTEN-7031][VL] Minimized backend API#7218
zhztheplayer merged 7 commits intoapache:mainfrom
zhztheplayer:wip-mini-be-api

Conversation

@zhztheplayer
Copy link
Member

@zhztheplayer zhztheplayer commented Sep 13, 2024

Part of #7031

Add minimized backend API in gluten-core. The new API will consist of:

  1. Static initializers to inject into Gluten's loading procedure;
  2. Planner rules to convert Spark query plan to backend's query plan;
  3. Backend's columnar batch type definition, so Gluten's query planner code make decisions to add transitions (C2R / R2C / C2C) among computations in this backend and vanilla Spark, Arrow, or other backends.

And will rename previous Backend in gluten-substrait module to SubstraitBackend and make it extend Backend. The rich APIs will still be extensible for backends relying on gluten-substrait.

Future works:

  1. Cleanup BackendApiManager code in gluten-substrait
  2. Remove SkipCondition
  3. gluten-data cleanups (will open ticket for this)

@github-actions github-actions bot added CORE works for Gluten Core VELOX CLICKHOUSE labels Sep 13, 2024
@github-actions
Copy link

#7031

@github-actions
Copy link

Run Gluten Clickhouse CI

@github-actions
Copy link

Run Gluten Clickhouse CI

@github-actions
Copy link

Run Gluten Clickhouse CI

@github-actions
Copy link

Run Gluten Clickhouse CI

@github-actions
Copy link

Run Gluten Clickhouse CI

@github-actions
Copy link

Run Gluten Clickhouse CI

@zhztheplayer zhztheplayer marked this pull request as ready for review September 13, 2024 02:32
@zhztheplayer zhztheplayer merged commit b69a6e8 into apache:main Sep 13, 2024
sharkdtu pushed a commit to sharkdtu/gluten that referenced this pull request Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLICKHOUSE CORE works for Gluten Core VELOX

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants