Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor(executor): implement time slicing for shared executor scheduling across queries #14594

Closed
wants to merge 11 commits into from
Closed

refactor(executor): implement time slicing for shared executor scheduling across queries #14594

wants to merge 11 commits into from

Conversation

dqhl76
Copy link
Collaborator

@dqhl76 dqhl76 commented Feb 4, 2024 •
edited
Loading

Still Work In Progress.

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

This PR introduces time slicing to enable different queries to share a single executor for scheduling.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@dqhl76 dqhl76 changed the title [WIP] refactor(executor): implement time slicing for shared executor scheduling across queries refactor(executor): implement time slicing for shared executor scheduling across queries Feb 4, 2024
@github-actions github-actions bot added the pr-refactor this PR changes the code base without new features or bugfix label Feb 4, 2024
@dqhl76
chore: fmt and clean code
dd90f23 
Signed-off-by: Liuqing Yue <[email protected]>
@dqhl76 dqhl76 force-pushed the add-basic-time-slice-patch2 branch from 45fb848 to dd90f23 Compare February 4, 2024 13:22
@zhang2014 zhang2014 added the ci-benchmark Benchmark: run all test label Feb 5, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 5, 2024

Docker Image for PR

  • tag: pr-14594-8331ed7

note: this image tag is only available for internal use,
please check the internal doc for more details.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 5, 2024

cloud cloud

ClickBench Report

@dqhl76 dqhl76 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Feb 5, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 5, 2024

Docker Image for PR

  • tag: pr-14594-815a0ad

note: this image tag is only available for internal use,
please check the internal doc for more details.

@dqhl76 dqhl76 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Feb 5, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 5, 2024

Docker Image for PR

  • tag: pr-14594-3c865f1

note: this image tag is only available for internal use,
please check the internal doc for more details.

@dqhl76 dqhl76 added ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits and removed ci-benchmark Benchmark: run all test labels Feb 5, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 5, 2024

Docker Image for PR

  • tag: pr-14594-b671567

note: this image tag is only available for internal use,
please check the internal doc for more details.

@dqhl76 dqhl76 added ci-benchmark Benchmark: run all test and removed ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits labels Feb 6, 2024
@dqhl76 dqhl76 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Feb 6, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 6, 2024

Docker Image for PR

  • tag: pr-14594-83ed889

note: this image tag is only available for internal use,
please check the internal doc for more details.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 6, 2024

Docker Image for PR

  • tag: pr-14594-14b3452

note: this image tag is only available for internal use,
please check the internal doc for more details.

@everpcpc everpcpc added ci-benchmark-local Benchmark: run only local test and removed ci-benchmark Benchmark: run all test labels Feb 6, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 6, 2024

Docker Image for PR

  • tag: pr-14594-7d5bad4

note: this image tag is only available for internal use,
please check the internal doc for more details.

@everpcpc everpcpc added ci-benchmark-local Benchmark: run only local test and removed ci-benchmark-local Benchmark: run only local test labels Feb 6, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 6, 2024

Docker Image for PR

  • tag: pr-14594-1666b78

note: this image tag is only available for internal use,
please check the internal doc for more details.

zhang2014
zhang2014 reviewed Feb 6, 2024
View reviewed changes
src/query/service/src/pipelines/executor/executor_tasks.rs
pub fn push_tasks(
&self,
worker_id: usize,
current_tasks: Option<VecDeque<ExecutorTask>>,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
current_tasks: Option<VecDeque<ExecutorTask>>,
current_tasks: VecDeque<ExecutorTask>,

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got That.

@dqhl76
Copy link
Collaborator Author

dqhl76 commented Feb 6, 2024

I am currently working on refactoring the modified logic of the Executor to make it more isolated and easier merging.

@dqhl76 dqhl76 mentioned this pull request Feb 19, 2024
Merged
11 tasks
@dqhl76 dqhl76 mentioned this pull request Feb 28, 2024
Merged
11 tasks
@dqhl76 dqhl76 closed this Feb 28, 2024
@dqhl76 dqhl76 deleted the add-basic-time-slice-patch2 branch February 29, 2024 00:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhang2014 zhang2014 zhang2014 left review comments

Assignees
No one assigned
Labels
ci-benchmark-local Benchmark: run only local test pr-refactor this PR changes the code base without new features or bugfix
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dqhl76 @zhang2014 @everpcpc