Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ray] Spread scheduling subtasks with empty dependencies #3281

Merged
merged 12 commits into from
Oct 18, 2022

Conversation

fyrestone
Copy link
Contributor

What do these changes do?

  • Spread scheduling subtasks with empty dependencies.
  • mars.new_ray_session support supervisor_cpu.
  • mars.new_ray_session(backend="ray", worker_cpu=n) can specify subtask num_cpus to n.
  • Log large messages (serialized bytes > 1GB).
  • Refine logs of clean func / restore func.
  • Fix new_ray_session bugs.

Related issue number

Fixes #3279 #3280

Check code requirements

  • tests added / passed (if needed)
  • Ensure all linting tests pass, see here for how to run them

@fyrestone fyrestone self-assigned this Oct 14, 2022
@fyrestone fyrestone added this to In Progress in Ray DAG Execution Backend via automation Oct 14, 2022
@fyrestone fyrestone marked this pull request as ready for review October 18, 2022 03:51
@fyrestone fyrestone requested a review from a team as a code owner October 18, 2022 03:51
Copy link
Contributor

@chaokunyang chaokunyang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@zhongchun zhongchun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@fyrestone fyrestone merged commit 8e7c0eb into mars-project:master Oct 18, 2022
Ray DAG Execution Backend automation moved this from In Progress to Done Oct 18, 2022
qianduoduo0904 pushed a commit to qianduoduo0904/mars that referenced this pull request Oct 24, 2022
…t#3281)

* Spread indep subtasks

* Refine logs

* Fix mars.new_ray_session(backend="ray") fetch data error

* Always stop_isolation at exit

* mars.new_ray_session() support supervisor_cpu

* Add test case

* Refine logs

* Add copyright

* Walkaround CI crash

* Fix CI

* Fix coverage

* Fix type annotation

Co-authored-by: 刘宝 <[email protected]>
(cherry picked from commit 8e7c0eb)
qianduoduo0904 pushed a commit to qianduoduo0904/mars that referenced this pull request Oct 24, 2022
…t#3281)

* Spread indep subtasks

* Refine logs

* Fix mars.new_ray_session(backend="ray") fetch data error

* Always stop_isolation at exit

* mars.new_ray_session() support supervisor_cpu

* Add test case

* Refine logs

* Add copyright

* Walkaround CI crash

* Fix CI

* Fix coverage

* Fix type annotation

Co-authored-by: 刘宝 <[email protected]>
(cherry picked from commit 8e7c0eb)
aresnow1 pushed a commit to xorbitsai/mars that referenced this pull request Oct 24, 2022
…t#3281)

* Spread indep subtasks

* Refine logs

* Fix mars.new_ray_session(backend="ray") fetch data error

* Always stop_isolation at exit

* mars.new_ray_session() support supervisor_cpu

* Add test case

* Refine logs

* Add copyright

* Walkaround CI crash

* Fix CI

* Fix coverage

* Fix type annotation

Co-authored-by: 刘宝 <[email protected]>
(cherry picked from commit 8e7c0eb)
qianduoduo0904 pushed a commit to qianduoduo0904/mars that referenced this pull request Oct 24, 2022
…t#3281)

* Spread indep subtasks

* Refine logs

* Fix mars.new_ray_session(backend="ray") fetch data error

* Always stop_isolation at exit

* mars.new_ray_session() support supervisor_cpu

* Add test case

* Refine logs

* Add copyright

* Walkaround CI crash

* Fix CI

* Fix coverage

* Fix type annotation

Co-authored-by: 刘宝 <[email protected]>
(cherry picked from commit 8e7c0eb)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants