Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Shuffle] Add n_mappers and n_reducers to ShuffleProxy #3160

Merged
merged 8 commits into from
Jun 24, 2022
Merged

[Shuffle] Add n_mappers and n_reducers to ShuffleProxy #3160

merged 8 commits into from
Jun 24, 2022

Conversation

chaokunyang
Copy link
Contributor

@chaokunyang chaokunyang commented Jun 21, 2022 •
edited
Loading

What do these changes do?

  • add n_mappers/n_reducers info to shuffle proxy, so it can be used for shuffle optimization such as fetch by index.
  • Refine n_reducers/reducer_ordinal by set those attributes in MapReduceOperand#new_chunks

Related issue number

Fixes #xxxx

Check code requirements

  • tests added / passed (if needed)
  • Ensure all linting tests pass, see here for how to run them

@chaokunyang chaokunyang requested a review from a team as a code owner June 21, 2022 11:43
@chaokunyang chaokunyang mentioned this pull request Jun 21, 2022
Merged
11 tasks
@wjsi
Copy link
Member

wjsi commented Jun 21, 2022

In my opinion, n_mappers and n_reducer shall be recorded on graph execution to reduce complexity for operand writers.

@chaokunyang
Copy link
Contributor Author

In my opinion, n_mappers and n_reducer shall be recorded on graph execution to reduce complexity for operand writers.

Some reducer chunks are not included in the chunk graph, we can't get all info when build graph.

@chaokunyang chaokunyang changed the title [Shuffle] Add shuffle info to proxy [Shuffle] Refine adding shuffle info to operands Jun 22, 2022
wjsi
wjsi approved these changes Jun 24, 2022
View reviewed changes
Copy link
Member

@wjsi wjsi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wjsi wjsi added type: enhancement request to be backported Indicate that the PR need to be backported to stable branch shuffle mod: tensor mod: dataframe labels Jun 24, 2022
@wjsi wjsi added this to In progress in DataFrame via automation Jun 24, 2022
@wjsi wjsi added this to In progress in Tensor via automation Jun 24, 2022
@wjsi wjsi added this to PR-In progress in v0.10 Release via automation Jun 24, 2022
@wjsi wjsi added this to the v0.10.0a2 milestone Jun 24, 2022
@wjsi wjsi changed the title [Shuffle] Refine adding shuffle info to operands [Shuffle] Add n_mappers and n_reducers to ShuffleProxy Jun 24, 2022
zhongchun
zhongchun approved these changes Jun 24, 2022
View reviewed changes
Copy link
Contributor

@zhongchun zhongchun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

DataFrame automation moved this from In progress to Reviewer approved Jun 24, 2022
Tensor automation moved this from In progress to Reviewer approved Jun 24, 2022
@wjsi wjsi merged commit 4bcfd7f into mars-project:master Jun 24, 2022
DataFrame automation moved this from Reviewer approved to Done Jun 24, 2022
Tensor automation moved this from Reviewer approved to Done Jun 24, 2022
v0.10 Release automation moved this from PR-In progress to PR-Done Jun 24, 2022
wjsi pushed a commit to wjsi/mars that referenced this pull request Jun 24, 2022
@chaokunyang @wjsi
[Shuffle] Add n_mappers and n_reducers to ShuffleProxy (mars-pr…
73b4d41 
…oject#3160)
@wjsi wjsi mentioned this pull request Jun 24, 2022
Merged
hekaisheng pushed a commit that referenced this pull request Jun 27, 2022
@wjsi @chaokunyang
[BACKPORT][Shuffle] Add n_mappers and n_reducers to ShuffleProxy (
9e0138f 
#3160) (#3169)

Co-authored-by: Shawn <[email protected]>
@hekaisheng hekaisheng added backported already PR has been backported and removed to be backported Indicate that the PR need to be backported to stable branch labels Jun 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhongchun zhongchun zhongchun approved these changes

@wjsi wjsi wjsi approved these changes

Assignees
No one assigned
Labels
backported already PR has been backported mod: dataframe mod: tensor shuffle type: enhancement request
Projects
DataFrame
  
Done
Tensor
  
Done
v0.10 Release
PR-Done
Milestone
v0.10.0a2
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@chaokunyang @wjsi @zhongchun @hekaisheng