[Shuffle] Skip store shuffle object refs to reduce meta overhead #3209
Conversation
chaokunyang
force-pushed
the
skip_store_shuffle_object_refs_when_autoscale_disabled
branch
2 times, most recently
from
1f99c3d
to
a037db0
Compare
chaokunyang
changed the title
| cluster_api = await ClusterAPI.create(address) | ||
| supervisor_address = (await cluster_api.get_supervisors())[0] | ||
| autoscaler = await mo.actor_ref( | ||
| AutoscalerActor.default_uid(), address=supervisor_address |
There was a problem hiding this comment.
Why not use address=address? The address is the current supervisor address.
There was a problem hiding this comment.
If supervisor created subpools and current process is subpool, then address won't be supervisor address
There was a problem hiding this comment.
But, the SubtaskQueueingActor, SubtaskManagerActor and AutoscalerActor are all created in the same address. So, the actor refs should be created from the same address, too. Please refer to: https://github.com/mars-project/mars/blob/master/mars/services/scheduling/supervisor/service.py#L122
Can you unify the address of actor ref creations of SubtaskQueueingActor, SubtaskManagerActor and AutoscalerActor?
There was a problem hiding this comment.
SubtaskQueueingActor and SubtaskManagerActor may be created in the subpool. The AutoscalerActor will always be created in the supervisor main pool, it's similar to GlobaleResourceManager.
There was a problem hiding this comment.
AutoscalerActor
It's different address, the address ofSubtaskQueueingActor/SubtaskManagerActor/AutoscalerActor may be subpool address, AutoscalerActor will always use main pool address.
I changed actror_ref creation of AutoscalerActor to:
cluster_api = await ClusterAPI.create(address)
[autoscaler] = await cluster_api.get_supervisor_refs(
[AutoscalerActor.default_uid()]
)| if self._has_shuffle: | ||
| mapper_chunks, proxy_chunks = [], [] | ||
| for c in result_chunks: | ||
| if ( |
There was a problem hiding this comment.
This logic is only for the ShuffleFetchType.FETCH_BY_INDEX? Record the shuffle proxy subtask reduces duplicate search shuffle proxy overhead, but introduces overhead for ShuffleFetchType.FETCH_BY_KEY.
There was a problem hiding this comment.
It's used by TaskStageProcessor
chaokunyang
force-pushed
the
skip_store_shuffle_object_refs_when_autoscale_disabled
branch
from
05c8b0b
to
6170d0b
Compare
chaokunyang
force-pushed
the
skip_store_shuffle_object_refs_when_autoscale_disabled
branch
from
6170d0b
to
4d4a4b4
Compare
chaokunyang
force-pushed
the
skip_store_shuffle_object_refs_when_autoscale_disabled
branch
from
4d4a4b4
to
3a93dc1
Compare
…s-project#3209) * disable shuffle in autoscale to skip shuffle meta * fix remove mapper data * refine autoscale in * fix SubtaskGraph add proxy chunks * add shuffle tests to autoscale * fxi mapper chunks check * remove unnecessary event * refine proxy_subtasks check * workaround versionner compatibility with PEP660 * fix get autoscaler * fix subtask graph building (cherry picked from commit b2d658e)
…s-project#3209) * disable shuffle in autoscale to skip shuffle meta * fix remove mapper data * refine autoscale in * fix SubtaskGraph add proxy chunks * add shuffle tests to autoscale * fxi mapper chunks check * remove unnecessary event * refine proxy_subtasks check * workaround versionner compatibility with PEP660 * fix get autoscaler * fix subtask graph building (cherry picked from commit b2d658e)
…s-project#3209) * disable shuffle in autoscale to skip shuffle meta * fix remove mapper data * refine autoscale in * fix SubtaskGraph add proxy chunks * add shuffle tests to autoscale * fxi mapper chunks check * remove unnecessary event * refine proxy_subtasks check * workaround versionner compatibility with PEP660 * fix get autoscaler * fix subtask graph building (cherry picked from commit b2d658e)
What do these changes do?
This PR skip store shuffle object refs to reduce meta overhead and supervisor serialization bottleneck by disable autoscale-in when shuffle is executing.
Related issue number
#2916
Check code requirements