Ray-based Shuffle For Mars

[chaokunyang]

This issue is used for tracking Ray-based Shuffle For Mars progress:

• Ray Future-based Shuffle MEP：MEP-0001: Future-based shuffle on Ray meps#2
• Shuffle Meta optimization Add n_reducers and reducer_ordinal to shuffle operands #3055
• Ray Future-based Shuffle
  • Basic Ray Shuffle Implementation [Ray] task based shuffle for ray #3040
  • Shuffle object GC
  • Shuffle operands compatibility
• Push-based Ray Shuffle
  • Push-based ray shuffle MEP
  • Scheduling all ray tasks in placement group
  • Pipelined push-based Shuffle
    • Push strategy optimization. Skip push shuffle blocks whose size exceed specified size.
    • Support configure merge factor.
    • Support configure concurrent running mapper subtasks.
    • Support adjust merge factor adaptively
    • Support adjust concurrent running mapper subtasks adaptively to reduce write amplification
• Handle shuffle slow sutbask by speculative execution
• Dynamic merge shuffle partitions
• Dynamic split shuffle partitions
  • Dynamic detect skewed partitions
  • Dynamic optimize skewed join
  • Support sub chunks
    [ ] Shuffle Failover based lineage reconstruction

[wjsi]

I've implemented a half-done branch https://github.com/wjsi/mars/tree/feature/shuffle_svc for shuffle. The reason why it is not merged is that currently storage service lacks the capability of remote readers and writers. Now working on it.

[chaokunyang]

https://github.com/wjsi/mars/tree/feature/shuffle_svc

Is there a design doc for this?

[wjsi]

https://github.com/wjsi/mars/tree/feature/shuffle_svc

Is there a design doc for this?

Not yet. I'll write it soon.