Slightly Smarter Task Shuffling#2057

Merged

ssalinas merged 20 commits intomasterfrom smarter-shuffle

Jan 28, 2020

+841-209

Commits on Jan 9, 2020

shuffle lowest memory utilization tasks instead of highest
William Hou
committed
modify unit test to reflect new shuffling logic
William Hou
committed

Commits on Jan 10, 2020

slight refactor, postpone larger refactor
William Hou
committed

Commits on Jan 13, 2020

raw refactor of shuffle logic
William Hou
committed
cleanup shuffle refactor, wire into SingularityUsagePoller
William Hou
committed
method consistency for CPU/memory shuffles
William Hou
committed
resolve checkstyle violations
William Hou
committed
uniform action id per task shuffle
William Hou
committed
minimal viable shuffle unit tests
William Hou
committed
fix off by one error in shuffle task count check
William Hou
committed
actually count initial # of shuffling tasks per host
William Hou
committed
correctly update overusage, attempt to ensure that slave reaches resource threshold
William Hou
committed
rearrange helper test methods
William Hou
committed

Commits on Jan 14, 2020

fix bug in initial CPU utilization calculation + memory shuffle unit test with CPU
William Hou
committed
more thorough shuffle unit test
William Hou
committed
attempts to use helper methods in shuffle unit testing have failed
William Hou
committed
memory shuffle should take into account external memory pressure
William Hou
committed
add slightly more logging if shuffling >= 1 slave
William Hou
committed

Commits on Jan 15, 2020

sanity check logging, remove after sanity regained
William Hou
committed
move shuffle debug logging
William Hou
committed