fix: the scheduler is now fair

[andaaron]

Generators are now ordered by rank in the priority queue.

The rank computation formula is:
- 100/(1+generated_task_count) for high priority tasks
- 10/(1+generated_task_count) for medium priority tasks
- 1/(1+generated_task_count) for low priority tasks

Note the ranks are used when comparing generators both with the same priority and with different priority.
So now we are:
- giving an opportunity to all generators with the same priority to take turns generating tasks
- giving roughly 1 low priority and 10 medium priority tasks the opportunity to run for every 100 high priority tasks running.

After a generator generates a task, the generators are reordered in the priority queue based on rank.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (355b1ee) 92.15% compared to head (aa21bd8) 92.20%.
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2158      +/-   ##
==========================================
+ Coverage   92.15%   92.20%   +0.05%     
==========================================
  Files         166      166              
  Lines       28770    28792      +22     
==========================================
+ Hits        26513    26549      +36     
+ Misses       1662     1652      -10     
+ Partials      595      591       -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rchincha]

So this is a dynamic heuristic -
Between generators of constant output rate, numerator determines priority and when one of them has a higher rate, priority decreases.

[rchincha]

lgtm

Tested in production, memory/cpu usage looking much better.