scheduler opt-in forwarding

[apfitzge]

Problem

• In scheduler forward packets #898 we re-added forwarding to support those with swqos side-deals to use the central scheduler.
• This behavior is useless for any operators using the central scheduler if they do not have such side-deals

Summary of Changes

• SchedulerController is now configurable and can enable forwarding or not.
• Add CLI argument enable-block-production-forwarding
  • this makes forwarding opt-in. Any operators with side-deals should enable this if they want it to actually do forwarding.

Fixes #

[apfitzge]

Diff is not too bad, relatively easy to make this opt-in. cc @diman-io for awareness.

@t-nelson I think if we merge #1663 we should strongly consider backporting this immediately after.
This PR basically returns scheduler (by default) to the current behavior that is on MNB.

[apfitzge]

@t-nelson is there anything else to do here?

[tao-stones]

looks great to me

[mergify]

Backports to the beta branch are to be avoided unless absolutely necessary for fixing bugs, security issues, and perf regressions. Changes intended for backport should be structured such that a minimum effective diff can be committed separately from any refactoring, plumbing, cleanup, etc that are not strictly necessary to achieve the goal. Any of the latter should go only into master and ride the normal stabilization schedule. Exceptions include CI/metrics changes, CLI improvements and documentation updates on a case by case basis.