Ability to supersede promotions to avoid Promotion queue backlog

[jessesuen]

Proposed Feature

Promotions and Verifications run at mutually exclusive points in time (e.g. when verification is happening, promotion does not and vice versa). Sometimes, new Freight might appear at a faster rate than can be verified. This might happen automatically -- if many image builds are produced quickly, but the analysis template is long-running. When this happens, the Stage will queue the Promotion job, which then might accumulate on the Stage:

Kargo could offer a feature that would allow a later promotion to cancel all currently Pending promotions so that when the new Promotion is created, it will be the next one to run.

Motivation

I think a common desire will be that a later Freight is the only one that matters to be verified and that intermediate versions be skipped (in terms of deploying into the Stage as well as being verified).

Suggested Implementation

We should support this manually as well as automatically. e.g. in the manual case, the UI/API could have an option to supersede the Pending promotions, and offer a checkbox to cancel them before creating the later one.

1. Stage and/or Projects should introduce a promotion policy preference with a "supersede" option.
2. If "supersede" is true, Pending promotions will be canceled before creating the new Promotion resource
3. Even though this will be a spec field in Stage and/or Project, the UI/API should still have the ability to override the Project/Stage setting and explicitly choose to supersede or queue the Promotion. (e.g. sometimes user might want to occasionally queue up a bunch of different Freight to be verified).