Build fails to resume if controller crashes before queue is saved

[Vlatombe]

This covers a case where a queue item for a node was filed, then the controller shut down without going through clean up.
Because of that, the queue wasn't saved, and when the controller restarts, the queue item is gone.

This modifies the behaviour of ExecutorStepExecution#onResume to detect this condition and re-schedule a new queue item if needed.

Testing done

See provided unit test.

Submitter checklist

• Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
• Ensure that the pull request title represents the desired changelog entry
• Please describe what you did
• Link to relevant issues in GitHub or Jira
• Link to relevant pull requests, esp. upstream and downstream changes
• Ensure you have provided tests - that demonstrates feature works or fixes the issue

[Vlatombe]

The same test passes with

rjr.stopJenkins();

[Vlatombe]

Need jenkinsci/jenkins-test-harness#876 for RealJenkinsRule#stopJenkinsForcibly

[Vlatombe]

The given test sometimes succeeds...

[Vlatombe]

The case where it succeeds is when the stop happens before entering the second node step (still in sleep), so I need a better assertion before stopping.

[jglick]

(For reference, this is a corner case which would previously trigger AnomalousStatus, as most recently touched in #405.)

[jglick]

To remind myself: this would have been added right away in

workflow-durable-task-step-plugin/src/main/java/org/jenkinsci/plugins/workflow/support/steps/ExecutorStepExecution.java

Lines 132 to 139 in 6ae8123

	public boolean start() throws Exception {
	final PlaceholderTask task = new PlaceholderTask(getContext(), step.getLabel());
	Queue.WaitingItem waitingItem = Queue.getInstance().schedule2(task, 0).getCreateItem();
	if (waitingItem == null) {
	// There can be no duplicates. But could be refused if a QueueDecisionHandler rejects it for some odd reason.
	throw new IllegalStateException("failed to schedule task");
	}
	getContext().get(FlowNode.class).addAction(new QueueItemActionImpl(waitingItem.getId()));

[jglick]

🤔 maybe we should start developing reusable Matchers in JTH so you could write for example

await().until(() -> b, hasMessage("Complete branch 2 ?"));

which would be more flexible & composable than the current assertions.

[Vlatombe]

Yes, would be much nicer than the current assertions available on JenkinsRule.

[jglick]

#408 (comment) seems like a real bug. Otherwise looks good.