[ML] Don't allocate persistent tasks for failed or closing jobs

[elasticmachine]

Original comment by @droberts195:

In LINK REDACTED it was noted that failed jobs would incorrectly reopen if the node they had been running on was restarted, and a fix for the problem was made in LINK REDACTED. The fix was to make the nodeOperation a no-op for failed or closing jobs.

However, @bleskes pointed out that a better fix might be to prevent allocation of persistent tasks for ML jobs if the status is failed or closing. This sounds like a better idea, but some research and testing is required to find out what all the implications and side effects are.

[droberts195]

One argument against changing the way we do this today is that it's really useful to have the persistent task with failed status in the cluster state in support diags as it means we can see which node a job was running on at the time it failed.

I think we should leave things as they are until there's an alternative way to achieve this.

[droberts195]

This also ties in with #48934. Just as it makes sense to completely cancel a stopping datafeed if the node it was running on is removed from the cluster, it makes sense to completely cancel a closing job if node it was running on is removed from the cluster.

In both cases if the code doing the graceful stop/close is killed part way through its work there's no way another node can take over. So the datafeed/job might as well be considered stopped/closed rather than left in limbo stopping/closing on a new node.

[droberts195]

Fixed by #55315