createpod doesn't delete dead pods, spawned by the old createpod instance

[d-uzlov]

Expected Behavior

createpod monitors all its pods' state. If a pod died, then this pod should be deleted by the createpod element to prevent pod list pollution.

Current Behavior

createpod only monitors the pods that it has created during the current session. If a server with createpod dies and respawns, then all of the pods that were spawned by the old server remains in the pod list forever, until they are removed manually.

Steps to Reproduce

1. Create a deployment with createpod element.
2. Use server with createpod to spawn a new pod.
3. Restart the deployment.
4. Wait until spawned server dies.
   The pod it was running in will not be removed from the list be the server with createpod element.

Solution

1. Add a label to each of the pods that createpod spawns.
2. Watch for all pods with this label, regardless of if it was spawned by this server of not.

[d-uzlov]

Other issues caused by createpod having transient state:

Issue 1: If a server with createpod creates a pod and then dies and respawns, and the new server gets a request, then a new pod is created, despite the fact that we already have the old pod.

Issue 2: If we accidentally or deliberately have several servers with createpod that create the same pod, then these servers will each create one pod on any node, thus one node will potentially have as many pods as we have servers spawning them.

Both of these issues can actually be solved exactly the same way as the issue with dead pods remaining in the list forever.
We could restore the list of created pods on the start using the label and then update our list using the pod events to synchronize with potentially existing other servers and minimize the chances of having several pods on one node.

@denis-tingaikin do you approve fixing this or is current behaviour desired?

[denis-tingaikin]

it is a good catch. Within this issue, we need to consider a scenario with a few suppliers on the cluster. So currently it is out of scope and the issue doesn't break the core scenario scale from zero.
Thus issue will be considered after release.