Central cleanup job for Shoot clusters on KCP DEV

[tobiscr]

Description

We use currently two cleanup mechanisms for removing shoot-clusters on KCP DEV.

One is executed by KEB and removes Shoot-clusters which were created by KEB. Another job was running decoupled from KEB and removed shoot-clusters created by Provisioner.

To simplify the cleanup approach, a single cleanup job should be preferred. This job retrieves all Shoots from Gardener cluster and verifies whether they are known by KEB. If they are unknown, the cleanup mechanism will delete this cluster, independent who created it:

Created by	Shoot deleted by cleanup job via
KEB	KEB executes its deletion logic
KIM	Removing the RuntimeCR
Gardener-UI	Removing the Shoot-CR on Gardener cluster

The deletion has to consider the case, that a Shoot-CR can be marked with the label owner.do-not-delete: '<bool>'. Before a cluster is deleted, the cleanup job has to check the existence respectively the value of the label on the Shoot CR.

AC:

• A job is implemented which retrieves all Shoot-CR on KCP DEV
• The job verifies if the Shoot-cluster is known by KEB, if the cluster is unknown or marked for deletion, the cluster-deletion logic will be trigger
• the deletion of a Shoot-cluster is only allowed if the be Shoot-CR does not contain the owner.do-not-delete label or if the value of the label is false.
• the deletion of the cluster happens independent to the creator, means the deletion logic tries to;
  1. delete the cluster in the provisioner
  2. delete the Runtime CR
  3. delete the Shoot-CR

Reasons

Simplify KCP DEV housekeeping by centralising the shoot-cluster deletion job.

Attachments

[tobiscr]

@kyma-project/gopher / @PK85 - will you take it over?