Wrong revision is picked up for traffic target marked as "latest"

[rhuss]

What version of Knative?

0.12.0

Expected Behavior

When deleting a revision and updating a service short after each other, it occurs that the revision that is deleted is picked for the latest traffic target

Actual Behavior

In the client E2E we have a flake that looks like this

┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🦆 kn revision delete hello-pptgc-1 --namespace kne2etests0
┃ Revision 'hello-pptgc-1' deleted in namespace 'kne2etests0'.
┃ 
┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🦆 kn service update hello --env TARGET=kn --port 8888 --namespace kne2etests0
┃ Updating Service 'hello' in namespace 'kne2etests0':
┃ 
┃   2.200s Revision "hello-pptgc-1" referenced in traffic not found.
┃ 
🔥 RevisionMissing: Revision "hello-pptgc-1" referenced in traffic not found.
🔥 

As you can see clearly, the revision that is just deleted is picked up again on the following update for the traffic section.

See also the full test log which includes the resources descriptors.

The same happens also sometimes, when deleting a service and recreating it with the same name immediately after the deletion.

I suspect, as delete is an async operation, that the internal lookup for the traffic target doesn't honor the meta.deletionTimestamp on the revision.

Steps to Reproduce the Problem

Try to delete revision/service and then perform a service create/update. In the client E2E tests I would say it happens in 10-15% of the test cases (but that is only a gut feeling).

[mattmoor]

/kind good-first-issue

[markusthoemmes]

Could this also be an issue where the informer isn't yet updated with the current deletion state of the revision? 🤔

[mattmoor]

I suspect that this is the state (briefly) when metadata.generation = status.observedGeneration for the first time because we've started work, but haven't yet cleared the Ready=False from the prior state from the ksvc.

[taragu]

Verified that #6592 fixes this problem