E2E test should cover upgrade case

[tnqn]

Describe the bug
There should be upgrade tests to ensure:

• Connectivity of existing Pods can continue work
• Existing Pods can be deleted and antrea-managed resources like ovs port, veth devices can be cleaned up
• Existing NetworkPolicies can continue work
• Existing Namespaces can be deleted, see bug Namespace cannot be deleted after upgrading antrea from 0.4.0 to 0.5.0-dev  #494

[tnqn]

@jianjuns @antoninbas @salv-orlando please share your opinions and see if we should have this ready before 0.5.0 as we have missed something that could cause issue after upgrading, like #494.

[jianjuns]

Yes, agreed we should have this in 0.5.0, since we had production deployment since 0.4.0.

[antoninbas]

I think that's a good start and we should try to have this in time for 0.5.

Maybe in the longer term we can verify other things as well (e.g. monitoring CRDs are still valid and Octant plugin still works as expected?).

What do you think the test should look like exactly? Maybe a specific folder under "test/e2e/" and the ability to provide 2 parameters when running go test: -from-sha and -to-sha. Then we can ensure that the test runs before a release with the correct parameters (e.g. -from-sha=v0.4.0 -to-sha=<sha for 0.5.0 release candidate>). If we can run the test in a Kind cluster, it should be very straightforward.
Instead of a single test, we can also verify a matrix of upgrades. Maybe to start with, we can check that 0.3.0, 0.4.0 and 0.4.1 can be upgraded to 0.5.0. If we keep adding tests and running this as a Github workflow is no an option anymore, we will migrate the test to VMC.

[tnqn]

Thanks @jianjuns and @antoninbas for your input.
@antoninbas I was just thinking some simple rolling update scenario from N-1 version. I think your proposal is very flexible and convenient. We get only a few days, do you want to implement the proposal yourself? If not, I would volunteer to be the coder.

[antoninbas]

@tnqn I can take a stab at it tomorrow. But have we figured out what to do about #494. I suggest keeping system as a dummy API for now (at least for 0.5.0), since we do not know yet for sure that we will not be needing it in the future. It would be great if you could drive this on your side?

[tnqn]

@antoninbas sure, @jianjuns and I are discussing adding controllerinfo API under this group, if we can make it 0.5.0, then we don't have to have a dummy one. If not, I will take care of adding a dummy APIGroup.