Splitting CSI Sanity unit and E2E tests

[davidz627]

Looking at CSI Sanity, it seems that it is running unit and e2e tests in the same framework.

For example in one suite "NodeUnpublishVolume" it will have unit tests like "should fail when no volume id is provided." That could be exercised on the driver with mocked out back-ends. This makes sure there are no side-effects or other unrelated errors that arise from this test hitting real back-ends.

And in the same suite "NodeUnpublishVolume" there is an e2e test "should return appropriate values (no optional values added)" which is exercising the entire workflow of creating, controller publish, node publish, node unpublish, controller unpublish, and deletevolume. This exact same e2e test is also duplicated in the "NodePublishVolume" suite as the "test should return appropriate values (no optional values added)" test.

I propose that we split the existing sanity testing framework into two parts, one for unit tests to exercise single RPC calls for their various error conditions and sanity check their basic functionality with mocked out backends. And a second e2e sanity testing framework designed to be run with a production backend that can exercise end to end workflows of creating actual volumes and various error cases with respect to that.

This would solve a few problems:

1. Unit testing is sped up because it will not hit production backends
2. E2E Tests will no longer be duplicated across different suites ("should return appropriate values (no optional values added)")
3. Unit/e2e tests can be run in different environments (unit with mock backend, e2e with production backend for example)
4. Frees up room to write more extensive e2e tests failure modes like "controller publish before createvolume" or "Node stage capability specified but NodeStageVolume not called"

[davidz627]

@lpabon @sbezverk @saad-ali

[sbezverk]

@davidz627 afaik there are two mock drivers, one used for unit tests and the other one used in e2e. I was under impression that unit tests are using @lpabon developed mock driver and e2e tests are using new mock driver we forked recently. Latter I thought would be sort of a reference driver for e2e tests as it will implement full driver's functionality. Does it go against to what you proposing?

[davidz627]

I am talking about splitting the CSI Sanity testing package into two pieces. A "csi-sanity-unit" and "csi-sanity-e2e". These could then be imported to any external driver and used for testing of that driver (mock, hostpath, gce, portworx, etc.).

My argument is that unit and e2e tests are different enough that they should be split up so that they can be deployed/run differently for each driver. As well as improving the developer experience on csi-sanity (unit tests do not have to worry about back-end side effects, e2e test can test full workflows without issue).

[davidz627]

Talked to @sbezverk and @lpabon offline. We will not be splitting the sanity test into unit/e2e parts because it is just meant to test conformance with the Spec. The Sanity package is not meant to actually E2E test data plane or internal behavior of the plugin.

However, I will still be cleaning up the code a little and factoring duplicated code into functions so even if we want to call the same test in multiple Describe suites it will be clear that this is the same test, as well as easier to modify and maintain the test.

[davidz627]

Fixed in: 53