Make mount and staging dir creation flexible #159

darkowlzz · 2019-01-27T12:27:31Z

This change adds an option to attach callback functions to CSI sanity
for mount target directory creation. This could be used to customize the
target and staging mount path creation.

Also, adds targetPath and stagingPath fields to SanityContext to
store the actual target and staging paths, derived from the sanity
config.

Adds an e2e test section to test the above setup.

Fixes #144

pohly · 2019-01-28T09:58:15Z

hack/_apitest2/api_test.go

+		},
+	}
+
+	sanity.Test(t, config)


This doesn't actually verify that CreateTargetDir and CreateStagingDir are getting used as intended, does it?

Admittedly a thorough check is hard, but perhaps record inside createTargetDir that it was called and then add some asserts against that after sanity.Test returns? If you invoke _apitest2/api_test.go with -test.run such that it executes a well-known set of tests, it might even become possible to have a deterministic set of created directories to compare against.

pohly · 2019-01-28T10:00:06Z

pkg/sanity/sanity.go

+	// If not defined, directories are created in the default way at TargetPath
+	// and StagingPath.
+	CreateTargetDir  func(path string) (string, error)
+	CreateStagingDir func(path string) (string, error)


I'm fine with not exposing this functionality in the command line tool.

If someone needs it, we could add it by letting the test command invoke an external tool.

pohly · 2019-02-13T15:24:48Z

pkg/sanity/sanity.go

+		if sc.Config.CreateStagingDir != nil {
+			newpath, err := sc.Config.CreateStagingDir(sc.Config.StagingPath)
+			Expect(err).NotTo(HaveOccurred())
+			sc.stagingPath = newpath


Always explain what the error is about, see kubernetes/kubernetes#34059. I know, the rest of the code doesn't do that either (would be a good enhancement), but it's never too late to do it better.

This code is rather repetitive. I would change the createMountTargetLocation method and then use it like this:

sc.targetPath = createMountTargetLocation(sc.Config.TargetPath, sc.Config.CreateTargetDir) sc.stagingPath = createMountTargetLocation(sc.Config.StagingPath, sc.Config.CreateStagingDir)

Speaking of createMountTargetLocation: there is a call to it in node.go which causes directories to be created on the host even when sc.Config.StagingPath is set. I think that call can be removed.

I can confirm that in scenario we use (staging and target directories are created on VMs where pods run, host acting only as test orchestrator) I see /var/tmp/tmp.XXXXXXXXXX directories created on host which are not used and remain undeleted, both with csi-test 1.0.2 and with current PR change.
Positive feedback is that tests pass same way with current PR changes (as with 1.0.2)

darkowlzz · 2019-02-25T17:22:08Z

Updated the PR addressing the suggestions.

Changed runTestAPIWithCustomTargetPaths() to run NodePublishVolume based tests only to have a deterministic number of target and staging directory creation calls.

Added more context to the test failure errors and changed the signature of createMountTargetLocation() to repeat repetition.

pohly · 2019-02-28T10:26:35Z

pkg/sanity/sanity.go

+func createMountTargetLocation(targetPath string, customCreateDir func(string) (string, error)) (string, error) {
+
+	// Return the target path if empty.
+	if len(targetPath) < 0 {


An empty string has length 0, so this check is not getting triggered as intended.

Better compare with if targetPath == "". It's more obvious what's meant (always most important) and there's no obvious performance benefit from the length comparison (performance doesn't matter here anyway).

Sorry, that's really a mistake 😅 . Changed it.
Thanks!

pohly · 2019-02-28T10:35:11Z

pkg/sanity/sanity.go

+	} else {
+		fileInfo, err := os.Stat(targetPath)
+		if err != nil && os.IsNotExist(err) {
+			return "", os.MkdirAll(targetPath, 0755)


Here you return an empty string also when os.MkdirAll succeeds. That can't be right? I wonder why it isn't breaking the Travis CI check. Ah, probably because the mock driver doesn't actually use the directories and doesn't check whether they exist (unverified). Might be worth adding such a check. Perhaps add such a check as part of the PR, verify "make test" fails, then fix this code?

Overall I find this version more readable:

fileInfo, err := os.Stat(targetPath) if err != nil { if !os.IsNotExist(err) { return "", err } if err := os.MkdirAll(targetPath, 0755); err != nil { return "", err } } if !fileInfo.IsDir() { return "", fmt.Errorf("Target location %s is not a directory", targetPath) }

oops! thanks for noticing that.
I've added new checks in NodePublishVolume and NodeStageVolume to fail if the target paths don't exist. And adopted the above code snippet, this is actually better.

pohly · 2019-03-02T20:36:46Z

I said earlier that adding support for this to the command line can wait until we have a need for it. I now found a use case, so can you add command line flags for this? They should accept a single parameter, the command to run, and then call that command with the path as argument. stdout of the command is the new path, stderr can go to stderr of the caller (i.e. don't redirect).

My use case is for adding sanity testing to kubernetes-csi/csi-driver-host-path#16: when building the hostpath driver (or soon, also other drivers), I want to check out a certain revision of csi-test and run the sanity command against the deployed driver, without having to built a custom test suite.

The custom command is then the name of a script which contains kubectl exec csi-hostpathplugin-0 mkdir -p "$@". That'll work because since kubernetes-csi/csi-driver-host-path#14 the pod has a deterministic name.

Access to the csi.sock will be via ssh socket forwarding, set up outside of the sanity command invocation.

darkowlzz · 2019-03-03T15:23:14Z

I've added command line support for passing external commands that can create the target path. An example usage is in the e2e test file.

I realized that it would be better if we change the names of a few flags and variables.
With block volume support, the target paths will no longer be directories always. The flags mountdir and stagingdir are file volume specific.
Maybe mountpath and stagingpath would be better names.
If we change this, we should also change the Config attributes that store custom target path creation callback functions CreateTargetDir and CreateStagingDir to CreateTargetPath and CreateStagingPath.

Maybe we need to maintain backwards compatibility?

darkowlzz · 2019-03-03T15:28:25Z

pkg/sanity/sanity.go

 }

 func (sc *SanityContext) teardown() {
+	// Delete the created paths if any.
+	os.RemoveAll(sc.targetPath)
+	os.RemoveAll(sc.stagingPath)


I added this in teardown because without this, some tests were using the directories created by the previous tests and that was resulting in the directory exists check in createMountTargetLocation() to be true and hence, not returning a target path even if it was created was passing, because that code block was being skipped.
With this, the directories are deleted for every test and not returning target path from createMountTargetLocation() results in failure.

While it's nicer if the test cleans up after itself, we then also need functions that correspond to CreateStagingDir and CreateTargetDir. The main use-case for these functions is for testing a driver that was deployed somewhere, and then those functions will not create the dirs locally.

Right! I wasn't thinking in that way. Thanks, I'll add the corresponding cleanup options as well.

pohly · 2019-03-04T11:42:42Z

Maybe mountpath and stagingpath would be better names.
If we change this, we should also change the Config attributes that store custom target path creation callback functions CreateTargetDir and CreateStagingDir to CreateTargetPath and CreateStagingPath.

This is a good point. The proposed new names make more sense, but then the next release will have to be v2.0.0 due to the API change. I'm a bit undecided myself.

@lpabon what do you think?

darkowlzz · 2019-03-17T11:04:40Z

Added support for custom target path removal callback function and command line flag. The same is being used in the e2e test.

pohly

/lgtm
/approve

Thanks for seeing this through to the end.

k8s-ci-robot · 2019-03-19T16:36:31Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: darkowlzz, pohly

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [pohly]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This change adds an option to attach callback functions to CSI sanity for mount target directory creation. This could be used to customize the target and staging mount path creation. Also, adds `targetPath` and `stagingPath` fields to `SanityContext` to store the actual target and staging paths, derived from the sanity config. Adds an e2e test section to test the above setup.

This adds target path exists checks in the mock driver in NodeStageVolume and NodePublishVolume to ensure that the paths actually exist. Also adds path cleanup in the teardown. This ensures that every test creates and deletes their target directories. Without this, the test setup function for Controller only tests created the target directory and the same was being used by subsequent tests.

This adds three new flags to support custom target path creation: - `createmountpathcmd`: Command to create a custom mount path. - `createstagingpathcmd`: Command to create a custom staging path. - `createpathcmdtimeout`: Timeout for the custom path creation commands. Also adds a new test section in e2e tests to test this custom command support.

This adds three new flags to support custom target path removal: - `removemountpathcmd`: Command to remove a custom mount path. - `removestagingpathcmd`: Command to remove a custom staging path. - `removepathcmdtimeout`: Timeout for the custom path removal commands.

pohly

/lgtm

kubernetes-csi#159 introduced the ability to use different paths for each tests via external commands or callbacks, but it missed two lines in the cleanup code which also have to use the actual paths instead of the configured paths.

…dirs Make mount and staging dir creation flexible

kubernetes-csi#159 introduced the ability to use different paths for each tests via external commands or callbacks, but it missed two lines in the cleanup code which also have to use the actual paths instead of the configured paths.

k8s-ci-robot requested review from lpabon and pohly January 27, 2019 12:27

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 27, 2019

pohly reviewed Jan 28, 2019

View reviewed changes

pohly reviewed Feb 13, 2019

View reviewed changes

darkowlzz force-pushed the flexible-target-dirs branch from 92fd1a2 to c8ac1b0 Compare February 25, 2019 17:15

pohly requested changes Feb 28, 2019

View reviewed changes

darkowlzz commented Mar 3, 2019

View reviewed changes

darkowlzz force-pushed the flexible-target-dirs branch from 700baf2 to 6edd078 Compare March 17, 2019 11:02

pohly approved these changes Mar 19, 2019

View reviewed changes

k8s-ci-robot assigned pohly Mar 19, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 19, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 19, 2019

pohly mentioned this pull request Mar 19, 2019

Make mount and staging dir creation flexible #179

Closed

darkowlzz added 4 commits March 20, 2019 14:04

darkowlzz force-pushed the flexible-target-dirs branch from 6edd078 to 506d128 Compare March 20, 2019 08:35

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 20, 2019

pohly approved these changes Mar 20, 2019

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 20, 2019

k8s-ci-robot merged commit ca4c526 into kubernetes-csi:master Mar 20, 2019

pohly mentioned this pull request Apr 4, 2019

sanity: use correct path during cleanup #187

Merged

suneeth51 pushed a commit to suneeth51/csi-test that referenced this pull request Sep 11, 2019

Merge pull request kubernetes-csi#159 from darkowlzz/flexible-target-…

b079f24

…dirs Make mount and staging dir creation flexible

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make mount and staging dir creation flexible #159

Make mount and staging dir creation flexible #159

darkowlzz commented Jan 27, 2019

pohly Jan 28, 2019

pohly Jan 28, 2019

pohly Feb 13, 2019 •

edited

Loading

okartau Feb 19, 2019

darkowlzz commented Feb 25, 2019

pohly Feb 28, 2019

darkowlzz Mar 3, 2019 •

edited

Loading

pohly Feb 28, 2019

darkowlzz Mar 3, 2019

pohly commented Mar 2, 2019

darkowlzz commented Mar 3, 2019 •

edited

Loading

darkowlzz Mar 3, 2019 •

edited

Loading

pohly Mar 4, 2019

darkowlzz Mar 11, 2019

pohly commented Mar 4, 2019

darkowlzz commented Mar 17, 2019

pohly left a comment

k8s-ci-robot commented Mar 19, 2019

pohly left a comment

Make mount and staging dir creation flexible #159

Make mount and staging dir creation flexible #159

Conversation

darkowlzz commented Jan 27, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pohly Feb 13, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darkowlzz commented Feb 25, 2019

Choose a reason for hiding this comment

darkowlzz Mar 3, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pohly commented Mar 2, 2019

darkowlzz commented Mar 3, 2019 • edited Loading

darkowlzz Mar 3, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pohly commented Mar 4, 2019

darkowlzz commented Mar 17, 2019

pohly left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Mar 19, 2019

pohly left a comment

Choose a reason for hiding this comment

pohly Feb 13, 2019 •

edited

Loading

darkowlzz Mar 3, 2019 •

edited

Loading

darkowlzz commented Mar 3, 2019 •

edited

Loading

darkowlzz Mar 3, 2019 •

edited

Loading