Merge pull request kubernetes-csi#27 from jsafrane/rebase-v1.6.0

Rebase to v1.6.0

CHANGELOG-1.1.md

@@ -15,7 +15,7 @@
 * [In-tree storage plugin to CSI Driver Migration](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190129-csi-migration.md) is now alpha ([#253](https://github.com/kubernetes-csi/external-provisioner/pull/253))
 * The external provisioner now tries to connect to the CSI driver indefinitely ([#234](https://github.com/kubernetes-csi/external-provisioner/pull/234))
 * A new --timeout parameter has been added for CSI operations ([#230](https://github.com/kubernetes-csi/external-provisioner/pull/230))
-* README.md has been signficantly enhanced ([#249](https://github.com/kubernetes-csi/external-provisioner/pull/249))
+* README.md has been significantly enhanced ([#249](https://github.com/kubernetes-csi/external-provisioner/pull/249))
 * Add support for  Lease based leader election. Enable this by setting
   `--leader-election-type=leases` ([#261](https://github.com/kubernetes-csi/external-provisioner/pull/261))
 

CHANGELOG-1.5.md

@@ -2,12 +2,12 @@
 
 ## Breaking Changes
 
-- Updates VolumeSnapshot CRD to v1beta1. v1alpha1 is no longer upported. ([#335](https://github.com/kubernetes-csi/external-provisioner/pull/335), [@xing-yang](https://github.com/xing-yang))
+- Updates VolumeSnapshot CRD to v1beta1. v1alpha1 is no longer supported. ([#335](https://github.com/kubernetes-csi/external-provisioner/pull/335), [@xing-yang](https://github.com/xing-yang))
 
 ## New Features
 
 - Add prometheus metrics to CSI external-provisioner under the /metrics endpoint. This can be enabled via the "--metrics-address" and "--metrics-path" options. ([#388](https://github.com/kubernetes-csi/external-provisioner/pull/388), [@saad-ali](https://github.com/saad-ali))
-- Updates VolumeSnapshot CRD to v1beta1. v1alpha1 is no longer upported. ([#335](https://github.com/kubernetes-csi/external-provisioner/pull/335), [@xing-yang](https://github.com/xing-yang))
+- Updates VolumeSnapshot CRD to v1beta1. v1alpha1 is no longer supported. ([#335](https://github.com/kubernetes-csi/external-provisioner/pull/335), [@xing-yang](https://github.com/xing-yang))
 
 ## Other Notable Changes
 

CHANGELOG-1.6.md

@@ -0,0 +1,79 @@
+# Changes since v1.5.0
+
+## Deprecations
+
+- Support for `storage.k8s.io/v1beta1.CSINode` and the `nodeid` annotation will be
+  removed in a future release. Only cluster versions that support
+  `storage.k8s.io/v1.CSINode` will be supported. The topology feature gate will
+  also be removed and will always be on if the plugin supports the
+  `VOLUME_ACCESSIBILITY_CONSTRAINTS` capability.
+  ([#385](https://github.com/kubernetes-csi/external-provisioner/issues/385),
+  [#257](https://github.com/kubernetes-csi/external-provisioner/issues/257))
+- The current leader election lock name will be changed in a future release.
+  Rolling upgrades to a newer sidecar will not maintain proper leadership, and a
+  full teardown and bringup will be required if the driver cannot handle
+  multiple leaders.
+  ([#295](https://github.com/kubernetes-csi/external-provisioner/issues/295))
+- The `--leader-election-type` argument will be removed in a future release.
+  Only Leases-type leader election will be supported. The
+  `--enable-leader-election` argument will also be renamed to
+  `--leader-election`.
+  ([#401](https://github.com/kubernetes-csi/external-provisioner/issues/401))
+- The default value of `--worker-threads` will be lowered to 10 to match the
+  other CSI sidecars in a future release.
+  ([#322](https://github.com/kubernetes-csi/external-provisioner/issues/322))
+- The external-provisioner will no longer default an empty fstype to `ext4`.
+  A new option to set the default will be added, otherwise the CSI driver
+  must be able to handle an empty fstype according to the CSI spec.
+  ([#328](https://github.com/kubernetes-csi/external-provisioner/issues/328))
+- Already deprecated arguments `--connection-timeout` and `--provisioner` will
+  be removed in a future release.
+
+## New Features
+
+- `StorageClass.allowedtopologies` can specify a subset of the topology keys
+  supported by the driver.
+  ([#421](https://github.com/kubernetes-csi/external-provisioner/pull/421),
+  [@pawanpraka1](https://github.com/pawanpraka1))
+- Added a new flag, `--cloning-protection-threads` which defaults to 1,
+  managing how many threads will simultaneously serve the
+  `provisioner.storage.kubernetes.io/cloning-protection` finalizer removal
+  ([#424](https://github.com/kubernetes-csi/external-provisioner/pull/424),
+  [@Danil-Grigorev](https://github.com/Danil-Grigorev))
+- finalizer `provisioner.storage.kubernetes.io/cloning-protection`
+  is now set on the source PVC preventing their removal before the cloning finishes.
+  ([#422](https://github.com/kubernetes-csi/external-provisioner/pull/422),
+  [@Danil-Grigorev](https://github.com/Danil-Grigorev))
+- Adds `--extraCreateMetadata` flag which, when enabled, will inject parameters onto CreateVolume driver requests with PVC and PV metadata.
+  Injected keys:
+  - csi.storage.k8s.io/pvc/name
+  - csi.storage.k8s.io/pvc/namespace
+  - csi.storage.k8s.io/pv/name ([#399](https://github.com/kubernetes-csi/external-provisioner/pull/399), [@zetsub0u](https://github.com/zetsub0u))
+- Default StorageClass secrets are added:
+  - csi.storage.k8s.io/secret-name: ${pvc.name}
+  - csi.storage.k8s.io/secret-namespace: ${pvc.namespace}
+
+  Note: Default secrets for storage class feature does work only when both parameters are added. ([#393](https://github.com/kubernetes-csi/external-provisioner/pull/393), [@taaraora](https://github.com/taaraora))
+
+## Bug Fixes
+
+- If a CSI driver returns ResourceExhausted for CreateVolume, supports topology,
+  and the `StorageClass.volumeBindingMode` is `WaitForFirstConsumer`,
+  then the pod will be rescheduled. This may then result in a CreateVolume retry
+  with a different topology.
+  ([#405](https://github.com/kubernetes-csi/external-provisioner/pull/405),
+  [@pohly](https://github.com/pohly))
+- CSI Migration: Fixes CSI migration issue where PVCs provisioned before
+  migration was turned on could not be deleted
+  ([#412](https://github.com/kubernetes-csi/external-provisioner/pull/412),
+  [@davidz627](https://github.com/davidz627))
+- Cloning: ensure source and target PVCs have matching volumeMode.
+  ([#410](https://github.com/kubernetes-csi/external-provisioner/pull/410),
+  [@j-griffith](https://github.com/j-griffith))
+
+
+## Other Notable Changes
+
+- Use informers for StorageClass ([#387](https://github.com/kubernetes-csi/external-provisioner/pull/387), [@bertinatto](https://github.com/bertinatto))
+
+

README.md

@@ -13,7 +13,7 @@ This information reflects the head of this branch.
 
 | Compatible with CSI Version                                                                | Container Image                | Recommended K8s Version |
 | ------------------------------------------------------------------------------------------ | -------------------------------| --------------- |
-| [CSI Spec v1.0.0](https://github.com/container-storage-interface/spec/releases/tag/v1.0.0) | quay.io/k8scsi/csi-provisioner | 1.17            |
+| [CSI Spec v1.0.0](https://github.com/container-storage-interface/spec/releases/tag/v1.0.0) | quay.io/k8scsi/csi-provisioner | 1.18            |
 
 ## Feature status
 
@@ -23,8 +23,6 @@ Following table reflects the head of this branch.
 
 | Feature        | Status  | Default | Description                                                                                   | Provisioner Feature Gate Required |
 | -------------- | ------- | ------- | --------------------------------------------------------------------------------------------- | --------------------------------- |
-| Topology       | Beta    | Off     | [Topology aware dynamic provisioning](https://kubernetes-csi.github.io/docs/topology.html)  (requires kubelet 1.14 on nodes). | Yes |
-| Cloning        | Beta    | On      | [Cloning](https://kubernetes-csi.github.io/docs/volume-cloning.html). | No |
 | Snapshots      | Beta    | On      | [Snapshots and Restore](https://kubernetes-csi.github.io/docs/snapshot-restore-feature.html). | No |
 | CSIMigration   | Beta    | On      | [Migrating in-tree volume plugins to CSI](https://kubernetes.io/docs/concepts/storage/volumes/#csi-migration). | No |
 
@@ -61,10 +59,14 @@ Note that the external-provisioner does not scale with more replicas. Only one e
 
 * `--worker-threads <num>`: Number of simultaneously running `ControllerCreateVolume` and `ControllerDeleteVolume` operations. Default value is `100`.
 
-* `--metrics-address`: The TCP network address address where the prometheus metrics endpoint will run (example: `:8080` which corresponds to port 8080 on local host). The default is empty string, which means metrics endpoint is disabled.
+* `--cloning-protection-threads <num>`: Number of simultaniously running threads, handling cloning finalizer removal. Defaults to `1`.
+
+* `--metrics-address`: The TCP network address where the prometheus metrics endpoint will run (example: `:8080` which corresponds to port 8080 on local host). The default is empty string, which means metrics endpoint is disabled.
 
 * `--metrics-path`: The HTTP path where prometheus metrics will be exposed. Default is `/metrics`.
 
+* `--extra-create-metadata`: Enables the injection of extra PVC and PV metadata as parameters when calling `CreateVolume` on the driver (keys: "csi.storage.k8s.io/pvc/name", "csi.storage.k8s.io/pvc/namespace", "csi.storage.k8s.io/pv/name")
+
 #### Other recognized arguments
 * `--feature-gates <gates>`: A set of comma separated `<feature-name>=<true|false>` pairs that describe feature gates for alpha/experimental features. See [list of features](#feature-status) or `--help` output for list of recognized features. Example: `--feature-gates Topology=true` to enable Topology feature that's disabled by default.
 

cmd/csi-provisioner/csi-provisioner.go

@@ -33,7 +33,7 @@ import (
 	"github.com/kubernetes-csi/csi-lib-utils/metrics"
 	ctrl "github.com/kubernetes-csi/external-provisioner/pkg/controller"
 	snapclientset "github.com/kubernetes-csi/external-snapshotter/pkg/client/clientset/versioned"
-	"sigs.k8s.io/sig-storage-lib-external-provisioner/controller"
+	"sigs.k8s.io/sig-storage-lib-external-provisioner/v5/controller"
 
 	"k8s.io/apimachinery/pkg/util/wait"
 	"k8s.io/client-go/kubernetes"
@@ -45,7 +45,7 @@ import (
 	utilfeature "k8s.io/apiserver/pkg/util/feature"
 	"k8s.io/client-go/informers"
 	v1 "k8s.io/client-go/listers/core/v1"
-	storagelisters "k8s.io/client-go/listers/storage/v1beta1"
+	storagelistersv1beta1 "k8s.io/client-go/listers/storage/v1beta1"
 	utilflag "k8s.io/component-base/cli/flag"
 	csitrans "k8s.io/csi-translation-lib"
 )
@@ -61,15 +61,17 @@ var (
 	retryIntervalStart   = flag.Duration("retry-interval-start", time.Second, "Initial retry interval of failed provisioning or deletion. It doubles with each failure, up to retry-interval-max.")
 	retryIntervalMax     = flag.Duration("retry-interval-max", 5*time.Minute, "Maximum retry interval of failed provisioning or deletion.")
 	workerThreads        = flag.Uint("worker-threads", 100, "Number of provisioner worker threads, in other words nr. of simultaneous CSI calls.")
+	finalizerThreads     = flag.Uint("cloning-protection-threads", 1, "Number of simultaniously running threads, handling cloning finalizer removal")
 	operationTimeout     = flag.Duration("timeout", 10*time.Second, "Timeout for waiting for creation or deletion of a volume")
 	_                    = deprecatedflags.Add("provisioner")
 
 	enableLeaderElection    = flag.Bool("enable-leader-election", false, "Enables leader election. If leader election is enabled, additional RBAC rules are required. Please refer to the Kubernetes CSI documentation for instructions on setting up these RBAC rules.")
 	leaderElectionType      = flag.String("leader-election-type", "endpoints", "the type of leader election, options are 'endpoints' (default) or 'leases' (strongly recommended). The 'endpoints' option is deprecated in favor of 'leases'.")
 	leaderElectionNamespace = flag.String("leader-election-namespace", "", "Namespace where the leader election resource lives. Defaults to the pod namespace if not set.")
 	strictTopology          = flag.Bool("strict-topology", false, "Passes only selected node topology to CreateVolume Request, unlike default behavior of passing aggregated cluster topologies that match with topology keys of the selected node.")
+	extraCreateMetadata     = flag.Bool("extra-create-metadata", false, "If set, add pv/pvc metadata to plugin create requests as parameters.")
 
-	metricsAddress = flag.String("metrics-address", "", "The TCP network address address where the prometheus metrics endpoint will listen (example: `:8080`). The default is empty string, which means metrics endpoint is disabled.")
+	metricsAddress = flag.String("metrics-address", "", "The TCP network address where the prometheus metrics endpoint will listen (example: `:8080`). The default is empty string, which means metrics endpoint is disabled.")
 	metricsPath    = flag.String("metrics-path", "/metrics", "The HTTP path where prometheus metrics will be exposed. Default is `/metrics`.")
 
 	featureGates        map[string]bool
@@ -171,13 +173,36 @@ func main() {
 	timeStamp := time.Now().UnixNano() / int64(time.Millisecond)
 	identity := strconv.FormatInt(timeStamp, 10) + "-" + strconv.Itoa(rand.Intn(10000)) + "-" + provisionerName
 
+	factory := informers.NewSharedInformerFactory(clientset, ctrl.ResyncPeriodOfCsiNodeInformer)
+
+	// -------------------------------
+	// Listers
+	// Create informer to prevent hit the API server for all resource request
+	scLister := factory.Storage().V1().StorageClasses().Lister()
+	claimLister := factory.Core().V1().PersistentVolumeClaims().Lister()
+
+	var csiNodeLister storagelistersv1beta1.CSINodeLister
+	var nodeLister v1.NodeLister
+	if ctrl.SupportsTopology(pluginCapabilities) {
+		csiNodeLister = factory.Storage().V1beta1().CSINodes().Lister()
+		nodeLister = factory.Core().V1().Nodes().Lister()
+	}
+
+	// -------------------------------
+	// PersistentVolumeClaims informer
+	rateLimiter := workqueue.NewItemExponentialFailureRateLimiter(*retryIntervalStart, *retryIntervalMax)
+	claimQueue := workqueue.NewNamedRateLimitingQueue(rateLimiter, "claims")
+	claimInformer := factory.Core().V1().PersistentVolumeClaims().Informer()
+
+	// Setup options
 	provisionerOptions := []func(*controller.ProvisionController) error{
 		controller.LeaderElection(false), // Always disable leader election in provisioner lib. Leader election should be done here in the CSI provisioner level instead.
 		controller.FailedProvisionThreshold(0),
 		controller.FailedDeleteThreshold(0),
-		controller.RateLimiter(workqueue.NewItemExponentialFailureRateLimiter(*retryIntervalStart, *retryIntervalMax)),
+		controller.RateLimiter(rateLimiter),
 		controller.Threadiness(int(*workerThreads)),
 		controller.CreateProvisionedPVLimiter(workqueue.DefaultControllerRateLimiter()),
+		controller.ClaimsInformer(claimInformer),
 	}
 
 	translator := csitrans.New()
@@ -192,16 +217,6 @@ func main() {
 		provisionerOptions = append(provisionerOptions, controller.AdditionalProvisionerNames([]string{supportsMigrationFromInTreePluginName}))
 	}
 
-	var csiNodeLister storagelisters.CSINodeLister
-	var nodeLister v1.NodeLister
-	var factory informers.SharedInformerFactory
-	if ctrl.SupportsTopology(pluginCapabilities) {
-		// Create informer to prevent hit the API server for all resource request
-		factory = informers.NewSharedInformerFactory(clientset, ctrl.ResyncPeriodOfCsiNodeInformer)
-		csiNodeLister = factory.Storage().V1beta1().CSINodes().Lister()
-		nodeLister = factory.Core().V1().Nodes().Lister()
-	}
-
 	// Create the provisioner: it implements the Provisioner interface expected by
 	// the controller
 	csiProvisioner := ctrl.NewCSIProvisioner(
@@ -218,8 +233,12 @@ func main() {
 		supportsMigrationFromInTreePluginName,
 		*strictTopology,
 		translator,
+		scLister,
 		csiNodeLister,
-		nodeLister)
+		nodeLister,
+		claimLister,
+		*extraCreateMetadata,
+	)
 
 	provisionController = controller.NewProvisionController(
 		clientset,
@@ -229,25 +248,31 @@ func main() {
 		provisionerOptions...,
 	)
 
+	csiClaimController := ctrl.NewCloningProtectionController(
+		clientset,
+		claimLister,
+		claimInformer,
+		claimQueue,
+	)
+
 	run := func(context.Context) {
-		if factory != nil {
-			stopCh := context.Background().Done()
-			factory.Start(stopCh)
-			cacheSyncResult := factory.WaitForCacheSync(stopCh)
-			for _, v := range cacheSyncResult {
-				if !v {
-					klog.Fatalf("Failed to sync Informers!")
-				}
+		stopCh := context.Background().Done()
+		factory.Start(stopCh)
+		cacheSyncResult := factory.WaitForCacheSync(stopCh)
+		for _, v := range cacheSyncResult {
+			if !v {
+				klog.Fatalf("Failed to sync Informers!")
 			}
 		}
 
+		go csiClaimController.Run(int(*finalizerThreads), stopCh)
 		provisionController.Run(wait.NeverStop)
 	}
 
 	if !*enableLeaderElection {
 		run(context.TODO())
 	} else {
-		// this lock name pattern is also copied from sigs.k8s.io/sig-storage-lib-external-provisioner/controller
+		// this lock name pattern is also copied from sigs.k8s.io/sig-storage-lib-external-provisioner/v5/controller
 		// to preserve backwards compatibility
 		lockName := strings.Replace(provisionerName, "/", "-", -1)
 

go.mod

@@ -8,18 +8,18 @@ require (
 	github.com/kubernetes-csi/csi-lib-utils v0.7.0
 	github.com/kubernetes-csi/csi-test v2.0.0+incompatible
 	github.com/kubernetes-csi/external-snapshotter v1.2.1-0.20191220180133-bba358438aee
-	github.com/miekg/dns v1.1.8 // indirect
 	github.com/spf13/pflag v1.0.5
 	google.golang.org/grpc v1.26.0
-	k8s.io/api v0.17.0
-	k8s.io/apimachinery v0.17.1-beta.0
+	k8s.io/api v0.17.3
+	k8s.io/apimachinery v0.17.3
 	k8s.io/apiserver v0.17.0
 	k8s.io/client-go v0.17.0
 	k8s.io/component-base v0.17.0
 	k8s.io/csi-translation-lib v0.17.0
 	k8s.io/klog v1.0.0
 	k8s.io/kubernetes v1.14.0
-	sigs.k8s.io/sig-storage-lib-external-provisioner v4.0.1+incompatible
+	sigs.k8s.io/sig-storage-lib-external-provisioner v4.1.0+incompatible // indirect
+	sigs.k8s.io/sig-storage-lib-external-provisioner/v5 v5.0.0
 )
 
 replace k8s.io/apiextensions-apiserver => k8s.io/apiextensions-apiserver v0.17.0