diff --git a/keps/prod-readiness/sig-storage/3314.yaml b/keps/prod-readiness/sig-storage/3314.yaml new file mode 100644 index 00000000000..b2568c9d30a --- /dev/null +++ b/keps/prod-readiness/sig-storage/3314.yaml @@ -0,0 +1,3 @@ +kep-number: 3314 +alpha: + approver: "@soltysh" diff --git a/keps/sig-storage/3314-csi-changed-block-tracking/README.md b/keps/sig-storage/3314-csi-changed-block-tracking/README.md new file mode 100644 index 00000000000..604626f83b6 --- /dev/null +++ b/keps/sig-storage/3314-csi-changed-block-tracking/README.md @@ -0,0 +1,1689 @@ + +# KEP-3314: CSI Changed Block Tracking + + + + + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories](#user-stories) + - [Full snapshot backup](#full-snapshot-backup) + - [Incremental snapshot backup](#incremental-snapshot-backup) + - [Notes/Constraints/Caveats](#notesconstraintscaveats) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [The CSI SnapshotMetadata Service API](#the-csi-snapshotmetadata-service-api) + - [Metadata Format](#metadata-format) + - [GetMetadataAllocated RPC](#getmetadataallocated-rpc) + - [GetMetadataAllocated Errors](#getmetadataallocated-errors) + - [GetMetadataDelta RPC](#getmetadatadelta-rpc) + - [GetMetadataDelta Errors](#getmetadatadelta-errors) + - [Kubernetes Components](#kubernetes-components) + - [The Kubernetes SnapshotMetadata Service API](#the-kubernetes-snapshotmetadata-service-api) + - [Snapshot Metadata Service Custom Resource](#snapshot-metadata-service-custom-resource) + - [The SP Snapshot Metadata Service](#the-sp-snapshot-metadata-service) + - [The External Snapshot Metadata Sidecar](#the-external-snapshot-metadata-sidecar) + - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit tests](#unit-tests) + - [Integration tests](#integration-tests) + - [e2e tests](#e2e-tests) + - [Graduation Criteria](#graduation-criteria) + - [Alpha](#alpha) + - [Beta](#beta) + - [GA](#ga) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed (Optional)](#infrastructure-needed-optional) + + +## Release Signoff Checklist + + + +Items marked with (R) are required *prior to targeting to a milestone / release*. + +- [x] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) +- [ ] (R) KEP approvers have approved the KEP status as `implementable` +- [x] (R) Design details are appropriately documented +- [x] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors) + - [ ] e2e Tests for all Beta API Operations (endpoints) + - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) + - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free +- [x] (R) Graduation criteria is in place + - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) +- [ ] (R) Production readiness review completed +- [ ] (R) Production readiness review approved +- [ ] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + + + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website + +## Summary + + + +This KEP proposes new CSI API that can be used to identify the list of changed +blocks between pairs of CSI volume snapshots. CSI drivers can implement this API +to expose their changed block tracking (CBT) services to enable efficient and +reliable differential backup of data stored in CSI volumes. + +Kubernetes backup applications directly use this API to stream changed +block information, bypassing and posing no additional load on the Kubernetes +API server. +The mechanism that enables this direct access utilizes a proxy service sidecar +to shield the CSI drivers from managing the individual Kubernetes clients. + +## Motivation + + + +Changed block tracking (CBT) techniques have been used by commercial backup +systems to efficiently back up large amount of data in block volumes. They +identify block-level changes between two arbitrary pair of snapshots of the +same block volume, and selectively back up what has changed between the two +checkpoints. This type of differential backup approach is a lot more efficient +than backing up the entire volume. + +This KEP proposes a design to extend the Kubernetes CSI framework to utilize +these CBT features to bring efficient, cloud-native data protection to +Kubernetes users. + +### Goals + + + +* Provide a secure, idiomatic CSI API to efficiently identify +the allocated blocks of a CSI volume snapshot, and +the changed blocks between two arbitrary pairs of CSI volume snapshots +of the same block volume. +* Relay large amount of snapshot metadata from the storage provider without +overloading the Kubernetes API server. +* This API is an optional component of the CSI framework. + +### Non-Goals + + + +* Specify how data is written to the block volume in the first place. + > *The volume could be attached to a pod with either `Block` or `Filesystem` + [volume modes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#volume-mode).* +* Provide an API to retrieve the data blocks of a snapshot. + > *It is assumed that a snapshot's data blocks can be retrieved by creating a + PersistentVolume for the snapshot, launching a pod with this volume + attached in `Block` volume mode, and then reading the individual + blocks from the raw block device.* + +* Support of file changed list tracking for network file shares is not +addressed by this proposal. + +## Proposal + + + +The proposal extends the CSI specification with a new, optional, +[CSI SnapshotMetadata gRPC service](#the-csi-snapshotmetadata-service-api), +that is used by Kubernetes to retrieve metadata on the allocated blocks of a +single snapshot, or the changed blocks between a pair of snapshots of the same +block volume. + +A [Kubernetes SnapshotMetadata gRPC service](#the-kubernetes-snapshotmetadata-service-api) +is an API that is used by a Kubernetes backup application client to retrieve +snapshot metadata. This API is implemented by the the community provided +[external-snapshot-metadata sidecar](#the-external-snapshot-metadata-sidecar), +which must be deployed by a CSI driver. +A Kubernetes backup application will retrieve snapshot metadata through a +TLS gRPC connection to such a service. +This direct connection results in a minimal load on the Kubernetes API server, +unrelated to the amount of metadata transferred or the sizes of the volumes +and snapshots involved. + +The [external-snapshot-metadata sidecar](#the-external-snapshot-metadata-sidecar) +communicates over a private UNIX domain socket with the CSI driver's +implementation of the [CSI SnapshotMetadata](#the-csi-snapshotmetadata-service-api) +gRPC service. +The CSI driver service only handles the retrieval of the metadata requested; +the sidecar is responsible for validating the Kubernetes authentication token, +authorizing the backup application, validating the parameters of the RPC calls +and fetching the provisioner secrets needed to complete the request. +The sidecar forwards the RPC call to the CSI driver service over the UNIX +domain socket, after translating Kubernetes object names into SP object names, +and re-streams the results back to its client. + +A CSI driver advertises the existence of the community sidecar's +[Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) gRPC service +to Kubernetes backup applications +by creating a [SnapshotMetadataService CR](#snapshot-metadata-service-custom-resource) +that contains the service's TCP endpoint address, CA certificate and +an audience string needed for token authentication. +The CSI driver name is specified in a metadata label in this +CR, so that a backup application can efficiently search for +the [Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) gRPC service +of the provisioner of the VolumeSnapshots to be backed up. + +Before accessing a +[Kubernetes SnapshotMetadata gRPC service](#the-kubernetes-snapshotmetadata-service-api) +a backup application must first obtain an authentication token using the Kubernetes +[TokenRequest API](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-request-v1/) +with the service's audience string. +It should establish trust with the specified CA for use in gRPC calls and +then directly make TLS gRPC calls to the +[Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) +gRPC service's TCP endpoint. +The audience-scoped authentication token must be passed in the `security_token` +field of each RPC request message; +it will be used to authorize the backup application's use of the service. +Every RPC returns a gRPC stream through which the metadata can be recovered. + +The process of accessing snapshot metadata via the sidecar is illustrated in the figure below. +Additional information is available in the [Design Details](#design-details) section. + +![metadata retrieval flow](./flow.drawio.svg) + +### User Stories + + + + +#### Full snapshot backup + +A backup application needs to perform a full backup on volumes of a specific +Kubernetes application. + +For each volume in the application: +1. The backup application creates a VolumeSnapshot of a PVC that needs to be +backed up. +2. The backup application queries the changed block tracking (CBT) service to identify +all the allocated data blocks in the snapshot. The CBT service returns the list +of allocated blocks. +3. Using the VolumeSnapshot as the source, the backup application creates a new +PVC and mounts it with `Block` VolumeMode in a pod. +4. The backup application uses the CBT metadata to identify the data that needs to +be backed up and reads these blocks from the mounted PVC in the pod. + +#### Incremental snapshot backup + +A backup application needs to perform an incremental backup on volumes of a specific +Kubernetes application. The backup application knows the identifiers of the +VolumeSnapshots it had backed up previously. + +For each volume in the application: +1. The backup application creates a VolumeSnapshot of a PVC that needs to be +backed up incrementally. +2. The backup application queries the changed block tracking (CBT) service to identify the +changes between the latest snapshot and the one it had previously backed up. +The CBT service returns the list of blocks changed between the snapshots. +3. Using the latest VolumeSnapshot as the source, the backup application creates a new +PVC and mounts it with `Block` VolumeMode in a pod. +4. The backup application uses the CBT metadata to find the only changed data to +backup and reads these blocks from the mounted PVC in the pod. + +### Notes/Constraints/Caveats + + + +- This proposal requires a backup application to directly connect to a +[Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) +gRPC service offered by the community provided +[external-snapshot-metadata sidecar](#the-external-snapshot-metadata-sidecar) +deployed by a CSI driver. +This was necessary to not place a load on the Kubernetes API server +that would be proportional to the number of allocated blocks in a volume +snapshot. + +- Each [Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) +gRPC service only operates on volume snapshots provisioned by the CSI driver +that deploys the related sidecar. +A backup application is responsible for locating the +service for a CSI driver by searching for its +[SnapshotMetadataService CR](#snapshot-metadata-service-custom-resource). +This search can fail as this service is optional. + +- A backup application must obtain a Kubernetes audience-scoped authentication +token in order to use a +[Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) +gRPC service. +This requires that the backup application be authorized to use the Kubernetes +[TokenRequest API](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-request-v1/). +The token can be obtained directly by a call to this API, or indirectly via +a projected volume in the Pod used to access the +[Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) +API. + +- The Kubernetes audience-scoped authentication token must be provided as the + `security_token` field in each gRPC request message made by a backup + application. + +- The [CSI SnapshotMetadata](#the-csi-snapshotmetadata-service-api) +gRPC service RPC calls +allow an application to ***restart*** an interrupted +stream from where it previously failed +by reissuing the RPC call with a starting byte offset. +The same functionality is available through the +[Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) +of the service. + +- The [CSI SnapshotMetadata](#the-csi-snapshotmetadata-service-api) +gRPC service +permits metadata to be returned in either an ***extent*** +or a ***block*** based format, at the discretion of the CSI driver. +A portable backup application is expected to handle both such formats. +This also applies to the +[Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) +of the service. + +- The [CSI SnapshotMetadata](#the-csi-snapshotmetadata-service-api) gRPC service +must be capable of serving metadata on a VolumeSnapshot +concurrently with the backup application's use of a PersistentVolume +created on that same VolumeSnapshot. +This is because a backup application would likely mount the PersistentVolume with +`Block` VolumeMode in a Pod in order to read and archive the raw snapshot data blocks, +and this read/archive loop will be driven by the stream of snapshot block metadata. + +- The proposal does not specify how its security model is to be implemented. + It is expected that the RBAC policies used by backup applications + and the existing CSI drivers will be extended for this purpose. + +### Risks and Mitigations + + + +> *A review by SIG-Auth (July 19, 2023) recommended the use of the +> [TokenRequest](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-request-v1/) and +> [TokenReview](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/) +> APIs to make authentication and authorization checks possible between +> authorized Kubernetes principals.* + +The following risks are identified: + +- Exposure of snapshot metadata by the use of the networked + [Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) + gRPC API. +- Uncontrolled access to the [Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) + service could lead to denial-of-service attacks. +- A principal with the authority to use a + [Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) + gRPC service indirectly gains access to the metadata of otherwise + inaccessible VolumeSnapshots. + +The risks are mitigated as follows: + +- The possible exposure of snapshot metadata by use of a network API + is addressed by using **encryption** and **mutual authentication** for the + direct gRPC call made by the backup application client. + The gRPC client is required to first establish trust with the service's CA, + and, while the direct TLS gRPC call itself does not perform mutual authentication, + an **audience-scoped authentication token** must be passed as a parameter in each RPC + call, which effectively provides the mechanism for the service to both authenticate + and authorize the client. + + The audience-scoped authentication token is obtained from the Kubernetes + [TokenRequest API](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-request-v1/) + and is validated with the Kubernetes + [TokenReview API](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/). + Its scope is narrowed to the just the target service (the "***audience***") by specifying + an audience string defined by and unique to the service, during token creation. + An authentication token has an expiry time so will not last forever; + additionally, it can be bound to the Pod used by the backup application to access the + service, to further constrain its effective lifetime. + +- Access to a [Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) + gRPC service, to the VolumeSnapshots referenced by through the service, + and the ability to use the + [TokenRequest](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-request-v1/), + [TokenReview](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/) and + [SubjectAccessReview](https://kubernetes.io/docs/reference/kubernetes-api/authorization-resources/subject-access-review-v1/) + APIs are controlled by Kubernetes security policy. + +The proposal requires the existence of security policy to establish the access +rights described below, and illustrated in the following figure: + +![Permissions needed](./perms.drawio.svg) + + +The proposal requires that Kubernetes security policy authorize access to: + +- The [SnapshotMetadataService CR](#snapshot-metadata-service-custom-resource) objects + that advertise the existence of + [Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) + gRPC services available in the cluster. + These objects do not contain secret information so limiting access just controls + the principals who obtain the service contact information. + At the least, backup applications should be permitted to read these objects. + +- Backup applications must be granted permission to use the Kubernetes + [TokenRequest API](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-request-v1/) + in order to obtain the audience-scoped authentication tokens that are passed + in each [Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) + gRPC service RPC call. + +- Backup applications must be granted access to view VolumeSnapshot objects in + the target namespaces. + Presumably they already have such permission if they were the ones initiating + the creation of the VolumeSnapshot objects. + +- The CSI driver service account must be granted permission to use the Kubernetes + [TokenReview API](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/) + in order for its + [Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) + gRPC service to validate the authentication token. + +- The CSI driver must be granted permission to use the Kubernetes + [SubjectAccessReview API](https://kubernetes.io/docs/reference/kubernetes-api/authorization-resources/subject-access-review-v1/) + in order for its + [Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) + gRPC service to validate that a security token authorizes access to + VolumeSnapshot objects in a namespace. + +- The CSI driver presumably already has access rights to the VolumeSnapshot + and VolumeSnapshot content objects as they are within its purview. + This is needed for its + [Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) + gRPC service. + +The proposal does not specify how such a security policy is to be configured. + +## Design Details + + + + +### The CSI SnapshotMetadata Service API + +> In this section we use the terminology of the gRPC specification where the +> word *service* is assumed to be a *gRPC service* and not a Kubernetes service, +> while the word *plugin* is +> used to refer to the software component that implements the gRPC service. + +The CSI specification will be extended with the addition of the following new, optional +**SnapshotMetadata** [gRPC service](https://grpc.io/docs/what-is-grpc/core-concepts/#service-definition). +The [SP Snapshot Metadata](#the-sp-snapshot-metadata-service) plugin implements +this service. + +The service is defined as follows, and will be described in the sub-sections below. +Refer to [CSI PR 551](https://github.com/container-storage-interface/spec/pull/551) +for the official specification. +``` +service SnapshotMetadata { + rpc GetMetadataAllocated(GetMetadataAllocatedRequest) + returns (stream GetMetadataAllocatedResponse) {} + rpc GetMetadataDelta(GetMetadataDeltaRequest) + returns (stream GetMetadataDeltaResponse) {} +} + +enum BlockMetadataType { + UNKNOWN=0; + FIXED_LENGTH=1; + VARIABLE_LENGTH=2; +} + +message BlockMetadata { + int64 byte_offset = 1; + int64 size_bytes = 2; +} + +message GetMetadataAllocatedRequest { + string snapshot_id = 1; + int64 starting_offset = 2; + int32 max_results = 3; + map secrets = 4; +} + + +message GetMetadataAllocatedResponse { + BlockMetadataType block_metadata_type = 1; + int64 volume_capacity_bytes = 2; + repeated BlockMetadata block_metadata = 3; +} + +message GetMetadataDeltaRequest { + string base_snapshot_id = 1; + string target_snapshot_id = 2; + int64 starting_offset = 3; + int32 max_results = 4; + map secrets = 5; +} + +message GetMetadataDeltaResponse { + BlockMetadataType block_metadata_type = 1; + int64 volume_capacity_bytes = 2; + repeated BlockMetadata block_metadata = 3; +} +``` + +#### Metadata Format +Block volume data ranges are specified by a sequence of `(ByteOffset, Length)` tuples, +with the tuples in ascending order of `ByteOffset` and no overlap between adjacent tuples. +There are two prevalent styles, *extent-based* or *block-based*, +which describe if the `Length` field of the tuples in a sequence can +**vary** or are **fixed** across all the tuples in the sequence. +The **SnapshotMetadata** service permits either style at the discretion of the plugin, +and it is required that a client of this service be able to handle both styles. + +The `BlockMetadataType` enumeration specifies the style used: `FIXED_LENGTH` or `VARIABLE_LENGTH`. +When the *block-based* style (`FIXED_LENGTH`) is used it is up to the SP plugin to define the +block size. + +An individual tuple is identified by the `BlockMetadata` message, and the sequence is +defined collectively across the tuple lists returned in the RPC +[message stream](https://grpc.io/docs/what-is-grpc/core-concepts/#server-streaming-rpc). +Note that the plugin must ensure that the style is not change mid-stream in any given RPC invocation. + +#### GetMetadataAllocated RPC +The `GetMetadataAllocated` RPC returns metadata on the ***allocated blocks*** of a snapshot - +i.e. this identifies the data ranges that have valid data as they were the target of +some previous write operation. +Backup applications typically make an initial **full** backup of a volume followed +by a series of **incremental** backups, and the size of the initial full backup can +be reduced considerably if only the allocated blocks are saved. + +The RPC's input arguments are specified by the `GetMetadataAllocatedRequest` message, +and it returns a +[stream](https://grpc.io/docs/what-is-grpc/core-concepts/#server-streaming-rpc) +of `GetMetadataAllocatedResponse` messages. +The fields of the `GetMetadataAllocatedRequest` message are defined as follows: + - `snapshot_id`
+ The identifier of a snapshot of the specified volume, in the nomenclature of the plugins. + + - `starting_offset`
+ This specifies the 0 based starting byte position in the volume snapshot from which the result should be computed. + It is intended to be used to continue a previously interrupted call. + The plugins may round down this offset to the nearest alignment boundary based on the + `BlockMetadataType` it will use. + + - `max_results`
+ This is an optional field. If non-zero it specifies the maximum length of the `block_metadata` list + that the client wants to process in a given `GetAllocateResponse` element. + The plugins will determine an appropriate value if 0, and is always free to send less than the requested + maximum. + + - `secrets`
+ This is an optional field. It should contain the provisioner secrets associated with + the volume snapshot, if any. + In Kubernetes such data is specified with the keys + `csi.storage.k8s.io/snapshotter-secret-[name|namespace]` +in the `VolumeSnapshotClass.Parameters` field. + +The fields of the `GetMetadataAllocatedResponse` message are defined as follows: +- `block_metadata_type`
+ This specifies the metadata format as described in the [Metadata Format](#metadata-format) section above. + +- `volume_capacity_bytes`
+ The size of the underlying volume, in bytes. + +- `block_metadata`
+ This is a list of `BlockMetadata` tuples as described in the + [Metadata Format](#metadata-format) section above. + The caller may request a maximum length of this list in the `max_results` field + of the `GetMetadataAllocatedRequest` message, otherwise the length is determined by the plugins. + +Note that while the `block_metadata_type` and `volume_capacity_bytes` fields are +repeated in each `GetMetadataAllocatedResponse` message by the nature of the syntax of the +specification language, their values in a given RPC invocation must be constant. +i.e. a plugin is not free to modify these value mid-stream. + +##### GetMetadataAllocated Errors +If the plugin is unable to complete the `GetMetadataAllocated` call successfully it +must return a non-OK gRPC code in the gRPC status. + +The following conditions are well defined: + +| Condition | gRPC Code | Description | Recovery Behavior | +|-----------|-----------|-------------|-------------------| +| Missing or otherwise invalid argument | 3 INVALID_ARGUMENT | Indicates that a required argument field was not specified or an argument value is invalid | The caller should correct the error and resubmit the call. | +| Invalid `snapshot` | 5 NOT_FOUND | Indicates that the snapshot specified was not found. | The caller should re-check that this object exists. | +| Invalid `starting_offset` | 11 OUT_OF_RANGE | The starting offset exceeds the volume size. | The caller should specify a `starting_offset` less than the volume's size. | + +#### GetMetadataDelta RPC +The `GetMetadataDelta` RPC returns the metadata on the blocks that have changed between +a pair of snapshots from the same volume. + +The RPC's input arguments are specified by the `GetMetadataDeltaRequest` message, +and it returns a +[stream](https://grpc.io/docs/what-is-grpc/core-concepts/#server-streaming-rpc) +of `GetMetadataDeltaResponse` messages. +The fields of the `GetMetadataDeltaRequest` message are defined as follows: + + - `base_snapshot_id`
+ The identifier of a snapshot of the specified volume, in the nomenclature of the plugins. + + - `target_snapshot_id`
+ The identifier of a second snapshot of the specified volume, in the nomenclature of the plugins. + This snapshot should have been created after the base snapshot, and the RPC will return the changes + made since the base snapshot was created. + + - `starting_offset`
+ This specifies the 0 based starting byte position in the `target_snapshot` from which the result should be computed. + It is intended to be used to continue a previously interrupted call. + The plugins may round down this offset to the nearest alignment boundary based on the + `BlockMetadataType` it will use. + + - `max_results`
+ This is an optional field. If non-zero it specifies the maximum length of the `block_metadata` list + that the client wants to process in a given `GetMetadataDeltaResponse` element. + The plugins will determine an appropriate value if 0, and is always free to send less than the requested + maximum. + + - `secrets`
+ This is an optional field. It should contain the provisioner secrets associated with + the volume snapshot, if any. + In Kubernetes such data is specified with the keys + `csi.storage.k8s.io/snapshotter-secret-[name|namespace]` +in the `VolumeSnapshotClass.Parameters` field. +This field should not be set by a Kubernetes backup application client. + +The fields of the `GetMetadataDeltaResponse` message are defined as follows: +- `block_metadata_type`
+ This specifies the metadata format as described in the [Metadata Format](#metadata-format) section above. + +- `volume_capacity_bytes`
+ The size of the underlying volume, in bytes. + +- `block_metadata`
+ This is a list of `BlockMetadata` tuples as described in the + [Metadata Format](#metadata-format) section above. + The caller may request a maximum length of this list in the `max_results` field + of the `GetMetadataDeltaRequest` message, otherwise the length is determined by the plugins. + +Note that while the `block_metadata_type` and `volume_capacity_bytes` fields are +repeated in each `GetMetadataDeltaResponse` message by the nature of the syntax of the +specification language, their values in a given RPC invocation must be constant. +i.e. a plugin is not free to modify these value mid-stream. + +##### GetMetadataDelta Errors +If the plugin is unable to complete the `GetMetadataDelta` call successfully it +must return a non-OK gRPC code in the gRPC status. + +The following conditions are well defined: + +| Condition | gRPC Code | Description | Recovery Behavior | +|-----------|-----------|-------------|-------------------| +| Missing or otherwise invalid argument | 3 INVALID_ARGUMENT | Indicates that a required argument field was not specified or an argument value is invalid | The caller should correct the error and resubmit the call. | +| Invalid `base_snapshot` or `target_snapshot` | 5 NOT_FOUND | Indicates that the snapshots specified were not found. | The caller should re-check that these objects exist. | +| Invalid `starting_offset` | 11 OUT_OF_RANGE | The starting offset exceeds the volume size. | The caller should specify a `starting_offset` less than the volume's size. | + +### Kubernetes Components +The following Kubernetes resources and components are involved at runtime: + +- The [Kubernetes SnapshotMetadata Service API](#the-kubernetes-snapshotmetadata-service-api) + is used by Kubernetes clients to retrieve snapshot metadata. + +- A [SnapshotMetadataService CR](#snapshot-metadata-service-custom-resource) + that advertises the existence of a + [Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) + gRPC service. + +- A CSI driver vendor provided + [SP Snapshot Metadata Service](#the-sp-snapshot-metadata-service) + that actually sources the snapshot metadata required through a + [CSI SnapshotMetadata](#the-csi-snapshotmetadata-service-api) + gRPC service. + +- A community provided + [external-snapshot-metadata sidecar](#the-external-snapshot-metadata-sidecar) + that provides a + [Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) + gRPC service that interacts with Kubernetes clients and **proxies** TCP TLS + gRPC requests from Kubernetes client applications to the + [CSI SnapshotMetadata](#the-csi-snapshotmetadata-service-api) + gRPC service implemented by a + [SP Snapshot Metadata Service](#the-sp-snapshot-metadata-service), + over a UNIX domain transport. + +#### The Kubernetes SnapshotMetadata Service API + +The proposal minimizes the use of the Kubernetes API server when retrieving +snapshot metadata. +Instead, it calls for a Kubernetes client to directly make a gRPC connection +with a community provided +[external-snapshot-metadata sidecar](#the-external-snapshot-metadata-sidecar) +which will proxy the calls to an SP provided service that implements the +[CSI SnapshotMetadata Service API](#the-csi-snapshotmetadata-service-api). + +The proposal could have called for client to use the +[CSI SnapshotMetadata Service API](#the-csi-snapshotmetadata-service-api) +itself when communicating with the sidecar, but there are a number of reasons +to specify a different API, the **Kubernetes SnapshotMetadata API**. +These reasons and any related modifications to the API are described below: + +- Introduction of a level of indirection between the CSI specification + and the Kubernetes implementation to decouple the life-cycle of the + [external-snapshot-metadata sidecar](#the-external-snapshot-metadata-sidecar) + from the life-cycle of the CSI specification. + +- The need to pass a Kubernetes audience-scoped authentication token to + the sidecar. + This is handled by introducing a `security_token` field in every + request message. + All Kubernetes SnapshotMetadata API calls will return the `UNAUTHENTICATED` + gRPC error code if this token is incorrect or adequate authority has not + been granted to the invoker. + +- Kubernetes Snapshots must be identified by `namespace` and `name`, while the + CSI specification uses a single identifier. + This difference is handled by the addition of a `namespace` parameter in + every Kubernetes SnapshotMetadata API message that identifies a Snapshot, + and the replacement of snapshot "id" parameters with "name" parameters. + +- The CSI specification requires provisioner secrets be passed + to the SP service if configured for the CSI driver. + These secrets are out of the purview of a backup application and are + fetched and inserted by the sidecar. + As such, all `secrets` parameters are removed from the + Kubernetes SnapshotMetadata API messages. + +The proposed Kubernetes SnapshotMetadata API is very similar to the +[CSI SnapshotMetadata Service API](#the-csi-snapshotmetadata-service-api). +The only structural differences between the two specifications are in some +message properties. +The messages modified in the Kubernetes SnapshotMetadata API are shown below: +``` +message GetMetadataAllocatedRequest { + string security_token = 1; + string namespace = 2; + string snapshot_name = 3; + int64 starting_offset = 4; + int32 max_results = 5; +} + +message GetMetadataDeltaRequest { + string security_token = 1; + string namespace = 2; + string base_snapshot_name = 3; + string target_snapshot_name = 4; + int64 starting_offset = 5; + int32 max_results = 6; +} +``` +The full specification of the Kubernetes SnapshotMetadata API will be +published in the source code repository of the +[external-snapshot-metadata sidecar](#the-external-snapshot-metadata-sidecar). + +#### Snapshot Metadata Service Custom Resource + +`SnapshotMetadataService` is a cluster-scoped Custom Resource that defines the +existence of, and contains information needed to connect to the +[Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) +gRPC service provided by the +[external-snapshot-metadata sidecar](#the-external-snapshot-metadata-sidecar) +deployed by a CSI driver. + +The CR name should be that of the associated CSI driver to ensure that only one such +CR is created for a given driver. + +The CR `spec` contains the following fields: + +- `address`
+ Specifies the IP address or DNS name of the gRPC service. + It should be provided in the format host:port, + without specifying the scheme (e.g., http or https). +- `caCert`
+ Specifies the CA certificate used to enable TLS (Transport Layer + Security) security for gRPC calls made to the service. +- `audience`
+ Specifies the audience string value expected in an audience-scoped authentication token + presented to the [sidecar](#the-external-snapshot-metadata-sidecar) by a Kubernetes client. + The value should be unique to the service if possible; for example, it could be the DNS name of the service. + +The full Custom Resource Definition is shown below: + +```yaml +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.11.1 + api-approved.kubernetes.io: unapproved + creationTimestamp: null + name: snapshotmetadataservices.cbt.storage.k8s.io +spec: + group: cbt.storage.k8s.io + names: + kind: SnapshotMetadataService + listKind: SnapshotMetadataServiceList + plural: snapshotmetadataservices + singular: snapshotmetadataservice + scope: Cluster + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: 'The presence of a SnapshotMetadataService CR advertises the existence of a CSI + driver's Kubernetes SnapshotMetadata gRPC service. + An audience scoped Kubernetes authentication bearer token must be passed in the + "security_token" field of each gRPC call made by a Kubernetes backup client.' + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: This contains data needed to connect to a Kubernetes SnapshotMetadata gRPC service. + properties: + address: + type: string + description: The TCP endpoint address of the gRPC service. + audience: + type: string + description: The audience string value expected in a client's authentication token passed in the + "security_token" field of each gRPC call. + caCert: + description: Certificate authority bundle needed by the client to validate the service. + format: byte + type: string + type: object + type: object + served: true + storage: true +``` + +#### The SP Snapshot Metadata Service + +The SP must provide a container that implements the +[CSI SnapshotMetadata](#the-csi-snapshotmetadata-service-api) +gRPC service that sources the snapshot metadata requested by a Kubernetes client. +The service runs under the authority of the CSI driver ServiceAccount, +which must be authorized as described in [Risks and Mitigations](#risks-and-mitigations). + +The CSI driver should delegate all Kubernetes client interaction to +the community provided [external-snapshot-metadata sidecar](#the-external-snapshot-metadata-sidecar) +which must be deployed in a pod alongside its service container, +and configured to communicate with it over a UNIX domain socket. +The sidecar will translate all Kubernetes object identifiers used by the Kubernetes +client to SP identifiers before forwarding the remote procedure calls. + +The SP service decides whether the metadata is returned in *block-based* format +(`block_metadata_type` is `FIXED_LENGTH`) +or *extent-based* format (`block_metadata_type` is `VARIABLE_LENGTH`). +In any given RPC call, the `block_metadata_type` and `volume_capacity_bytes` return +properties should be constant; +likewise the `size_bytes` of all `BlockMetadata` entries if the `block_metadata_type` +value returned is `FIXED_LENGTH`. + +#### The External Snapshot Metadata Sidecar + +The `external-snapshot-metadata` sidecar is a community provided container +that handles all aspects of Kubernetes client interaction for +a [SP Snapshot Metadata Service](#the-sp-snapshot-metadata-service). +The sidecar should be configured to run under the authority of the CSI driver ServiceAccount, +which must be authorized as described in [Risks and Mitigations](#risks-and-mitigations). + +A Service object must be created for the TCP based +[Kubernetes SnapshotMetadata](#the-kubernetes-snapshotmetadata-service-api) +gRPC service implemented by the sidecar. +A [SnapshotMetadataService CR](#snapshot-metadata-service-custom-resource) +must be created for the gRPC service within the sidecar. +The CR contains the CA certificate and Service endpoint address of the sidecar and +the audience string needed for the client's authentication token. +The sidecar must be configured with the name of this CR object. + +The sidecar must be deployed in the same pod as the +[SP Snapshot Metadata Service](#the-sp-snapshot-metadata-service) +and must be configured to communicate with it through a UNIX domain socket. + +The sidecar acts as a proxy for the +[SP Snapshot Metadata Service](#the-sp-snapshot-metadata-service), +handling all aspects of the Kubernetes client interaction described +in this proposal, including +- Authenticating the client. +- Authorizing the client. +- Validating individual RPC arguments. +- Translating RPC arguments from the Kubernetes domain to the SP domain at runtime. +- Fetching the vendor secrets identified by the VolumeSnapshot object, if any. + +A Kubernetes client must provide an audience scoped authentication token in the +`security_token` field of every remote procedure call request message. +The sidecar will use the +[TokenReview API](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/) +to validate this authentication token, using the audience string specified in the +[SnapshotMetadataService CR](#snapshot-metadata-service-custom-resource). + +If the client is authenticated, the sidecar will then use the returned **UserInfo** in the +[SubjectAccessReview API](https://kubernetes.io/docs/reference/kubernetes-api/authorization-resources/subject-access-review-v1/) +to verify that the client has access to VolumeSnapshot objects in the `namespace` +specified in the remote procedure call. + +The sidecar will attempt to load the VolumeSnapshots specified +along with their associated VolumeSnapshotContent objects, to ensure that they still +exist, belong to the CSI driver, and to obtain their SP identifiers. +Additional checks may be performed depending on the RPC; +for example, in the case of a [GetMetadataDelta](#getdelta-rpc) RPC, it will check +that all the snapshots come from the same volume and that the +snapshot order is correct. + +If all checks are successful, the RPC call is forwarded to the +[CSI SnapshotMetadata](#the-csi-snapshotmetadata-service-api) gRPC service in the +[SP Snapshot Metadata Service](#the-sp-snapshot-metadata-service) container +over the UNIX domain socket, with its input parameters appropriately +translated. The metadata result stream is sent back to the calling +Kubernetes client. + +### Test Plan + + + +[x] I/we understand the owners of the involved components may require updates to +existing tests to make this code solid enough prior to committing the changes necessary +to implement this enhancement. + +##### Prerequisite testing updates + + + +##### Unit tests + + + + + +All unit tests will be included in the out-of-tree CSI repositories, with no +impact on the test coverage of the core packages. + +These include: + +* Unit tests to cover the logic around retrieving the audience-scoped token and +volume snapshot name and namespace from the gGRPC request metadata. +* Unit tests to cover the logic around getting the snapshot, snapshot content, +snapshot class, volume handle, and metadata service resources. +* Unit tests to cover the logic around creating token review and subject access +review resources. +* Unit tests to ensure no tokens/secrets are logged by k8s-csi logging methods. + +##### Integration tests + + + + + +No integration tests are required. This feature is better tested with e2e tests. + +##### e2e tests + + + +The prototype project of this KEP contains a [sample gRPC client][2] that can be +used to simulate gRPC requests to the `SnapshotMetadata` service. + +The e2e tests will test the `SnapshotMetadata` service ability to: + +* Handle and authenticate gRPC requests from the sample client to: + * Get allocated blocks of a PVC + * Get changed blocks between a pair of snapshots +* Get the following kinds of API resources, based on the gRPC requests and CSI +provisioner: + * `SnapshotMetadataService` + * `TokenReview` + * `SubjectAccessReview` + * `VolumeSnapshot` + * `VolumeSnapshotContent` +* Marshal and stream responses from the mock SP service to the sample client + +### Graduation Criteria + + + +#### Alpha + +* Approval of the proposed CRDs and GRPC specification. +* The `SnapshotMetadata` service can handle gRPC requests to get allocated and +changed blocks, and stream responses back to client, without imposing load on +the K8s API server. +* Initial e2e tests completed and enabled. + +#### Beta + +* Involve 2 different storage providers and backup applications in the +implementations of the `SnapshotMetadata` service to enable successful e2e +backup workflow. +* Increase e2e test coverage. +* Gather feedback from CSI driver maintainers and backup users, especially +on performance metrics. + +#### GA + +* CSI drivers include the `SnapshotMetadata` service as part of their distros. +* Allowing time for user feedback. + + + +### Upgrade / Downgrade Strategy + + + +### Version Skew Strategy + + + +If the `external-snapshot-metadata` sidecar is added to an older CSI driver that +doesn't implement the CBT feature, the sidecar returns a gRPC `UNIMPLEMENTED` +[status code][3] (HTTP 501) to the backup application. + +The snapshot metadata components support the v1 snapshot objects (VolumeSnapshot, +VolumeSnapshotContent, VolumeSnapshotClass etc.), starting with snapshot +controller v7.0. There are no plans to support earlier beta and alpha versions of +the snapshot objects. + +This enhancement focuses only on the control plane with no effects on the kubelet. + +## Production Readiness Review Questionnaire + + + +### Feature Enablement and Rollback + + + +###### How can this feature be enabled / disabled in a live cluster? + + + +- [ ] Feature gate (also fill in values in `kep.yaml`) + - Feature gate name: + - Components depending on the feature gate: +- [x] Other + - Describe the mechanism: +The new components will be implemented as part of the out-of-tree CSI framework. +Storage providers can embed the sidecar component in their CSI drivers, if they +choose to support this feature. + - Will enabling / disabling the feature require downtime of the control + plane? +No. + - Will enabling / disabling the feature require downtime or reprovisioning + of a node? +No. + +###### Does enabling the feature change any default behavior? + + + +No. + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? + + + +The feature can be disabled by removing the sidecar from the CSI driver. + +###### What happens if we reenable the feature if it was previously rolled back? + +No effects. + +###### Are there any tests for feature enablement/disablement? + + + +No. + +### Rollout, Upgrade and Rollback Planning + + + +###### How can a rollout or rollback fail? Can it impact already running workloads? + + + +###### What specific metrics should inform a rollback? + + + +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? + + + +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? + + + +### Monitoring Requirements + + + +###### How can an operator determine if the feature is in use by workloads? + + + +###### How can someone using this feature know that it is working for their instance? + + + +- [ ] Events + - Event Reason: +- [ ] API .status + - Condition name: + - Other field: +- [ ] Other (treat as last resort) + - Details: + +###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? + + + +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + + + +- [ ] Metrics + - Metric name: + - [Optional] Aggregation method: + - Components exposing the metric: +- [ ] Other (treat as last resort) + - Details: + +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + + + +### Dependencies + + + +###### Does this feature depend on any specific services running in the cluster? + + + +### Scalability + + + +###### Will enabling / using this feature result in any new API calls? + + + +- API call type (e.g. PATCH pods): + - GET VolumeSnapshot, VolumeSnapshotContent, VolumeSnapshotClass, SnapshotMetadataService + - CREATE TokenRequest, TokenReview, SubjectAccessReview, SnapshotMetadataService +- estimated throughput: one object of each kind, per request +- originating component(s) (e.g. Kubelet, Feature-X-controller): + - user's backup application + - snapshot metadata sidecar +- components listing and/or watching resources they didn't before: N/A +- API calls that may be triggered by changes of some Kubernetes resources + (e.g. update of object X triggers new updates of object Y): N/A. All API calls +are initiated by the user's backup application +- periodic API calls to reconcile state (e.g. periodic fetching state, + heartbeats, leader election, etc.): N/A + +###### Will enabling / using this feature result in introducing new API types? + + + +- API type: SnapshotMetadataService +- Supported number of objects per cluster: One object for every CSI driver that +supports CBT +- Supported number of objects per namespace (for namespace-scoped objects): N/A + +###### Will enabling / using this feature result in any new calls to the cloud provider? + + + +The CSI driver plugin will call the cloud provider's CBT API to retrieve the CBT +snapshot metadata. + +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + + + +Existing API objects will not be affected. + +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + + + +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + + + +###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)? + + + +### Troubleshooting + + + +###### How does this feature react if the API server and/or etcd is unavailable? + +###### What are other known failure modes? + + + +###### What steps should be taken if SLOs are not being met to determine the problem? + +## Implementation History + + + +## Drawbacks + + + +## Alternatives + + + +The aggregated API server solution described in [#3367][0] was deemed unsuitable +because of the potentially large amount of CBT payloads that will be proxied +through the K8s API server. Further discussion can be found in this [thread][1]. + +An approach based on using volume populator to store the CBT payloads on +intermediary storage, instead of sending them over the network was also +considered. But the amount of pod creation/deletion churns and latency incurred +made this solution inappropriate. + +The previous design which involved generating and returning a RESTful callback +endpoint to the caller, to serve CBT payloads was superceded by the aggregation +extension mechanism as described in [#3367][0], due to the requirement for more +structured request and response payloads. + +## Infrastructure Needed (Optional) + + + +[0]: https://github.com/kubernetes/enhancements/pull/3367 +[1]: https://github.com/kubernetes/enhancements/pull/3367#pullrequestreview-1133441329 +[2]: https://github.com/PrasadG193/external-snapshot-metadata/blob/c8da6e4ed6a206bcc68c55a1592c8f321f21cae4/cmd/client/main.go +[3]: https://github.com/googleapis/googleapis/blob/254e61ae3eb7ce8660c7cd3f15c8d30f1f46bfd3/google/rpc/code.proto#L162 diff --git a/keps/sig-storage/3314-csi-changed-block-tracking/flow.drawio.svg b/keps/sig-storage/3314-csi-changed-block-tracking/flow.drawio.svg new file mode 100644 index 00000000000..9894d8d10c6 --- /dev/null +++ b/keps/sig-storage/3314-csi-changed-block-tracking/flow.drawio.svg @@ -0,0 +1,1654 @@ + + + + + + + + + + + +
+
+
+ + + csi driver namespace + + +
+
+
+ + + csi driver namespace + + + + + + + + +
+
+
+ + Deployment: Create SA, +
+ Role bindings, +
+ + SnapshotMetadataService + + CR and + + Service + + for the sidecar + +
+
+
+ + + Deployment: Create SA,... + + + + + + + +
+
+
+ + + apiVersion: v1 + +
+ + kind: + + Service + + +
+ + metadata: + +
+ + name: + + + + + sidecar-service + + + +
+ + namespace: + + + + csi-driver-ns +
+ + + + + +
+ + +
+
+
+ + + apiVersion: v1... + + + + + + + +
+
+
+
+ + + ... + + +
+
+ + + status: + + +
+
+
+
+ + + ...... + + + + + + +
+
+
+ + authenticated: true + +
+ + user: + + + + user-info + + + + +
+ + +
+
+
+ + + authenticated: true... + + + + + + + + + + +
+
+
+ + Kubernetes API Server + +
+
+
+ + + Kubernetes API Server + + + + + + + + +
+
+
+ + GetMetadataDelta() + +
+
+
+ + + GetMetadataDel... + + + + + + + + +
+
+
+ + Send + +
+
+
+ + + Send + + + + + + +
+
+
+ + + GetMetadataDeltaResponse message + +
+ +
+
+
+ + + GetMetadataDeltaResponse message + + + + + + + + +
+
+
+ Send +
+
+
+ + + Send + + + + + + +
+
+
+ + + GetMetadataDeltaResponse message + +
+ +
+
+
+ + + GetMetadataDeltaResponse message + + + + + + +
+
+
+ + + + Snapshot Metadata Service Pod +
+ + + +
+
+
+ + + Snapshot Metadata Service Pod + + + + + + + + +
+
+
+ + Send + +
+
+
+ + + Send + + + + + + + + +
+
+
+ + Send + +
+
+
+ + + Send + + + + + + + + +
+
+
+ + GetMetadataDelta() + +
+
+
+ + + GetMetadataDel... + + + + + + + + +
+
+
+ + + SP Snapshot Metadata Service + +
+ + A CSI SnapshotMetadata +
+ gRPC Server + +
+ +
+
+
+ + + SP Snapshot Metadata Servic... + + + + + + + +
+
+
+ + Assemble next page of data + +
+
+
+ + + Assemble next p... + + + + + + + +
+
+
+ + Initiate operation + +
+
+
+ + + Initiate operat... + + + + + + + +
+
+
+ + Assemble next page of data + +
+
+
+ + + Assemble next p... + + + + + + + + +
+
+
+ + external-snapshot-metadata + + + + + sidecar + + + +
+ + + A Kubernetes SnapshotMetadata +
+ gRPC Server + + +
+
+
+ + + external-snapshot-metadata sidecar... + + + + + + + +
+
+
+ + + Proxy the RPC call after translating IDs + + +
+
+
+ + + Proxy the RPC call aft... + + + + + + + +
+
+
+ + + global (no namespace) + + +
+
+
+ + + global (no namespace) + + + + + + + +
+
+
+ + + apiVersion: cbt.storage.k8s.io/v1alpha1 + +
+ + kind: + + SnapshotMetadataService + + +
+ + metadata: + +
+ + + +
+
+
+ spec: + +
+ +
+ +
+ +
+
+
+ + + apiVersion: cbt.storage.k8s.io/v1alp... + + + + + + +
+
+
+ + + audience: + + + audience-string + + +
+ caCert: + + + + ca-cert + + +
+ + address: + + + + sidecar-service.csi-driver-ns + + + +
+
+
+ + + audience: audience-string... + + + + + + +
+
+
+ + + name: + + + + XXX + + +
+ + labels: + +
+ + - cbt.storage.k8s.io/driver: + + + + driver-name + +
+ + +
+
+
+ + + name: XXX... + + + + + + + +
+
+
+ + + Authorize the caller's access to + + VolumeSnapshots + + in the specified namespace using the + + SubjectAccessReview + + API + + +
+
+
+ + + Authorize the caller's access... + + + + + + + +
+
+
+ + + Load + + VolumeSnapshot + + and + + VolumeSnapshotContent + + objects to validate and fetch SP snapshot IDs +
+ Load the + + VolumeSnapshotClass + + to fetch + + Secrets + + +
+ +
+
+
+ + + Load VolumeSnapshot and Volume... + + + + + + + +
+
+
+ + + Validate the K8s audience scoped token using the + + TokenReview + + API and obtain the + + UserInfo + + for the caller + + +
+
+
+ + + Validate the K8s audience sco... + + + + + + + + +
+
+
+ + Backup Client + +
+
+
+ + + Backup Client + + + + + + + +
+
+
+ + + + Pass + + + + + the K8s audience scoped authentication token as the value of the + + security_token + + field in every gRPC request message +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ + + + +
+ Process the returned changed block metadata stream + + +
+ +
+
+
+ + + Pass the K8s audience scoped authe... + + + + + + + +
+
+
+ + Trust the + + ca-cert + +
+ +
+
+
+ + + Trust the ca-cert + + + + + + + +
+
+
+ + + Determine the CSI driver name from, say, the + + VolumeSnapshotClass + + . + +
+ +
+
+
+ + + Determine the CSI driver name from,... + + + + + + + +
+
+
+ + + Obtain an audience scoped token for the snapshot session service, either directly using the + + TokenRequest + + API, or indirectly via a projected service account token volume + +
+ +
+
+
+ + + Obtain an audience scoped token for... + + + + + + + +
+
+
+ + + Search for the +
+ + SnapshotMetadataService + +
+ CR for the CSI driver + +
+ +
+
+
+ + + Search for the... + + + + + + + + +
+
+
+ GET +
+
+
+ + + GET + + + + + + + + +
+
+
+ CREATE +
+
+
+ + + CREATE + + + + + + +
+
+
+ + apiVersion: authentication.k8s.io/v1 +
+ kind: + + TokenRequest + +
+ metadata: ... +
+ spec: +
+
+
+
+
+ +
+
+
+ + + apiVersion: authentication.k8s.i... + + + + + + +
+
+
+ + + audiences: +
+ - + + + audience-string + + +
+ expirationSeconds: + + optional + + boundObjectRef: + + optional + +
+ + +
+
+
+ + + audiences:... + + + + + + + + + + +
+
+
+ + apiVersion: authentication.k8s.io/v1 +
+ kind: + + TokenReview + +
+ metadata: ... +
+ spec: +
+
+
+
+ +
+
+
+ + + apiVersion: authentication.k8s... + + + + + + +
+
+
+ + + audiences: +
+ - + + + audience-string + + +
+ + + + token: + + + + security_token + + + + +
+ + +
+
+
+ + + audiences:... + + + + + + + + +
+
+
+ CREATE +
+
+
+ + + CREATE + + + + + + + + + +
+
+
+ + + New community provided component + + +
+
+
+ + + New community provided c... + + + + + + + +
+
+
+ + + New SP component + + +
+
+
+ + + New SP component + + + + + + +
+
+
+ + + Key: + + +
+
+
+ + + Key: + + + + + + + +
+
+
+ + + K8s API + + +
+
+
+ + + K8s API + + + + + + + +
+
+
+ + + gRPC TLS + + +
+
+
+ + + gRPC TLS + + + + + + + +
+
+
+ + + gRPC Unix socket + + +
+
+
+ + + gRPC Unix socket + + + + + + + + + + + + + + +
+
+
+ + apiVersion: authorization.k8s.io/v1 +
+ kind: + + SubjectAccessReview + +
+ metadata: ... +
+ spec: +
+
+
+
+
+
+ +
+
+
+ + + apiVersion: authorization.k8s.io... + + + + + + +
+
+
+ + + resourcesAttributes: ... +
+ user: + + + user-info.username + + +
+ uid: + + + + + user-info.uid + + + + +
+ groups: + + + + + user-info.groups +
+ + + + + extra: + + + + + user-info.extra + + + + +
+ + +
+
+
+ + + resourcesAttributes: ...... + + + + + + + + +
+
+
+ CREATE +
+
+
+ + + CREATE + + + + + + + + + + +
+
+
+ GET +
+
+
+ + + GET + + + + + + + + + + +
+
+
+ LIST(label search by driver name) +
+
+
+ + + LIST(label search by driver name) + + + + + + + + + + +
+
+
+
+ + + ... + + +
+
+ + + status: + + +
+
+
+
+ + + ...... + + + + + + +
+
+
+ + allowed: true + + + +
+ + +
+
+
+ + + allowed: true + + + + + + + +
+
+
+
+ + + ... + + +
+
+ + + status: + + +
+
+
+
+ + + ...... + + + + + + +
+
+
+ + expirationTimestamp: ... + +
+ + token: + + + + audience-scoped-token + + + + +
+ + +
+
+
+ + + expirationTimestamp: ...... + + + + + + + +
+
+
+ + + GetMetadataDeltaRequest message: +
+ - security_token +
+ - namespace +
+ - base_snapshot_name +
+ - target_snapshot_name +
+ + - + + starting_offset +
+ - max_results + +
+ +
+
+
+ + + GetMetadataDeltaRequest message... + + + + + + +
+
+
+ + + Params: +
+ - base_snapshot_id +
+ - target_snapshot_id +
+ - starting_offset +
+ - max_results +
+ + + - secrets + +
+ +
+
+
+ + + Params:... + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/keps/sig-storage/3314-csi-changed-block-tracking/kep.yaml b/keps/sig-storage/3314-csi-changed-block-tracking/kep.yaml new file mode 100644 index 00000000000..9b77e4ad4cc --- /dev/null +++ b/keps/sig-storage/3314-csi-changed-block-tracking/kep.yaml @@ -0,0 +1,53 @@ +title: KEP Template +kep-number: 3314 +authors: + - "@carlbraganza" + - "@ihcsim" + - "@PrasadG193" +owning-sig: sig-storage +participating-sigs: + - sig-storage + - sig-auth +status: implementable +creation-date: 2023-06-12 +reviewers: + - "@xing-yang" + - "@bswartz" +approvers: + - "@xing-yang" + - "@msau42" + - "@thockin" + +##### WARNING !!! ###### +# prr-approvers has been moved to its own location +# You should create your own in keps/prod-readiness +# Please make a copy of keps/prod-readiness/template/nnnn.yaml +# to keps/prod-readiness/sig-xxxxx/00000.yaml (replace with kep number) +#prr-approvers: + +see-also: +replaces: + +# The target maturity stage in the current dev cycle for this KEP. +stage: alpha + +# The most recent milestone for which work toward delivery of this KEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "v1.31" + +# The milestone at which this feature was, or is targeted to be, at each stage. +milestone: + alpha: "v1.31" + beta: "" + stable: "" + +# The following PRR answers are required at alpha release +# List the feature gate name and the components for which it must be enabled +feature-gates: + - name: + components: +disable-supported: true + +# The following PRR answers are required at beta release +metrics: diff --git a/keps/sig-storage/3314-csi-changed-block-tracking/perms.drawio.svg b/keps/sig-storage/3314-csi-changed-block-tracking/perms.drawio.svg new file mode 100644 index 00000000000..f260fb53366 --- /dev/null +++ b/keps/sig-storage/3314-csi-changed-block-tracking/perms.drawio.svg @@ -0,0 +1,621 @@ + + + + + + + + + +
+
+
+ + snapshot app namespace + +
+
+
+ + + snapshot app namespace + + + + + + + + + +
+
+
+ GET, LIST +
+
+
+ + + GET, LIST + + + + + + + + + +
+
+
+ Backup Application +
+ + ServiceAccount + +
+
+
+ + + Backup A... + + + + + + + + +
+
+
+ GET +
+
+
+ + + GET + + + + + + + + +
+
+
+ GET +
+
+
+ + + GET + + + + + + + + +
+
+
+ GET +
+
+
+ + + GET + + + + + + + + +
+
+
+ GET +
+
+
+ + + GET + + + + + + + + + +
+
+
+ + + SnapshotMetadataService + + +
+
+
+ + + SnapshotM... + + + + + + + + + +
+
+
+ + + VolumeSnapshotContent + + +
+
+
+ + + VolumeSna... + + + + + + + + +
+
+
+

+ + + Key + + +

+
+
+
+ + + Key + + + + + + + + + +
+
+
+ + Object + +
+
+
+ + + Obje... + + + + + + + + +
+
+
+ + Permissions +
+ Needed +
+ +
+
+
+ + + Permissions... + + + + + + + + + +
+
+
+ + Security Principal + +
+
+
+ + + Secu... + + + + + + + + + +
+
+
+ CSI Driver +
+ + ServiceAccount + +
+
+
+ + + CSI Dri... + + + + + + + + + +
+
+
+ + + TokenReview + + +
+
+
+ + + TokenRevi... + + + + + + + + + +
+
+
+ + + SubjectAccessReview + + +
+
+
+ + + SubjectAc... + + + + + + + + +
+
+
+ CREATE +
+
+
+ + + CREATE + + + + + + + + +
+
+
+ CREATE +
+
+
+ + + CREATE + + + + + + + + + +
+
+
+ + + TokenRequest + + +
+
+
+ + + TokenRequ... + + + + + + + + +
+
+
+ CREATE +
+
+
+ + + CREATE + + + + + + + + + +
+
+
+ + + VolumeSnapshot + + +
+
+
+ + + VolumeSna... + + + + + + +
+
+
+ + Required to access namespaced snapshots + +
+
+
+ + + Required to acces... + + + + + + +
+
+
+ + Required to get snapshot metadata + +
+
+
+ + + Required to get s... + + + + + + +
+
+
+ + Required to use the snapshot metadata service + +
+
+
+ + + Required to use t... + + + + + + +
+
+
+ + Required to validate an authentication token + +
+
+
+ + + Required to valid... + + + + + + +
+
+
+ + Required to get an authentication token + +
+
+
+ + + Required to get a... + + + + + + +
+
+
+ + Required to authorize a security principal + +
+
+
+ + + Required to author... + + + + + + + + + +
+
+
+ + + VolumeSnapshotClass + + +
+
+
+ + + VolumeSna... + + + + + + +
+
+
+ + Required to get provisioner secrets + +
+
+
+ + + Required to get p... + + + + + + + + +
+
+
+ GET +
+
+
+ + + GET + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file