Added a KEP for the Stateful Application Data Management API #1051
Conversation
k8s-ci-robot
added
the
sig/apps
k8s-ci-robot
added
sig/storage
|
/cc @jingxu97 @xing-yang |
| } | ||
|
|
||
| // HookSpec defines a hook for quiescing or unquiescing an application. | ||
| type HookSpec struct { |
There was a problem hiding this comment.
Why creating another HookSpec struct? Can you use the ExecutionHook API object designed in this KEP (https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190120-execution-hook-design.md)?
There was a problem hiding this comment.
Discussed about this in today's review meeting. Can you please add some description to explain that ExecutionHook will be created based on HookSpec?
There was a problem hiding this comment.
I updated the KEP to use ExecutionHookSpec instead of the HookSpec struct.
There was a problem hiding this comment.
Actually stepping back a bit, I tried to run through the user flow if we use ExecutionHookSpec instead, and it seemed to need more typing from the users. I think it makes sense to keep the HookSpec struct here which is tailed for the needs here and is less verbose. I added descriptions to make it clear that an ExecutionHook object will be created at runtime for each hook based on information provided in the HookSpec.
| ## Summary | ||
|
|
||
| This KEP proposes a Kubernetes Stateful Application Data Management API consisting of a set of [CustomResourceDefinitions (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) that collectively define the notion of stateful applications, i.e., applications that maintain persistent state, and a set of data management semantics on stateful applications such as snapshot, backup, restoration, and clone. A snapshot of a stateful application is defined as a point-in-time capture of the state of the application, taken in an application-consistent manner. It captures both the application configurations (definitions of Kubernetes resources that make up the application, e.g., StatefulSets, Services, ConfigMaps, Secrets, etc.) and persistent data contained within the application (via persistent volumes). A snasphot can also include optional tags in the form of key-value pairs for extra information about itself. | ||
|
|
There was a problem hiding this comment.
typo: s/snasphot/snapshot
| With the [VolumeSnapshot](https://kubernetes.io/docs/concepts/storage/volume-snapshots/) API and [ExecutionHook](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190120-execution-hook-design.md) API, we have some of the necessary building blocks to perform application-level snapshot and restoration operations. However, application-level data management operations require higher-level workflows and automation. For example, taking an application snapshot is a multi-step workflow that does more than orchestrating volume snapshots and hook executions. As of today, there's no API in Kubernetes for application data management truly at the application-level. | ||
|
|
||
| This proposal tries to close the gap by introducing a new Kubernetes API for supporting snapshot, backup, restoration, and clone semantics at the application-level in an application-consistent manner. This new API exposes application-level data management semantics through Kubernetes custom resources and provides a way to orchestrate volume-level snapshot operations and hook executions by automatically managing the lifecycle of `VolumeSnapshot` and `ExevcutionHook` API objects as part of some higher-level workflows. By modeling application snapshots and backups as declarative custom resources, they can be used as data sources for application restoration and cloning. Similarly, connections between snapshots and backups can be expressed to support operations such as exporting snapshots as backups and importing of backups into snapshots. | ||
|
|
There was a problem hiding this comment.
s/ExevcutionHook/ExecutionHook
| // resource to be created if set. The name of this StatefulApplication resource will be used if this is | ||
| // not set. It’s assumed that the Application resource is in the same namespace of it exists. | ||
| // +optional | ||
| ApplicationName *string `json:"applicationName,omitempty"` |
There was a problem hiding this comment.
Will there be any validation on whether the application specified is really a stateful application or not? Will the validation done by the controller or can it be handled in API validation?
There was a problem hiding this comment.
No. There's no validation on this. I'm thinking that the same mechanism could also be used for taking snapshots of resource definitions of applications that do not use PVCs.
| ValueSubstitutionRuleTypeEnvVar ValueSubstitutionRuleType = "EnvVar" | ||
| ) | ||
|
|
||
| // ValueSubstitutionRule defines a rule for value sibstitution. |
There was a problem hiding this comment.
s/sibstitution/substitution
| const ( | ||
| DataSourceSnapshot DataSourceType = "Snapshot" | ||
| DataSourceBackup DataSourceType = "Backup" | ||
| ) |
There was a problem hiding this comment.
There is work going on to support volume cloning. A data source of "PVC" will be possible for creating a PVC when that work is complete. So here you may have a third type ("Application"?) in the future.
There was a problem hiding this comment.
That's good to know. Yes, when that feature is available, it makes sense to add a third option Application to support cloning an existing application.
There was a problem hiding this comment.
Both Snapshot and PVC DataSource are in beta now.
https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/core/types.go#L426
| // StatefulApplication of this class is to be provisioned. | ||
| Handler string `json:"handler"` | ||
|
|
||
| // Parameters are opaque parameters to the snapsohtter. |
There was a problem hiding this comment.
s/snapsohtter/snapshotter
| } | ||
|
|
||
| type ApplicationSnapshotStatus struct { | ||
| // Phease is the current phase of the snapshot process. |
| ``` | ||
|
|
||
| `ApplicationSnapshotContent` is a cluster-scoped CRD that describes the details about a local application snapshot. An `ApplicationSnapshotContent` object is created as the result of successfully processing an `ApplicationSnapshot`. The relationship between an `ApplicationSnapshot` object and an `ApplicationSnapshotContent` object is like the relationship between a `VolumeSnapshot` object and a `VolumeSnapshotContent` object. Below is the definition of `ApplicationSnapshotContent`. | ||
|
|
There was a problem hiding this comment.
How is the binding of individual VolumeSnapshot/VolumeSnapshotContent related to the binding of ApplicationSnapshot/ApplicationSnapshotContent? Is static provisioning supported for ApplicationSnapshot?
There was a problem hiding this comment.
We support static binding of an ApplicationSnapshot to an existing ApplicationSnapshotContent.
| metav1.ObjectMeta `json:"metadata,omitempty"` | ||
|
|
||
| Spec ApplicationSnapshotContentSpec `json:"spec,omitempty"` | ||
| Status ApplicationSnapshotContentStatus `json:"status,omitempty"` |
There was a problem hiding this comment.
Will this status be different from ApplicationSnapshotStatus?
There was a problem hiding this comment.
Yes, ApplicationSnapshotStatus is what users see and care about. This status is not and is currently empty as there's really nothing to put in this status section. Things may change in the future though.
There was a problem hiding this comment.
I deleted this status given that it's empty and not really used. There's also no foreseeable use of this in the future.
liyinan926
force-pushed
the
master
branch
5 times, most recently
from
87eef91
to
589059d
Compare
|
/cc |
|
|
||
| ## Motivation | ||
|
|
||
| The addition of the [VolumeSnapshot](https://kubernetes.io/docs/concepts/storage/volume-snapshots/) API in Kubernetes 1.12 enables backup and restoration support for persistent volumes. With the [VolumeSnapshot](https://kubernetes.io/docs/concepts/storage/volume-snapshots/) API, users can take a snapshot of a persistent volume or restore a volume fron a snapshot. When it comes to application snapshot and restoration semantics, however, more things need to be considered than just volume snapshots. As stated above, an application snapshot captures both its configurations and persistent data. The key is that an application snapshot contains sufficient information to completely create (from scratch) an instance of a stateful application as captured at a particular point in time. |
There was a problem hiding this comment.
users can take a snapshot of a persistent volume or restore a volume fron a snapshot.
Small typo of 'fron' instead of 'from'
|
|
||
| ### Stateful Application | ||
|
|
||
| The API is centered around the notion of stateful applications modeled using a CRD named `StatefulApplication`. The `StatefulApplication` CRD leverages the [Application CRD](https://github.com/kubernetes-sigs/application) for description and aggregation of components and status of an application. Additionally, it supports data management semantics such as snapshot, backup, restoration, and clone for the application it models. |
There was a problem hiding this comment.
Is there a plan for managing the dependency between this work and the Application CRD, e.g. across versions of that CRD? It might be difficult to deal with changes in the App CRD spec, and users will have to install that CRD separately to the ones described here in their cluster. The same concerns probably go for the ExecutionHook CRD.
There was a problem hiding this comment.
That's a good question. We don't have one currently, but I agree that we need a proposal for dependency management between related CRDs.
|
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-contributor-experience at kubernetes/community. |
k8s-ci-robot
added
the
lifecycle/stale
|
/cc |
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
k8s-ci-robot
added
lifecycle/rotten
|
/remove-lifecycle rotten |
k8s-ci-robot
removed
the
lifecycle/rotten
|
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
k8s-ci-robot
added
the
lifecycle/stale
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
k8s-ci-robot
added
lifecycle/rotten
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close |
|
@k8s-triage-robot: Closed this PR. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/remove-lifecycle rotten |
k8s-ci-robot
removed
the
lifecycle/rotten
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: liyinan926 The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@liyinan926: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
k8s-ci-robot
added
the
lifecycle/stale
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
k8s-ci-robot
added
lifecycle/rotten
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close |
|
@k8s-triage-robot: Closed this PR. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
This is the KEP that introduces the Kubernetes Stateful Application Data Management API for supporting application-level snapshot, restoration, and clone semantics for applications that maintain persistent data state, hence the name
Stateful Application./sig apps
/sig storage
/assign @kow3ns @saad-ali @thockin