Add docs for ApplySet-based pruning in kubectl apply

content/en/docs/reference/labels-annotations-taints/_index.md

@@ -92,6 +92,63 @@ Common forms of values include:
 
 One of the [recommended labels](/docs/concepts/overview/working-with-objects/common-labels/#labels).
 
+### applyset.k8s.io/additional-namespaces
+
+Example: `applyset.k8s.io/additional-namespaces: "namespace1,namespace2"`
+
+Used on: Objects being used as ApplySet parents.
+
+For Kubernetes version {{< skew currentVersion >}}, you can use this label on Secrets, ConfigMaps, or custom resources if the {{< glossary_tooltip term_id="CustomResourceDefinition" text="CustomResourceDefinition" >}} defining them has the `applyset.k8s.io/is-parent-type` label.
+
+Part of the specification used to implement [ApplySet-based pruning in kubectl](/docs/tasks/manage-kubernetes-objects/declarative-config/#alternative-kubectl-apply-f-directory-prune). This annotation is applied to the parent object used to track an ApplySet to extend the scope of the ApplySet beyond the parent object's own namespace (if any). The value is a comma-separated list of the names of namespaces other than the parent's namespace in which objects are found.
+
+### applyset.k8s.io/contains-group-resources
+
+Example: `applyset.k8s.io/contains-group-resources: "certificates.cert-manager.io,configmaps,deployments.apps,secrets,services"`
+
+Used on: Objects being used as ApplySet parents.
+
+For Kubernetes version {{< skew currentVersion >}}, you can use this label on Secrets, ConfigMaps, or custom resources if the {{< glossary_tooltip term_id="CustomResourceDefinition" text="CustomResourceDefinition" >}} defining them has the `applyset.k8s.io/is-parent-type` label.
+
+Part of the specification used to implement [ApplySet-based pruning in kubectl](/docs/tasks/manage-kubernetes-objects/declarative-config/#alternative-kubectl-apply-f-directory-prune). This annotation is applied to the parent object used to track an ApplySet to optimize listing of ApplySet member objects. It is optional in the ApplySet specification, as tools can perform discovery or use a different optimization. However, as of Kubernetes version {{< skew currentVersion >}}, it is required by kubectl. When present, the value of this annotation must be a comma separated list of the group-kinds, in the fully-qualified name format, i.e. `<resource>.<group>`.
+
+
+### applyset.k8s.io/id
+
+Example: `applyset.k8s.io/id: "applyset-0eFHV8ySqp7XoShsGvyWFQD3s96yqwHmzc4e0HR1dsY-v1"`
+
+Used on: Objects being used as ApplySet parents.
+
+For Kubernetes version {{< skew currentVersion >}}, you can use this label on Secrets, ConfigMaps, or custom resources if the {{< glossary_tooltip term_id="CustomResourceDefinition" text="CustomResourceDefinition" >}} defining them has the `applyset.k8s.io/is-parent-type` label.
+
+Part of the specification used to implement [ApplySet-based pruning in kubectl](/docs/tasks/manage-kubernetes-objects/declarative-config/#alternative-kubectl-apply-f-directory-prune). This label is what makes an object an ApplySet parent object. Its value is the unique ID of the ApplySet, which is derived from the identity of the parent object itself. This ID **must** be the base64 encoding (using the URL safe encoding of RFC4648) of the hash of the group-kind-name-namespace of the object it is on, in the form: `<base64(sha256(<name>.<namespace>.<kind>.<group>))>`. There is no relation between the value of this label and object UIDs.
+
+### applyset.k8s.io/is-parent-type
+
+Example: `applyset.k8s.io/is-parent-type: "true"`
+
+Used on: Custom Resource Definition (CRD)
+
+Part of the specification used to implement [ApplySet-based pruning in kubectl](/docs/tasks/manage-kubernetes-objects/declarative-config/#alternative-kubectl-apply-f-directory-prune). You can set this label on a {{< glossary_tooltip term_id="CustomResourceDefinition" text="CustomResourceDefinition" >}} (CRD) to identify the custom resource type it defines (not the CRD itself) as an allowed parent for an ApplySet. The only permitted value for this label is `"true"`; if you want to mark a CRD as not being a valid parent for ApplySets, omit this label.
+
+### applyset.k8s.io/part-of
+
+Example: `applyset.k8s.io/part-of: "applyset-0eFHV8ySqp7XoShsGvyWFQD3s96yqwHmzc4e0HR1dsY-v1"`
+
+Used on: All objects.
+
+Part of the specification used to implement [ApplySet-based pruning in kubectl](/docs/tasks/manage-kubernetes-objects/declarative-config/#alternative-kubectl-apply-f-directory-prune). This label is what makes an object a member of an ApplySet. The value of the label **must** match the value of the `applyset.k8s.io/id` label on the parent object.
+
+### applyset.k8s.io/tooling
+
+Example: `applyset.k8s.io/tooling: "kubectl/v{{< skew currentVersion >}}"`
+
+Used on: Objects being used as ApplySet parents.
+
+For Kubernetes version {{< skew currentVersion >}}, you can use this label on Secrets, ConfigMaps, or custom resources if the {{< glossary_tooltip term_id="CustomResourceDefinition" text="CustomResourceDefinition" >}} defining them has the `applyset.k8s.io/is-parent-type` label.
+
+Part of the specification used to implement [ApplySet-based pruning in kubectl](/docs/tasks/manage-kubernetes-objects/declarative-config/#alternative-kubectl-apply-f-directory-prune). This label is applied to the parent object used to track an ApplySet to indicate which tooling manages that ApplySet. Tooling should refuse to mutate ApplySets belonging to other tools. The value must be in the format `<toolname>/<semver>`.
+
 ### cluster-autoscaler.kubernetes.io/safe-to-evict
 
 Example: `cluster-autoscaler.kubernetes.io/safe-to-evict: "true"`

content/en/docs/tasks/manage-kubernetes-objects/declarative-config.md

@@ -371,44 +371,82 @@ to result in the user deleting something unintentionally:
 kubectl delete -f <filename>
 ```
 
-### Alternative: `kubectl apply -f <directory/> --prune -l your=label`
+### Alternative: `kubectl apply -f <directory/> --prune`
 
-Only use this if you know what you are doing.
+As an alternative to `kubectl delete`, you can use `kubectl apply` to identify objects to be deleted after
+their manifests have been removed from a directory in the local filesystem.
 
-{{< warning >}}
-`kubectl apply --prune` is in alpha, and backwards incompatible
-changes might be introduced in subsequent releases.
-{{< /warning >}}
+In Kubernetes {{< skew currentVersion >}}, there are two pruning modes available in kubectl apply:
+- Allowlist-based pruning: This mode has existed since kubectl v1.5 but is still in alpha due to usability, correctness and performance issues with its design. The ApplySet-based mode is designed to replace it, and upon its graduation, allowlist-based pruning will be deprecated using beta timelines.
+- ApplySet-based pruning: An ApplySet is a server-side object (by default, a Secret) that kubectl can use to accurately and efficiently track set membership across `apply` operations. This mode was introduced in alpha in kubectl v1.27 as a replacement for allowlist-based pruning. Unlike allowlist-based pruning, it is under active development and is expected to ultimately become generally available.
+
+{{< tabs name="kubectl_apply_prune" >}}
+{{% tab name="Allowlist" %}}
+
+{{< feature-state for_k8s_version="v1.5" state="alpha" >}}
 
 {{< warning >}}
-You must be careful when using this command, so that you
-do not delete objects unintentionally.
+Only use this if you know what you are doing. You must be careful when using this command, so that you do not delete objects unintentionally. For more information on the problems with this command, see the [Current Implementation](git.k8s.io/enhancements/keps/sig-cli/3659-kubectl-apply-prune#current-implementation) section of the ApplySet KEP.
 {{< /warning >}}
 
-As an alternative to `kubectl delete`, you can use `kubectl apply` to identify objects to be deleted after their
-configuration files have been removed from the directory. Apply with `--prune`
-queries the API server for all objects matching a set of labels, and attempts
-to match the returned live object configurations against the object
+To use allowlist-based pruning, add the following flags to your `kubectl apply` invocation:
+- `--prune`: Delete previously applied objects that are not in the set passed to the current invocation.
+- `--prune-allowlist`: A list of group-version-kinds (GVKs) to consider for pruning. This flag is optional but strongly encouraged, as its [default value](https://git.k8s.io/kubectl/pkg/util/prune/prune.go#L28-L50) is a partial list of both namespaced and cluster-scoped types, which can lead to surprising results.
+- `--selector/-l`: Use a label selector to constrain the set of objects selected for pruning. This flag is optional but strongly encouraged.
+- `--all`: use instead of `--selector/-l` to explicitly select all previously applied objects of the allowlisted types.
+
+Allowlist-based pruning queries the API server for all objects of the allowlisted GVKs that match the given labels (if any), and attempts to match the returned live object configurations against the object
 configuration files. If an object matches the query, and it does not have a
 configuration file in the directory, and it has a `last-applied-configuration` annotation,
 it is deleted.
 
-{{< comment >}}
-TODO(pwittrock): We need to change the behavior to prevent the user from running apply on subdirectories unintentionally.
-{{< /comment >}}
 
 ```shell
-kubectl apply -f <directory/> --prune -l <labels>
+kubectl apply -f <directory/> --prune -l <labels> --prune-allowlist=<gvk-list>
 ```
 
 {{< warning >}}
 Apply with prune should only be run against the root directory
 containing the object configuration files. Running against sub-directories
-can cause objects to be unintentionally deleted if they are returned
-by the label selector query specified with `-l <labels>` and
-do not appear in the subdirectory.
+can cause objects to be unintentionally deleted if they were previously applied, 
+have the labels given (if any), and do not appear in the subdirectory.
+{{< /warning >}}
+
+{{% /tab %}}
+
+{{% tab name="ApplySet" %}}
+
+{{< feature-state for_k8s_version="v1.27" state="alpha" >}}
+
+{{< warning >}}
+`kubectl apply --prune --applyset` is in alpha, and backwards incompatible
+changes might be introduced in subsequent releases.
 {{< /warning >}}
 
+To use ApplySet-based pruning, set the `KUBECTL_APPLYSET=true` environment variable, and add the following flags to your `kubectl apply` invocation:
+- `--prune`: Delete previously applied objects that are not in the set passed to the current invocation.
+- `--applyset`: The name of an object that kubectl can use to accurately and efficiently track set membership across `apply` operations.
+
+```shell
+KUBECTL_APPLYSET=true kubectl apply -f <directory/> --prune --applyset=<name>
+```
+
+By default, the type of the ApplySet parent object used is a Secret. However, ConfigMaps can also be used in the format: `--applyset=configmaps/<name>`. When using a Secret or ConfigMap, kubectl will create the object if it does not already exist.
+
+It is also possible to use custom resources as ApplySet parent objects. To enable this, label the Custom Resource Definition (CRD) that defines the resource you want to use with the following: `applyset.k8s.io/is-parent-type: true`. Then, create the object you want to use as an ApplySet parent (kubectl does not do this automatically for custom resources). Finally, refer to that object in the applyset flag as follows: `--applyset=<resource>.<group>/<name>` (for example, `widgets.custom.example.com/widget-name`).
+
+With ApplySet-based pruning, kubectl adds the `applyset.k8s.io/part-of=<parentID>` label to each object in the set before they are sent to the server. For performance reasons, it also collects the list of resource types and namespaces that the set contains and adds these in annotations on the live parent object. Finally, at the end of the apply operation, it queries the API server for objects of those types in those namespaces (or in the cluster scope, as applicable) that belong to the set, as defined by the `applyset.k8s.io/part-of=<parentID>` label.
+
+Caveats and restrictions:
+- Each object must be a member of at most one set.
+- The `--namespace` flag is required when using any namespaced parent, including the default Secret.  This means that ApplySets spanning multiple namespaces must use a cluster-scoped custom resource as the parent object.
+- To safely use ApplySet-based pruning with multiple directories, use a unique ApplySet name for each.
+
+{{% /tab %}}
+
+{{< /tabs >}}
+
+
 ## How to view an object
 
 You can use `kubectl get` with `-o yaml` to view the configuration of a live object:
@@ -1007,5 +1045,3 @@ template:
 * [Imperative Management of Kubernetes Objects Using Configuration Files](/docs/tasks/manage-kubernetes-objects/imperative-config/)
 * [Kubectl Command Reference](/docs/reference/generated/kubectl/kubectl-commands/)
 * [Kubernetes API Reference](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/)
-
-