Introduce ResourceSet API

[stefanprodan]

ResourceSet is a declarative API for generating a group of Kubernetes objects
based on a matrix of input values and a set of templated resources.

The ResourceSet API offers a high-level abstraction for defining and managing
Flux resources and related Kubernetes objects as a single unit. It is designed
to reduce the complexity of Kustomize overlays by providing a compact way
of defining different configurations for a set of workloads per tenant and/or environment.

Use cases:

• Application definition: Bundle a set of Kubernetes resources (Flux HelmRelease, OCIRepository, Alert, Provider, Receiver, ImagePolicy) into a single deployable unit.
• Dependency management: Define dependencies between apps to ensure that the resources are applied in the correct order. The dependencies are more flexible than in Flux, they can be for other ResourceSets, CRDs, or any other Kubernetes object.
• Multi-instance provisioning: Generate multiple instances of the same application with different configurations.
• Multi-cluster provisioning: Generate multiple instances of the same application for each target cluster that are deployed by Flux from a management cluster.
• Multi-tenancy provisioning: Generate a set of resources (Namespace, ServiceAccount, RoleBinding) for each tenant with specific roles and permissions.

Example

The following example shows a ResourceSet that generates an application instance consisting of a
Flux HelmRelease and OCIRepository for each tenant with a specific version and replica count.

apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSet
metadata:
  name: podinfo
  namespace: default
  annotations:
    fluxcd.controlplane.io/reconcile: "enabled"
    fluxcd.controlplane.io/reconcileEvery: "30m"
    fluxcd.controlplane.io/reconcileTimeout: "5m"
spec:
  commonMetadata:
    labels:
      app.kubernetes.io/name: podinfo
  inputs:
    - tenant: "team1"
      app:
       version: "6.7.x"
       replicas: 2
    - tenant: "team2"
      app:
       version: "6.6.x"
       replicas: 3
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1beta2
      kind: OCIRepository
      metadata:
        name: podinfo-<< inputs.tenant >>
        namespace: default
      spec:
        interval: 10m
        url: oci://ghcr.io/stefanprodan/charts/podinfo
        ref:
          semver: << inputs.app.version | quote >>
    - apiVersion: helm.toolkit.fluxcd.io/v2
      kind: HelmRelease
      metadata:
        name: podinfo-<< inputs.tenant >>
        namespace: default
      spec:
        interval: 1h
        releaseName: podinfo-<< inputs.tenant >>
        chartRef:
          kind: OCIRepository
          name: podinfo-<< inputs.tenant >>
        values:
          replicaCount: << inputs.app.replicas | int >>

You can run this example by saving the manifest into podinfo.yaml.

1. Apply the ResourceSet on the cluster:

   kubectl apply -f podinfo.yaml

2. Wait for the ResourceSet to reconcile the generated resources:

   kubectl wait resourceset/podinfo --for=condition=ready --timeout=5m

3. Run kubectl get resourceset to see the status of the resource:

    $ kubectl get resourceset
    NAME      AGE   READY   STATUS
    podinfo   59s   True    Reconciliation finished in 52s

4. Run kubectl describe resourceset to see the reconciliation status conditions and events:

   $ kubectl describe resourceset podinfo
   Status:
     Conditions:
       Last Transition Time:  2024-09-24T09:58:53Z
       Message:               Reconciliation finished in 52s
       Observed Generation:   1
       Reason:                ReconciliationSucceeded
       Status:                True
       Type:                  Ready
   Events:
     Type    Reason          Age   From           Message
     ----    ------          ----  ----           -------
     Normal  ApplySucceeded  72s   flux-operator  HelmRelease/default/podinfo-team1 created
                                                  HelmRelease/default/podinfo-team2 created
                                                  OCIRepository/default/podinfo-team1 created
                                                  OCIRepository/default/podinfo-team2 created
     Normal  ReconciliationSucceeded  72s  flux-operator  Reconciliation finished in 52s

5. Run kubectl events to see the events generated by the flux-operator:

   kubectl events --for resourceset/podinfo

6. Run kubectl delete to remove the ResourceSet and its generated resources:

   kubectl delete resourceset podinfo

Writing a ResourceSet spec

As with all other Kubernetes config, a ResourceSet needs apiVersion,
kind, metadata.name and metadata.namespace fields.
The name of a ResourceSet object must be a valid DNS subdomain name.
A ResourceSet also needs a .spec section.

Inputs configuration

The .spec.inputs field is optional and specifies a list of input values
to be used in the resources templates.

Example inputs:

spec:
  inputs:
   - tenant: team1
     role: restricted
   - tenant: team2
     role: privileged

An input value is a key-value pair of strings and structs, where the key is the input name
which can be referenced in the resource templates using the << inputs.name >> syntax.

Resources configuration

The .spec.resources field is optional and specifies the list of Kubernetes resource
to be generated and reconciled on the cluster.

Example of plain resources without any templating:

spec:
  resources:
   - apiVersion: v1
     kind: Namespace
     metadata:
      name: apps
   - apiVersion: v1
     kind: ServiceAccount
     metadata:
      name: flux
      namespace: apps

Templating resources

The resources can be templated using the << inputs.name >> syntax. The templating engine
is based on Go text template. The << >> delimiters are used instead of {{ }} to avoid
conflicts with Helm templating and allow ResourceSets to be included in Helm charts.

Example of templated resources:

spec:
  inputs:
   - tenant: team1
     role: admin
   - tenant: team2
     role: cluster-admin
  resources:
   - apiVersion: v1
     kind: Namespace
     metadata:
      name: << inputs.tenant >>
   - apiVersion: v1
     kind: ServiceAccount
     metadata:
      name: flux
      namespace: << inputs.tenant >>
   - apiVersion: rbac.authorization.k8s.io/v1
     kind: RoleBinding
     metadata:
      name: flux
      namespace: << inputs.tenant >>
     subjects:
      - kind: ServiceAccount
        name: flux
        namespace: << inputs.tenant >>
        roleRef:
         kind: ClusterRole
         name: << inputs.role >>
         apiGroup: rbac.authorization.k8s.io

The above example will generate a Namespace, ServiceAccount and RoleBinding for each tenant
with the specified role.

Templating functions

The templating engine supports slim-sprig functions.

It is recommended to use the quote function when templating strings to avoid issues with
special characters e.g. << inputs.version | quote >>.

When templating integers, use the int function to convert the string to an integer
e.g. << inputs.replicas | int >>.

When templating booleans, use the bool function to convert the string to a boolean
e.g. << inputs.enabled | bool >>.

When using integer or boolean inputs as metadata label values, use the quote function to convert
the value to a string e.g. << inputs.enabled | quote >>.

When templating nested fields, use the toYaml and nindent functions
to properly format the string e.g.:

spec:
  inputs:
    - tenant: team1
      layerSelector:
        mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip"
        operation: copy
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1beta2
      kind: OCIRepository
      metadata:
        name: << inputs.tenant >>
      spec:
        layerSelector: << inputs.layerSelector | toYaml | nindent 4 >>

In addition to the slim-sprig functions, a slugify function is available to normalize a string for use in a Kubernetes
label value
e.g. << inputs.tenant | slugify >>.

Resource deduplication

The flux-operator deduplicates resources based on the
apiVersion, kind, metadata.name and metadata.namespace fields.

This allows defining shared resources that are applied only once, regardless of the number of inputs.

Example of a shared Flux source:

spec:
  inputs:
    - tenant: "team1"
      replicas: "2"
    - tenant: "team2"
      replicas: "3"
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1beta2
      kind: OCIRepository
      metadata:
        name: podinfo
        namespace: default
      spec:
        interval: 10m
        url: oci://ghcr.io/stefanprodan/charts/podinfo
        ref:
          semver: '*'
    - apiVersion: helm.toolkit.fluxcd.io/v2
      kind: HelmRelease
      metadata:
        name: podinfo-<< inputs.tenant >>
        namespace: default
      spec:
        interval: 1h
        releaseName: podinfo-<< inputs.tenant >>
        chartRef:
          kind: OCIRepository
          name: podinfo
        values:
          replicaCount: << inputs.replicas | int >>

In the above example, the OCIRepository resource is created only once
and referred by all HelmRelease resources.

Conditionally resource exclusion

To exclude a resource based on input values, the fluxcd.controlplane.io/reconcile annotation can be set
to disabled on the resource metadata. This will prevent the resource from being reconciled by the operator.

Example of excluding a resource based on an input value:

spec:
  inputs:
    - tenant: "team1"
    - tenant: "team2"
  resources:
    - apiVersion: v1
      kind: Namespace
      metadata:
        name: << inputs.tenant >>
    - apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: flux
        namespace: << inputs.tenant >>
        annotations:
          fluxcd.controlplane.io/reconcile: << if eq inputs.tenant "team1" >>enabled<< else >>disabled<< end >>

In the above example, the ServiceAccount resource is generated only for the team1 tenant.

Resources template

The .spec.resourcesTemplate field is optional and offers an alternative to the .spec.resources.
The .spec.resourcesTemplate is a single string that contains the multi-document YAML of the resources
definitions. This field can be used for complex templating scenarios with the trade-off of reduced readability.

Note that when both .spec.resources and .spec.resourcesTemplate are set, the resulting resources
are the union of the two. If duplicate resources are defined in both fields, the resources from
.spec.resources take precedence.

Example of a template containing conditional and repeated blocks:

spec:
  inputs:
    - bundle: addons
      decryption: false
      components:
        - ingress-nginx
        - cert-manager
    - bundle: apps
      decryption: true
      components:
        - frontend
        - backend
  resourcesTemplate: |
    ---
    apiVersion: source.toolkit.fluxcd.io/v1beta2
    kind: OCIRepository
    metadata:
      name: << inputs.bundle >>
      namespace: flux-system
    spec:
      interval: 10m
      url: oci://registry.example.com/<< inputs.bundle >>
    <<- range $component := inputs.components >>
    ---
    apiVersion: kustomize.toolkit.fluxcd.io/v1
    kind: Kustomization
    metadata:
      name: << $component >>
      namespace: flux-system
    spec:
      interval: 1h
      prune: true
      <<- if inputs.decryption >>
      decryption:
        provider: sops
        secretRef:
          name: << inputs.bundle >>-sops
      <<- end >>
      sourceRef:
        kind: OCIRepository
        name: << inputs.bundle >>
      path: ./<< $component >>
    <<- end >>

The above example generates two OCIRepository resources (one for each bundle) and four
Kustomization resources (one for each component in each bundle).

Common metadata

The .spec.commonMetadata field is optional and specifies common metadata to be applied to all resources.

It has two optional fields:

• labels: A map used for setting labels
  on an object. Any existing label will be overridden if it matches with a key in
  this map.
• annotations: A map used for setting annotations
  on an object. Any existing annotation will be overridden if it matches with a key
  in this map.

Example common metadata:

spec:
  commonMetadata:
    labels:
      app.kubernetes.io/name: podinfo
    annotations:
      fluxcd.controlplane.io/prune: disabled

In the above example, all resources generated by the ResourceSet
will not be pruned by the garbage collection process as
the fluxcd.controlplane.io/prune annotation is set to disabled.

Dependency management

.spec.dependsOn is an optional list used to refer to Kubernetes
objects that the ResourceSet depends on. If specified, then the ResourceSet
is reconciled after the referred objects exist in the cluster.

A dependency is a reference to a Kubernetes object with the following fields:

• apiVersion: The API version of the referred object (required).
• kind: The kind of the referred object (required).
• name: The name of the referred object (required).
• namespace: The namespace of the referred object (optional).
• ready: A boolean indicating if the referred object must have the Ready status condition set to True (optional, default is false).
• readyExpr: A CEL expression that evaluates to a boolean indicating if the referred object is ready (optional).

Example of conditional reconciliation based on the existence of CustomResourceDefinitions
and the readiness of a ResourceSet:

spec:
  dependsOn:
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: helmreleases.helm.toolkit.fluxcd.io
      ready: true
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: servicemonitors.monitoring.coreos.com
    - apiVersion: fluxcd.controlplane.io/v1
      kind: ResourceSet
      name: cluster-addons
      namespace: flux-system
      ready: true

Note that is recommended to define dependencies on CustomResourceDefinitions if the ResourceSet
deploys Flux HelmReleases which contain custom resources.

When the dependencies are not met, the flux-operator will reevaluate the requirements
every five seconds and reconcile the ResourceSet when the dependencies are satisfied.
Failed dependencies are reported in the ResourceSet Ready status condition,
in log messages and Kubernetes events.

CEL readiness expressions

The readyExpr field allows for more complex readiness checks and
can be used for gating the reconciliation of a ResourceSet based on the evaluation
of the CEL expression.

The expression is evaluated in the context of the referred object and has access to all the fields of the object,
including the status conditions and the status subfields. The expression must evaluate to a boolean value, any syntax
or runtime errors will be reported in the ResourceSet status conditions.

Example readiness expression:

spec:
  dependsOn:
    - apiVersion: cluster.x-k8s.io/v1beta1
      kind: Cluster
      name: my-cluster
      namespace: dev
      ready: true
      readyExpr: |
        metadata.generation == status.observedGeneration &&
        status.controlPlaneReady == true
    - apiVersion: v1
      kind: Secret
      name: my-gate
      namespace: dev
      ready: true
      readyExpr: |
        string(base64.decode(data.gate)) == 'opened'

For testing the CEL expressions, you can use the CEL playground.

Reconciliation configuration

The reconciliation of behaviour of a ResourceSet can be configured using the following annotations:

• fluxcd.controlplane.io/reconcile: Enable or disable the reconciliation loop. Default is enabled, set to disabled to pause the reconciliation.
• fluxcd.controlplane.io/reconcileEvery: Set the reconciliation interval used for drift detection and correction. Default is 1h.
• fluxcd.controlplane.io/reconcileTimeout: Set the reconciliation timeout including health checks. Default is 5m.

Health check configuration

The .spec.wait field is optional and instructs the flux-operator to perform
a health check on all applied resources and waits for them to become ready. The health
check is disabled by default and can be enabled by setting the .spec.wait field to true.

The health check is performed for the following resources types:

• Kubernetes built-in kinds: Deployment, DaemonSet, StatefulSet,
  PersistentVolumeClaim, Service, Ingress, CustomResourceDefinition.
• Flux kinds: HelmRelease, OCIRepository, Kustomization, GitRepository, etc.
• Custom resources that are compatible with kstatus.

By default, the wait timeout is 5m and can be changed with the
fluxcd.controlplane.io/reconcileTimeout annotation, set on the ResourceSet object.

Role-based access control

The .spec.serviceAccountName field is optional and specifies the name of the
Kubernetes ServiceAccount used by the flux-operator to reconcile the ResourceSet.
The ServiceAccount must exist in the same namespace as the ResourceSet
and must have the necessary permissions to create, update and delete
the resources defined in the ResourceSet.

On multi-tenant clusters, it is recommended to use a dedicated ServiceAccount per tenant namespace
with the minimum required permissions. To enforce a ServiceAccount for all ResourceSets,
the --default-service-account=flux-operatorflag can be set in the flux-operator container arguments.
With this flag set, only the ResourceSets created in the same namespace as the flux-operator
will run with cluster-admin permissions.

Garbage collection

The operator performs garbage collection of the resources previously generated by a ResourceSet
that are no longer present in the current revision. Garbage collection is also performed
when a ResourceSet object is deleted, triggering a removal of all Kubernetes
objects previously applied on the cluster.

The garbage collection process removes stale resources in stages, first it deletes the Flux
custom resources and waits for the Flux Kustomizations and HelmReleases to be finalized by
the controllers. After the Flux resources are removed (max wait is one minute), the operator
proceeds with the deletion of the remaining Kubernetes objects. This ensures that the
Flux controllers have a chance to clean up the resources they manage before the operator
deletes the Kubernetes ServiceAccount and RoleBinding used by Flux impersonation.

After the garbage collection process is completed, the operator issues a Kubernetes event
containing the list of removed resources and the duration of the cleanup.

The garbage collection is enabled by default and can be disabled for certain resources
by setting the fluxcd.controlplane.io/prune annotation to disabled. To fully disable
the garbage collection for a ResourceSet, the annotation must be set on all resources
using the .spec.commonMetadata field.

ResourceSet Status

Conditions

A ResourceSet enters various states during its lifecycle, reflected as Kubernetes Conditions.
It can be reconciling while applying the
resources on the cluster, it can be ready, or it can fail during
reconciliation.

The ResourceSet API is compatible with the kstatus specification,
and reports Reconciling and Stalled conditions where applicable to
provide better (timeout) support to solutions polling the ResourceSet to
become Ready.

Reconciling ResourceSet

The flux-operator marks a ResourceSet as reconciling when it starts
the reconciliation of the same. The Condition added to the ResourceSet's
.status.conditions has the following attributes:

• type: Reconciling
• status: "True"
• reason: Progressing | reason: ProgressingWithRetry

The Condition message is updated during the course of the reconciliation to
report the action being performed at any particular moment such as
building manifests, detecting drift, etc.

The Ready Condition's status is also marked as Unknown.

Ready ResourceSet

The flux-operator marks a ResourceSet as ready when the resources were
built and applied on the cluster and all health checks are observed to be passing.

When the ResourceSet is "ready", the flux-operator sets a Condition with the
following attributes in the ResourceSet’s .status.conditions:

• type: Ready
• status: "True"
• reason: ReconciliationSucceeded

Failed ResourceSet

The flux-operator may get stuck trying to reconcile and apply a
ResourceSet without completing. This can occur due to some of the following factors:

• The dependencies are not ready.
• The templating of the resources fails.
• The resources are invalid and cannot be applied.
• Garbage collection fails.
• Running health checks fails.

When this happens, the flux-operator sets the Ready Condition status to False
and adds a Condition with the following attributes to the ResourceSet’s
.status.conditions:

• type: Ready
• status: "False"
• reason: DependencyNotReady | BuildFailed | ReconciliationFailed | HealthCheckFailed

The message field of the Condition will contain more information about why
the reconciliation failed.

While the ResourceSet has one or more of these Conditions, the flux-operator
will continue to attempt a reconciliation with an
exponential backoff, until it succeeds and the ResourceSet is marked as ready.

Inventory status

In order to perform operations such as drift detection, garbage collection, upgrades, etc.,
the flux-operator needs to keep track of all Kubernetes objects that are
reconciled as part of a ResourceSet. To do this, it maintains an inventory
containing the list of Kubernetes resource object references that have been
successfully applied and records it in .status.inventory. The inventory
records are in the format Id: <namespace>_<name>_<group>_<kind>, V: <version>.

Example:

Status:
  Inventory:
    Entries:
      Id: default_podinfo__ServiceAccount
      V:  v1
      Id: default_podinfo__Service
      V:  v1
      Id: default_podinfo_apps_Deployment
      V:  v1

[matheuscscp]

First batch of comments 🤗

[matheuscscp]

LGTM, after resolving the remaining comments I think we can merge 👍

[matheuscscp]

🚀 🚀 🚀