reconcile for externally managed secret is not trigerred for update event

[AshfaqMH]

I am trying to watch an externally managed secret which has a spec field in the CR. Every CRUD on the CR triggers a reconcile to validate the fields in the CR and also validates the content of the secret.

The secret is not owned by the CR. It is externally created and managed. For every CRUD on the secret , would like check in which CR the secret name is reference and trigger reconcile on that CR for the validations to through.

Below is the details :

operator-sdk version: "v1.25.2", commit: "b63b921837de8dd6ce480033e427ecfc5e34abcc", kubernetes version: "1.25.0", go version: "go1.19.2", GOOS: "linux", GOARCH: "amd64"

This is almost the same use case which is shown in the below document,

https://book.kubebuilder.io/reference/watching-resources/externally-managed.html

I have implemented exactly same as shown in the above document. The document is based on the confimap but my use case is for secret watch.

On a CR CRUD event the reconcile triggers and work perfectly fine.

But ,

• If the secret name is referenced in the CR and not created/found in the k8s cluster , the reconcile throws error and update the CR to "ERROR" status. Once the secret is created the loop reconcile picks the secret and validates and update the CR to "READY" status.

• If CR is in READY state and I delete the secret , reconcile is triggered and throws error secret not found and update the CR to "ERROR" status

• If the secret is updated/edited , no reconcile is triggered.

Am I missing something ?

Below is the implementation

const (
	secretNameField         = ".spec.secretRef"
)

// SetupWithManager sets up the controller with the Manager.
func (r *ResourceReconciler) SetupWithManager(mgr ctrl.Manager) error {
	if err := mgr.GetFieldIndexer().IndexField(context.Background(), &v1.Resource{}, secretNameField, func(rawObj client.Object) []string {
		// Extract the SecretRef name from the Resource Spec, if one is provided
		nResource := rawObj.(*v1.Resource)
		if nResource.Spec.SecretRef == "" {
			return nil
		}
		return []string{nResource.Spec.SecretRef}
	}); err != nil {
		return err
	}
	manager := ctrl.NewControllerManagedBy(mgr).
		For(&v1.Resource{}).
		WithEventFilter(ignoreStatusUpdatePredicate()).
		Watches(
			&source.Kind{Type: &corev1.Secret{}},
			handler.EnqueueRequestsFromMapFunc(r.findObjectsForSecret),
			builder.WithPredicates(predicate.ResourceVersionChangedPredicate{}))
	return manager.Complete(r)
}
func (r *ResourceReconciler) findObjectsForSecret(secretName client.Object) []reconcile.Request {
	nCRs := &v1.ResourceList{}
	listOps := &client.ListOptions{
		Namespace:     secretName.GetNamespace(),
		FieldSelector: fields.OneTermEqualSelector(secretNameField, secretName.GetName()),
	}
	err := r.List(context.TODO(), nCRs, listOps)
	if err != nil {
		return []reconcile.Request{}
	}
requests := make([]reconcile.Request, len(nCRs.Items))
	for i, item := range nCRs.Items {
		requests[i] = reconcile.Request{
			NamespacedName: types.NamespacedName{
				Name:      item.GetName(),
				Namespace: item.GetNamespace(),
			},
		}
	}
	return requests
}
func ignoreStatusUpdatePredicate() predicate.Predicate {
	return predicate.Funcs{
		UpdateFunc: func(e event.UpdateEvent) bool {
			// Ignore updates to CR status in which case metadata.Generation does not change.
			return e.ObjectOld.GetGeneration() != e.ObjectNew.GetGeneration()
		},
	}
}

[camilamacedo86]

From the information provided, you're leveraging the capability to watch externally managed resources, especially focusing on Secret objects. This methodology is consistent with Kubebuilder's recommendations, as seen in their documentation. Please check this one.

Let's break down your current situation and address the concerns:

1. CRUD operations on the CR: Every CRUD operation on the CR successfully triggers the reconcile function. This is as expected.

2. Reference to a missing Secret: When the CR references a Secret that doesn't exist, it's logical that your reconcile function throws an error and updates the CR status to "ERROR". This behavior correctly signifies an unresolved reference. Once the missing Secret is created, the reconcile loop appropriately changes the status to "READY".

3. Deletion of a referenced Secret: If a referenced Secret gets deleted while the CR is in "READY" state, the reconcile function identifies this and updates the CR status to "ERROR". This too is expected and signifies the absence of a previously existing resource.

4. Secret Updates: The crux of the problem seems to be here. When the Secret undergoes an update/edit, it's not triggering a reconcile. The mechanism you've implemented using the predicate.ResourceVersionChangedPredicate{} should typically detect these changes and trigger reconciliation.

Key Points to Consider:

• ownerRef: In Kubernetes, the ownerRef mechanism is powerful. When controllers create resources, they often set themselves as the "owner" of these resources. This allows Kubernetes to manage the lifecycle of the resources in tandem. If the owner is deleted, Kubernetes can automatically garbage collect the owned resources. Here's a simple example:

  // Set the ownerRef for the Deployment
  if err := ctrl.SetControllerReference(myresource, dep, r.Scheme); err != nil {
      return nil, err
  }

  In your use case, the Secret is externally managed, so you wouldn't use ownerRef directly in this context. However, understanding this concept might be useful for other scenarios where your controller might create and manage the secret.

• Watching Exclusively the Secret: If the resource (Secret in your case) is managed externally and not created by the controller, you can set up your controller to watch this resource explicitly:

  // SetupWithManager sets up the controller with the Manager.
  func (r *BusyboxReconciler) SetupWithManager(mgr ctrl.Manager) error {
      return ctrl.NewControllerManagedBy(mgr).
          For(&examplecomv1alpha1.MyKind{}).
          Owns(<put here the secret if it is created by the controller>).
          Complete(r)
  }

  Here, the Owns method is used to watch Secret objects managed by MyKind. This can be adapted to your use case.

Recommendation:

For the issue with the Secret updates, ensure that no other parts of your reconcile function or setup might be interfering with the watch mechanism. Sometimes, nuances in the logic or error handling can cause unexpected behavior. Also, look into the operator's logs to check if there are any errors or information related to the watch functionality on the Secret.

You can also try

• Remove the Predicate: As a first step, try removing builder.WithPredicates(predicate.ResourceVersionChangedPredicate{}) from your Watches method call. This will let you observe if any update to the Secret triggers a reconcile, even without the ResourceVersion changing.
• Custom Predicate: If you're concerned about reducing unnecessary reconciliations and want to filter some events out, you might need to craft a custom predicate that's more suited to your needs.

Here's a modified SetupWithManager method after removing the mentioned predicate:

// SetupWithManager sets up the controller with the Manager.
func (r *ResourceReconciler) SetupWithManager(mgr ctrl.Manager) error {
    if err := mgr.GetFieldIndexer().IndexField(context.Background(), &v1.Resource{}, secretNameField, func(rawObj client.Object) []string {
        // Extract the SecretRef name from the Resource Spec, if one is provided
        nResource := rawObj.(*v1.Resource)
        if nResource.Spec.SecretRef == "" {
            return nil
        }
        return []string{nResource.Spec.SecretRef}
    }); err != nil {
        return err
    }
    manager := ctrl.NewControllerManagedBy(mgr).
        For(&v1.Resource{}).
        WithEventFilter(ignoreStatusUpdatePredicate()).
        Watches(
            &source.Kind{Type: &corev1.Secret{}},
            handler.EnqueueRequestsFromMapFunc(r.findObjectsForSecret))
    return manager.Complete(r)
}

Try this modification, then update a Secret and see if it triggers reconciliation of the corresponding CR. If removing the predicate solves your issue, then it confirms that the ResourceVersionChangedPredicate was indeed filtering out the updates that you wanted to act upon. If you still face the issue, there might be some other factors at play and would require further investigation.

Closing this one as sorted out
However, if you see that is required feel free to re-open.