A CRD is missing when deploying the kube-prometheus-stack with Argo CD

[lucj]

When deploying the kube-prometheus-stack with the following definition:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kube-prometheus-stack
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    server: https://kubernetes.default.svc
    namespace: monitoring
  project: default
  source:
    repoURL: https://prometheus-community.github.io/helm-charts
    targetRevision: 45.9.1
    chart: kube-prometheus-stack
    helm:
      releaseName: kube-prometheus
      parameters:
      - name: prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues
        value: "false"
      - name: prometheus.prometheusSpec.serviceMonitorSelector
        value: "{}"
      - name: prometheus.prometheusSpec.serviceMonitorNamespaceSelector
        value: "{}"
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

I can see the following CRDs are created:

$ k get crds | grep coreos
prometheusrules.monitoring.coreos.com       2023-04-13T12:11:36Z
servicemonitors.monitoring.coreos.com       2023-04-13T12:11:36Z
alertmanagerconfigs.monitoring.coreos.com   2023-04-13T12:11:36Z
probes.monitoring.coreos.com                2023-04-13T12:11:36Z
podmonitors.monitoring.coreos.com           2023-04-13T12:11:36Z
alertmanagers.monitoring.coreos.com         2023-04-13T12:11:36Z
thanosrulers.monitoring.coreos.com          2023-04-13T12:11:36Z

but I cannot see metrics in the dashboard.
From the logs, the prometheuses.monitoring.coreos.com CRD is missing:

level=error ts=2023-04-13T12:15:16.836941424Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Prometheus: failed to list *v1.Prometheus: the server could not find the requested resource (get prometheuses.monitoring.coreos.com)"

Any idea what could be wrong ?

[KGB33]

I ran into (what I think is) the same issue. After digging into the logs it looks like ArgoCD (and kubectl apply) can't add the previous definition in the last-applied-configuration annotation. I had the error message below:

Error from server (Invalid): error when creating "manifests/setup/prometheus-operator-0prometheusCustomResourceDefinition.yaml": CustomResourceDefinition.apiextensions.k8s.io "prom
etheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

The solution (found here) is to set the ServerSideApply=true syncOperation for Argo, or use kubectl create (instead of apply).

[lucj]

Thanks, I’ll try that 👍Currently I managed to create the stack with https://github.com/lucj/k8sapps/blob/main/apps/observability/kube-prometheus-stack.yamlLe 20 avr. 2023 à 18:42, Kelton Bassingthwaite ***@***.***> a écrit : I ran into (what I think is) the same issue. After digging into the logs it looks like ArgoCD (and kubectl apply) can't add the previous definition in the last-applied-configuration annotation. I had the error message below: Error from server (Invalid): error when creating "manifests/setup/prometheus-operator-0prometheusCustomResourceDefinition.yaml": CustomResourceDefinition.apiextensions.k8s.io "prom etheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes The solution (found here) is to set the ServerSideApply=true syncOperation for Argo, or use kubectl create (instead of apply). —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[github-actions]

This issue has been automatically marked as stale because it has not had any activity in the last 60 days. Thank you for your contributions.

[github-actions]

This issue was closed because it has not had any activity in the last 120 days. Please reopen if you feel this is still valid.