The ConfigMap is invalid: metadata.annotations: Too long: must have at most 262144 characters

[kuznero]

Does it apply to Argo CD managed helm charts? Can it be overwritten/overcome somehow?

[jessesuen]

This error is happening during kubectl apply phase for one of your ConfigMap resources, and is not just specific to helm. We use kubectl apply to perform deployments. One of the downsides of using kubectl apply is that it stores the entire spec as an annotation in the object (which it uses to understand how handle defaulted vs. deleted fields). What's happening is that the data fields in the ConfiMap is likely exceeding 262144 characters enforced by the K8s API server, and thus it cannot fit in the last-applied-configuration kubectl annotation.

Theres not a workaround other than a Sync hook which performs the deploy in a manner different than apply (e.g. kubectl replace). Which helm chart is causing problems?

[kuznero]

This is a custom made helm chart with a bit config map that I currently need to deploy with kubectl create. Could you please reference an example with a Sync hook you mentioned?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[basvandenbrink]

This error is happening during kubectl apply phase for one of your ConfigMap resources, and is not just specific to helm. We use kubectl apply to perform deployments. One of the downsides of using kubectl apply is that it stores the entire spec as an annotation in the object (which it uses to understand how handle defaulted vs. deleted fields). What's happening is that the data fields in the ConfiMap is likely exceeding 262144 characters enforced by the K8s API server, and thus it cannot fit in the last-applied-configuration kubectl annotation.

Theres not a workaround other than a Sync hook which performs the deploy in a manner different than apply (e.g. kubectl replace). Which helm chart is causing problems?

@jessesuen This issue occurs when install Grafana with a large dashboard using ConfigMaps.

Wouldn't it be better to use kubectl create or kubectl replace?

I am also curious to the sync hook workaround.

[phantomnat]

I also have problem with grafana.
For the workaround, try adding these annotations to the resources that need to be deleted before apply.

  annotations:
    "helm.sh/hook": pre-install
    "helm.sh/hook-weight": "-1"
    "helm.sh/hook-delete-policy": before-hook-creation

[cabrinha]

I also have problem with grafana.
For the workaround, try adding these annotations to the resources that need to be deleted before apply.

  annotations:
    "helm.sh/hook": pre-install
    "helm.sh/hook-weight": "-1"
    "helm.sh/hook-delete-policy": before-hook-creation

Using these annotations seems to have kinda-worked... I still get caught up on a few of the larger config-maps. They just won't sync with the same error.

[Thakurvaibhav]

I also have problem with grafana.
For the workaround, try adding these annotations to the resources that need to be deleted before apply.

annotations:
"helm.sh/hook": pre-install
"helm.sh/hook-weight": "-1"
"helm.sh/hook-delete-policy": before-hook-creation

This does not seem to work in case of a grafana dashboards configmap. Please suggest a workaround / fix

[mprzygrodzki]

I also have problem with grafana.
For the workaround, try adding these annotations to the resources that need to be deleted before apply.
annotations:
"helm.sh/hook": pre-install
"helm.sh/hook-weight": "-1"
"helm.sh/hook-delete-policy": before-hook-creation

This does not seem to work in case of a grafana dashboards configmap. Please suggest a workaround / fix

Same problem, annotations doesnt seem to work in case of Grafana chart.

[jdomag]

Same here. @kuznero can you reopen the issue?

[ebuildy]

Hello @jessesuen where did you see argocd can use "create" instead of "apply"?

Tried with

annotations:
   argocd.argoproj.io/hook: PostSync
   argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
   argocd.argoproj.io/sync-options: Validate=false

[abdennour]

same issue here . exactly same.

[abdennour]

any deadline for fixing this issue ?
Could you , for example, patching the YAML before applying it ( removing the annotation last-applied-configuration , .. so on) ?
If the resource does not work because of this error, try to run k replace instead of k apply ?

[diranged]

We just ran into this on a CRD from Strimzi.io:

apiextensions.k8s.io/v1beta1/CustomResourceDefinition
strimzi-operator
kafkas.kafka.strimzi.io
 SyncFailed
CustomResourceDefinition.apiextensions.k8s.io "kafkas.kafka.strimzi.io" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

We obviously cannot have the CRD deleted and re-created for an upgrade.. what should we do? This is the resource it's trying to apply: https://github.com/strimzi/strimzi-kafka-operator/blob/0.22.1/install/cluster-operator/040-Crd-kafka.yaml

[alexmt]

The argoproj/gitops-engine#252 adds special handling for CRDs. Argo CD will try to just use K8S update request for CRDs if Replace=true option present. The update won't try to recreate resource so it is safe to use with CRDs.

[blackliner]

Similar thing happens with kube-prometheus-stack from https://prometheus-community.github.io/helm-charts. The prometheuses.**** CRD is HUGE, makes ArgoCD webUI lag like 1969

[dlemfh]

The argoproj/gitops-engine#252 adds special handling for CRDs. Argo CD will try to just use K8S update request for CRDs if Replace=true option present. The update won't try to recreate resource so it is safe to use with CRDs.

Please mention that ArgoCD v2.5 is introducing an option to perform a manifest sync with side-server apply.

#9711

With ArgoCD 2.5+

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  syncPolicy:
    syncOptions:
    - ServerSideApply=true

would resolve this issue in ArgoCD, because side-server apply would not create an annotation like kubectl.kubernetes.io/last-applied-configuration anymore.

Until its release, you can read about this feature here and why it would resolve this issue inside ArgoCD: master/docs/user-guide/sync-options.md#server-side-apply

Hey @alexmt, @crenshaw-dev,

now that argocd v2.5 is out, which solution would you recommend for the problem with CRD's being too long?

CustomResourceDefinition.apiextensions.k8s.io "kafkas.kafka.strimzi.io" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

• Replace=true

or

• ServerSideApply=true

?

[nallenscott]

In my experience, Replace is substantially faster than ServerSideApply when deploying lots of large CRDs, especially when you have a local copy. My CRDs spin up almost instantly with the following setup (if you deploy to the same cluster as Argo, replace the ApplicationSet with an Application):

1. Use braid to copy the CRDs to a local directory (simply braid update to trigger a sync in Argo):

$ braid add --branch main --path bitnami/kube-prometheus/crds https://github.com/bitnami/charts <local_path>/crds

2. Create an ApplicationSet pointing to the crds in your repo:

source:
  path: <path_to_crds>
  repoURL: [email protected]:<username>/<repo>.git
  targetRevision: <branch_or_commit>
syncPolicy:
  syncOptions:
    - Replace=true

3. Create an Application to deploy the AppSet, or place the AppSet in the path of an existing App:

source:
  path: <path_to_app_set>
  repoURL: [email protected]:<username>/<repo>.git
  targetRevision: <branch_or_commit>

[jkroepke]

@nallenscott Nice catch, but it seems a bit outdated. There is an semi official helm chart now that holds the Prometheus CRDs

https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-operator-crds

which natively allows setup annotations, if required.

Local local copy required

[nallenscott]

@jkroepke I tried that chart with and without Replace and ServerSideApply and CRDs take minutes to sync, if they sync at all. That's been consistent with all of the charts I've tried.

[jkroepke]

Not talking about Replace and ServerSideApply. But using the Chart replaces the braid hack

[nallenscott]

@jkroepke Clarified my comment.

[nallenscott]

Additionally, it seems wise to manage CRDs externally, seeing as helm is unable to upgrade or delete them after they're deployed. From the argo-cd helm repo: https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd#custom-resource-definitions

[davhdavh]

Couldnt argo just recognize this specific error message and use create on it?

[crenshaw-dev]

@davhdavh theoretically, but it's not always safe to Replace resources. I think it's better to leave it to the user to decide.

Although, I do think it would be worth adding a note in the UI to say, "Maybe you can use Replace (with the usual safety caveats) or ServerSideApply."

[matan-legit]

ServerSideApply doesn't fix the issue for me is there another way to fix the issue? specifically when using a helm type application