CustomResourceDefinition.apiextensions.k8s.io "runners.actions.summerwind.dev" is invalid: metadata.annotations: Too long

[donovanmuller]

Describe the bug

After updating to the 0.15.3 Chart I am getting the following error when sync'ing the Chart via ArgoCD:

Failed sync attempt to f4b6f0a64ab13ed7478264478306ca231b22ff44: one or more objects failed to apply, reason: CustomResourceDefinition.apiextensions.k8s.io "runners.actions.summerwind.dev" is invalid: metadata.annotations: Too long: must have at most 262144 bytes,unable to recognize "/dev/shm/471866979": no matches for kind "RunnerDeployment" in version "actions.summerwind.dev/v1alpha1",CustomResourceDefinition.apiextensions.k8s.io "runnerdeployments.actions.summerwind.dev" is invalid: metadata.annotations: Too long: must have at most 262144 bytes,CustomResourceDefinition.apiextensions.k8s.io "runnerreplicasets.actions.summerwind.dev" is invalid: metadata.annotations: Too long: must have at most 262144 bytes (retried 5 times).

Checks

• My actions-runner-controller version (v0.x.y) does support the feature
• I'm using an unreleased version of the controller I built from HEAD of the default branch

To Reproduce
Steps to reproduce the behavior:

1. Update ArgoCD Application with the 0.15.3 Chart version
2. After a failed sync, the error message above presented
3. Controller fails to start with no matches for kind \"RunnerReplicaSet\"

Expected behavior
ArgoCD Application syncs successfully and controller is running.

Environment (please complete the following information):

• Controller Version: 0.20.4
• Deployment Method: Helm via ArgoCD Application
• Helm Chart Version: 0.15.3

Additional context
This is a similar/same issue as faced in prometheus-community/helm-charts#1500

[mumoshu]

@donovanmuller This is a limitation of K8s. You need to use kubectl replace instead of kubectl apply to upgrade CRDs.

[donovanmuller]

@mumoshu agreed, however, using the Helm chart and then especially when using in conjunction with ArgoCD, makes this more challenging than it should be. However, I appreciate that this isn't really an issue in this project.

I've worked around it by applying the advice from the Prometheus thread and using a Helm + Kustomize setup to add the argocd.argoproj.io/sync-options: Replace=true annotation to the affected CRD's.

If there's nothing to action, happy to close.

[mumoshu]

@donovanmuller Hey, thanks for sharing your work-around!
JFYI, having two ArgoCD apps, one for skipCRDs=true and another for Replace=true is said to work in recent RC version of ArgoCD. Perhaps that's a cleaner solution once released.

prometheus-operator/prometheus-operator#4439 (comment)

[jimmyh85]

@donovanmuller , would you mind sharing your solution? How did you include the CRDs into your kustomiziation folder? Do they need to be added manually or can they be referenced in a smarter way? Might be helpful for everyone who is using ArgoCD and the actions runner controller :)

[yohangdev]

advice

I'm following solution at prometheus-operator/prometheus-operator#4439 (comment) at get this problem solved.

    syncOptions:
    - Replace=true

[donovanmuller]

@donovanmuller , would you mind sharing your solution? How did you include the CRDs into your kustomiziation folder? Do they need to be added manually or can they be referenced in a smarter way? Might be helpful for everyone who is using ArgoCD and the actions runner controller :)

@jimmyh85 Using the Helm + Kustomize approach, we just use patchesStrategicMerge to add just the annotation to the CRD's rendered from the Helm Chart.

E.g.

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../helm

patchesStrategicMerge:
  - patch/runners-crd.yaml

patch/runners-crd.yaml

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    argocd.argoproj.io/sync-options: Replace=true
  name: runners.actions.summerwind.dev

[KENNYSOFT]

advice

I'm following solution at prometheus-operator/prometheus-operator#4439 (comment) at get this problem solved.

    syncOptions:
    - Replace=true

Thanks, can achieve the same result by setting in web dashboard:

[norman-zon]

@donovanmuller, what does your kustomized-helm plugin config look like?
If I use the default config CRDs are not rendered. Do you just add --include-crds ?

[jimmyh85]

@norman-zon, I also had some issues in setting it up via kustomized-helm and solved it by using below plugin config

  configManagementPlugins: |
    - name: kustomized-helm
      init:
        command: ["/bin/sh", "-c"]
        args: ["helm dependency build"]
      generate:
        command: [sh, -c]
        args: ["helm template . --name-template $ARGOCD_APP_NAME --namespace $ARGOCD_APP_NAMESPACE --include-crds > all.yaml && kustomize build"]

Main problem was that the example of helm-kustomized did not consider the namespace. Hope this helps, but the approach to use the sync option replace=true might be easier

[donovanmuller]

@donovanmuller, what does your kustomized-helm plugin config look like? If I use the default config CRDs are not rendered. Do you just add --include-crds ?

@norman-zon 👍

configManagementPlugins: |
  - name: kustomized-helm
    init:
      command: ["/bin/sh", "-c"]
      args: ["helm dependency build ../../helm || true"]
    generate:
      command: ["/bin/sh", "-c"]
      args: ["echo \"$HELM_VALUES\" | helm template ../../helm --name-template $ARGOCD_APP_NAME --namespace $ARGOCD_APP_NAMESPACE $HELM_ARGS -f - --include-crds > ../../helm/all.yaml && kustomize build"]

[imrajdas]

Is this version released?

[KENNYSOFT]

Is this version released?

Nothing was changed