After update to Helm release 2.6.4 `Error updating SealedSecret "secret" not found` message in log #961

3deep5me · 2022-09-15T11:25:38Z

Which component:
Helm release 2.6.4

Describe the bug
After the update from chart 2.6.2 to 2.6.4 i get following error on recreated/new created sealedsecrets:

2022/09/15 10:26:11 Updating namespace/secret
2022/09/15 10:26:11 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"namespace", Name:"secret", UID:"5d1436b0-3655-4e69-9a9dc774c0dd", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"379770364", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
2022/09/15 10:26:11 Error updating SealedSecret namespace/secret status: sealedsecrets.bitnami.com "secret" not found

To Reproduce
Steps to reproduce the behavior:

Install Helm release 2.6.4
Create a new or recreate a sealed-secret
check the logs
See error

Expected behavior
No not found message in the log

Version of Kubernetes:
v1.21.6 (Openshift)
v1.22.6 (AKS)

Output of kubectl version:

(AKS)

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.6", GitCommit:"ece9ecf2f9aecbd86d3eba31f0be62e4b6353a5a", GitTreeState:"clean", BuildDate:"2022-07-28T23:33:17Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

(Openshift)

WCAlRiNeInNtG :V evresrisoinon :d iffveerresnicoen .bIentfwoeen{ clieMnatj o(r1.2:3") a1nd "s,e rveMri n(o1r.21:)" ex2c3eed"s,  theG istuVpeprosriteodn mi:n"orv 1v.e2r3s.i0o"n,  skeGwi toCfo m+m/i-t1:
"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.6+4b61f94", GitCommit:"cefce093e4e5bc9a1916eb5a489ed37c7d467f6f", GitTreeState:"clean", BuildDate:"2022-02-09T02:16:16Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}

Additional context
It seems like this also breaks the argocd health-check which caused that the resources is always on status "processing" in argocd with health-detail "Waiting for Sealed Secret to be decrypted" even its decrypted.
https://github.com/argoproj/argo-cd/blob/master/resource_customizations/bitnami.com/SealedSecret/health.lua

The text was updated successfully, but these errors were encountered:

josvazg · 2022-09-15T15:17:39Z

Could you provide more details on the steps to reproduce the error?

Here is an example we tried with that version, and we could not reproduce:

$ helm install sealed-secrets -n kube-system --set-string fullnameOverride=sealed-secrets-controller --version 2.6.4 sealed-secrets/sealed-secrets
...
$ helm ls -A
NAME          	NAMESPACE  	REVISION	UPDATED                                 	STATUS  	CHART               	APP VERSION
sealed-secrets	kube-system	1       	2022-09-15 17:09:13.055100484 +0200 CEST	deployed	sealed-secrets-2.6.4	v0.18.2
...
# test
$ echo -n foobar | kubectl create secret generic secret --dry-run=client --from-file=foo=/dev/stdin -o json > /tmp/secret.json
$ kubeseal < /tmp/secret.json > /tmp/sealedsecret.json
$ kubectl create -f /tmp/sealedsecret.json
sealedsecret.bitnami.com/secret created
$ kubectl get sealedsecrets
NAME     AGE
secret   9s
$ kubectl get secrets
NAME     TYPE     DATA   AGE
secret   Opaque   1      14s
$ kubectl get secrets/secret -oyaml
apiVersion: v1
data:
  foo: Zm9vYmFy
kind: Secret
...
$ echo Zm9vYmFy | base64 -d
foobar

josvazg · 2022-09-15T15:19:49Z

Sending us the logs from the controller would also help:

For example, for the above test:

$ kubectl get pods -A |grep sealed
kube-system   sealed-secrets-controller-68dd8fb447-jtt25   1/1     Running   0              8m49s

$ kubectl logs -n kube-system sealed-secrets-controller-68dd8fb447-jtt25
controller version: 0.18.2
2022/09/15 15:09:19 Starting sealed-secrets controller version: 0.18.2
2022/09/15 15:09:19 Searching for existing private keys
2022/09/15 15:09:19 ----- sealed-secrets-keyt2jq9
2022/09/15 15:09:19 HTTP server serving on :8080
2022/09/15 15:12:03 Updating default/secret
2022/09/15 15:12:03 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"default", Name:"secret", UID:"fb4954ac-5dee-4f56-98f4-eeab95ab0824", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"25299", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
2022/09/15 15:12:03 Updating default/secret
2022/09/15 15:12:03 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"default", Name:"secret", UID:"fb4954ac-5dee-4f56-98f4-eeab95ab0824", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"25301", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully

3deep5me · 2022-09-15T16:01:14Z

Thanks for trying to reproduce.
I tried to do the same

$ echo -n foobar | kubectl create secret generic secret --dry-run=client --from-file=foo=/dev/stdin -o json > /tmp/secret.json
$ kubeseal --controller-name sealed-secrets < /tmp/secret.json > /tmp/sealedsecret.json
$ kubectl create -f /tmp/sealedsecret.json
sealedsecret.bitnami.com/secret created
$ kubectl get sealedsecrets
NAME     AGE
secret   10s
$ kubectl get secret secret
NAME     TYPE     DATA   AGE
secret   Opaque   1      29s
$ kubectl get secrets/secret -oyaml
apiVersion: v1
data:
  foo: ITxzeW1saW5rPv/+LwBwAHIAbwBjAC8AcwBlAGwAZgAvAGYAZAAvADAAAAA=
kind: Secret
metadata:
  creationTimestamp: "2022-09-15T15:54:31Z"
  name: secret
  namespace: monitoring
  ownerReferences:
  - apiVersion: bitnami.com/v1alpha1
    controller: true
    kind: SealedSecret
    name: secret
    uid: f9e580ff-9bb3-47af-add7-972a9a75f046
  resourceVersion: "32118004"
  uid: f6326167-5415-4650-ad09-b6ab8db44687
type: Opaque
$ echo ITxzeW1saW5rPv/+LwBwAHIAbwBjAC8AcwBlAGwAZgAvAGYAZAAvADAAAAA= | base64 -d
!<symlink>□□/proc/self/fd/0

The logs from the controller:

$ k logs -n kube-system sealed-secrets-6dcc9776cf-57dfz
controller version: 0.18.2
2022/09/15 15:42:01 Starting sealed-secrets controller version: 0.18.2
2022/09/15 15:42:01 Searching for existing private keys
2022/09/15 15:42:01 ----- sealed-secrets-key4flrz
2022/09/15 15:42:01 ----- sealed-secrets-keyghwdt
2022/09/15 15:42:01 ----- sealed-secrets-keymhtw5
2022/09/15 15:42:01 HTTP server serving on :8080
2022/09/15 15:54:31 Updating monitoring/secret
2022/09/15 15:54:31 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"monitoring", Name:"secret", UID:"f9e580ff-9bb3-47af-add7-972a9a75f046", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"32118003", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
2022/09/15 15:54:31 Error updating SealedSecret monitoring/secret status: sealedsecrets.bitnami.com "secret" not found

The only difference is that i did the install with argocd:
(No values modified)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sealed-secrets
  namespace: argocd
  annotations:
spec:
  destination:
    namespace: kube-system
    server: https://kubernetes.default.svc
  project: infra
  source:
    path: helm/sealed-secrets
    repoURL: https://github.com/bitnami-labs/sealed-secrets.git
    targetRevision: helm-v2.6.4
  syncPolicy:
    automated: {}

I will try this also on a new cluster with a all new install tomorrow.

martin-jensen · 2022-09-15T16:28:37Z

If it can help, i have been having the same issue, tried with both a clean install of k3s or microk8s.

For extra info I am on windows and using the windows version of kubectl and kubeseal.

So with a new ubuntu vm (22.04) created with multipass and a default install of k3s.

helm install sealed-secrets -n kube-system --set-string fullnameOverride=sealed-secrets-controller --version 2.6.4 sealed-secrets/sealed-secrets
kubectl create secret generic test --dry-run=client --output=json --from-literal=password=supersecret | kubeseal > mysealedsecret.json
kubectl apply -f .\mysealedsecret.json

Results in the following log from the controller

controller version: 0.18.2
2022/09/15 15:52:26 Starting sealed-secrets controller version: 0.18.2
2022/09/15 15:52:26 Searching for existing private keys
2022/09/15 15:52:28 New key written to kube-system/sealed-secrets-keyj6s66
2022/09/15 15:52:28 Certificate is
-----BEGIN CERTIFICATE-----
xxxx
-----END CERTIFICATE-----

2022/09/15 15:52:28 HTTP server serving on :8080
2022/09/15 15:53:14 Updating default/test
2022/09/15 15:53:14 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"default", Name:"test", UID:"fdb2491d-3877-4f49-aac9-559fc4591dff", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"698", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
2022/09/15 15:53:14 Error updating SealedSecret default/test status: sealedsecrets.bitnami.com "test" not found

Output of "kubectl version"

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"windows/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.4+k3s1", GitCommit:"c3f830e9b9ed8a4d9d0e2aa663b4591b923a296e", GitTreeState:"clean", BuildDate:"2022-08-25T03:45:26Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}

Output of "kubeseal --version"
kubeseal version: 0.18.2

rtsp · 2022-09-15T20:24:48Z

This also happen to me on all secrets that unsealed after 2.6.4 upgrade.

2022/09/15 20:13:20 Updating gitlab-runner/gitlab-runner-secret
2022/09/15 20:13:20 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"kube-prometheus-stack", Name:"kube-prometheus-stack-grafana", UID:"abd6bae2-f1af-40b7-aa85-3152a70f7588", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"3082499", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
2022/09/15 20:13:20 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"gitlab-runner", Name:"gitlab-runner-secret", UID:"35c57d22-693d-430e-ba69-424ec1921405", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"3734948", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
2022/09/15 20:13:20 Error updating SealedSecret gitlab-runner/gitlab-runner-secret status: sealedsecrets.bitnami.com "gitlab-runner-secret" not found

2022/09/15 20:18:50 Updating gitlab-runner/gitlab-runner-secret
2022/09/15 20:18:50 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"gitlab-runner", Name:"gitlab-runner-secret", UID:"35c57d22-693d-430e-ba69-424ec1921405", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"3737826", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
2022/09/15 20:18:50 Error updating SealedSecret gitlab-runner/gitlab-runner-secret status: sealedsecrets.bitnami.com "gitlab-runner-secret" not found

2022/09/15 20:18:50 Updating gitlab-runner/gitlab-runner-secret
2022/09/15 20:18:50 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"gitlab-runner", Name:"gitlab-runner-secret", UID:"83934bde-5235-48ac-8979-ffa5bc3973e8", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"3737842", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
2022/09/15 20:18:50 Error updating SealedSecret gitlab-runner/gitlab-runner-secret status: sealedsecrets.bitnami.com "gitlab-runner-secret" not found

The secrets unsealed succesfully and working good except the status field of the parent SealedSecrets resource is missing. This cause Argo CD to stuck in "Progressing" state.

vizv · 2022-09-17T16:02:54Z

Found the issue, working on the fix...

alemorcuq · 2022-09-20T14:07:15Z

We will release a new chart including the patched CRD shortly.

alemorcuq · 2022-09-20T14:23:39Z

Chart 2.6.7 released including the patched CRD. Thanks, @vizv !

github-actions bot added the triage Issues/PRs that need to be reviewed label Sep 15, 2022

3deep5me changed the title ~~After update to Helm release 2.6.4 Error updating SealedSecret message in log~~ After update to Helm release 2.6.4 Error updating SealedSecret "secret" not found message in log Sep 15, 2022

vizv mentioned this issue Sep 17, 2022

Fix status sub-resource not found error on updating status #964

Merged

alemorcuq added bug and removed triage Issues/PRs that need to be reviewed labels Sep 20, 2022

alemorcuq mentioned this issue Sep 20, 2022

Check SealedSecret status in tests #967

Merged

alemorcuq closed this as completed in #964 Sep 20, 2022

alemorcuq mentioned this issue Sep 20, 2022

Release chart v2.6.7 #968

Merged

alemorcuq mentioned this issue Sep 23, 2022

updateStatus not being set properly in Helm-based install? #975

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

After update to Helm release 2.6.4 `Error updating SealedSecret "secret" not found` message in log #961

After update to Helm release 2.6.4 `Error updating SealedSecret "secret" not found` message in log #961

3deep5me commented Sep 15, 2022

josvazg commented Sep 15, 2022

josvazg commented Sep 15, 2022

3deep5me commented Sep 15, 2022

martin-jensen commented Sep 15, 2022

rtsp commented Sep 15, 2022 •

edited

Loading

vizv commented Sep 17, 2022

alemorcuq commented Sep 20, 2022

alemorcuq commented Sep 20, 2022

After update to Helm release 2.6.4 Error updating SealedSecret "secret" not found message in log #961

After update to Helm release 2.6.4 Error updating SealedSecret "secret" not found message in log #961

Comments

3deep5me commented Sep 15, 2022

josvazg commented Sep 15, 2022

josvazg commented Sep 15, 2022

3deep5me commented Sep 15, 2022

martin-jensen commented Sep 15, 2022

rtsp commented Sep 15, 2022 • edited Loading

vizv commented Sep 17, 2022

alemorcuq commented Sep 20, 2022

alemorcuq commented Sep 20, 2022

After update to Helm release 2.6.4 `Error updating SealedSecret "secret" not found` message in log #961

After update to Helm release 2.6.4 `Error updating SealedSecret "secret" not found` message in log #961

rtsp commented Sep 15, 2022 •

edited

Loading