Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)

[sjiekak]

What happened?

We are using crossplane with provider-gcp-kms:v1.8.0 and we experience panics

 k logs pods/gcp-provider-kms-xxxxx -n crossplane
E0901 23:07:06.785441       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 1052684 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x64b7ba0?, 0xc069ea0})
	k8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x85
github.com/crossplane/upjet/pkg/controller.(*panicHandler).recoverIfPanic(0xc0087e3f60)
	github.com/crossplane/[email protected]/pkg/controller/external_async_tfpluginfw.go:154 +0x87
panic({0x64b7ba0?, 0xc069ea0?})
	runtime/panic.go:914 +0x21f
github.com/json-iterator/go.(*OptionalDecoder).Decode(0xc00512a9e0, 0x0, 0x83d42e?)
	github.com/json-iterator/[email protected]/reflect_optional.go:32 +0x59
github.com/json-iterator/go.(*placeholderDecoder).Decode(0xc0087e3a58?, 0x410a05?, 0xc0087e3a90?)
	github.com/json-iterator/[email protected]/reflect.go:324 +0x1b
github.com/json-iterator/go.(*structFieldDecoder).Decode(0xc00512ab80, 0x407758?, 0xc004d330e0)
	github.com/json-iterator/[email protected]/reflect_struct_decoder.go:1054 +0x50
github.com/json-iterator/go.(*twoFieldsStructDecoder).Decode(0xc005133d70, 0xc00511c340?, 0xc004d330e0)
	github.com/json-iterator/[email protected]/reflect_struct_decoder.go:615 +0xc6
github.com/json-iterator/go.(*sliceDecoder).doDecode(0xc005126e58, 0xc0085d6518, 0xc004d330e0)
	github.com/json-iterator/[email protected]/reflect_slice.go:86 +0xa2
github.com/json-iterator/go.(*sliceDecoder).Decode(0xc005126e58, 0x415b8b?, 0xc004d330e0)
	github.com/json-iterator/[email protected]/reflect_slice.go:60 +0x25
github.com/json-iterator/go.(*structFieldDecoder).Decode(0xc00512b120, 0xc00513e780?, 0xc004d330e0)
	github.com/json-iterator/[email protected]/reflect_struct_decoder.go:1054 +0x50
github.com/json-iterator/go.(*generalStructDecoder).decodeOneField(0xc00512b280, 0xc001e65ba0?, 0xc004d330e0)
	github.com/json-iterator/[email protected]/reflect_struct_decoder.go:552 +0x29c
github.com/json-iterator/go.(*generalStructDecoder).Decode(0xc00512b280, 0x5f363a0?, 0xc004d330e0)
	github.com/json-iterator/[email protected]/reflect_struct_decoder.go:508 +0x8b
github.com/json-iterator/go.(*Iterator).ReadVal(0xc004d330e0, {0x666f840, 0xc0085d64e0})
	github.com/json-iterator/[email protected]/reflect.go:79 +0x123
github.com/json-iterator/go.(*frozenConfig).Unmarshal(0xc0004210e0, {0xc008662380?, 0xc005767830?, 0x70d6420?}, {0x666f840, 0xc0085d64e0})
	github.com/json-iterator/[email protected]/config.go:348 +0x99
github.com/upbound/provider-gcp/apis/kms/v1beta1.(*CryptoKey).SetObservation(0xc0085d62c0, 0xc0087e3eb8?)
	github.com/upbound/provider-gcp/apis/kms/v1beta1/zz_cryptokey_terraformed.go:43 +0x7e
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKExternal).Update(0xc008f81880, {0x81822b0, 0xc004a135e0}, {0x81ceb28?, 0xc0085d62c0})
	github.com/crossplane/[email protected]/pkg/controller/external_tfpluginsdk.go:699 +0x27b
github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKAsyncExternal).Update.func1()
	github.com/crossplane/[email protected]/pkg/controller/external_async_tfpluginsdk.go:199 +0x1fa
created by github.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKAsyncExternal).Update in goroutine 1524
	github.com/crossplane/[email protected]/pkg/controller/external_async_tfpluginsdk.go:178 +0x167
E0901 23:07:06.787757       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 1524 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x64b7ba0?, 0xc069ea0})
	k8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x85
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:108 +0xb2
panic({0x64b7ba0?, 0xc069ea0?})
	runtime/panic.go:914 +0x21f
reflect.typedslicecopy(0x7255a5d?, {0xc006d9bd80?, 0x81a29d0?, 0x68ca2c0?}, {0x0?, 0x68ca2c0?, 0x410a65?})
	runtime/mbarrier.go:295 +0x47
reflect.Copy({0x5e60b80?, 0xc005738eb8?, 0x1?}, {0x5e60b80?, 0xc0085d6518?, 0x18?})
	reflect/value.go:2953 +0x416
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).array(0xc006f54f70, {0x5e60b80?, 0xc0085d6518?, 0x52b?})
	sigs.k8s.io/[email protected]/internal/golang/encoding/json/decode.go:585 +0x576
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).value(0xc006f54f70, {0x5e60b80?, 0xc0085d6518?, 0x7?})
	sigs.k8s.io/[email protected]/internal/golang/encoding/json/decode.go:391 +0x74
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).object(0xc006f54f70, {0x7076220?, 0xc0085d64e0?, 0x68ca400?})
	sigs.k8s.io/[email protected]/internal/golang/encoding/json/decode.go:867 +0x1277
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).value(0xc006f54f70, {0x7076220?, 0xc0085d64e0?, 0xa?})
	sigs.k8s.io/[email protected]/internal/golang/encoding/json/decode.go:401 +0x3e
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).object(0xc006f54f70, {0x68ca400?, 0xc0085d64c0?, 0xfce?})
	sigs.k8s.io/[email protected]/internal/golang/encoding/json/decode.go:867 +0x1277
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).value(0xc006f54f70, {0x68ca400?, 0xc0085d64c0?, 0x6?})
	sigs.k8s.io/[email protected]/internal/golang/encoding/json/decode.go:401 +0x3e
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).object(0xc006f54f70, {0x7214c00?, 0xc0085d62c0?, 0x7214c00?})
	sigs.k8s.io/[email protected]/internal/golang/encoding/json/decode.go:867 +0x1277
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).value(0xc006f54f70, {0x7214c00?, 0xc0085d62c0?, 0x7a68a71a95b8?})
	sigs.k8s.io/[email protected]/internal/golang/encoding/json/decode.go:401 +0x3e
sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).unmarshal(0xc006f54f70, {0x7214c00?, 0xc0085d62c0?})
	sigs.k8s.io/[email protected]/internal/golang/encoding/json/decode.go:188 +0x133
sigs.k8s.io/json/internal/golang/encoding/json.Unmarshal({0xc006a90000, 0xdb8, 0x1000}, {0x7214c00, 0xc0085d62c0}, {0xc002c67da0, 0x2, 0xc002c67dc0?})
	sigs.k8s.io/[email protected]/internal/golang/encoding/json/decode.go:113 +0x159
sigs.k8s.io/json.UnmarshalCaseSensitivePreserveInts(...)
	sigs.k8s.io/[email protected]/json.go:62
k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).unmarshal(0xc0043b0c80?, {0x816a6c0?, 0xc0085d62c0?}, {0xc006a90000?, 0xc0066bbe30?, 0x9?}, {0xc006a90000?, 0xc00043e9a0?, 0x813f060?})
	k8s.io/[email protected]/pkg/runtime/serializer/json/json.go:258 +0x38f
k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode(0xc001325450, {0xc006a90000, 0xdb8, 0x1000}, 0x0, {0x816a6c0, 0xc0085d62c0?})
	k8s.io/[email protected]/pkg/runtime/serializer/json/json.go:206 +0xa25
k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode({{0x8146340?, 0xc001325450?}}, {0xc006a90000?, 0x199?, 0xc0085d62c0?}, 0x6b9c340?, {0x816a6c0?, 0xc0085d62c0?})
	k8s.io/[email protected]/pkg/runtime/helper.go:252 +0x4f
sigs.k8s.io/controller-runtime/pkg/client/apiutil.targetZeroingDecoder.Decode({{0x8142020?, 0xc002dda210?}}, {0xc006a90000, 0xdb8, 0x1000}, 0x0?, {0x816a6c0?, 0xc0085d62c0?})
	sigs.k8s.io/[email protected]/pkg/client/apiutil/apimachinery.go:215 +0xb0
k8s.io/client-go/rest.Result.Into({{0xc006a90000, 0xdb8, 0x1000}, {0x0, 0x0, 0x0}, {0xc003694e60, 0x10}, {0x0, 0x0}, ...}, ...)
	k8s.io/[email protected]/rest/request.go:1373 +0x97
sigs.k8s.io/controller-runtime/pkg/client.(*typedClient).UpdateSubResource(0xc000790ab0, {0x81822b0, 0xc007b9db90}, {0x81a4480?, 0xc0085d62c0}, {0x72427c5, 0x6}, {0x0, 0x0, 0x0})
	sigs.k8s.io/[email protected]/pkg/client/typed_client.go:247 +0x4e5
sigs.k8s.io/controller-runtime/pkg/client.(*subResourceClient).Update(0xc0081bf0e0, {0x81822b0, 0xc007b9db90}, {0x81a4480?, 0xc0085d62c0}, {0x0, 0x0, 0x0})
	sigs.k8s.io/[email protected]/pkg/client/client.go:590 +0x287
github.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile(0xc002eef100, {0x8182208, 0xc004b23050}, {{{0x0, 0x0}, {0xc002926f60, 0x24}}})
	github.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:1224 +0x56eb
github.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile(0xc0017ea730, {0x8182208, 0xc004b23050}, {{{0x0?, 0x5?}, {0xc002926f60?, 0xc007224d08?}}})
	github.com/crossplane/[email protected]/pkg/ratelimiter/reconciler.go:54 +0x151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x8189938?, {0x8182208?, 0xc004b23050?}, {{{0x0?, 0xb?}, {0xc002926f60?, 0x0?}}})
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc005b79540, {0x8182240, 0xc0013259a0}, {0x6915100?, 0xc0039412e0?})
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316 +0x3cc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc005b79540, {0x8182240, 0xc0013259a0})
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266 +0x1af
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 1266
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:223 +0x56

How can we reproduce it?

Unfortunately I can't. Is it possible this is the result of concurrent access to the same object ?

[mergenci]

For reference, the panic stems from this Upjet call:

upjet/pkg/controller/external_tfpluginsdk.go

Line 699 in 2e361ad

err = mg.(resource.Terraformed).SetObservation(stateValueMap)

Judging by the following line in the panic output above, it looks like the problem is with the “terraformed” resource.

github.com/upbound/provider-gcp/apis/kms/v1beta1.(*CryptoKey).SetObservation(0xc0085d62c0, 0xc0087e3eb8?)
	github.com/upbound/provider-gcp/apis/kms/v1beta1/zz_cryptokey_terraformed.go:43 +0x7e

I wonder if a faulty resource configuration might cause such a problem.