Plugin registration mechanism of CSI is broken

[gnufied]

It looks like plugin registration mechanism of CSI might be broken:

Oct 31 13:57:11 foobar-csi-test-minion-group-xxx kubelet[1987]: E1031 13:57:11.226647    1987 plugin_watcher.go:115] error failed to get plugin info using RPC GetInfo at socket /var/lib/kubelet/plugins/com.google.csi.gcepd/csi.sock, err: rpc error: code = Unimplemented desc = unkno
wn service pluginregistration.Registration when handling create event: "/var/lib/kubelet/plugins/com.google.csi.gcepd/csi.sock": CREATE

[gnufied]

/sig storage
/kind bug
/assign

[gnufied]

cc @verult @msau42

[msau42]

Is this with/without feature gates? On head or 1.12? Does this impact other functionality like attach, mount, etc?

[jsafrane]

I think that it's dynamic registration that discovers all sockets in /var/lib/kubelet/plugins, including a socket created by a driver in a subdirectory. It tries to talk to it with dynamic registration gRPC and it fails.

Driver registrar exposes a different socket (/var/lib/kubelet/plugins/*.sock, no subdirectory) and that one should be discovered and used by kubelet.

The message is harmless, at least on my machine, however, it pollutes logs and we should do something about it. Should the dynamic registration watch only /var/lib/kubelet/plugins without subdirectories?

/sig node

[jsafrane]

cc @RenaudWasTaken for dynamic registration expertise.

[gnufied]

Okay, so this could not be reason of volume limits not being registered for csi plugins. hmm. it looked suspicious.

[RenaudWasTaken]

Hello!

Yep this seems to be related to #69015 which is a known harmless bug.

Basically what happens is that the plugin watcher will iterate recursively over that directory and try to register every file. In your case the csi.sock doesn't seem to implement the gRPC service, since this is the case, the pluginwatcher mechanism will ignore this file.

We are considering limiting the recursive behavior to one level, and making sure that the files we try to contact always end with .sock in the next release.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[msau42]

/close

[k8s-ci-robot]

@msau42: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[msau42]

Red herring

[kremers]

Same issue with trident v19.10.0