Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Node Feature Discovery Master random crashes #1640

Closed
tatodorov opened this issue Mar 27, 2024 · 10 comments
Closed

Node Feature Discovery Master random crashes #1640

tatodorov opened this issue Mar 27, 2024 · 10 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@tatodorov
Copy link

tatodorov commented Mar 27, 2024 •
edited
Loading

What happened:
Ocasionally, NFD Master crashes with the following error:

fatal error: concurrent map read and map write

Followed by a go routine dump.

Go routine dump 
goroutine 358442 [running]:
k8s.io/apimachinery/pkg/runtime.(*Scheme).ObjectKinds(0xc0004439d0, {0x24401f8?, 0xc003598c60})
        /go/pkg/mod/k8s.io/[email protected]/pkg/runtime/scheme.go:267 +0x1ca
k8s.io/apimachinery/pkg/runtime.(*parameterCodec).EncodeParameters(0xc0003f8900, {0x24401f8, 0xc003598c60}, {{0x2148806, 0xf}, {0x213da3e, 0x8}})
        /go/pkg/mod/k8s.io/[email protected]/pkg/runtime/codec.go:190 +0x6e
k8s.io/client-go/rest.(*Request).SpecificallyVersionedParams(0xc000afa6c0, {0x24401f8?, 0xc003598c60?}, {0x2440450?, 0xc0003f8900?}, {{0x2148806?, 0x11?}, {0x213da3e?, 0xc0016559a0?}})
        /go/pkg/mod/k8s.io/[email protected]/rest/request.go:377 +0x83
k8s.io/client-go/rest.(*Request).VersionedParams(...)
        /go/pkg/mod/k8s.io/[email protected]/rest/request.go:370
sigs.k8s.io/node-feature-discovery/pkg/generated/clientset/versioned/typed/nfd/v1alpha1.(*nodeFeatures).List(0xc0003cef00, {0x2453fe8, 0xc00012e008}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, ...}, ...})
        /go/node-feature-discovery/pkg/generated/clientset/versioned/typed/nfd/v1alpha1/nodefeature.go:89 +0x145
sigs.k8s.io/node-feature-discovery/pkg/generated/informers/externalversions/nfd/v1alpha1.NewFilteredNodeFeatureInformer.func1({{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0, ...})
        /go/node-feature-discovery/pkg/generated/informers/externalversions/nfd/v1alpha1/nodefeature.go:65 +0x145
k8s.io/client-go/tools/cache.(*ListWatch).List(0x6?, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, 0x0, ...})
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/listwatch.go:106 +0x55
k8s.io/client-go/tools/cache.(*Reflector).list.func1.2({{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0, ...})
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:488 +0x55
k8s.io/client-go/tools/pager.SimplePageFunc.func1({0x10?, 0x10?}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, ...})
        /go/pkg/mod/k8s.io/[email protected]/tools/pager/pager.go:40 +0x55
k8s.io/client-go/tools/pager.(*ListPager).list(0xc001655f20, {0x2453fe8, 0xc00012e000}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, ...}, ...}, ...)
        /go/pkg/mod/k8s.io/[email protected]/tools/pager/pager.go:108 +0x168
k8s.io/client-go/tools/pager.(*ListPager).ListWithAlloc(...)
        /go/pkg/mod/k8s.io/[email protected]/tools/pager/pager.go:89
k8s.io/client-go/tools/cache.(*Reflector).list.func1()
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:513 +0x216
created by k8s.io/client-go/tools/cache.(*Reflector).list
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:479 +0x3db

goroutine 1 [runnable]:
k8s.io/klog/v2.(*loggingT).printDepth(0x357d0c0?, 0x0?, 0x0, {0x0, 0x0}, 0x2?, {0xc000597d90, 0x1, 0x1})
        /go/pkg/mod/k8s.io/klog/[email protected]/klog.go:698 +0x225
k8s.io/klog/v2.(*loggingT).printS(0xa6e3a4?, {0x0, 0x0}, 0xa6e3e0?, 0x1, {0x217bbe7, 0x29}, {0x0, 0x0, 0x0})
        /go/pkg/mod/k8s.io/klog/[email protected]/klog.go:799 +0x1f8
k8s.io/klog/v2.(*loggingT).infoS(0x0?, 0x34630b8a000?, {0x0?, 0x0?}, 0x0?, {0x217bbe7?, 0xc00167fd60?}, {0x0, 0x0, 0x0})
        /go/pkg/mod/k8s.io/klog/[email protected]/klog.go:783 +0x13f
k8s.io/klog/v2.InfoS(...)
        /go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1512
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).stop(0xc0004f3710)
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:106 +0x146
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nfdMaster).Run(0xc0001f5b00)
        /go/node-feature-discovery/pkg/nfd-master/nfd-master.go:301 +0x754
main.main()
        /go/node-feature-discovery/cmd/nfd-master/main.go:109 +0x445

goroutine 358393 [chan receive, 2 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:969 +0x57
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0022ebf38?, {0x2438720, 0xc0013c83f0}, 0x1, 0xc0024edce0)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0011c8b40)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:968 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 358506 [IO wait]:
internal/poll.runtime_pollWait(0x7f2836b70e18, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0004a5680?, 0xc001549ed0?, 0x1)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0004a5680, {0xc001549ed0, 0x10000, 0x10000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299
os.(*File).read(...)
        /usr/local/go/src/os/file_posix.go:31
os.(*File).Read(0xc0003ae1f0, {0xc001549ed0?, 0x0?, 0x0?})
        /usr/local/go/src/os/file.go:118 +0x5e
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc003976190)
        /go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:356 +0xdf
created by github.com/fsnotify/fsnotify.NewWatcher
        /go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:150 +0x1b0

goroutine 163 [IO wait, 2815 minutes]:
internal/poll.runtime_pollWait(0x7f2836b70d28, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc000212580?, 0x16b?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000212580)
        /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd
net.(*netFD).accept(0xc000212580)
        /usr/local/go/src/net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc000729140)
        /usr/local/go/src/net/tcpsock_posix.go:148 +0x25
net.(*TCPListener).Accept(0xc000729140)
        /usr/local/go/src/net/tcpsock.go:297 +0x3d
net/http.(*Server).Serve(0xc0003921e0, {0x2452b20, 0xc000729140})
        /usr/local/go/src/net/http/server.go:3059 +0x385
net/http.(*Server).ListenAndServe(0xc0003921e0)
        /usr/local/go/src/net/http/server.go:2988 +0x7d
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.runMetricsServer(0x0?)
        /go/node-feature-discovery/pkg/nfd-master/metrics.go:117 +0x599
created by sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nfdMaster).Run
        /go/node-feature-discovery/pkg/nfd-master/nfd-master.go:251 +0x4e5

goroutine 358363 [semacquire]:
sync.runtime_Semacquire(0x10000c000c02500?)
        /usr/local/go/src/runtime/sema.go:62 +0x27
sync.(*WaitGroup).Wait(0xc000b75838?)
        /usr/local/go/src/sync/waitgroup.go:116 +0x4b
k8s.io/apimachinery/pkg/util/wait.(*Group).Wait(...)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:48
k8s.io/client-go/tools/cache.(*controller).Run(0xc000b757c0, 0xc002acbda0)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:160 +0x35d
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc0003a6580, 0x0?)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:504 +0x224
sigs.k8s.io/node-feature-discovery/pkg/generated/informers/externalversions.(*sharedInformerFactory).Start.func1()
        /go/node-feature-discovery/pkg/generated/informers/externalversions/factory.go:132 +0x6b
created by sigs.k8s.io/node-feature-discovery/pkg/generated/informers/externalversions.(*sharedInformerFactory).Start
        /go/node-feature-discovery/pkg/generated/informers/externalversions/factory.go:130 +0x21b

goroutine 139 [select]:
sigs.k8s.io/node-feature-discovery/pkg/utils.(*FsWatcher).watch(0xc00012c880)
        /go/node-feature-discovery/pkg/utils/fswatcher.go:123 +0xe5
created by sigs.k8s.io/node-feature-discovery/pkg/utils.CreateFsWatcher
        /go/node-feature-discovery/pkg/utils/fswatcher.go:50 +0x10a

goroutine 125 [IO wait]:
internal/poll.runtime_pollWait(0x7f2836b70f08, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc00075c100?, 0xc000888000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00075c100, {0xc000888000, 0xc000, 0xc000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc00075c100, {0xc000888000?, 0xc000888005?, 0x22?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc00074a048, {0xc000888000?, 0x66fda3?, 0xc0007661b0?})
        /usr/local/go/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc000729ba8, {0xc000888000?, 0xc000729ba8?, 0x0?})
        /usr/local/go/src/crypto/tls/conn.go:788 +0x3d
bytes.(*Buffer).ReadFrom(0xc000766290, {0x24339a0, 0xc000729ba8})
        /usr/local/go/src/bytes/buffer.go:202 +0x98
crypto/tls.(*Conn).readFromUntil(0xc000766000, {0x2438d20?, 0xc00074a048}, 0xc000?)
        /usr/local/go/src/crypto/tls/conn.go:810 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc000766000, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:617 +0x116
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0xc000766000, {0xc00048c000, 0x1000, 0xc000078948?})
        /usr/local/go/src/crypto/tls/conn.go:1316 +0x16f
bufio.(*Reader).Read(0xc0007af0e0, {0xc000054580, 0x9, 0x7f285f7005b8?})
        /usr/local/go/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x24337c0, 0xc0007af0e0}, {0xc000054580, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:332 +0x9a
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc000054580?, 0x9?, 0xc00175ee00?}, {0x24337c0?, 0xc0007af0e0?})
        /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000054540)
        /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:498 +0x95
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc00175ef98)
        /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:2275 +0x12e
golang.org/x/net/http2.(*ClientConn).readLoop(0xc000770300)
        /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:2170 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
        /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:821 +0xc1f

goroutine 164 [select, 2815 minutes]:
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nfdMaster).runGrpcServer(0xc0001f5b00, 0x0?)
        /go/node-feature-discovery/pkg/nfd-master/nfd-master.go:357 +0x754
created by sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nfdMaster).Run
        /go/node-feature-discovery/pkg/nfd-master/nfd-master.go:258 +0x58a

goroutine 165 [select]:
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nfdMaster).nfdAPIUpdateHandler(0xc0001f5b00)
        /go/node-feature-discovery/pkg/nfd-master/nfd-master.go:381 +0x149
created by sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nfdMaster).Run
        /go/node-feature-discovery/pkg/nfd-master/nfd-master.go:265 +0x63c

goroutine 175 [IO wait, 2815 minutes]:
internal/poll.runtime_pollWait(0x7f2836b70b48, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0007af9e0?, 0xc000663ed0?, 0x1)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0007af9e0, {0xc000663ed0, 0x10000, 0x10000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299
os.(*File).read(...)
        /usr/local/go/src/os/file_posix.go:31
os.(*File).Read(0xc00074a348, {0xc000663ed0?, 0x0?, 0x0?})
        /usr/local/go/src/os/file.go:118 +0x5e
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc0003b1040)
        /go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:356 +0xdf
created by github.com/fsnotify/fsnotify.NewWatcher
        /go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:150 +0x1b0

goroutine 176 [select, 2815 minutes]:
sigs.k8s.io/node-feature-discovery/pkg/utils.(*FsWatcher).watch(0xc0000217c0)
        /go/node-feature-discovery/pkg/utils/fswatcher.go:123 +0xe5
created by sigs.k8s.io/node-feature-discovery/pkg/utils.CreateFsWatcher
        /go/node-feature-discovery/pkg/utils/fswatcher.go:50 +0x10a

goroutine 177 [IO wait]:
internal/poll.runtime_pollWait(0x7f2836b70c38, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc000212700?, 0x4?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000212700)
        /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd
net.(*netFD).accept(0xc000212700)
        /usr/local/go/src/net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc000729158)
        /usr/local/go/src/net/tcpsock_posix.go:148 +0x25
net.(*TCPListener).Accept(0xc000729158)
        /usr/local/go/src/net/tcpsock.go:297 +0x3d
google.golang.org/grpc.(*Server).Serve(0xc000602000, {0x2452b20?, 0xc000729158})
        /go/pkg/mod/google.golang.org/[email protected]/server.go:852 +0x475
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nfdMaster).runGrpcServer.func1()
        /go/node-feature-discovery/pkg/nfd-master/nfd-master.go:353 +0x85
created by sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nfdMaster).runGrpcServer
        /go/node-feature-discovery/pkg/nfd-master/nfd-master.go:351 +0x69b

goroutine 358441 [select]:
k8s.io/client-go/tools/cache.(*Reflector).list(0xc0008722a0, 0xc00010ec60)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:526 +0x465
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0008722a0, 0xc00010ec60)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:348 +0x22c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:291 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000881dc0?, {0x2438700, 0xc003deb860}, 0x1, 0xc00010ec60)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0008722a0, 0xc00010ec60)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:290 +0x17d
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:55 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 358508 [runnable]:
sigs.k8s.io/node-feature-discovery/pkg/generated/informers/externalversions.(*sharedInformerFactory).Start.func1()
        /go/node-feature-discovery/pkg/generated/informers/externalversions/factory.go:130
created by sigs.k8s.io/node-feature-discovery/pkg/generated/informers/externalversions.(*sharedInformerFactory).Start
        /go/node-feature-discovery/pkg/generated/informers/externalversions/factory.go:130 +0x21b

goroutine 358391 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc001308fd0, 0x14)
        /usr/local/go/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc001ad68c0?)
        /usr/local/go/src/sync/cond.go:70 +0x8c
k8s.io/client-go/util/workqueue.(*Type).Get(0xc00007f1a0)
        /go/pkg/mod/k8s.io/[email protected]/util/workqueue/queue.go:200 +0xa5
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).processNodeUpdateRequest(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:44 +0x62
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).runNodeUpdater(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:67 +0x39
created by sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).start
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:93 +0x3e6

goroutine 358304 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc001308fd0, 0xc)
        /usr/local/go/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc001ad68e0?)
        /usr/local/go/src/sync/cond.go:70 +0x8c
k8s.io/client-go/util/workqueue.(*Type).Get(0xc00007f1a0)
        /go/pkg/mod/k8s.io/[email protected]/util/workqueue/queue.go:200 +0xa5
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).processNodeUpdateRequest(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:44 +0x62
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).runNodeUpdater(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:67 +0x39
created by sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).start
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:93 +0x3e6

goroutine 358365 [chan receive, 2 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc0028a6690, 0xc000602000?)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:803 +0x58
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:55 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 358439 [runnable]:
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 358390 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc001308fd0, 0x10)
        /usr/local/go/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc001ad6930?)
        /usr/local/go/src/sync/cond.go:70 +0x8c
k8s.io/client-go/util/workqueue.(*Type).Get(0xc00007f1a0)
        /go/pkg/mod/k8s.io/[email protected]/util/workqueue/queue.go:200 +0xa5
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).processNodeUpdateRequest(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:44 +0x62
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).runNodeUpdater(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:67 +0x39
created by sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).start
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:93 +0x3e6

goroutine 358392 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc001308fd0, 0xd)
        /usr/local/go/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc001ad68b0?)
        /usr/local/go/src/sync/cond.go:70 +0x8c
k8s.io/client-go/util/workqueue.(*Type).Get(0xc00007f1a0)
        /go/pkg/mod/k8s.io/[email protected]/util/workqueue/queue.go:200 +0xa5
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).processNodeUpdateRequest(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:44 +0x62
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).runNodeUpdater(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:67 +0x39
created by sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).start
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:93 +0x3e6

goroutine 358305 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc001308fd0, 0x13)
        /usr/local/go/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc001ad6910?)
        /usr/local/go/src/sync/cond.go:70 +0x8c
k8s.io/client-go/util/workqueue.(*Type).Get(0xc00007f1a0)
        /go/pkg/mod/k8s.io/[email protected]/util/workqueue/queue.go:200 +0xa5
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).processNodeUpdateRequest(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:44 +0x62
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).runNodeUpdater(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:67 +0x39
created by sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).start
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:93 +0x3e6

goroutine 358379 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:969 +0x57
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc001599f38?, {0x2438720, 0xc0016808d0}, 0x1, 0xc002c58b40)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000652a50?, 0x3b9aca00, 0x0, 0x5?, 0xc001599f88?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0011c8ab0)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:968 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 358388 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc001308fd0, 0xe)
        /usr/local/go/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc001ad68f0?)
        /usr/local/go/src/sync/cond.go:70 +0x8c
k8s.io/client-go/util/workqueue.(*Type).Get(0xc00007f1a0)
        /go/pkg/mod/k8s.io/[email protected]/util/workqueue/queue.go:200 +0xa5
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).processNodeUpdateRequest(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:44 +0x62
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).runNodeUpdater(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:67 +0x39
created by sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).start
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:93 +0x3e6

goroutine 358397 [sync.Mutex.Lock]:
sync.runtime_SemacquireMutex(0x1f71740?, 0x40?, 0x3?)
        /usr/local/go/src/runtime/sema.go:77 +0x26
sync.(*Mutex).lockSlow(0xc00012c740)
        /usr/local/go/src/sync/mutex.go:171 +0x165
sync.(*Mutex).Lock(...)
        /usr/local/go/src/sync/mutex.go:90
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).Stop(0xc00012c740)
        /go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:88 +0x4c
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc00012c740)
        /go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:125 +0x4ac
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
        /go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:76 +0x130

goroutine 358389 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc001308fd0, 0xb)
        /usr/local/go/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc0023749c0?)
        /usr/local/go/src/sync/cond.go:70 +0x8c
k8s.io/client-go/util/workqueue.(*Type).Get(0xc00007f1a0)
        /go/pkg/mod/k8s.io/[email protected]/util/workqueue/queue.go:200 +0xa5
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).processNodeUpdateRequest(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:44 +0x62
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).runNodeUpdater(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:67 +0x39
created by sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).start
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:93 +0x3e6

goroutine 358387 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc001308fd0, 0x12)
        /usr/local/go/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc0023749d0?)
        /usr/local/go/src/sync/cond.go:70 +0x8c
k8s.io/client-go/util/workqueue.(*Type).Get(0xc00007f1a0)
        /go/pkg/mod/k8s.io/[email protected]/util/workqueue/queue.go:200 +0xa5
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).processNodeUpdateRequest(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:44 +0x62
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).runNodeUpdater(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:67 +0x39
created by sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).start
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:93 +0x3e6

goroutine 358367 [runnable]:
golang.org/x/net/http2.(*dataBuffer).Len(0x0?)
        /go/pkg/mod/golang.org/x/[email protected]/http2/databuffer.go:114
golang.org/x/net/http2.(*pipe).closeWithError(0xc000c144b0, 0xc000c14518, {0x2433bc0, 0xc0001f8c90}, 0x9?)
        /go/pkg/mod/golang.org/x/[email protected]/http2/pipe.go:130 +0x164
golang.org/x/net/http2.(*pipe).BreakWithError(...)
        /go/pkg/mod/golang.org/x/[email protected]/http2/pipe.go:107
golang.org/x/net/http2.transportResponseBody.Close({0x408699?})
        /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:2609 +0x5b
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Close(0x0?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/framer/framer.go:169 +0x22
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Close(0xc0019783b8?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/streaming/streaming.go:110 +0x22
k8s.io/client-go/rest/watch.(*Decoder).Close(0xc001978360?)
        /go/pkg/mod/k8s.io/[email protected]/rest/watch/decoder.go:71 +0x22
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).Stop(0xc0014b2d40)
        /go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:95 +0xaa
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc000127960, {0x0?, 0x0?}, 0xc002acbda0, 0xc000b58ca0?)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:435 +0x56e
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000127960, 0xc002acbda0)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:358 +0x34e
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:291 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0015f6340?, {0x2438700, 0xc0028a6780}, 0x1, 0xc002acbda0)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000127960, 0xc002acbda0)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:290 +0x17d
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:55 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 358396 [select, 2 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc000c14480, 0xc002c75c00)
        /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1464 +0xb27
golang.org/x/net/http2.(*clientStream).doRequest(0xc002303fa0?, 0xc002303fb0?)
        /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1326 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
        /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1232 +0x34a

goroutine 358362 [semacquire]:
sync.runtime_Semacquire(0x10000c000bdc700?)
        /usr/local/go/src/runtime/sema.go:62 +0x27
sync.(*WaitGroup).Wait(0xc001537838?)
        /usr/local/go/src/sync/waitgroup.go:116 +0x4b
k8s.io/apimachinery/pkg/util/wait.(*Group).Wait(...)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:48
k8s.io/client-go/tools/cache.(*controller).Run(0xc0015377c0, 0xc002acbda0)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:160 +0x35d
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc0003a64d0, 0x0?)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:504 +0x224
sigs.k8s.io/node-feature-discovery/pkg/generated/informers/externalversions.(*sharedInformerFactory).Start.func1()
        /go/node-feature-discovery/pkg/generated/informers/externalversions/factory.go:132 +0x6b
created by sigs.k8s.io/node-feature-discovery/pkg/generated/informers/externalversions.(*sharedInformerFactory).Start
        /go/node-feature-discovery/pkg/generated/informers/externalversions/factory.go:130 +0x21b

goroutine 358302 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000524720)
        /go/pkg/mod/k8s.io/[email protected]/util/workqueue/delaying_queue.go:276 +0x305
created by k8s.io/client-go/util/workqueue.newDelayingQueue
        /go/pkg/mod/k8s.io/[email protected]/util/workqueue/delaying_queue.go:113 +0x22a

goroutine 358402 [sync.Cond.Wait, 2 minutes]:
sync.runtime_notifyListWait(0xc000c144c8, 0x0)
        /usr/local/go/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0x0?)
        /usr/local/go/src/sync/cond.go:70 +0x8c
golang.org/x/net/http2.(*pipe).Read(0xc000c144b0, {0xc00194e800, 0x200, 0x200})
        /go/pkg/mod/golang.org/x/[email protected]/http2/pipe.go:76 +0xeb
golang.org/x/net/http2.transportResponseBody.Read({0xc003e43c98?}, {0xc00194e800?, 0x417d10?, 0xc00075cf80?})
        /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:2558 +0x75
encoding/json.(*Decoder).refill(0xc0015c6dc0)
        /usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc0015c6dc0)
        /usr/local/go/src/encoding/json/stream.go:140 +0xbb
encoding/json.(*Decoder).Decode(0xc0015c6dc0, {0x1e3d0c0, 0xc000c08420})
        /usr/local/go/src/encoding/json/stream.go:63 +0x78
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc001744ea0, {0xc000e94400, 0x400, 0x400})
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/framer/framer.go:152 +0x1a5
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0028a6960, 0x14d0546?, {0x2440360, 0xc0014b2d80})
        /go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/streaming/streaming.go:77 +0xa7
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000b58f80)
        /go/pkg/mod/k8s.io/[email protected]/rest/watch/decoder.go:49 +0x4f
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc0014b2d40)
        /go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:105 +0xdc
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
        /go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:76 +0x130

goroutine 358394 [select, 2 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0011c8b40)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:939 +0x119
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 358438 [runnable]:
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 358377 [select]:
golang.org/x/net/http2.transportResponseBody.Close({0x408699?})
        /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:2630 +0x232
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Close(0x0?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/framer/framer.go:169 +0x22
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Close(0xc0000e6298?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/streaming/streaming.go:110 +0x22
k8s.io/client-go/rest/watch.(*Decoder).Close(0xc0000e6240?)
        /go/pkg/mod/k8s.io/[email protected]/rest/watch/decoder.go:71 +0x22
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).Stop(0xc00012c740)
        /go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:95 +0xaa
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc000055500, {0x0?, 0x0?}, 0xc002acbda0, 0xc000d5e720?)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:435 +0x56e
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000055500, 0xc002acbda0)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:358 +0x34e
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:291 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00256dcc0?, {0x2438700, 0xc003e76500}, 0x1, 0xc002acbda0)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000055500, 0xc002acbda0)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:290 +0x17d
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:55 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 358386 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc001308fd0, 0x11)
        /usr/local/go/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc001ad6900?)
        /usr/local/go/src/sync/cond.go:70 +0x8c
k8s.io/client-go/util/workqueue.(*Type).Get(0xc00007f1a0)
        /go/pkg/mod/k8s.io/[email protected]/util/workqueue/queue.go:200 +0xa5
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).processNodeUpdateRequest(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:44 +0x62
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).runNodeUpdater(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:67 +0x39
created by sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).start
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:93 +0x3e6

goroutine 358440 [runnable]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:131
created by k8s.io/client-go/tools/cache.(*controller).Run
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:131 +0xc5

goroutine 358303 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc001308fd0, 0xf)
        /usr/local/go/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0xc001ad68d0?)
        /usr/local/go/src/sync/cond.go:70 +0x8c
k8s.io/client-go/util/workqueue.(*Type).Get(0xc00007f1a0)
        /go/pkg/mod/k8s.io/[email protected]/util/workqueue/queue.go:200 +0xa5
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).processNodeUpdateRequest(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:44 +0x62
sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).runNodeUpdater(0xc0004f3710, {0x2461bf8, 0xc001340200})
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:67 +0x39
created by sigs.k8s.io/node-feature-discovery/pkg/nfd-master.(*nodeUpdaterPool).start
        /go/node-feature-discovery/pkg/nfd-master/node-updater-pool.go:93 +0x3e6

goroutine 358375 [chan receive, 2 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc0028a6640, 0xc001597fd0?)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:803 +0x58
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:55 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

goroutine 358507 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0004b6028, 0x0)
        /usr/local/go/src/runtime/sema.go:527 +0x14c
sync.(*Cond).Wait(0x4ffa47?)
        /usr/local/go/src/sync/cond.go:70 +0x8c
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0004b6000, 0xc0008bdfb0)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/delta_fifo.go:575 +0x256
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc00071cfa0)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:188 +0x36
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xa67825?, {0x2438720, 0xc0021d18f0}, 0x1, 0xc00010ec60)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00071d018?, 0x3b9aca00, 0x0, 0x0?, 0x34630b8a000?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc00071cfa0, 0xc00010ec60)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:159 +0x34f
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc002b760b0, 0xc002303fd0?)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:504 +0x224
sigs.k8s.io/node-feature-discovery/pkg/generated/informers/externalversions.(*sharedInformerFactory).Start.func1()
        /go/node-feature-discovery/pkg/generated/informers/externalversions/factory.go:132 +0x6b
created by sigs.k8s.io/node-feature-discovery/pkg/generated/informers/externalversions.(*sharedInformerFactory).Start
        /go/node-feature-discovery/pkg/generated/informers/externalversions/factory.go:130 +0x21b

goroutine 358404 [runnable]:
golang.org/x/net/http2.(*ClientConn).forgetStreamID(0xc000c15b30, 0xc15b88)
        /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:2132
golang.org/x/net/http2.(*clientStream).cleanupWriteRequest(0xc000c15b00, {0x2433bc0, 0xc0001f8c90})
        /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1586 +0x39f
golang.org/x/net/http2.(*clientStream).doRequest(0xc000602000?, 0xc000b4f9f4?)
        /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1327 +0x2e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
        /go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1232 +0x34a

goroutine 358380 [select, 1 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0011c8ab0)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:939 +0x119
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x85

What you expected to happen:
The NFD Master not crashing.

How to reproduce it (as minimally and precisely as possible):
Unfortunately, I don't know how to reproduce it.
I have it running for 11 days, and it crashed 6 times already.

Anything else we need to know?:
From the log of NFD Master, I can see this is encountered after:

I0325 22:34:43.794369       1 nfd-master.go:287] "stopping the nfd api controller"
I0325 22:34:43.794383       1 nfd-master.go:1338] "starting the nfd api controller"
I0325 22:34:43.794548       1 node-updater-pool.go:106] "stopping the NFD master node updater pool"
I0325 22:34:43.794601       1 node-updater-pool.go:81] "starting the NFD master node updater pool" parallelism=10
I0325 22:34:45.509250       1 nfd-master.go:694] "will process all nodes in the cluster"
I0325 22:35:45.833878       1 nfd-master.go:280] "reloading configuration"
I0325 22:35:45.834130       1 nfd-master.go:1214] "configuration file parsed" path="/etc/kubernetes/node-feature-discovery/nfd-master.conf"
I0325 22:35:45.834409       1 nfd-master.go:1274] "configuration successfully updated" configuration=<
        DenyLabelNs: {}
        EnableTaints: false
        ExtraLabelNs:
          nvidia.com: {}
        Klog: {}
        LabelWhiteList: {}
        LeaderElection:
          LeaseDuration:
            Duration: 15000000000
          RenewDeadline:
            Duration: 10000000000
          RetryPeriod:
            Duration: 2000000000
        NfdApiParallelism: 10
        NoPublish: false
        ResourceLabels: {}
        ResyncPeriod:
          Duration: 3600000000000
 >
I0325 22:35:45.834430       1 nfd-master.go:287] "stopping the nfd api controller"
I0325 22:35:45.834446       1 nfd-master.go:1338] "starting the nfd api controller"
fatal error: concurrent map read and map write

Environment:

  • Node Feature Discovery version: 0.14.5
  • Kubernetes version (use kubectl version): 1.24.6
  • Cloud provider or hardware configuration: 4x AMD EPYC 7543 - 128 cores in total
  • OS (e.g: cat /etc/os-release): Ubuntu 22.04.3 LTS (Jammy Jellyfish)
  • Kernel (e.g. uname -a): 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
  • Network plugin and version (if this is a network-related bug):
  • Others: NVIDIA GPU Operator 23.9.2
@tatodorov tatodorov added the kind/bug Categorizes issue or PR as related to a bug. label Mar 27, 2024
@marquiz
Copy link
Contributor

marquiz commented Mar 27, 2024

Thanks @tatodorov for reporting the issue.

I'm looking into it.

@marquiz marquiz mentioned this issue Mar 27, 2024
Merged
@marquiz
Copy link
Contributor

marquiz commented Mar 27, 2024

I believe #1641 should fix the issue. @tatodorov do you have possiblity to run/test a staging image if/when we merge the fix and backport it to v0.14 branch?

@tatodorov
Copy link
Author

@marquiz , thanks for the prompt reply!
Yes, I can test the staging image.
Just provide me the one I should use!

@marquiz
Copy link
Contributor

marquiz commented Mar 28, 2024

@tatodorov now we have a v0.14 staging image with the fix backported

gcr.io/k8s-staging-nfd/node-feature-discovery:v0.14.5-2-g3cbf845

Really appreciated if you could test this and report back. We can then cut new patch releases

This was referenced Mar 28, 2024
Merged
Merged
@tatodorov
Copy link
Author

@tatodorov now we have a v0.14 staging image with the fix backported

gcr.io/k8s-staging-nfd/node-feature-discovery:v0.14.5-2-g3cbf845

Really appreciated if you could test this and report back. We can then cut new patch releases

@marquiz , thank you very much for the update!
I already deployed the new image, but in order to confirm if it fixes the issue, I would prefer to leave it running for a week.
The problem is that I don't know how to reproduce it, and also it doesnt happen too often.
The last time when I faced it was 2 days ago.

@marquiz
Copy link
Contributor

marquiz commented Mar 28, 2024

Thanks @tatodorov 👍 I'd suggest let's wait for a few days. I think most of the NFD team will be having an Easter break until next Tuesday or so. So if we don't have any crashes until then, we'll cut a patch release.

@marquiz
Copy link
Contributor

marquiz commented Apr 2, 2024

@tatodorov anything to report?

@tatodorov
Copy link
Author

@tatodorov anything to report?

@marquiz , I haven't observed any crashes after switching to a staging image with the fix.
From my point of view, seems the problem has been resolved.

Thank you very much for the assistance!

@marquiz
Copy link
Contributor

marquiz commented Apr 5, 2024

Fix merged in master and released in v0.14.6 and v0.15.4
/close

@k8s-ci-robot
Copy link
Contributor

@marquiz: Closing this issue.

In response to this:

Fix merged in master and released in v0.14.6 and v0.15.4
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@marquiz @tatodorov @k8s-ci-robot