Segmentation violation Error with confluent kafka GO sdk v2

[jayeshvaltix]

Description

Team,
This is regarding a crash issue we are facing with confluent Kafka go SDK. Basically, we use GO “github.com/confluentinc/confluent-kafka-go/v2/kafka" SDK to connect to Kafka Admin client and then get all inventory topics name based on group name passed (Later we use this topics name to get all information related to Kafka topics consumer lags and we monitor those lags for each topics).

The. SDK crashes with below error.

SIGSEGV: segmentation violation
PC=0x599c240 m=3 sigcode=1 addr=0x0
signal arrived during cgo execution

This error is with GO confluent Kafka SDK V2.

How to reproduce

create kafka.NewAdminClient

// get all inventory topics name based on group name
describeGroupsResult, err := adminClient.DescribeConsumerGroups(ctx, []string{groupID}, kafka.SetAdminRequestTimeout(2*time.Second))
if err != nil {
log.Printf("Error while calling Describe consumer groups: %v", err)
return nil, err
}
This raise this segmentation error

Checklist

Please provide the following information:

• confluent-kafka-go and librdkafka version (`github.com/confluentinc/confluent-kafka-go/v2/kafka'):
• Apache Kafka broker version:
• Client configuration: ConfigMap{...}
• Operating system: Centos 9
• Provide client logs (

SIGSEGV: segmentation violation
PC=0x599c240 m=3 sigcode=1 addr=0x0
signal arrived during cgo execution

goroutine 990 gp=0xc002534540 m=3 mp=0xc00008d008 [syscall]:
runtime.cgocall(0x58d8a50, 0xc028058550)
/home/ec2-user/go1.22.3/src/runtime/cgocall.go:157 +0x4b fp=0xc028058510 sp=0xc0280584d8 pc=0x40fe0b
github.com/confluentinc/confluent-kafka-go/v2/kafka._Cfunc_rd_kafka_Node_id(0x0)
_cgo_gotypes.go:2382 +0x4b fp=0xc028058550 sp=0xc028058510 pc=0x145e0eb
github.com/confluentinc/confluent-kafka-go/v2/kafka.(*AdminClient).cToConsumerGroupDescriptions.func8(0x0)
/home/ec2-user/valtix-sw/gosrc/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/adminapi.go:853 +0x4d fp=0xc028058598 sp=0xc028058550 pc=0x146980d
[github.com/confluentinc/confluent-kafka-go/v2/kafka.(*AdminClient).cToConsumerGroupDescriptions(0xc028473e30](https://github.com/confluentinc/confluent-kafka-go/v2/kafka.(*AdminClient).cToConsumerGroupDescriptions(0xc028473e30), 0x7f1580058170, 0x1)
/home/ec2-user/valtix-sw/gosrc/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/adminapi.go:853 +0x1f1 fp=0xc028058880 sp=0xc028058598 pc=0x1468bb1
[github.com/confluentinc/confluent-kafka-go/v2/kafka.(*AdminClient).DescribeConsumerGroups(0xc028473e30](https://github.com/confluentinc/confluent-kafka-go/v2/kafka.(*AdminClient).DescribeConsumerGroups(0xc028473e30), {0x7783608, 0x9f39c60}, {0xc028059190, 0x1, 0x1}, {0xc028059180, 0x1, 0x1})
/home/ec2-user/valtix-sw/gosrc/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/adminapi.go:2078 +0x7b5 fp=0xc028058b40 sp=0xc028058880 pc=0x147b6b5
[github.com/valtixinc/valtix-sw/controller/kafkautils.GetConsumerLagForKafkaTopics({0x7783608](https://github.com/valtixinc/valtix-sw/controller/kafkautils.GetConsumerLagForKafkaTopics(%7B0x7783608), 0x9f39c60}, 0xc028473e30, {0xc002342100, 0x10})

• )

• Provide broker log excerpts

• Critical issue Yes

[nkostoulas]

We are also having similar issues with the AdminClient. I'm curious how did you get those logs? We don't get anything unfortunately so I could not debug, possible because we use the Logs() channel?

Also what version are you using? Could this be related to confluentinc/librdkafka#4689 (and fixed in librdkafka v2.5)? I saw it mentions the .NET client crashing but it might be the Go one too.

[nkostoulas]

It doesn't look like librdkafka v2.5 fixes the issue.

[nkostoulas]

Downgrading to v2.3.0 seems to fix our issue.

[Jictyvoo]

I'm facing the same problem. But in my case, I'm only using the package github.com/confluentinc/confluent-kafka-go/v2.

When upgrading from v2.4.0 to v2.5.0/v2.5.3, I started to have these errors occurring on my applications when the container receives the signal to kill the process. I have a graceful shutdown operation to do some stuff before the process finish, but now it just crashes.

I'm able to fix the problem by downgrading the confluent-kafka-go version to 2.4.0 again.

Is this error some kind on the roadmap?

[Jictyvoo]

What I discover here is that my Docker Images are built using musl in a alpine based image with static link.
Here the build command GOOS=linux GOARCH=amd64 go build -ldflags='-w -s -linkmode external -extldflags "-static"' -tags="release musl"

When I build in this way, if I press Ctrl+C on terminal that is running the built binary, the crash happens.
Below some logs to help

SIGSEGV: segmentation violation
PC=0xd95321 m=0 sigcode=1 addr=0x70ca785a4a20
signal arrived during cgo execution

goroutine 1 gp=0xc0000061c0 m=0 mp=0x1e4f5a0 [syscall]:
runtime.cgocall(0xcdde50, 0xc000163388)
        /usr/local/go/src/runtime/cgocall.go:167 +0x4b fp=0xc000163360 sp=0xc000163328 pc=0x46c94b
github.com/confluentinc/confluent-kafka-go/v2/kafka._Cfunc__rk_queue_poll(0x70ca78ae04d0, 0xf1c, 0xc0002566c0, 0xc0002dd5c0, 0x0)
        _cgo_gotypes.go:1039 +0x4c fp=0xc000163388 sp=0xc000163360 pc=0x9d06ec
github.com/confluentinc/confluent-kafka-go/v2/kafka.(*handle).eventPoll.func1(0x13f5f00?, 0xc0001635c8?, 0xc0002566c0, 0xc0002dd5c0, 0xc0000061c0?)
        /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/event.go:168 +0xcb fp=0xc0001633f0 sp=0xc000163388 pc=0x9e102b
github.com/confluentinc/confluent-kafka-go/v2/kafka.(*handle).eventPoll(0xc0000dc5b8, 0x0, 0xf1c, 0xc1ae303f408b6111?, 0x0)
        /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/event.go:168 +0xf6 fp=0xc000163660 sp=0xc0001633f0 pc=0x9dfc96
github.com/confluentinc/confluent-kafka-go/v2/kafka.(*Consumer).Poll(...)
        /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/consumer.go:451
github.com/confluentinc/confluent-kafka-go/v2/kafka.(*Consumer).ReadMessage(0xc0000dc5b0, 0x12a05f200)
        /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/consumer.go:502 +0x17a fp=0xc000163728 sp=0xc000163660 pc=0x9da65a
github.com/myprivateORG/myprivateORG-go/privatekafkalib-wrapper/internal/conflukafka.(*ConfluentEventConsumer).run(0xc000234310)
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/[email protected]/internal/conflukafka/consumer.go:53 +0x3b fp=0xc000163870 sp=0xc000163728 pc=0x9ed29b
github.com/myprivateORG/myprivateORG-go/privatekafkalib-wrapper/internal/conflukafka.(*ConfluentEventConsumer).Listen(0xc000234310, {0xc002402300?, 0xc?, 0x721400?}, {0x15ab960, 0xc002404730})
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/[email protected]/internal/conflukafka/consumer.go:121 +0xec fp=0xc000163950 sp=0xc000163870 pc=0x9ed7cc
github.com/myprivateORG/myprivateORG-go/privatekafkalib-wrapper/thegreatconsumer.(*HighConsumer).Listen(0xc0002f0000, {0x15ab8f0, 0x1e79da0})
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/[email protected]/thegreatconsumer/consumer.go:230 +0x274 fp=0xc000163a70 sp=0xc000163950 pc=0x9f1ff4
main.main()
        /go/src/cordastest/apps/kfkapplication/main.go:38 +0x34f fp=0xc000163f50 sp=0xc000163a70 pc=0xcd9ccf
runtime.main()
        /usr/local/go/src/runtime/proc.go:272 +0x28b fp=0xc000163fe0 sp=0xc000163f50 pc=0x43e8cb
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000163fe8 sp=0xc000163fe0 pc=0x47ae81

goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00008afa8 sp=0xc00008af88 pc=0x472d4e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:337 +0xb3 fp=0xc00008afe0 sp=0xc00008afa8 pc=0x43ec13
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00008afe8 sp=0xc00008afe0 pc=0x47ae81
created by runtime.init.7 in goroutine 1
        /usr/local/go/src/runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00008b780 sp=0xc00008b760 pc=0x472d4e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.bgsweep(0xc00004a080)
        /usr/local/go/src/runtime/mgcsweep.go:317 +0xdf fp=0xc00008b7c8 sp=0xc00008b780 pc=0x42829f
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00008b7e0 sp=0xc00008b7c8 pc=0x41c985
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00008b7e8 sp=0xc00008b7e0 pc=0x47ae81
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000007340 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x1598e08?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00008bf78 sp=0xc00008bf58 pc=0x472d4e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x1e4db60)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00008bfa8 sp=0xc00008bf78 pc=0x425c89
runtime.bgscavenge(0xc00004a080)
        /usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00008bfc8 sp=0xc00008bfa8 pc=0x426219
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00008bfe0 sp=0xc00008bfc8 pc=0x41c925
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00008bfe8 sp=0xc00008bfe0 pc=0x47ae81
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000007c00 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x14a0138?, 0x20?, 0xe2?, 0x2000000020?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00008a620 sp=0xc00008a600 pc=0x472d4e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc00008a7e0 sp=0xc00008a620 pc=0x41ba07
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00008a7e8 sp=0xc00008a7e0 pc=0x47ae81
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:163 +0x3d

goroutine 6 gp=0xc000224e00 m=nil [chan receive]:
runtime.gopark(0xc00008c760?, 0x5bf905?, 0xe0?, 0x33?, 0x15bdb40?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00008c718 sp=0xc00008c6f8 pc=0x472d4e
runtime.chanrecv(0xc000046380, 0x0, 0x1)
        /usr/local/go/src/runtime/chan.go:639 +0x41c fp=0xc00008c790 sp=0xc00008c718 pc=0x40c03c
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:489 +0x12 fp=0xc00008c7b8 sp=0xc00008c790 pc=0x40bbf2
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
        /usr/local/go/src/runtime/mgc.go:1732
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1735 +0x2f fp=0xc00008c7e0 sp=0xc00008c7b8 pc=0x41f98f
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00008c7e8 sp=0xc00008c7e0 pc=0x47ae81
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1730 +0x96

goroutine 35 gp=0xc000225180 m=nil [chan receive]:
runtime.gopark(0x40b64c?, 0x12e0660?, 0x80?, 0x51?, 0xc00008ce60?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00008ce30 sp=0xc00008ce10 pc=0x472d4e
runtime.chanrecv(0xc00029f500, 0xc00008cf88, 0x1)
        /usr/local/go/src/runtime/chan.go:639 +0x41c fp=0xc00008cea8 sp=0xc00008ce30 pc=0x40c03c
runtime.chanrecv2(0x40b500?, 0xc0000061c0?)
        /usr/local/go/src/runtime/chan.go:494 +0x12 fp=0xc00008ced0 sp=0xc00008cea8 pc=0x40bc12
github.com/myprivateORG/myprivateORG-go/privatekafkalib-wrapper.runKafkaMessageDeliveryReport(0xc00029f500, {0x15b4d68, 0xc0002a8370})
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/[email protected]/producer_helpers.go:13 +0x52 fp=0xc00008cfb8 sp=0xc00008ced0 pc=0x9f47f2
github.com/myprivateORG/myprivateORG-go/privatekafkalib-wrapper.NewProducer.gowrap1()
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/[email protected]/producer.go:19 +0x28 fp=0xc00008cfe0 sp=0xc00008cfb8 pc=0x9f4768
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00008cfe8 sp=0xc00008cfe0 pc=0x47ae81
created by github.com/myprivateORG/myprivateORG-go/privatekafkalib-wrapper.NewProducer in goroutine 1
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/[email protected]/producer.go:19 +0x11b

goroutine 36 gp=0xc0002b8fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000086738 sp=0xc000086718 pc=0x472d4e
runtime.gcBgMarkWorker(0xc0002b20e0)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000867c8 sp=0xc000086738 pc=0x41eca9
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0000867e0 sp=0xc0000867c8 pc=0x41eb85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000867e8 sp=0xc0000867e0 pc=0x47ae81
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 37 gp=0xc0002b9180 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000086f38 sp=0xc000086f18 pc=0x472d4e
runtime.gcBgMarkWorker(0xc0002b20e0)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc000086fc8 sp=0xc000086f38 pc=0x41eca9
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc000086fe0 sp=0xc000086fc8 pc=0x41eb85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000086fe8 sp=0xc000086fe0 pc=0x47ae81
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 38 gp=0xc0002b9340 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000087738 sp=0xc000087718 pc=0x472d4e
runtime.gcBgMarkWorker(0xc0002b20e0)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000877c8 sp=0xc000087738 pc=0x41eca9
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0000877e0 sp=0xc0000877c8 pc=0x41eb85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000877e8 sp=0xc0000877e0 pc=0x47ae81
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 39 gp=0xc0002b9500 m=nil [GC worker (idle)]:
runtime.gopark(0x2586b902f07?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000087f38 sp=0xc000087f18 pc=0x472d4e
runtime.gcBgMarkWorker(0xc0002b20e0)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc000087fc8 sp=0xc000087f38 pc=0x41eca9
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc000087fe0 sp=0xc000087fc8 pc=0x41eb85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000087fe8 sp=0xc000087fe0 pc=0x47ae81
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 40 gp=0xc0002b96c0 m=nil [GC worker (idle)]:
runtime.gopark(0x2586b85473c?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000088738 sp=0xc000088718 pc=0x472d4e
runtime.gcBgMarkWorker(0xc0002b20e0)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000887c8 sp=0xc000088738 pc=0x41eca9
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0000887e0 sp=0xc0000887c8 pc=0x41eb85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000887e8 sp=0xc0000887e0 pc=0x47ae81
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 41 gp=0xc0002b9880 m=nil [GC worker (idle)]:
runtime.gopark(0x1e7ba80?, 0x1?, 0x12?, 0x3?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000088f38 sp=0xc000088f18 pc=0x472d4e
runtime.gcBgMarkWorker(0xc0002b20e0)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc000088fc8 sp=0xc000088f38 pc=0x41eca9
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc000088fe0 sp=0xc000088fc8 pc=0x41eb85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000088fe8 sp=0xc000088fe0 pc=0x47ae81
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 42 gp=0xc0002b9a40 m=nil [GC worker (idle)]:
runtime.gopark(0x2586b825d4a?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000089738 sp=0xc000089718 pc=0x472d4e
runtime.gcBgMarkWorker(0xc0002b20e0)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0000897c8 sp=0xc000089738 pc=0x41eca9
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0000897e0 sp=0xc0000897c8 pc=0x41eb85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000897e8 sp=0xc0000897e0 pc=0x47ae81
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 43 gp=0xc0002b9c00 m=nil [GC worker (idle)]:
runtime.gopark(0x2586b84a2c8?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000089f38 sp=0xc000089f18 pc=0x472d4e
runtime.gcBgMarkWorker(0xc0002b20e0)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc000089fc8 sp=0xc000089f38 pc=0x41eca9
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc000089fe0 sp=0xc000089fc8 pc=0x41eb85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000089fe8 sp=0xc000089fe0 pc=0x47ae81
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 44 gp=0xc0002b9dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x2586b8468f8?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc0002fa738 sp=0xc0002fa718 pc=0x472d4e
runtime.gcBgMarkWorker(0xc0002b20e0)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0002fa7c8 sp=0xc0002fa738 pc=0x41eca9
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0002fa7e0 sp=0xc0002fa7c8 pc=0x41eb85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0002fa7e8 sp=0xc0002fa7e0 pc=0x47ae81
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 45 gp=0xc0002fe000 m=nil [GC worker (idle)]:
runtime.gopark(0x2586b835d31?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc0002faf38 sp=0xc0002faf18 pc=0x472d4e
runtime.gcBgMarkWorker(0xc0002b20e0)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0002fafc8 sp=0xc0002faf38 pc=0x41eca9
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0002fafe0 sp=0xc0002fafc8 pc=0x41eb85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0002fafe8 sp=0xc0002fafe0 pc=0x47ae81
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 46 gp=0xc0002fe1c0 m=nil [GC worker (idle)]:
runtime.gopark(0x2586b921142?, 0x3?, 0xc8?, 0xae?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc0002fb738 sp=0xc0002fb718 pc=0x472d4e
runtime.gcBgMarkWorker(0xc0002b20e0)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0002fb7c8 sp=0xc0002fb738 pc=0x41eca9
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0002fb7e0 sp=0xc0002fb7c8 pc=0x41eb85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0002fb7e8 sp=0xc0002fb7e0 pc=0x47ae81
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 47 gp=0xc0002fe380 m=nil [GC worker (idle)]:
runtime.gopark(0x1e7ba80?, 0x1?, 0x31?, 0xdd?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc0002fbf38 sp=0xc0002fbf18 pc=0x472d4e
runtime.gcBgMarkWorker(0xc0002b20e0)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0002fbfc8 sp=0xc0002fbf38 pc=0x41eca9
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0002fbfe0 sp=0xc0002fbfc8 pc=0x41eb85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0002fbfe8 sp=0xc0002fbfe0 pc=0x47ae81
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 48 gp=0xc0002fe540 m=nil [GC worker (idle)]:
runtime.gopark(0x1e7ba80?, 0x1?, 0x50?, 0x8b?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc0002fc738 sp=0xc0002fc718 pc=0x472d4e
runtime.gcBgMarkWorker(0xc0002b20e0)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0002fc7c8 sp=0xc0002fc738 pc=0x41eca9
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0002fc7e0 sp=0xc0002fc7c8 pc=0x41eb85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0002fc7e8 sp=0xc0002fc7e0 pc=0x47ae81
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 49 gp=0xc0002fe700 m=nil [GC worker (idle)]:
runtime.gopark(0x2586b8464ca?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc0002fcf38 sp=0xc0002fcf18 pc=0x472d4e
runtime.gcBgMarkWorker(0xc0002b20e0)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0002fcfc8 sp=0xc0002fcf38 pc=0x41eca9
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0002fcfe0 sp=0xc0002fcfc8 pc=0x41eb85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0002fcfe8 sp=0xc0002fcfe0 pc=0x47ae81
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 50 gp=0xc0002fe8c0 m=nil [GC worker (idle)]:
runtime.gopark(0x2586b90252f?, 0x3?, 0x27?, 0xe1?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc0002fd738 sp=0xc0002fd718 pc=0x472d4e
runtime.gcBgMarkWorker(0xc0002b20e0)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0002fd7c8 sp=0xc0002fd738 pc=0x41eca9
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0002fd7e0 sp=0xc0002fd7c8 pc=0x41eb85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0002fd7e8 sp=0xc0002fd7e0 pc=0x47ae81
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 51 gp=0xc0002fea80 m=nil [GC worker (idle)]:
runtime.gopark(0x2586b8465ec?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc0002fdf38 sp=0xc0002fdf18 pc=0x472d4e
runtime.gcBgMarkWorker(0xc0002b20e0)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0002fdfc8 sp=0xc0002fdf38 pc=0x41eca9
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0002fdfe0 sp=0xc0002fdfc8 pc=0x41eb85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0002fdfe8 sp=0xc0002fdfe0 pc=0x47ae81
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 20 gp=0xc001384380 m=9 mp=0xc001300008 [syscall]:
runtime.cgocall(0xcdde50, 0xc00009fca8)
        /usr/local/go/src/runtime/cgocall.go:167 +0x4b fp=0xc00009fc80 sp=0xc00009fc48 pc=0x46c94b
github.com/confluentinc/confluent-kafka-go/v2/kafka._Cfunc__rk_queue_poll(0x70cabf403d70, 0x64, 0xc00029d510, 0xc000123c20, 0x0)
        _cgo_gotypes.go:1039 +0x4c fp=0xc00009fca8 sp=0xc00009fc80 pc=0x9d06ec
github.com/confluentinc/confluent-kafka-go/v2/kafka.(*handle).eventPoll.func1(0x0?, 0x0?, 0xc00029d510, 0xc000123c20, 0x0?)
        /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/event.go:168 +0xcb fp=0xc00009fd10 sp=0xc00009fca8 pc=0x9e102b
github.com/confluentinc/confluent-kafka-go/v2/kafka.(*handle).eventPoll(0xc0002f20d0, 0xc00029f570, 0x64, 0x0?, 0xc0000461c0)
        /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/event.go:168 +0xf6 fp=0xc00009ff80 sp=0xc00009fd10 pc=0x9dfc96
github.com/confluentinc/confluent-kafka-go/v2/kafka.poller(0xc0002f20c0, 0xc0000461c0)
        /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/producer.go:688 +0x4d fp=0xc00009ffb8 sp=0xc00009ff80 pc=0x9e9d4d
github.com/confluentinc/confluent-kafka-go/v2/kafka.NewProducer.func5()
        /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/producer.go:594 +0x25 fp=0xc00009ffe0 sp=0xc00009ffb8 pc=0x9e9545
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00009ffe8 sp=0xc00009ffe0 pc=0x47ae81
created by github.com/confluentinc/confluent-kafka-go/v2/kafka.NewProducer in goroutine 1
        /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/producer.go:593 +0x68f

goroutine 21 gp=0xc001384540 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc0002f7ec0 sp=0xc0002f7ea0 pc=0x472d4e
runtime.chanrecv(0xc00029f5e0, 0xc0002f7f88, 0x1)
        /usr/local/go/src/runtime/chan.go:639 +0x41c fp=0xc0002f7f38 sp=0xc0002f7ec0 pc=0x40c03c
runtime.chanrecv2(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:494 +0x12 fp=0xc0002f7f60 sp=0xc0002f7f38 pc=0x40bc12
github.com/confluentinc/confluent-kafka-go/v2/kafka.channelProducer(0xc0002f20c0)
        /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/producer.go:617 +0x3d fp=0xc0002f7fb8 sp=0xc0002f7f60 pc=0x9e97bd
github.com/confluentinc/confluent-kafka-go/v2/kafka.NewProducer.func6()
        /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/producer.go:608 +0x22 fp=0xc0002f7fe0 sp=0xc0002f7fb8 pc=0x9e94e2
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0002f7fe8 sp=0xc0002f7fe0 pc=0x47ae81
created by github.com/confluentinc/confluent-kafka-go/v2/kafka.NewProducer in goroutine 1
        /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/producer.go:607 +0x6ff

goroutine 22 gp=0xc001384700 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc0002f86b0 sp=0xc0002f8690 pc=0x472d4e
runtime.chanrecv(0xc0002660e0, 0xc0002f87a8, 0x1)
        /usr/local/go/src/runtime/chan.go:639 +0x41c fp=0xc0002f8728 sp=0xc0002f86b0 pc=0x40c03c
runtime.chanrecv2(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:494 +0x12 fp=0xc0002f8750 sp=0xc0002f8728 pc=0x40bc12
github.com/signalfx/splunk-otel-go/instrumentation/github.com/confluentinc/confluent-kafka-go/v2/kafka/splunkkafka.(*Producer).traceProduceChannel.func1()
        /go/pkg/mod/github.com/signalfx/splunk-otel-go/instrumentation/github.com/confluentinc/confluent-kafka-go/v2/kafka/[email protected]/producer.go:83 +0xd8 fp=0xc0002f87e0 sp=0xc0002f8750 pc=0xa0e7f8
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0002f87e8 sp=0xc0002f87e0 pc=0x47ae81
created by github.com/signalfx/splunk-otel-go/instrumentation/github.com/confluentinc/confluent-kafka-go/v2/kafka/splunkkafka.(*Producer).traceProduceChannel in goroutine 1
        /go/pkg/mod/github.com/signalfx/splunk-otel-go/instrumentation/github.com/confluentinc/confluent-kafka-go/v2/kafka/[email protected]/producer.go:81 +0xb9

goroutine 23 gp=0xc001384a80 m=nil [select]:
runtime.gopark(0xc00013ff30?, 0x3?, 0xd0?, 0xa2?, 0xc00013fef2?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00013fd80 sp=0xc00013fd60 pc=0x472d4e
runtime.selectgo(0xc00013ff30, 0xc00013feec, 0xc0002a8000?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:335 +0x7a5 fp=0xc00013fea8 sp=0xc00013fd80 pc=0x4505c5
go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0xc0002b66e0)
        /go/pkg/mod/go.opentelemetry.io/otel/[email protected]/trace/batch_span_processor.go:301 +0x114 fp=0xc00013ffa0 sp=0xc00013fea8 pc=0xa1e0f4
go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1()
        /go/pkg/mod/go.opentelemetry.io/otel/[email protected]/trace/batch_span_processor.go:117 +0x4e fp=0xc00013ffe0 sp=0xc00013ffa0 pc=0xa1d3ae
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00013ffe8 sp=0xc00013ffe0 pc=0x47ae81
created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor in goroutine 1
        /go/pkg/mod/go.opentelemetry.io/otel/[email protected]/trace/batch_span_processor.go:115 +0x2e5

goroutine 24 gp=0xc001385340 m=nil [sync.Cond.Wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc002357d08 sp=0xc002357ce8 pc=0x472d4e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
sync.runtime_notifyListWait(0xc000138410, 0x2)
        /usr/local/go/src/runtime/sema.go:587 +0x159 fp=0xc002357d58 sp=0xc002357d08 pc=0x474279
sync.(*Cond).Wait(0xa3?)
        /usr/local/go/src/sync/cond.go:71 +0x85 fp=0xc002357d98 sp=0xc002357d58 pc=0x485da5
go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*pool).createConnections.func2()
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/pool.go:1156 +0xcb fp=0xc002357e38 sp=0xc002357d98 pc=0xc20f2b
go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*pool).createConnections(0xc0002ba180, {0x15ab960, 0xc00024e410}, 0x0?)
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/pool.go:1178 +0x104 fp=0xc002357fb0 sp=0xc002357e38 pc=0xc208c4
go.mongodb.org/mongo-driver/x/mongo/driver/topology.newPool.gowrap1()
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/pool.go:241 +0x2c fp=0xc002357fe0 sp=0xc002357fb0 pc=0xc1b0ec
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc002357fe8 sp=0xc002357fe0 pc=0x47ae81
created by go.mongodb.org/mongo-driver/x/mongo/driver/topology.newPool in goroutine 1
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/pool.go:241 +0x6dc

goroutine 25 gp=0xc001385500 m=nil [sync.Cond.Wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc0002f9d08 sp=0xc0002f9ce8 pc=0x472d4e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
sync.runtime_notifyListWait(0xc000138410, 0x1)
        /usr/local/go/src/runtime/sema.go:587 +0x159 fp=0xc0002f9d58 sp=0xc0002f9d08 pc=0x474279
sync.(*Cond).Wait(0x0?)
        /usr/local/go/src/sync/cond.go:71 +0x85 fp=0xc0002f9d98 sp=0xc0002f9d58 pc=0x485da5
go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*pool).createConnections.func2()
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/pool.go:1156 +0xcb fp=0xc0002f9e38 sp=0xc0002f9d98 pc=0xc20f2b
go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*pool).createConnections(0xc0002ba180, {0x15ab960, 0xc00024e410}, 0x0?)
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/pool.go:1178 +0x104 fp=0xc0002f9fb0 sp=0xc0002f9e38 pc=0xc208c4
go.mongodb.org/mongo-driver/x/mongo/driver/topology.newPool.gowrap1()
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/pool.go:241 +0x2c fp=0xc0002f9fe0 sp=0xc0002f9fb0 pc=0xc1b0ec
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0002f9fe8 sp=0xc0002f9fe0 pc=0x47ae81
created by go.mongodb.org/mongo-driver/x/mongo/driver/topology.newPool in goroutine 1
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/pool.go:241 +0x6dc

goroutine 26 gp=0xc0013856c0 m=nil [select]:
runtime.gopark(0xc000271f28?, 0x3?, 0x8?, 0x31?, 0xc000271ec2?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000271d50 sp=0xc000271d30 pc=0x472d4e
runtime.selectgo(0xc000271f28, 0xc000271ebc, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:335 +0x7a5 fp=0xc000271e78 sp=0xc000271d50 pc=0x4505c5
go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*pool).maintain(0xc0002ba180, {0x15ab960, 0xc00024e410}, 0x0?)
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/pool.go:1290 +0x1d0 fp=0xc000271fb0 sp=0xc000271e78 pc=0xc21570
go.mongodb.org/mongo-driver/x/mongo/driver/topology.newPool.gowrap2()
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/pool.go:248 +0x2c fp=0xc000271fe0 sp=0xc000271fb0 pc=0xc1b08c
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000271fe8 sp=0xc000271fe0 pc=0x47ae81
created by go.mongodb.org/mongo-driver/x/mongo/driver/topology.newPool in goroutine 1
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/pool.go:248 +0x857

goroutine 27 gp=0xc001385880 m=nil [IO wait]:
runtime.gopark(0x10?, 0x30?, 0x78?, 0xb3?, 0xb?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc0013cd738 sp=0xc0013cd718 pc=0x472d4e
runtime.netpollblock(0x491618?, 0x409446?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:575 +0xf7 fp=0xc0013cd770 sp=0xc0013cd738 pc=0x4371f7
internal/poll.runtime_pollWait(0x70cabf4506b8, 0x72)
        /usr/local/go/src/runtime/netpoll.go:351 +0x85 fp=0xc0013cd790 sp=0xc0013cd770 pc=0x472045
internal/poll.(*pollDesc).wait(0xc0002ae480?, 0xc000256408?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0013cd7b8 sp=0xc0013cd790 pc=0x51f047
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0002ae480, {0xc000256408, 0x4, 0x4})
        /usr/local/go/src/internal/poll/fd_unix.go:165 +0x27a fp=0xc0013cd850 sp=0xc0013cd7b8 pc=0x52033a
net.(*netFD).Read(0xc0002ae480, {0xc000256408?, 0x4724e5?, 0x70cabf450700?})
        /usr/local/go/src/net/fd_posix.go:55 +0x25 fp=0xc0013cd898 sp=0xc0013cd850 pc=0x5e3f45
net.(*conn).Read(0xc00008e128, {0xc000256408?, 0x25d4f39bfee?, 0xc001300708?})
        /usr/local/go/src/net/net.go:189 +0x45 fp=0xc0013cd8e0 sp=0xc0013cd898 pc=0x5f3b45
net.(*TCPConn).Read(0x41067f?, {0xc000256408?, 0x14176e0?, 0x412901?})
        <autogenerated>:1 +0x25 fp=0xc0013cd910 sp=0xc0013cd8e0 pc=0x609505
io.ReadAtLeast({0x15a16c0, 0xc00008e128}, {0xc000256408, 0x4, 0x4}, 0x4)
        /usr/local/go/src/io/io.go:335 +0x90 fp=0xc0013cd958 sp=0xc0013cd910 pc=0x4b97b0
io.ReadFull(...)
        /usr/local/go/src/io/io.go:354
go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*connection).read(0xc00025a588, {0x15ab960, 0xc0000c5ae0})
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/connection.go:467 +0x1f3 fp=0xc0013cda50 sp=0xc0013cd958 pc=0xc13433
go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*connection).readWireMessage(0xc00025a588, {0x15ab960, 0xc0000c5ae0})
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/connection.go:420 +0x1ad fp=0xc0013cdb60 sp=0xc0013cda50 pc=0xc12ead
go.mongodb.org/mongo-driver/x/mongo/driver/topology.initConnection.ReadWireMessage({0xc000036270?}, {0x15ab960?, 0xc0000c5ae0?})
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/connection.go:589 +0x1d fp=0xc0013cdb88 sp=0xc0013cdb60 pc=0xc13d9d
go.mongodb.org/mongo-driver/x/mongo/driver.Operation.readWireMessage({0xc000236270, {0x144009d, 0x5}, {0x15a80d8, 0xc000236250}, 0xc000236290, {0x0, 0x0}, 0x0, 0x0, ...}, ...)
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/operation.go:1078 +0x52 fp=0xc0013cdcf8 sp=0xc0013cdb88 pc=0xb88f72
go.mongodb.org/mongo-driver/x/mongo/driver.Operation.roundTrip({0xc000236270, {0x144009d, 0x5}, {0x15a80d8, 0xc000236250}, 0xc000236290, {0x0, 0x0}, 0x0, 0x0, ...}, ...)
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/operation.go:1074 +0xf8 fp=0xc0013cde18 sp=0xc0013cdcf8 pc=0xb88e98
go.mongodb.org/mongo-driver/x/mongo/driver.Operation.roundTrip-fm({0x15ab960?, 0xc0000c5ae0?}, {0x15b28a0?, 0xc00025a588?}, {0xc00248c000?, 0xc000236290?, 0x0?})
        <autogenerated>:1 +0x7f fp=0xc0013cdf50 sp=0xc0013cde18 pc=0xb9221f
go.mongodb.org/mongo-driver/x/mongo/driver.Operation.Execute({0xc000236270, {0x144009d, 0x5}, {0x15a80d8, 0xc000236250}, 0xc000236290, {0x0, 0x0}, 0x0, 0x0, ...}, ...)
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/operation.go:807 +0x1932 fp=0xc0013cece8 sp=0xc0013cdf50 pc=0xb862d2
go.mongodb.org/mongo-driver/x/mongo/driver/operation.(*Hello).Execute(0xc0002526e0, {0x15ab960, 0xc0000c5ae0})
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/operation/hello.go:566 +0x188 fp=0xc0013ceee8 sp=0xc0013cece8 pc=0xbb4e88
go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*Server).check(_)
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/server.go:881 +0x754 fp=0xc0013cf6b8 sp=0xc0013ceee8 pc=0xc29954
go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*Server).update(0xc000124120)
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/server.go:605 +0x29a fp=0xc0013cffc8 sp=0xc0013cf6b8 pc=0xc271fa
go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*Server).Connect.gowrap1()
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/server.go:252 +0x25 fp=0xc0013cffe0 sp=0xc0013cffc8 pc=0xc24f45
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0013cffe8 sp=0xc0013cffe0 pc=0x47ae81
created by go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*Server).Connect in goroutine 1
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/server.go:252 +0x187

goroutine 15 gp=0xc0013848c0 m=nil [select, locked to thread]:
runtime.gopark(0xc0013c07a8?, 0x2?, 0x2?, 0x0?, 0xc0013c0794?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc0013c0630 sp=0xc0013c0610 pc=0x472d4e
runtime.selectgo(0xc0013c07a8, 0xc0013c0790, 0x0?, 0x0, 0x100010000?, 0x1)
        /usr/local/go/src/runtime/select.go:335 +0x7a5 fp=0xc0013c0758 sp=0xc0013c0630 pc=0x4505c5
runtime.ensureSigM.func1()
        /usr/local/go/src/runtime/signal_unix.go:1060 +0x19f fp=0xc0013c07e0 sp=0xc0013c0758 pc=0x46a5bf
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0013c07e8 sp=0xc0013c07e0 pc=0x47ae81
created by runtime.ensureSigM in goroutine 61
        /usr/local/go/src/runtime/signal_unix.go:1043 +0xc8

goroutine 56 gp=0xc001384c40 m=nil [select]:
runtime.gopark(0xc0013bff80?, 0x2?, 0x8?, 0x31?, 0xc0013bff44?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc002499de8 sp=0xc002499dc8 pc=0x472d4e
runtime.selectgo(0xc002499f80, 0xc0013bff40, 0x1e4d9c0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:335 +0x7a5 fp=0xc002499f10 sp=0xc002499de8 pc=0x4505c5
github.com/myprivateORG/myprivateORG-go/dbdrivers/mongodb-wrapper/connpool.ScheduleConnectionClose({0x15a1fc0, 0xc00025e358}, {0x15ab960, 0xc0024042d0})
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/dbdrivers/[email protected]/connpool/close_scheduler.go:19 +0xe6 fp=0xc002499fb0 sp=0xc002499f10 pc=0xc73c66
github.com/myprivateORG/myprivateORG-go/dbdrivers/mongodb-wrapper.(*ManagerDB[...]).Connect.gowrap1()
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/dbdrivers/[email protected]/mongo.go:131 +0x2c fp=0xc002499fe0 sp=0xc002499fb0 pc=0xcb9c4c
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc002499fe8 sp=0xc002499fe0 pc=0x47ae81
created by github.com/myprivateORG/myprivateORG-go/dbdrivers/mongodb-wrapper.(*ManagerDB[...]).Connect in goroutine 1
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/dbdrivers/[email protected]/mongo.go:131 +0x316

goroutine 10 gp=0xc0002fefc0 m=nil [select]:
runtime.gopark(0xc000053ea8?, 0x2?, 0x60?, 0x0?, 0xc000053e5c?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000053cf8 sp=0xc000053cd8 pc=0x472d4e
runtime.selectgo(0xc000053ea8, 0xc000053e58, 0xc00003e588?, 0x0, 0xc000256450?, 0x1)
        /usr/local/go/src/runtime/select.go:335 +0x7a5 fp=0xc000053e20 sp=0xc000053cf8 pc=0x4505c5
go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*rttMonitor).runHellos(0xc00013a1b0, 0xc00025a848)
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/rtt_monitor.go:156 +0x116 fp=0xc000053ef0 sp=0xc000053e20 pc=0xc23456
go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*rttMonitor).start(0xc00013a1b0)
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/rtt_monitor.go:130 +0xfc fp=0xc000053fa8 sp=0xc000053ef0 pc=0xc2319c
go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*rttMonitor).connect.func1()
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/rtt_monitor.go:90 +0x49 fp=0xc000053fe0 sp=0xc000053fa8 pc=0xc22e69
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000053fe8 sp=0xc000053fe0 pc=0x47ae81
created by go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*rttMonitor).connect in goroutine 27
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/rtt_monitor.go:87 +0xc5

goroutine 12 gp=0xc0002ff180 m=nil [select]:
runtime.gopark(0xc00248a780?, 0x2?, 0x0?, 0x0?, 0xc00248a77c?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00248a620 sp=0xc00248a600 pc=0x472d4e
runtime.selectgo(0xc00248a780, 0xc00248a778, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:335 +0x7a5 fp=0xc00248a748 sp=0xc00248a620 pc=0x4505c5
go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*cancellListener).Listen(0xc000036270, {0x15ab960, 0xc0000c5ae0}, 0xc000236310)
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/connection.go:887 +0x72 fp=0xc00248a7b0 sp=0xc00248a748 pc=0xc16032
go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*connection).read.gowrap1()
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/connection.go:448 +0x31 fp=0xc00248a7e0 sp=0xc00248a7b0 pc=0xc13971
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00248a7e8 sp=0xc00248a7e0 pc=0x47ae81
created by go.mongodb.org/mongo-driver/x/mongo/driver/topology.(*connection).read in goroutine 27
        /go/pkg/mod/go.mongodb.org/[email protected]/x/mongo/driver/topology/connection.go:448 +0x150

goroutine 60 gp=0xc000104a80 m=nil [IO wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00026fbb8 sp=0xc00026fb98 pc=0x472d4e
runtime.netpollblock(0x10?, 0x409446?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:575 +0xf7 fp=0xc00026fbf0 sp=0xc00026fbb8 pc=0x4371f7
internal/poll.runtime_pollWait(0x70cabf4503a0, 0x72)
        /usr/local/go/src/runtime/netpoll.go:351 +0x85 fp=0xc00026fc10 sp=0xc00026fbf0 pc=0x472045
internal/poll.(*pollDesc).wait(0xc00005ac80?, 0x10?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc00026fc38 sp=0xc00026fc10 pc=0x51f047
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00005ac80)
        /usr/local/go/src/internal/poll/fd_unix.go:620 +0x295 fp=0xc00026fce0 sp=0xc00026fc38 pc=0x524415
net.(*netFD).accept(0xc00005ac80)
        /usr/local/go/src/net/fd_unix.go:172 +0x29 fp=0xc00026fd98 sp=0xc00026fce0 pc=0x5e5f09
net.(*TCPListener).accept(0xc000138500)
        /usr/local/go/src/net/tcpsock_posix.go:159 +0x1e fp=0xc00026fde8 sp=0xc00026fd98 pc=0x5fcb5e
net.(*TCPListener).Accept(0xc000138500)
        /usr/local/go/src/net/tcpsock.go:372 +0x30 fp=0xc00026fe18 sp=0xc00026fde8 pc=0x5fbd30
net/http.(*onceCloseListener).Accept(0x15ab8f0?)
        <autogenerated>:1 +0x24 fp=0xc00026fe30 sp=0xc00026fe18 pc=0x74b244
net/http.(*Server).Serve(0xc0000ec690, {0x15a9cc0, 0xc000138500})
        /usr/local/go/src/net/http/server.go:3330 +0x30c fp=0xc00026ff60 sp=0xc00026fe30 pc=0x72236c
net/http.(*Server).ListenAndServe(0xc0000ec690)
        /usr/local/go/src/net/http/server.go:3259 +0x71 fp=0xc00026ff90 sp=0xc00026ff60 pc=0x722031
main.main.func1()
        /go/src/cordastest/apps/kfkapplication/main.go:31 +0x2d fp=0xc00026ffe0 sp=0xc00026ff90 pc=0xcd9e8d
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00026ffe8 sp=0xc00026ffe0 pc=0x47ae81
created by main.main in goroutine 1
        /go/src/cordastest/apps/kfkapplication/main.go:30 +0x286

goroutine 61 gp=0xc000104c40 m=nil [semacquire]:
runtime.gopark(0x1e58dc0?, 0xc0002f0088?, 0x80?, 0x7?, 0x70ca785a4000?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000141dd8 sp=0xc000141db8 pc=0x472d4e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.semacquire1(0xc0002f0088, 0x0, 0x1, 0x0, 0x12)
        /usr/local/go/src/runtime/sema.go:178 +0x225 fp=0xc000141e40 sp=0xc000141dd8 pc=0x451665
sync.runtime_Semacquire(0xc000141e90?)
        /usr/local/go/src/runtime/sema.go:71 +0x25 fp=0xc000141e78 sp=0xc000141e40 pc=0x473f05
sync.(*WaitGroup).Wait(0xc00029a6c0?)
        /usr/local/go/src/sync/waitgroup.go:118 +0x48 fp=0xc000141ea0 sp=0xc000141e78 pc=0x489448
github.com/myprivateORG/myprivateORG-go/privatekafkalib-wrapper/thegreatconsumer.(*HighConsumer).Shutdown(0xc0002f0000)
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/[email protected]/thegreatconsumer/consumer.go:261 +0x55 fp=0xc000141ec8 sp=0xc000141ea0 pc=0x9f24d5
github.com/myprivateORG/cordastest/kfkapplication/bootstrap.GracefulShutdown({0x15ac010, 0xc0002f0000}, 0xc0000ec690, 0xc0002344d0, {0x15b4d68, 0xc0002a8370})
        /go/src/cordastest/apps/kfkapplication/bootstrap/shutdown.go:27 +0x1c2 fp=0xc000141fa0 sp=0xc000141ec8 pc=0xcd8102
main.main.gowrap1()
        /go/src/cordastest/apps/kfkapplication/main.go:37 +0x34 fp=0xc000141fe0 sp=0xc000141fa0 pc=0xcd9e34
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000141fe8 sp=0xc000141fe0 pc=0x47ae81
created by main.main in goroutine 1
        /go/src/cordastest/apps/kfkapplication/main.go:37 +0x328

goroutine 62 gp=0xc000104e00 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc002486570 sp=0xc002486550 pc=0x472d4e
runtime.chanrecv(0xc000234380, 0xc002486710, 0x1)
        /usr/local/go/src/runtime/chan.go:639 +0x41c fp=0xc0024865e8 sp=0xc002486570 pc=0x40c03c
runtime.chanrecv2(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:494 +0x12 fp=0xc002486610 sp=0xc0024865e8 pc=0x40bc12
github.com/myprivateORG/myprivateORG-go/privatekafkalib-wrapper/thegreatconsumer.(*HighConsumer).consumeTopic(0xc0002f0000, {0x1452fea, 0x18}, 0xc0000db560, 0xc000234380, {0xc0024022c0, 0x4, 0x4})
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/[email protected]/thegreatconsumer/consumer_async.go:14 +0xed fp=0xc002486790 sp=0xc002486610 pc=0x9f260d
github.com/myprivateORG/myprivateORG-go/privatekafkalib-wrapper/thegreatconsumer.(*HighConsumer).initThreadsByTopic.gowrap1()
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/[email protected]/thegreatconsumer/consumer.go:200 +0x3c fp=0xc0024867e0 sp=0xc002486790 pc=0x9f1d5c
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0024867e8 sp=0xc0024867e0 pc=0x47ae81
created by github.com/myprivateORG/myprivateORG-go/privatekafkalib-wrapper/thegreatconsumer.(*HighConsumer).initThreadsByTopic in goroutine 1
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/[email protected]/thegreatconsumer/consumer.go:200 +0x31d

goroutine 63 gp=0xc000104fc0 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc002486d70 sp=0xc002486d50 pc=0x472d4e
runtime.chanrecv(0xc0002343f0, 0xc002486f10, 0x1)
        /usr/local/go/src/runtime/chan.go:639 +0x41c fp=0xc002486de8 sp=0xc002486d70 pc=0x40c03c
runtime.chanrecv2(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:494 +0x12 fp=0xc002486e10 sp=0xc002486de8 pc=0x40bc12
github.com/myprivateORG/myprivateORG-go/privatekafkalib-wrapper/thegreatconsumer.(*HighConsumer).consumeTopic(0xc0002f0000, {0x1451cb7, 0x17}, 0xc0000db620, 0xc0002343f0, {0xc0024022e0, 0x4, 0x4})
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/[email protected]/thegreatconsumer/consumer_async.go:14 +0xed fp=0xc002486f90 sp=0xc002486e10 pc=0x9f260d
github.com/myprivateORG/myprivateORG-go/privatekafkalib-wrapper/thegreatconsumer.(*HighConsumer).initThreadsByTopic.gowrap1()
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/[email protected]/thegreatconsumer/consumer.go:200 +0x3c fp=0xc002486fe0 sp=0xc002486f90 pc=0x9f1d5c
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc002486fe8 sp=0xc002486fe0 pc=0x47ae81
created by github.com/myprivateORG/myprivateORG-go/privatekafkalib-wrapper/thegreatconsumer.(*HighConsumer).initThreadsByTopic in goroutine 1
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/[email protected]/thegreatconsumer/consumer.go:200 +0x31d

goroutine 64 gp=0xc000105180 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc002487508 sp=0xc0024874e8 pc=0x472d4e
runtime.chanrecv(0xc0002342a0, 0xc002487730, 0x1)
        /usr/local/go/src/runtime/chan.go:639 +0x41c fp=0xc002487580 sp=0xc002487508 pc=0x40c03c
runtime.chanrecv2(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:494 +0x12 fp=0xc0024875a8 sp=0xc002487580 pc=0x40bc12
github.com/myprivateORG/myprivateORG-go/privatekafkalib-wrapper/thegreatconsumer.(*HighConsumer).consumeMessages(0xc0002f0000)
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/[email protected]/thegreatconsumer/consumer.go:172 +0x8a fp=0xc002487798 sp=0xc0024875a8 pc=0x9f166a
github.com/myprivateORG/myprivateORG-go/privatekafkalib-wrapper/thegreatconsumer.(*HighConsumer).Listen.func1()
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/[email protected]/thegreatconsumer/consumer.go:226 +0x5e fp=0xc0024877e0 sp=0xc002487798 pc=0x9f237e
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0024877e8 sp=0xc0024877e0 pc=0x47ae81
created by github.com/myprivateORG/myprivateORG-go/privatekafkalib-wrapper/thegreatconsumer.(*HighConsumer).Listen in goroutine 1
        /go/pkg/mod/github.com/myprivateORG/myprivateORG-go/[email protected]/thegreatconsumer/consumer.go:224 +0x125

goroutine 102 gp=0xc001385c00 m=17 mp=0xc000272008 [syscall]:
runtime.notetsleepg(0x1e7ac80, 0xffffffffffffffff)
        /usr/local/go/src/runtime/lock_futex.go:246 +0x29 fp=0xc0013c0fa0 sp=0xc0013c0f78 pc=0x4118a9
os/signal.signal_recv()
        /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc0013c0fc0 sp=0xc0013c0fa0 pc=0x474a49
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc0013c0fe0 sp=0xc0013c0fc0 pc=0xcd5533
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0013c0fe8 sp=0xc0013c0fe0 pc=0x47ae81
created by os/signal.Notify.func1.1 in goroutine 61
        /usr/local/go/src/os/signal/signal.go:151 +0x1f

rax    0x0
rbx    0x70ca785a4010
rcx    0x2
rdx    0x70cabf746e00
rdi    0x70cabf404940
rsi    0x0
rbp    0x0
rsp    0x7ffd20cd3ab0
r8     0x0
r9     0x0
r10    0x7ffd20cd39e0
r11    0x212
r12    0x4
r13    0x70cabf404940
r14    0x0
r15    0x10
rip    0xd95321
rflags 0x10202
cs     0x33
fs     0x0
gs     0x0

[nkostoulas]

Still an issue in 2.6.1 it seems 😞 @Jictyvoo you sure it works in 2.4.0? I think I had issues with that release too.

Has anyone found any workaround other than downgrading?