Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support the ROLE command so that Redis.Sentinel is fully supported. #686

Closed
Kilowhisky opened this issue May 10, 2023 · 6 comments
Closed

Support the ROLE command so that Redis.Sentinel is fully supported. #686

Kilowhisky opened this issue May 10, 2023 · 6 comments

Comments

@Kilowhisky
Copy link
Contributor

Kilowhisky commented May 10, 2023 •
edited
Loading

Is your feature request related to a problem? Please describe.
Redis Sentinel is a popular way to handle high availability with Redis. It also happens to work with Tile38 except when the Redis client doesn't support the fallback to the INFO replication command. This is the case for the StackExchange.Redis (C#) a Redis client, Unfortunately this client is the defacto standard client for C# based applications.

Fortunately the ROLE command is a very small command that outputs the same information that INFO replication does.

See bug with StackExchange.Redis. StackExchange/StackExchange.Redis#2461

Describe the solution you'd like
I would like Tile38 to support the ROLE command.

Describe alternatives you've considered
I've tried to get the other repo to support the fallback command, but considering the changeover happened LONG ago in the Redis command list they see supporting it as unlikely.
@Kilowhisky Kilowhisky mentioned this issue May 10, 2023
Closed
@tidwall
Copy link
Owner

tidwall commented May 11, 2023

I have no problem adding the ROLE command. The output looks cleaner that what INFO replication provides so I can see when StackExchange.Redis chose to use it.

tidwall added a commit that referenced this issue May 21, 2023
@tidwall
Added ROLE command
add9873 
See #686
@tidwall
Copy link
Owner

tidwall commented May 21, 2023

The ROLE command has been added to Tile38. It present the basic information that 'INFO replication' would otherwise provide in the format shown here.

@Kilowhisky
Copy link
Contributor Author

Works!

@iwpnd
Copy link
Contributor

iwpnd commented May 26, 2023 •
edited
Loading

@Kilowhisky are you at liberty you share how you setup Tile38 with sentinel?

@Kilowhisky
Copy link
Contributor Author

Kilowhisky commented Jun 12, 2023 •
edited
Loading

@iwpnd I'll post it up once i get it stable.

I keep having random problems where the failover causes the AOF to get corrupted or straight up crashes the followers with...

{"level":"error","message":"follow: cannot follow: dial tcp 10.134.72.151:9851: connect: connection refused"}
"{""level"":""debug"",""message"":""Expired 3 objects
""}"
fatal error: sync: Unlock of unlocked RWMutex
goroutine 73 [running]:
sync.fatal({0x11f0898?, 0x42?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/panic.go:1031 +0x1e"
sync.(*RWMutex).Unlock(0xc0000047b0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/rwmutex.go:209 +0x4a"
github.com/tidwall/tile38/internal/server.(*Server).handleInputCommand(0xc000004600, 0xc000678000, 0xc000093ae0)
"	/home/runner/work/tile38/tile38/internal/server/server.go:1099 +0x1f45"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc00012c000})
"	/home/runner/work/tile38/tile38/internal/server/server.go:595 +0xec8"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1 [chan receive, 49 minutes]:
github.com/tidwall/tile38/internal/server.Serve({{0x0, 0x0}, 0x267b, {0x7fff185f5b5f, 0x5}, 0x0, {0x7fff185f5b94, 0xc}, {0x0, 0x0}, ...})
"	/home/runner/work/tile38/tile38/internal/server/server.go:438 +0x1a68"
main.main()
"	/home/runner/work/tile38/tile38/cmd/tile38-server/main.go:487 +0x307e"
goroutine 6 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0000d3a00)
"	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:276 +0xad"
created by go.opencensus.io/stats/view.init.0
"	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x96"
goroutine 33 [syscall, 49 minutes]:
os/signal.signal_recv()
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sigqueue.go:152 +0x2f"
os/signal.loop()
"	/opt/hostedtoolcache/go/1.20.4/x64/src/os/signal/signal_unix.go:23 +0x19"
created by os/signal.Notify.func1.1
"	/opt/hostedtoolcache/go/1.20.4/x64/src/os/signal/signal.go:151 +0x2a"
goroutine 34 [chan receive, 49 minutes]:
main.main.func6()
"	/home/runner/work/tile38/tile38/cmd/tile38-server/main.go:420 +0x78"
created by main.main
"	/home/runner/work/tile38/tile38/cmd/tile38-server/main.go:419 +0x29cd"
goroutine 35 [sleep]:
time.Sleep(0x3b9aca00)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/time.go:195 +0x135"
github.com/tidwall/tile38/internal/endpoint.(*Manager).run(0xc000092320)
"	/home/runner/work/tile38/tile38/internal/endpoint/endpoint.go:183 +0x66"
created by github.com/tidwall/tile38/internal/endpoint.NewManager
"	/home/runner/work/tile38/tile38/internal/endpoint/endpoint.go:161 +0xe5"
goroutine 36 [IO wait]:
internal/poll.runtime_pollWait(0x7ff1895b7b58, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc0000d2000?, 0x5?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Accept(0xc0000d2000)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:614 +0x2bd"
net.(*netFD).accept(0xc0000d2000)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_unix.go:172 +0x35"
net.(*TCPListener).accept(0xc0000100c0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/tcpsock_posix.go:148 +0x25"
net.(*TCPListener).Accept(0xc0000100c0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/tcpsock.go:297 +0x3d"
github.com/tidwall/tile38/internal/server.(*Server).netServe(0xc000004600)
"	/home/runner/work/tile38/tile38/internal/server/server.go:488 +0x345"
github.com/tidwall/tile38/internal/server.Serve.func2()
"	/home/runner/work/tile38/tile38/internal/server/server.go:306 +0x26"
created by github.com/tidwall/tile38/internal/server.Serve
"	/home/runner/work/tile38/tile38/internal/server/server.go:304 +0x113b"
goroutine 37 [sleep]:
time.Sleep(0xee6b280)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/time.go:195 +0x135"
github.com/tidwall/tile38/internal/server.Serve.func3()
"	/home/runner/work/tile38/tile38/internal/server/server.go:313 +0x3e"
created by github.com/tidwall/tile38/internal/server.Serve
"	/home/runner/work/tile38/tile38/internal/server/server.go:310 +0x11af"
goroutine 38 [chan receive, 49 minutes]:
github.com/tidwall/tile38/internal/server.Serve.func4()
"	/home/runner/work/tile38/tile38/internal/server/server.go:318 +0x4d"
created by github.com/tidwall/tile38/internal/server.Serve
"	/home/runner/work/tile38/tile38/internal/server/server.go:317 +0x122c"
goroutine 39 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000000480)
"	/home/runner/go/pkg/mod/github.com/tidwall/[email protected]/buntdb.go:547 +0xc5"
created by github.com/tidwall/buntdb.Open
"	/home/runner/go/pkg/mod/github.com/tidwall/[email protected]/buntdb.go:169 +0x257"
goroutine 49 [IO wait, 49 minutes]:
internal/poll.runtime_pollWait(0x7ff1895b7a68, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc000513280?, 0x5?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Accept(0xc000513280)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:614 +0x2bd"
net.(*netFD).accept(0xc000513280)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_unix.go:172 +0x35"
net.(*TCPListener).accept(0xc0000100d8)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/tcpsock_posix.go:148 +0x25"
net.(*TCPListener).Accept(0xc0000100d8)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/tcpsock.go:297 +0x3d"
net/http.(*Server).Serve(0xc0000f04b0, {0x1495c50, 0xc0000100d8})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/http/server.go:3059 +0x385"
net/http.Serve(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/http/server.go:2581"
github.com/tidwall/tile38/internal/server.Serve.func8()
"	/home/runner/work/tile38/tile38/internal/server/server.go:404 +0x197"
created by github.com/tidwall/tile38/internal/server.Serve
"	/home/runner/work/tile38/tile38/internal/server/server.go:399 +0x16db"
goroutine 50 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0005926d0, 0x2e94)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:527 +0x14c"
sync.(*Cond).Wait(0xc0005926c0?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/cond.go:70 +0x8c"
github.com/tidwall/tile38/internal/server.(*Server).processLives(0xc000004600, 0xc00058e6c0)
"	/home/runner/work/tile38/tile38/internal/server/live.go:61 +0x189"
created by github.com/tidwall/tile38/internal/server.Serve
"	/home/runner/work/tile38/tile38/internal/server/server.go:414 +0x176a"
goroutine 51 [sleep]:
time.Sleep(0xbebc200)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/time.go:195 +0x135"
github.com/tidwall/tile38/internal/server.(*Server).loopUntilServerStops(0xc000004600, 0xee6b2800, 0xc000dc5f98)
"	/home/runner/work/tile38/tile38/internal/server/server.go:781 +0x4b"
github.com/tidwall/tile38/internal/server.(*Server).watchOutOfMemory(0x0?, 0x0?)
"	/home/runner/work/tile38/tile38/internal/server/server.go:787 +0x75"
created by github.com/tidwall/tile38/internal/server.Serve
"	/home/runner/work/tile38/tile38/internal/server/server.go:416 +0x17dc"
goroutine 52 [sleep]:
time.Sleep(0xbebc200)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/time.go:195 +0x135"
github.com/tidwall/tile38/internal/server.(*Server).loopUntilServerStops(0xc000004600, 0x2540be400, 0xc0005b6f98)
"	/home/runner/work/tile38/tile38/internal/server/server.go:781 +0x4b"
github.com/tidwall/tile38/internal/server.(*Server).watchLuaStatePool(0x0?, 0x0?)
"	/home/runner/work/tile38/tile38/internal/server/server.go:794 +0x7a"
created by github.com/tidwall/tile38/internal/server.Serve
"	/home/runner/work/tile38/tile38/internal/server/server.go:418 +0x184e"
goroutine 53 [sleep]:
time.Sleep(0xbebc200)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/time.go:195 +0x135"
github.com/tidwall/tile38/internal/server.(*Server).loopUntilServerStops(0xc000004600, 0x3b9aca00, 0xc000163f90)
"	/home/runner/work/tile38/tile38/internal/server/server.go:781 +0x4b"
github.com/tidwall/tile38/internal/server.(*Server).watchAutoGC(0xc000004600, 0x0?)
"	/home/runner/work/tile38/tile38/internal/server/server.go:727 +0xa6"
created by github.com/tidwall/tile38/internal/server.Serve
"	/home/runner/work/tile38/tile38/internal/server/server.go:420 +0x18ca"
goroutine 54 [sleep]:
time.Sleep(0xbebc200)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/time.go:195 +0x135"
github.com/tidwall/tile38/internal/server.(*Server).loopUntilServerStops(0xc000004600, 0x5f5e100, 0xc000dc9f98)
"	/home/runner/work/tile38/tile38/internal/server/server.go:781 +0x4b"
github.com/tidwall/tile38/internal/server.(*Server).backgroundExpiring(0x0?, 0x0?)
"	/home/runner/work/tile38/tile38/internal/server/expire.go:18 +0x75"
created by github.com/tidwall/tile38/internal/server.Serve
"	/home/runner/work/tile38/tile38/internal/server/server.go:422 +0x193c"
goroutine 55 [sleep]:
time.Sleep(0xbebc200)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/time.go:195 +0x135"
github.com/tidwall/tile38/internal/server.(*Server).loopUntilServerStops(0xc000004600, 0x3b9aca00, 0xc000177f98)
"	/home/runner/work/tile38/tile38/internal/server/server.go:781 +0x4b"
github.com/tidwall/tile38/internal/server.(*Server).backgroundSyncAOF(0x0?, 0x0?)
"	/home/runner/work/tile38/tile38/internal/server/server.go:802 +0x75"
created by github.com/tidwall/tile38/internal/server.Serve
"	/home/runner/work/tile38/tile38/internal/server/server.go:424 +0x19ae"
goroutine 20 [sleep]:
time.Sleep(0xee6b280)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/time.go:195 +0x135"
github.com/tidwall/tile38/internal/server.(*Server).processLives.func1()
"	/home/runner/work/tile38/tile38/internal/server/live.go:36 +0x7d"
created by github.com/tidwall/tile38/internal/server.(*Server).processLives
"	/home/runner/work/tile38/tile38/internal/server/live.go:29 +0x10a"
goroutine 11 [sync.Cond.Wait, 49 minutes]:
sync.runtime_notifyListWait(0xc000560090, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:527 +0x14c"
sync.(*Cond).Wait(0xc00057a0c0?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/cond.go:70 +0x8c"
github.com/tidwall/tile38/internal/server.(*Hook).manager(0xc00057a0c0)
"	/home/runner/work/tile38/tile38/internal/server/hooks.go:623 +0xc5"
created by github.com/tidwall/tile38/internal/server.(*Hook).Open
"	/home/runner/work/tile38/tile38/internal/server/hooks.go:564 +0x13b"
goroutine 12 [sync.Cond.Wait, 49 minutes]:
sync.runtime_notifyListWait(0xc000560110, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:527 +0x14c"
sync.(*Cond).Wait(0xc00057a300?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/cond.go:70 +0x8c"
github.com/tidwall/tile38/internal/server.(*Hook).manager(0xc00057a300)
"	/home/runner/work/tile38/tile38/internal/server/hooks.go:623 +0xc5"
created by github.com/tidwall/tile38/internal/server.(*Hook).Open
"	/home/runner/work/tile38/tile38/internal/server/hooks.go:564 +0x13b"
goroutine 13 [sync.Cond.Wait, 49 minutes]:
sync.runtime_notifyListWait(0xc000560190, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:527 +0x14c"
sync.(*Cond).Wait(0xc00057a540?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/cond.go:70 +0x8c"
github.com/tidwall/tile38/internal/server.(*Hook).manager(0xc00057a540)
"	/home/runner/work/tile38/tile38/internal/server/hooks.go:623 +0xc5"
created by github.com/tidwall/tile38/internal/server.(*Hook).Open
"	/home/runner/work/tile38/tile38/internal/server/hooks.go:564 +0x13b"
goroutine 14 [IO wait]:
internal/poll.runtime_pollWait(0x7ff1895b7978, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc00030a000?, 0xc00033e000?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc00030a000, {0xc00033e000, 0x1000, 0x1000})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc00030a000, {0xc00033e000?, 0x4e3926?, 0x0?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc000322000, {0xc00033e000?, 0x0?, 0xc00009d3e8?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
net/http.(*connReader).Read(0xc00009d3e0, {0xc00033e000, 0x1000, 0x1000})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/http/server.go:782 +0x171"
bufio.(*Reader).fill(0xc000324000)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/bufio/bufio.go:106 +0xff"
bufio.(*Reader).Peek(0xc000324000, 0x4)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/bufio/bufio.go:144 +0x5d"
net/http.(*conn).serve(0xc00056e1b0, {0x1496818, 0xc000116930})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/http/server.go:2030 +0x77c"
created by net/http.(*Server).Serve
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/http/server.go:3089 +0x5ed"
goroutine 21 [semacquire, 49 minutes]:
sync.runtime_Semacquire(0xc000140c80?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:62 +0x27"
sync.(*WaitGroup).Wait(0xc0002b9e70?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/waitgroup.go:116 +0x4b"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc000322010})
"	/home/runner/work/tile38/tile38/internal/server/server.go:627 +0x12f2"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1190 [IO wait]:
internal/poll.runtime_pollWait(0x7ff18931bba0, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc00030a480?, 0xc0074aa000?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc00030a480, {0xc0074aa000, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc00030a480, {0xc0074aa000?, 0xf29e66?, 0xc0000047b0?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc00012c0b0, {0xc0074aa000?, 0xc0000047b0?, 0xc00065dc70?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc00012c0b0})
"	/home/runner/work/tile38/tile38/internal/server/server.go:551 +0x52e"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1218 [IO wait]:
internal/poll.runtime_pollWait(0x7ff18931b9c0, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc00030a580?, 0xc00914c060?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc00030a580, {0xc00914c060, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc00030a580, {0xc00914c060?, 0xc000151790?, 0xc0001518f8?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc0005d20c0, {0xc00914c060?, 0xc000151af8?, 0x0?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*PipelineReader).ReadMessages(0xc00914c040)
"	/home/runner/work/tile38/tile38/internal/server/server.go:1550 +0x5c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription(0xc000004600, {0x149eb30?, 0xc000054980}, 0xc00a0a21e0?, 0xc0000939a0, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:366 +0x59f"
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc000004600, {0x1490780?, 0x1c205b0}, {0x149eb30, 0xc000054980?}, 0xc00914c040, 0xc0000939a0, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/live.go:105 +0x274"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
"	/home/runner/work/tile38/tile38/internal/server/server.go:616 +0x165"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2
"	/home/runner/work/tile38/tile38/internal/server/server.go:614 +0x12e5"
goroutine 72 [sleep]:
time.Sleep(0xbebc200)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/time.go:195 +0x135"
github.com/tidwall/tile38/internal/server.readMemStats.func1()
"	/home/runner/work/tile38/tile38/internal/server/stats.go:38 +0x4c"
created by github.com/tidwall/tile38/internal/server.readMemStats
"	/home/runner/work/tile38/tile38/internal/server/stats.go:31 +0x9a"
goroutine 22 [IO wait]:
internal/poll.runtime_pollWait(0x7ff1895b76a8, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc00030a200?, 0xc00012e000?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc00030a200, {0xc00012e000, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc00030a200, {0xc00012e000?, 0xf29e66?, 0xc0000047b0?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc000322018, {0xc00012e000?, 0xc0000047b0?, 0xc00015fc70?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc000322018})
"	/home/runner/work/tile38/tile38/internal/server/server.go:551 +0x52e"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 23 [IO wait, 49 minutes]:
internal/poll.runtime_pollWait(0x7ff1895b7798, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc00030a180?, 0xc000600060?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc00030a180, {0xc000600060, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc00030a180, {0xc000600060?, 0xc00011e790?, 0xc00011e8f8?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc000322010, {0xc000600060?, 0x40dec8?, 0xc0005a80d0?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*PipelineReader).ReadMessages(0xc000600040)
"	/home/runner/work/tile38/tile38/internal/server/server.go:1550 +0x5c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription(0xc000004600, {0x149eb30?, 0xc0000547e0}, 0xc0003000f0?, 0xc00056c460, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:366 +0x59f"
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc000004600, {0x1490780?, 0x1c205b0}, {0x149eb30, 0xc0000547e0?}, 0xc000600040, 0xc00056c460, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/live.go:105 +0x274"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
"	/home/runner/work/tile38/tile38/internal/server/server.go:616 +0x165"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2
"	/home/runner/work/tile38/tile38/internal/server/server.go:614 +0x12e5"
goroutine 24 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000140cd0, 0x1168)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:527 +0x14c"
sync.(*Cond).Wait(0xc00055a200?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/cond.go:70 +0x8c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription.func8()
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:323 +0xc5"
created by github.com/tidwall/tile38/internal/server.(*Server).liveSubscription
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:304 +0x553"
goroutine 74 [semacquire, 49 minutes]:
sync.runtime_Semacquire(0xc000592780?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:62 +0x27"
sync.(*WaitGroup).Wait(0xc000677e70?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/waitgroup.go:116 +0x4b"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc00012c008})
"	/home/runner/work/tile38/tile38/internal/server/server.go:627 +0x12f2"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 75 [IO wait, 49 minutes]:
internal/poll.runtime_pollWait(0x7ff1895b74c8, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc0000d2180?, 0xc000628060?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc0000d2180, {0xc000628060, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc0000d2180, {0xc000628060?, 0xc0005fc790?, 0xc0005fc8f8?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc00012c008, {0xc000628060?, 0xc0005fcaf8?, 0x0?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*PipelineReader).ReadMessages(0xc000628040)
"	/home/runner/work/tile38/tile38/internal/server/server.go:1550 +0x5c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription(0xc000004600, {0x149eb30?, 0xc000596120}, 0xc0000445a0?, 0xc00059a140, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:366 +0x59f"
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc000004600, {0x1490780?, 0x1c205b0}, {0x149eb30, 0xc000596120?}, 0xc000628040, 0xc00059a140, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/live.go:105 +0x274"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
"	/home/runner/work/tile38/tile38/internal/server/server.go:616 +0x165"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2
"	/home/runner/work/tile38/tile38/internal/server/server.go:614 +0x12e5"
goroutine 76 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0005927d0, 0x1168)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:527 +0x14c"
sync.(*Cond).Wait(0xc00042c600?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/cond.go:70 +0x8c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription.func8()
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:323 +0xc5"
created by github.com/tidwall/tile38/internal/server.(*Server).liveSubscription
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:304 +0x553"
goroutine 1188 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00009b990, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:527 +0x14c"
sync.(*Cond).Wait(0x0?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/cond.go:70 +0x8c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription.func8()
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:323 +0xc5"
created by github.com/tidwall/tile38/internal/server.(*Server).liveSubscription
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:304 +0x553"
goroutine 1192 [semacquire]:
sync.runtime_Semacquire(0xc00029a680?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:62 +0x27"
sync.(*WaitGroup).Wait(0xc000619e70?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/waitgroup.go:116 +0x4b"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc0005d20c0})
"	/home/runner/work/tile38/tile38/internal/server/server.go:627 +0x12f2"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1201 [IO wait]:
internal/poll.runtime_pollWait(0x7ff1895b72e8, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc00030a380?, 0xc0008f0060?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc00030a380, {0xc0008f0060, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc00030a380, {0xc0008f0060?, 0xc000176790?, 0xc0001768f8?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc000012180, {0xc0008f0060?, 0xc000176af8?, 0x414ba5?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*PipelineReader).ReadMessages(0xc0008f0040)
"	/home/runner/work/tile38/tile38/internal/server/server.go:1550 +0x5c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription(0xc000004600, {0x149eb30?, 0xc0005964e0}, 0xc001f5c348?, 0xc00056c550, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:366 +0x59f"
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc000004600, {0x1490780?, 0x1c205b0}, {0x149eb30, 0xc0005964e0?}, 0xc0008f0040, 0xc00056c550, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/live.go:105 +0x274"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
"	/home/runner/work/tile38/tile38/internal/server/server.go:616 +0x165"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2
"	/home/runner/work/tile38/tile38/internal/server/server.go:614 +0x12e5"
goroutine 1079 [IO wait]:
internal/poll.runtime_pollWait(0x7ff1895b7018, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc0000d2200?, 0xc00962e060?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc0000d2200, {0xc00962e060, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc0000d2200, {0xc00962e060?, 0xc000765790?, 0xc0007658f8?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc000322078, {0xc00962e060?, 0x7ff1b096ff18?, 0x10?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*PipelineReader).ReadMessages(0xc00962e040)
"	/home/runner/work/tile38/tile38/internal/server/server.go:1550 +0x5c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription(0xc000004600, {0x149eb30?, 0xc0000546c0}, 0xc00685df68?, 0xc0092e7b80, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:366 +0x59f"
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc000004600, {0x1490780?, 0x1c205b0}, {0x149eb30, 0xc0000546c0?}, 0xc00962e040, 0xc0092e7b80, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/live.go:105 +0x274"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
"	/home/runner/work/tile38/tile38/internal/server/server.go:616 +0x165"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2
"	/home/runner/work/tile38/tile38/internal/server/server.go:614 +0x12e5"
goroutine 1050 [IO wait]:
internal/poll.runtime_pollWait(0x7ff1895b6d48, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc0005d8500?, 0xc005d0e000?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc0005d8500, {0xc005d0e000, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc0005d8500, {0xc005d0e000?, 0x4?, 0x7ff1b096f108?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc00012c028, {0xc005d0e000?, 0xc008079c88?, 0x4d5657?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc00012c028})
"	/home/runner/work/tile38/tile38/internal/server/server.go:551 +0x52e"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1219 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00029a6d0, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:527 +0x14c"
sync.(*Cond).Wait(0x0?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/cond.go:70 +0x8c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription.func8()
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:323 +0xc5"
created by github.com/tidwall/tile38/internal/server.(*Server).liveSubscription
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:304 +0x553"
goroutine 1036 [semacquire, 5 minutes]:
sync.runtime_Semacquire(0xc00009b040?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:62 +0x27"
sync.(*WaitGroup).Wait(0xc004c31e70?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/waitgroup.go:116 +0x4b"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc0005d2058})
"	/home/runner/work/tile38/tile38/internal/server/server.go:627 +0x12f2"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1204 [IO wait]:
internal/poll.runtime_pollWait(0x7ff18931bab0, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc0000d2580?, 0xc0083b2000?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc0000d2580, {0xc0083b2000, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc0000d2580, {0xc0083b2000?, 0x4?, 0x7ff1b096f5b8?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc000012218, {0xc0083b2000?, 0xc000711c88?, 0x4d5657?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc000012218})
"	/home/runner/work/tile38/tile38/internal/server/server.go:551 +0x52e"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1120 [IO wait]:
internal/poll.runtime_pollWait(0x7ff1895b73d8, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc00030a400?, 0xc00098e060?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc00030a400, {0xc00098e060, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc00030a400, {0xc00098e060?, 0xc000e3e790?, 0xc000e3e8f8?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc00012c0a8, {0xc00098e060?, 0xc000e3eaf8?, 0x0?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*PipelineReader).ReadMessages(0xc00098e040)
"	/home/runner/work/tile38/tile38/internal/server/server.go:1550 +0x5c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription(0xc000004600, {0x149eb30?, 0xc000596620}, 0xc009a3c0d8?, 0xc00056c730, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:366 +0x59f"
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc000004600, {0x1490780?, 0x1c205b0}, {0x149eb30, 0xc000596620?}, 0xc00098e040, 0xc00056c730, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/live.go:105 +0x274"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
"	/home/runner/work/tile38/tile38/internal/server/server.go:616 +0x165"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2
"	/home/runner/work/tile38/tile38/internal/server/server.go:614 +0x12e5"
goroutine 1202 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0007942d0, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:527 +0x14c"
sync.(*Cond).Wait(0x0?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/cond.go:70 +0x8c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription.func8()
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:323 +0xc5"
created by github.com/tidwall/tile38/internal/server.(*Server).liveSubscription
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:304 +0x553"
goroutine 1168 [semacquire]:
sync.runtime_Semacquire(0xc000794280?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:62 +0x27"
sync.(*WaitGroup).Wait(0xc00a6f5e70?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/waitgroup.go:116 +0x4b"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc000012180})
"	/home/runner/work/tile38/tile38/internal/server/server.go:627 +0x12f2"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1078 [sync.Cond.Wait, 3 minutes]:
sync.runtime_notifyListWait(0xc00029a190, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:527 +0x14c"
sync.(*Cond).Wait(0x0?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/cond.go:70 +0x8c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription.func8()
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:323 +0xc5"
created by github.com/tidwall/tile38/internal/server.(*Server).liveSubscription
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:304 +0x553"
goroutine 1167 [IO wait]:
internal/poll.runtime_pollWait(0x7ff1895b71f8, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc00030a300?, 0xc000954000?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc00030a300, {0xc000954000, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc00030a300, {0xc000954000?, 0x4?, 0x0?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc000012148, {0xc000954000?, 0xc0003cdc88?, 0x4d5657?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc000012148})
"	/home/runner/work/tile38/tile38/internal/server/server.go:551 +0x52e"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1189 [semacquire]:
sync.runtime_Semacquire(0xc00009bb00?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:62 +0x27"
sync.(*WaitGroup).Wait(0xc00061de70?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/waitgroup.go:116 +0x4b"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc00012c0a8})
"	/home/runner/work/tile38/tile38/internal/server/server.go:627 +0x12f2"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1038 [sync.Cond.Wait, 5 minutes]:
sync.runtime_notifyListWait(0xc00009b190, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:527 +0x14c"
sync.(*Cond).Wait(0x0?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/cond.go:70 +0x8c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription.func8()
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:323 +0xc5"
created by github.com/tidwall/tile38/internal/server.(*Server).liveSubscription
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:304 +0x553"
goroutine 1186 [semacquire]:
sync.runtime_Semacquire(0xc00009b940?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:62 +0x27"
sync.(*WaitGroup).Wait(0xc0066afe70?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/waitgroup.go:116 +0x4b"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc00012c048})
"	/home/runner/work/tile38/tile38/internal/server/server.go:627 +0x12f2"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1217 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00009bb50, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:527 +0x14c"
sync.(*Cond).Wait(0x0?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/cond.go:70 +0x8c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription.func8()
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:323 +0xc5"
created by github.com/tidwall/tile38/internal/server.(*Server).liveSubscription
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:304 +0x553"
goroutine 1053 [IO wait]:
internal/poll.runtime_pollWait(0x7ff1895b6f28, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc0000d2280?, 0xc009eb6000?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc0000d2280, {0xc009eb6000, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc0000d2280, {0xc009eb6000?, 0xf29e66?, 0xc0000047b0?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc000322080, {0xc009eb6000?, 0xc0000047b0?, 0xc000819c70?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc000322080})
"	/home/runner/work/tile38/tile38/internal/server/server.go:551 +0x52e"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1187 [IO wait]:
internal/poll.runtime_pollWait(0x7ff1895b7108, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc0000d2080?, 0xc0006c2060?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc0000d2080, {0xc0006c2060, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc0000d2080, {0xc0006c2060?, 0xc000175790?, 0xc0001758f8?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc00012c048, {0xc0006c2060?, 0xc000175af8?, 0x0?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*PipelineReader).ReadMessages(0xc0006c2040)
"	/home/runner/work/tile38/tile38/internal/server/server.go:1550 +0x5c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription(0xc000004600, {0x149eb30?, 0xc0004e29c0}, 0xc00a7981e0?, 0xc002b80640, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:366 +0x59f"
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc000004600, {0x1490780?, 0x1c205b0}, {0x149eb30, 0xc0004e29c0?}, 0xc0006c2040, 0xc002b80640, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/live.go:105 +0x274"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
"	/home/runner/work/tile38/tile38/internal/server/server.go:616 +0x165"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2
"	/home/runner/work/tile38/tile38/internal/server/server.go:614 +0x12e5"
goroutine 1080 [sync.Cond.Wait, 3 minutes]:
sync.runtime_notifyListWait(0xc000592890, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:527 +0x14c"
sync.(*Cond).Wait(0x0?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/cond.go:70 +0x8c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription.func8()
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:323 +0xc5"
created by github.com/tidwall/tile38/internal/server.(*Server).liveSubscription
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:304 +0x553"
goroutine 1077 [IO wait]:
internal/poll.runtime_pollWait(0x7ff1895b6e38, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc0005d8680?, 0xc006b5a060?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc0005d8680, {0xc006b5a060, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc0005d8680, {0xc006b5a060?, 0xc000171790?, 0xc0001718f8?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc00012c040, {0xc006b5a060?, 0x7ff1b096f108?, 0x10?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*PipelineReader).ReadMessages(0xc006b5a040)
"	/home/runner/work/tile38/tile38/internal/server/server.go:1550 +0x5c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription(0xc000004600, {0x149eb30?, 0xc0004e25e0}, 0xc002e1a1e0?, 0xc002b80140, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:366 +0x59f"
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc000004600, {0x1490780?, 0x1c205b0}, {0x149eb30, 0xc0004e25e0?}, 0xc006b5a040, 0xc002b80140, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/live.go:105 +0x274"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
"	/home/runner/work/tile38/tile38/internal/server/server.go:616 +0x165"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2
"	/home/runner/work/tile38/tile38/internal/server/server.go:614 +0x12e5"
goroutine 564 [IO wait]:
internal/poll.runtime_pollWait(0x7ff18931be70, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc0005d8080?, 0xc004bd2000?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc0005d8080, {0xc004bd2000, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc0005d8080, {0xc004bd2000?, 0xf29e66?, 0xc0000047b0?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc000322020, {0xc004bd2000?, 0xc0000047b0?, 0xc0054e7c70?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc000322020})
"	/home/runner/work/tile38/tile38/internal/server/server.go:551 +0x52e"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1146 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7ff18931bc90, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc0000d2380?, 0xc007c96060?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc0000d2380, {0xc007c96060, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc0000d2380, {0xc007c96060?, 0xc000e41790?, 0xc000e418f8?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc00012c060, {0xc007c96060?, 0xc000e41af8?, 0x0?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*PipelineReader).ReadMessages(0xc007c96040)
"	/home/runner/work/tile38/tile38/internal/server/server.go:1550 +0x5c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription(0xc000004600, {0x149eb30?, 0xc000114600}, 0xc0071861e0?, 0xc00576fe50, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:366 +0x59f"
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc000004600, {0x1490780?, 0x1c205b0}, {0x149eb30, 0xc000114600?}, 0xc007c96040, 0xc00576fe50, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/live.go:105 +0x274"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
"	/home/runner/work/tile38/tile38/internal/server/server.go:616 +0x165"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2
"	/home/runner/work/tile38/tile38/internal/server/server.go:614 +0x12e5"
goroutine 1145 [semacquire, 2 minutes]:
sync.runtime_Semacquire(0xc00009a880?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:62 +0x27"
sync.(*WaitGroup).Wait(0xc005981e70?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/waitgroup.go:116 +0x4b"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc00012c060})
"	/home/runner/work/tile38/tile38/internal/server/server.go:627 +0x12f2"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1125 [IO wait]:
internal/poll.runtime_pollWait(0x7ff18931bd80, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc0005d8780?, 0xc00685e000?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc0005d8780, {0xc00685e000, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc0005d8780, {0xc00685e000?, 0xf29e66?, 0xc0000047b0?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc000012090, {0xc00685e000?, 0xc0000047b0?, 0xc0066b3c70?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc000012090})
"	/home/runner/work/tile38/tile38/internal/server/server.go:551 +0x52e"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1147 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00009a990, 0x105)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:527 +0x14c"
sync.(*Cond).Wait(0x0?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/cond.go:70 +0x8c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription.func8()
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:323 +0xc5"
created by github.com/tidwall/tile38/internal/server.(*Server).liveSubscription
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:304 +0x553"
goroutine 858 [semacquire, 13 minutes]:
sync.runtime_Semacquire(0xc000794100?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:62 +0x27"
sync.(*WaitGroup).Wait(0xc006987e70?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/waitgroup.go:116 +0x4b"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc000322030})
"	/home/runner/work/tile38/tile38/internal/server/server.go:627 +0x12f2"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1119 [IO wait]:
internal/poll.runtime_pollWait(0x7ff1895b7888, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc0005d8200?, 0xc0006b2000?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc0005d8200, {0xc0006b2000, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc0005d8200, {0xc0006b2000?, 0x4?, 0x0?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc0005d2028, {0xc0006b2000?, 0xc00a2f3c88?, 0x4d5657?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc0005d2028})
"	/home/runner/work/tile38/tile38/internal/server/server.go:551 +0x52e"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1035 [IO wait]:
internal/poll.runtime_pollWait(0x7ff1895b6c58, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc0005d8180?, 0xc005e32000?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc0005d8180, {0xc005e32000, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc0005d8180, {0xc005e32000?, 0xf29e66?, 0xc0000047b0?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc0005d2008, {0xc005e32000?, 0xc0000047b0?, 0xc008075c70?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc0005d2008})
"	/home/runner/work/tile38/tile38/internal/server/server.go:551 +0x52e"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1052 [semacquire, 3 minutes]:
sync.runtime_Semacquire(0xc000592780?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:62 +0x27"
sync.(*WaitGroup).Wait(0xc00952fe70?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/waitgroup.go:116 +0x4b"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc000322078})
"	/home/runner/work/tile38/tile38/internal/server/server.go:627 +0x12f2"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 799 [IO wait]:
internal/poll.runtime_pollWait(0x7ff18931c050, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc00030a100?, 0xc0066be060?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc00030a100, {0xc0066be060, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc00030a100, {0xc0066be060?, 0xc000760790?, 0xc0007608f8?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc000322030, {0xc0066be060?, 0x7ff1b096f5b8?, 0x10?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*PipelineReader).ReadMessages(0xc0066be040)
"	/home/runner/work/tile38/tile38/internal/server/server.go:1550 +0x5c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription(0xc000004600, {0x149eb30?, 0xc000114540}, 0xc006607518?, 0xc0067b3770, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:366 +0x59f"
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc000004600, {0x1490780?, 0x1c205b0}, {0x149eb30, 0xc000114540?}, 0xc0066be040, 0xc0067b3770, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/live.go:105 +0x274"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
"	/home/runner/work/tile38/tile38/internal/server/server.go:616 +0x165"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2
"	/home/runner/work/tile38/tile38/internal/server/server.go:614 +0x12e5"
goroutine 1173 [sleep]:
time.Sleep(0x3b9aca00)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/time.go:195 +0x135"
github.com/tidwall/tile38/internal/server.(*Server).follow(0xc005ad3e90?, {0xc0077ab050, 0xd}, 0x149ef50?, 0xc000322030?)
"	/home/runner/work/tile38/tile38/internal/server/follow.go:326 +0x51"
created by github.com/tidwall/tile38/internal/server.(*Server).cmdFollow
"	/home/runner/work/tile38/tile38/internal/server/follow.go:89 +0xa45"
goroutine 1051 [semacquire, 3 minutes]:
sync.runtime_Semacquire(0xc00029a140?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:62 +0x27"
sync.(*WaitGroup).Wait(0xc003501e70?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/waitgroup.go:116 +0x4b"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc00012c040})
"	/home/runner/work/tile38/tile38/internal/server/server.go:627 +0x12f2"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 1037 [IO wait]:
internal/poll.runtime_pollWait(0x7ff18931c140, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc0005d8480?, 0xc005c68060?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc0005d8480, {0xc005c68060, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc0005d8480, {0xc005c68060?, 0xc000e3d790?, 0xc000e3d8f8?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc0005d2058, {0xc005c68060?, 0x7ff1b096f108?, 0x10?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*PipelineReader).ReadMessages(0xc005c68040)
"	/home/runner/work/tile38/tile38/internal/server/server.go:1550 +0x5c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription(0xc000004600, {0x149eb30?, 0xc0005962e0}, 0xc0078c0030?, 0xc0049fee10, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:366 +0x59f"
github.com/tidwall/tile38/internal/server.(*Server).goLive(0xc000004600, {0x1490780?, 0x1c205b0}, {0x149eb30, 0xc0005962e0?}, 0xc005c68040, 0xc0049fee10, 0x0)
"	/home/runner/work/tile38/tile38/internal/server/live.go:105 +0x274"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2.2()
"	/home/runner/work/tile38/tile38/internal/server/server.go:616 +0x165"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe.func2
"	/home/runner/work/tile38/tile38/internal/server/server.go:614 +0x12e5"
goroutine 857 [IO wait]:
internal/poll.runtime_pollWait(0x7ff18931bf60, 0x72)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89"
internal/poll.(*pollDesc).wait(0xc00030a080?, 0xc006db0000?, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32"
internal/poll.(*pollDesc).waitRead(...)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89"
internal/poll.(*FD).Read(0xc00030a080, {0xc006db0000, 0xffff, 0xffff})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299"
net.(*netFD).Read(0xc00030a080, {0xc006db0000?, 0x4?, 0x7ff1b096f108?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29"
net.(*conn).Read(0xc000322008, {0xc006db0000?, 0xc00081fc88?, 0x4d5657?})
"	/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45"
github.com/tidwall/tile38/internal/server.(*Server).netServe.func2({0x149ef50, 0xc000322008})
"	/home/runner/work/tile38/tile38/internal/server/server.go:551 +0x52e"
created by github.com/tidwall/tile38/internal/server.(*Server).netServe
"	/home/runner/work/tile38/tile38/internal/server/server.go:498 +0x50c"
goroutine 800 [sync.Cond.Wait, 13 minutes]:
sync.runtime_notifyListWait(0xc000794150, 0x0)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:527 +0x14c"
sync.(*Cond).Wait(0x0?)
"	/opt/hostedtoolcache/go/1.20.4/x64/src/sync/cond.go:70 +0x8c"
github.com/tidwall/tile38/internal/server.(*Server).liveSubscription.func8()
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:323 +0xc5"
created by github.com/tidwall/tile38/internal/server.(*Server).liveSubscription
"	/home/runner/work/tile38/tile38/internal/server/pubsub.go:304 +0x553"

@samar-sajnani
Copy link

@Kilowhisky did you ever get it stable, I'm seeing issues where tile38 keeps showing redis-sentine going from live to not live and causing issues with RESP initial AOF bulk replication. I also had issues with followers crashing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tidwall @iwpnd @Kilowhisky @samar-sajnani