Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZK session expiry shuts down Astra. Don't make ZK updates in the shutdown path #653

Open
vthacker opened this issue Aug 24, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@vthacker
Copy link
Contributor

vthacker commented Aug 24, 2023

To Reproduce

I noticed this in our production cluster

16:52:09 - The ZK session has expired CuratorEventImpl
16:52:09 - Curator connection state changed to LOST and Runtime halter is called probably on an unrecoverable error. Stopping the VM.

and the bug is - when ZK is already in a bad state the Astra close protocol should not try to make a ZK fetch like it does today. The ZK operations fails and and the exception ends up calling the Runtime falter again

Another side note, when this happens then in the next 20 seconds we see 450+ log messages with CuratorCache error like this

java.lang.IllegalStateException: Expected state [STARTED] was [STOPPED]
	at org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:821)
	at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkState(CuratorFrameworkImpl.java:457)
	at org.apache.curator.framework.imps.CuratorFrameworkImpl.getData(CuratorFrameworkImpl.java:491)
	at org.apache.curator.framework.recipes.cache.CuratorCacheImpl.nodeChanged(CuratorCacheImpl.java:266)
@vthacker vthacker added the bug Something isn't working label Aug 24, 2023
@bryanlb bryanlb changed the title [BUG] ZK session expiry shuts down kaldb. Don't make ZK updates in the shutdown path ZK session expiry shuts down kaldb. Don't make ZK updates in the shutdown path Oct 30, 2023
@bryanlb bryanlb changed the title ZK session expiry shuts down kaldb. Don't make ZK updates in the shutdown path ZK session expiry shuts down Astra. Don't make ZK updates in the shutdown path Mar 11, 2024
Copy link

github-actions bot commented Sep 8, 2024

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 30 days.

@github-actions github-actions bot added the Stale label Sep 8, 2024
@bryanlb bryanlb added Stale and removed Stale labels Sep 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants