-
-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"failed to release lock" #27
Comments
Do you have more context what you did and when this error message occurred? How many concurrent Caddy servers are running, what traffic they have to handle how many certificates? However, it can happen that there is a problem releasing a Consul distributed lock without any further issues. |
I have a 3 consul cluster setup and the consul plugin does appear to be storing data however every new certificate that gets requested results in the above failure. Below is an example It seems that it created the lock without the prefix but attempts to release it with the prefix? consul kv does appear to have the correct thing setup
|
I came across this while working on #28 --- you can see this if you run Line 67 in 3811ba6
which doesn't log the Lines 84 to 90 in 3811ba6
which logs the error returned by Consul, which does happen to include the full path the plugin asked for. Rather, I think that error is coming from the logic in that goroutine to cleanup in the event a lock is lost. Line 78 in 3811ba6
Per the Consul SDK documentation, this is "... a channel that is closed if our lock is lost or an error." (https://pkg.go.dev/github.com/hashicorp/consul/api#Lock). It also seems that channel is closed if the original locker also unlocked the lock. So I think what is happening is:
I think that bit of logic needs to be rethought and a couple things done:
|
Work in progress, I think this addresses pteich#27 but it needs a sanity check and there's one edge case in there that needs some more thinking.
Do you know if the issue would cause a problem with renewing the certificate in the 90 day renewal period if consul kv still has the setting? |
I don't think this will cause a direct problem with renewing a certificate. When Caddy calls Line 122 in 3811ba6
You don't have that, so Caddy successfully asked Consul to release the lock, and Consul did. That, however, also triggers the goroutine Lines 83 to 90 in 3811ba6
where it tries to Lines 115 to 118 in 3811ba6
which when returned to Lines 86 to 88 in 3811ba6
gives you So in fact, I don't think we're actually asking Consul to unlock a second time, because when As further proof, when you look up that K/V entry you may see it there, but since there's no |
As @kula points out it is more or less an unnecessary double unlock. But it should and will be no problem with renewing. We use it for several years and many hundreds of domains without any issue regarding renewal. |
Is this the expected behavior for thus plugin?
The text was updated successfully, but these errors were encountered: