-
Notifications
You must be signed in to change notification settings - Fork 172
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LockError: Unable to fully release the lock on resource #65
Comments
me too! |
Same issue on my side! |
Hi all! Just now getting a chance to look at this. I've combed through the logic and tests and I haven't found anything suspicious. A few questions:
Is there any chance one of you can create a minimal reproduction of this, perhaps using docker-compose to set up the environment? |
Not sure about the latter. Btw Redis is on AWS (cache), if this is helpful. |
I still have problem with following error: Please help |
@mike-marcacci please check this error. |
@mike-marcacci Why dont you follow up issues? we are facing much problems with this error.... If you are not able to resolve it, please let us know to change the component. |
@amirzandi Hi, have you tried to roll back to version |
I finally found out the problem and I resolved it myself. I will have some plan to use another component in order to get rid of such issues. |
@amirzandi Could you please create a pull request so that the fix is merged back to the project, and anyone can use it. |
I only increased the lock time to 60000 milliseconds and it is now working fine. |
I had this same error in version 4.1.0, which went away after I removed the |
For me this goes away if going back to 3.1.2. But having a look at the timing it appears that this "new" error message pops up when the lock expired. Eg. if locking for 250 ms and the actual processing while the lock is acquired takes 300 ms an error gets thrown. Maybe this was enforced with 4.0.0 and before was handled silently? |
Hey folks, sorry for the slow replies: I've been unbelievably busy this year so far. I think that @b00tsy may be onto something here, especially given @amirzandi's workaround. Version 3 should have rejected with the same error, but perhaps it wasn't? Unlocking was originally a best-effort operation that wouldn't report an error, but that was changed quite early on. For those of you who have reverted to version 3, can you watch your redis instances for key expirations? Something like this:
If all is well and you are acquiring or extending locks with adequate durations, you shouldn't see any keys that match your locks expiring. |
Well I print the time my tasks have taken and sometimes they’re above the ttl I have set for the lock, so I know quite sure, that the error message comes when the key in redis probably has expired. In fact the error gave me a clue that something in my processing pipeline wasn’t running smooth of which I was not aware of before.
The question probably is whether unlock should result in an error if the key expired or whether it should maybe return 0 or 1 (or true or false) depending on whether unlocking was successful and if not print a warning or info message that the key expired before it could be unlocked...
… On 1. May 2020, at 00:31, Mike Marcacci ***@***.***> wrote:
Hey folks, sorry for the slow replies: I've been unbelievably busy this year so far. I think that @b00tsy <https://github.com/b00tsy> may be onto something here, especially given @amirzandi <https://github.com/amirzandi>'s workaround.
Version 3 should have rejected with the same error, but perhaps it wasn't? Unlocking was originally a best-effort operation that wouldn't report an error, but that was changed quite early on.
For those of you who have reverted to version 3, can you watch your redis instances for key expirations? Something like this:
config set notify-keyspace-events Ex
psubscribe ***@***.***__:expired
If all is well and you are acquiring or extending locks with adequate durations, you shouldn't see any keys that match your locks expiring.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#65 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACCYSGGRQERAJ2XSDK5UZTRPH33LANCNFSM4KVGSSFQ>.
|
I can reproduce this issue w 100% reliability w these steps:
|
@bohendo do you only experience this with these are separate processes? Following your algorithm does not produce any issues in my tests: const Redlock = require('./redlock');
const IORedis = require('ioredis');
const client = new IORedis();
const redlock = new Redlock([client], {
retryCount: 10,
retryDelay: 100,
retryJitter: 50
});
async function main (){
console.log("a");
await redlock.lock("regression", 200);
console.log("b");
const second = await redlock.lock("regression", 400);
console.log("c");
await second.unlock();
console.log("d");
}
main().then(console.log, console.error); And just to be 100% sure, there appear to be ~3.5 seconds in your logs between acquiring the lock and releasing it: I assume the lock duration is greater than this? |
same here. the error occurs sometimes but until now I only lock inside the same process. any solution? alternatives? |
The only input I can provide is that, you should not simply increase the lock/unlock time/timeout blindly. This is not really a solution and it can only mask other problems of the library or your system. If there is something wrong with your system, this can result in lock time exceptions. Either acquiring or releasing. For example, if your lengthy processes take more time than the total lock acquiring time, then any other attempt to acquire the lock will expire and throw an error. Such cases should be considered as false positives as regards to this issue I initially opened. As they are related with misconfiguration or bugs of the system using the lock mechanism. For my use-case, the system was tested; with each operation reaching a total lock time of 1-2 seconds as a maximum. After upgrading we started hitting the error, consistently. The only workaround so far was to roll back to redlock version 3.1.2. I believe that there is something wrong with the changes introduced |
@mike-marcacci Here is a script for the bug reproduction:
You can try run it with different library tags and make sure it works for 3.1.2 but not for 4.1.0. I personally used Redis server version 3.2.12 into a docker container and ioredis 4.14.2. My suggestion here is that you have broken that functionality by changing Lua scripts for locking and unlocking. Seems like a critical bug since an application code throws an error on an attempt to unlock a resource after a timeout has exceeded. |
@mike-marcacci So the problem is that in 3.1.2:
that passes for a case if an unlocked value does not exist and incremented As for 4.1.0 the algorithm has been broken:
that does not pass for the case. |
@gnitsenko93 perhaps I'm missing something here (I'm still on my first coffee), but in your reproduction the lock has already expired and the keys have been evicted by redis... so calling unlock should error. This way your application can alert you to the fact that the lock expired and became re-acquirable before you unlocked it. If this is the scenario that is being described in this issue, this is expected behavior in v4 as described in CHANGELOG.md. This would make sense given my inability to reproduce an issue that folks are clearly seeing in the wild. Now, perhaps this isn't a scenario that you would consider an error (and initially I didn't either, given this library's earlier behavior). However, given that the purpose of this library is to guarantee exclusivity of a particular resource, I would consider it an error where the specified expiration breaks this guarantee. |
@mike-marcacci Got ya. Closing my pull request. |
@mike-marcacci Still there is a small inaccuracy in the changelog.
Should be actually:
|
@mike-marcacci could the error returned get more details? Right now it says (mostly asking since we're seeing these errors now when we didn't before, and more context on what is failing would help us narrow down the issue 🙂) |
I think we could just catch that error and handle it not to propagate because it doesn't harm anything just like a warning. We just need to set |
I started having this issue too. We have been on version 4 for a while but today out of no where, I started to get this error where unable to fully release the lock. I listened for expired event and do not see any event with the locking key. |
I now also see the bug in 4.2.0, I want to know which version should I use to avoid this bug? |
+1 |
Same issue when upgrading to the latest. |
does the new v5 fix this even you don't use the new |
I have the same issue when acquire with a less ttl than task executing time. |
This error occurs when we attempt to release an already expired lock. A condition like below while releasing the lock would help
|
When you received an error as "Unable to fully release the lock on resource" the operation after acquiring the lock took more time than mentioned in TTL in the lock argument. Try to increase the TTL value and verify it again. |
Hi folks! I'm going to go ahead and close this, as the newest published versions are effectively full rewrites and any residual bugs from the former versions are unlikely to persist. It's still not entirely clear whether there is a but here at all or just expected behavior when a routine takes longer than the lock. Regardless, if anybody encounters this again, please open a new issue that we can constrain to a particular version. |
shouldn't it be lock.expiration > Date.now()? |
I suppose It should be > |
Not true... :( Still encountering this. |
After upgrading to v4, I started getting this error
LockError: Unable to fully release the lock on resource
.Redis server v=5.0.7
. (appears to have the same problems in newer versions as well)I rolled back to version 3.1.2 for now.
This could be related with #39
The text was updated successfully, but these errors were encountered: