Auto extend lock

[kusumo1920]

Hi, after reading documentation on npm package page, is there a way to auto extend ttl of lock, to prevent code executed outside the lock? Considering it's possible that some async work haven't finished. Thank you.

[mike-marcacci]

This is a fantastic idea, and one that I will look into implementing. I've been exceedingly busy elsewhere, but have a lot of changes planned for this library, which I hope to make shortly.

[kusumo1920]

Great! Looking forward to it.

[Sytten]

Also looking forward to that, it would be super useful

[onichandame]

+1 for this feature too. Btw, I am using setInterval(()=>lock.extend(TTL), TTL/2) as a workaround

[clouedoc]

Hello, we've encountered this issue in production with long-running tasks.

We are going to fix it with a variant of @onichandame's suggestion.

Locking code

import Redlock from "redlock"
import { redisClient } from "@force-adverse/database"
import { PROXY_LOCK_TIMEOUT } from "../constants/proxy-lock-constants"
import { logger } from "@force-adverse/logger"

/**
 * Get the lock key for a given proxy ID
 * @param proxyId the proxy ID for which we want to get a Redlock/Mutex ID
 */
function getProxyLockKey(proxyId: string): string {
  return `proxy-lock-${proxyId}`
}

/**
 * A RedLock proxy corresponding to a proxy
 */
export interface IProxyLock {
  unlock: () => Promise<void>
  key: string
}

/**
 * Return a Redlock.Lock for the given proxy ID
 * @param isTaskStillRunning a function that tells the lock if the task is still running. This is used to extend the lock
 * @param ttl the base time to live of the lock. Used for testing
 */
export async function getProxyLock(
  proxyId: string,
  ttl: number = PROXY_LOCK_TIMEOUT
): Promise<IProxyLock> {
  const redlock = new Redlock([redisClient()], { retryCount: 1 })
  const key = getProxyLockKey(proxyId)
  let lock = await redlock.lock(key, ttl)

  let timeout: NodeJS.Timeout

  // automatically extend the lock
  const extendLater = (): void => {
    timeout = setTimeout(async () => {
      let isExpired = false
      try {
        // eslint-disable-next-line require-atomic-updates
        lock = await lock.extend(ttl)
      } catch (err) {
        isExpired = err.message.includes("Cannot extend lock on resource")
        if (isExpired) {
          logger.error("The lock expired", { key })
        } else {
          throw err
        }
      }

      if (!isExpired) {
        extendLater()
      }
    }, ttl / 2) as unknown as NodeJS.Timeout // otherwise, we get a compilation error with the testing tsconfig
  }

  extendLater()

  return {
    unlock: async () => {
      clearTimeout(timeout)
      return lock.unlock()
    },
    key
  }
}

Testing code

import { delay, uuid } from "@force-adverse/utils"
import { getProxyLock } from "./proxy-lock"

const testTtl = 1000

jest.setTimeout(30 * testTtl)

/**
 * Note: you should have a Redis client ready
 */
it("does not crash when unlocking after TTL", async () => {
  const lock = await getProxyLock(uuid(), testTtl)
  await delay(testTtl + testTtl * 5)
  await expect(lock.unlock()).resolves.not.toThrow()
})

Related issue

#103

[mike-marcacci]

Hi there @clouedoc – thanks for this contribution!

For everybody here: in the v5 alpha (the current main branch), I have added a "using" method which wraps a routine within an auto-extending lock. This leverages the AbortController standard to communicate any extension failures into the running routine. This would be used like:

await redlock.using([senderId, recipientId], 5000, async (signal) => {
  // Do something...
  await something();

  // Make sure any necessary lock extension has not failed.
  if (signal.aborted) {
    throw signal.error;
  }

  // Do something else...
  await somethingElse();
});

I'm going to close this issue, but do feel free to continue discussing here if you have questions or further suggestions!