Queue Mode - Occasional missing locks in queue leading to noisy logs / stuck threads #5171
Description
Describe the bug
Version:
3.0.2
Errors:
Error: could not renew lock for job 242063
11:07:38.632
at /usr/src/node_modules/.pnpm/[email protected]/node_modules/bullmq/dist/cjs/classes/worker.js:766:40
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Worker.extendLocks (/usr/src/node_modules/.pnpm/[email protected]/node_modules/bullmq/dist/cjs/classes/worker.js:756:9)
at async Timeout._onTimeout (/usr/src/node_modules/.pnpm/[email protected]/node_modules/bullmq/dist/cjs/classes/worker.js:706:29)
11:07:46.133
Error: could not renew lock for job 242063
at /usr/src/node_modules/.pnpm/[email protected]/node_modules/bullmq/dist/cjs/classes/worker.js:766:40
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Worker.extendLocks (/usr/src/node_modules/.pnpm/[email protected]/node_modules/bullmq/dist/cjs/classes/worker.js:756:9)
at async Timeout._onTimeout (/usr/src/node_modules/.pnpm/[email protected]/node_modules/bullmq/dist/cjs/classes/worker.js:706:29)
Repeats on the worker ad-nauseum.
Worker configuration:
- name: MODE
value: queue
- name: QUEUE_REDIS_EVENT_STREAM_MAX_LEN
value: "2000"
- name: REMOVE_ON_COUNT
value: "2000"
- name: REMOVE_ON_AGE
value: "86400"
- name: ENABLE_BULLMQ_DASHBOARD
value: "false"
To Reproduce
Unable to identify root cause, although bullmq do have a troubleshooting page related to this issue - https://docs.bullmq.io/guide/troubleshooting#missing-locks
Expected behavior
- No lock clashes
- If lock clash, single log and cleanup / does not become an ongoing issue
- Log identifies problematic chatflow if due to single chatflow config [seems unlikely in this case]
Screenshots
No response
Flow
No response
Use Method
pnpm start
Flowise Version
3.0.2
Operating System
Linux
Browser
None
Additional context
No response