Distributed locks to prevent multiple pods executing Reload and Refresh

[lukhipolito-nexxbiz]

Purpose

Addition of Distributed locks to prevent multiple pods from triggering Reload or Refresh at the same time.

---

Scope

Select one primary concern:

• New feature

---

Description

Distributed lock with generic lock has been added to IWorkflowDefinitionsReloader interface as a decorator. The goal is to prevent multiple pods from ever executing this action at the same time.

The same was done for IWorkflowDefinitionsRefresher, with the exception that the key to the distributed lock uses a combination of definition Ids from the request to allow it to be called by different pods with with different Requests.

Problem

Double or multiple triggering of change notifications and updates/publishing of workflows.

Solution

Distributed locks ensure no race between pods and only one action that should be singleton accross pods is taken.

[greptile-apps]

Greptile Summary

Adds distributed locking decorators to prevent concurrent workflow definition reloads and refreshes across multiple pods.

• Decorator pattern correctly applied in WorkflowRuntimeFeature.cs for both IWorkflowDefinitionsRefresher and IWorkflowDefinitionsReloader
• Uses Medallion.Threading's IDistributedLockProvider with 1-minute timeout
• Lock keys: constant for reloader, composite from sorted definition IDs for refresher
• Critical: Both decorator classes incorrectly marked as abstract, preventing instantiation at runtime
• Critical: WorkflowDefinitionsRefresherDistributedLocking returns early when DefinitionIds is null/empty, but this should mean "refresh all workflows", not "skip operation"

Confidence Score: 1/5

• This PR has critical issues that will cause runtime failures
• The abstract keyword on both decorator classes will cause instantiation failures when the DI container tries to create them. Additionally, the null/empty check in the refresher changes intended behavior
• Both WorkflowDefinitionsRefresherDistributedLocking.cs and WorkflowDefinitionsReloaderDistributedLocking.cs require immediate fixes

Important Files Changed

Filename	Overview
src/modules/Elsa.Workflows.Runtime/Services/WorkflowDefinitionsRefresherDistributedLocking.cs	Adds distributed locking decorator, but has critical issues: abstract keyword prevents instantiation, and null/empty check changes behavior
src/modules/Elsa.Workflows.Runtime/Services/WorkflowDefinitionsReloaderDistributedLocking.cs	Adds distributed locking decorator, but abstract keyword prevents instantiation
src/modules/Elsa.Workflows.Runtime/Features/WorkflowRuntimeFeature.cs	Correctly registers decorators for distributed locking using standard decorator pattern

Sequence Diagram

sequenceDiagram
    participant Client
    participant Decorator as WorkflowDefinitionsRefresher<br/>DistributedLocking
    participant LockProvider as IDistributedLockProvider
    participant Inner as WorkflowDefinitionsRefresher
    participant Store as IWorkflowDefinitionStore

    Client->>Decorator: RefreshWorkflowDefinitionsAsync(request)
    Decorator->>Decorator: Generate lock key from DefinitionIds
    Decorator->>LockProvider: TryAcquireLockAsync(key, 1min)
    alt Lock acquired
        LockProvider-->>Decorator: Lock handle
        Decorator->>Inner: RefreshWorkflowDefinitionsAsync(request)
        Inner->>Store: FindManyAsync(filter)
        Store-->>Inner: Workflow definitions
        Inner->>Inner: Index triggers
        Inner-->>Decorator: Response (refreshed, notFound)
        Decorator-->>Client: Response
        Decorator->>LockProvider: Release lock (via await using)
    else Lock not acquired
        LockProvider-->>Decorator: null
        Decorator-->>Client: Empty response (skip refresh)
    end

Loading

_{Last reviewed commit: a0e6a4a}

[greptile-apps]

_{3 files reviewed, 4 comments}

_{Edit Code Review Agent Settings | Greptile}

[Copilot]

Pull request overview

This pull request adds distributed locking decorators to prevent race conditions when multiple pods concurrently execute workflow definition reload or refresh operations. The implementation uses Medallion.Threading's IDistributedLockProvider to ensure only one pod can execute these operations at a time.

Changes:

• Added WorkflowDefinitionsReloaderDistributedLocking decorator with a global lock to serialize reload operations across all pods
• Added WorkflowDefinitionsRefresherDistributedLocking decorator with per-definition-set locks to allow concurrent refreshes of different workflow sets
• Updated WorkflowRuntimeFeature to register these decorators in the dependency injection container

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
WorkflowDefinitionsReloaderDistributedLocking.cs	New decorator that adds distributed locking to workflow definition reload operations using a single global lock key
WorkflowDefinitionsRefresherDistributedLocking.cs	New decorator that adds distributed locking to workflow definition refresh operations using lock keys based on the set of definition IDs being refreshed
WorkflowRuntimeFeature.cs	Registers the new distributed locking decorators using the Decorate<> pattern after the base service registrations

[sfmskywalker]

Please move the distributed decorators to Elsa.Workflows.Runtime.Distributed.

[sfmskywalker]

This is the remaining issue to look into.

[sfmskywalker]

Looks good in principle, just one question and one code quality change request.