Distributed Background Jobs: Improve distributed background job locking behavior and performance

[nikolajlauridsen]

Summary

This PR improves the distributed background job system with better locking behavior and performance optimizations:

• Prevent picking up already running jobs: Jobs that are currently being executed by another server are no longer picked up, preventing duplicate execution
• Handle stale/timed-out jobs: Added a configurable MaximumExecutionTime setting (default 5 minutes) to detect and recover from jobs that appear stuck (server crashed during execution)
• Optimize EnsureJobsAsync: Reduced time spent holding the database lock by batching inserts and deletes, and pre-computing registered job data outside the critical section
• Add bulk repository operations: New batch Add and Delete methods on IDistributedJobRepository for improved performance
• Comprehensive test coverage: Added integration tests covering all job lifecycle scenarios

Changes

DistributedJobSettings.cs

• Added MaximumExecutionTime configuration option (default: 5 minutes)

IDistributedJobRepository.cs

• Added Add(IEnumerable<DistributedBackgroundJobModel>) for bulk inserts
• Added Delete(IEnumerable<DistributedBackgroundJobModel>) for bulk deletes
• Default implementations provided for backwards compatibility (TODO: remove in V18)

DistributedJobRepository.cs

• Implemented bulk Add using InsertBulk
• Implemented bulk Delete using WhereIn SQL for efficient deletion

DistributedJobService.cs

• TryTakeRunnableAsync: Now checks IsRunning flag and only picks up running jobs if they've exceeded Period + MaximumExecutionTime (stale job recovery)
• EnsureJobsAsync: Optimized to minimize lock hold time:
  • Pre-computes registered job data outside the lock
  • Collects all changes first, then executes batch operations
  • Uses bulk insert/delete instead of individual operations

DistributedJobServiceTests.cs (new)

Integration tests covering:

• Job is due and not running → returns job
• Job is not due → returns null
• Job is running but not timed out → returns null (prevents duplicate execution)
• Job is running but timed out → returns job (stale recovery)
• Finish marks job as not running
• EnsureJobs registers new jobs
• EnsureJobs removes unregistered jobs
• EnsureJobs updates job period when changed
• EnsureJobs is idempotent
• Boundary condition: job running just before timeout threshold

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR enhances the distributed background job system to prevent duplicate job execution in multi-server scenarios and adds recovery mechanisms for stale jobs. The implementation adds a MaximumExecutionTime configuration setting (default: 5 minutes) to detect timed-out jobs, optimizes the job registration process by batching database operations, and introduces comprehensive integration tests to validate all job lifecycle scenarios.

Key changes:

• Added stale job recovery logic that allows jobs exceeding their configured timeout to be reclaimed by another server
• Optimized EnsureJobsAsync to reduce database lock hold time through batch operations and pre-computation
• Added bulk repository methods to enable efficient batch insertions and deletions

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/Umbraco.Tests.Integration/Umbraco.Infrastructure/Services/DistributedJobServiceTests.cs	New comprehensive integration test suite covering job lifecycle, timeout behavior, idempotence, and edge cases
src/Umbraco.Infrastructure/Services/Implement/DistributedJobService.cs	Updated job claiming logic to check running status and timeout, optimized EnsureJobsAsync with batching, added new constructor accepting settings
src/Umbraco.Infrastructure/Persistence/Repositories/Implement/DistributedJobRepository.cs	Implemented bulk Add and Delete operations using InsertBulk and WhereIn for efficient batch operations
src/Umbraco.Infrastructure/Persistence/Repositories/IDistributedJobRepository.cs	Added bulk Add and Delete interface methods with backward-compatible default implementations
src/Umbraco.Infrastructure/BackgroundJobs/DistributedBackgroundJobHostedService.cs	Removed unused using statement and cleaned up XML documentation comments
src/Umbraco.Core/Configuration/Models/DistributedJobSettings.cs	Added MaximumExecutionTime configuration property with 5-minute default

[Zeegaan]

Looks and tests good, just have a few minor nitpicks here and there that aren't mandatory 🙈