HybridCache: Optimize content type change cache rebuild to resolve SQL timeouts

[AndyButland]

Summary

This PR addresses SQL timeout issues that can occur when saving document types with large amounts of existing content. The issue manifests as database lock contention during the cache rebuild process.

Changes included:

• Fix scope completion in distributed job service: Complete the database scope when no runnable job is found, preventing lock contention with concurrent document type save operations
• Parallel serialization during cache rebuild: Run content serialization steps in parallel for faster rebuild performance
• Lightweight queries for memory cache invalidation: Avoid loading full ContentCacheNode objects when only keys are needed
  • Added GetDocumentKeysWithPublishedStatus for documents
  • Added GetMediaKeysByContentTypeKeys for media
  • Updated DocumentCacheService, MediaCacheService, and MemberCacheService to use these optimized methods
• Optimization and custom data access for database cache rebuild: Avoiding use of document/media/member repositories that will return fully hydrated IContent/IMedia/IMember objects that we don't fully need, and replacing with dedicated SQL to get just the data we require.
• Add integration tests: Tests for DatabaseCacheRepository.Rebuild() covering documents, media, and members including composition scenarios, to verify that the optimizations don't change the behaviour.

Performance impact:

I've been testing with a database containing around 70,000 nodes in two languages, that second-line support use as a sample database for a large site. There's one document type that almost every node depends on, so changing this causes the database cache to be rebuilt for all practically all the content.

Stage	Current	New (partial implementing, using document/media/member repositories)	New (full implementation, using custom data access)	Improvement
Database cache rebuild	119s	107s	87s	26.9%
Memory cache clear	37s	0.25s	0.25s	99.3%
Overall time for document type save	193s	107s	87s	54.9%

The other benefit is that I'm no longer seeing locking issues and timeouts as were previously found (with both the current and partial implementation). To get these to run to completion with the older implementations, I often needed to disable distributed jobs and examine indexing.

Testing

I've added various integration tests to verify that the optimisation in behaviour doesn't impact the functionality. These all pass on main (with the one exception of property order in the JSON serialization, which I don't believe matters).

Manual checks would include making sure on save of a document type (e.g. to remove a property) that an existing rendered document continues to have existing properties displayed and the removed one not.

[Copilot]

Pull request overview

This PR optimizes HybridCache's content type change cache rebuild process to prevent SQL timeouts on sites with large amounts of content. The primary improvements include fixing database scope completion in the distributed job service, implementing parallel serialization during cache rebuilds, using lightweight queries for memory cache invalidation, and ensuring composed properties are included in rebuilt cache entries.

Key Changes:

• Fixed scope completion bug in DistributedJobService that could cause lock contention
• Optimized cache rebuild with parallel processing and direct SQL queries instead of entity hydration
• Added lightweight query methods to avoid loading serialized data when only keys are needed
• Ensured property aliases from compositions are included in cache rebuild

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/Umbraco.Infrastructure/Services/Implement/DistributedJobService.cs	Fixes scope completion when no job is found and adds debug logging
src/Umbraco.PublishedCache.HybridCache/Services/DocumentCacheService.cs	Updates to use lightweight queries for memory cache invalidation
src/Umbraco.PublishedCache.HybridCache/Services/MediaCacheService.cs	Updates to use lightweight queries and consolidates rebuild logic
src/Umbraco.PublishedCache.HybridCache/Services/MemberCacheService.cs	Adds Rebuild method implementation
src/Umbraco.PublishedCache.HybridCache/Services/IMemberCacheService.cs	Adds Rebuild method to interface
src/Umbraco.PublishedCache.HybridCache/Persistence/IDatabaseCacheRepository.cs	Adds new lightweight query methods for keys
src/Umbraco.PublishedCache.HybridCache/Persistence/DatabaseCacheRepository.cs	Major refactoring to use direct SQL queries with parallel processing instead of entity hydration
src/Umbraco.Infrastructure/Umbraco.Infrastructure.csproj	Adds InternalsVisibleTo for HybridCache project
tests/Umbraco.Tests.Integration/Umbraco.PublishedCache.HybridCache/DocumentCacheServiceTests.cs	New integration tests for document cache rebuild
tests/Umbraco.Tests.Integration/Umbraco.PublishedCache.HybridCache/MediaCacheServiceTests.cs	New integration tests for media cache rebuild including composition tests
tests/Umbraco.Tests.Integration/Umbraco.PublishedCache.HybridCache/MemberCacheServiceTests.cs	New integration tests for member cache rebuild

[AndyButland]

I asked Claude for an analysis of the data access changes as compared to the current, repository based approach.

---

OLD APPROACH: Repository-Based

The previous implementation used IDocumentRepository, IMediaRepository, and IMemberRepository:

1. Full Entity Hydration
The GetPage() method in DocumentRepository.cs:1534-1571 called MapDtosToContent() which:

• Created complete IContent objects via ContentBaseFactory.BuildEntity() (line 282)
• Loaded content types via _contentTypeRepository.Get()
• Loaded all properties into PropertyCollection via GetPropertyCollections()
• Loaded culture variations, templates, and enabled change tracking

2. SQL Generated (inefficient pattern)

-- Complex base query with many JOINs
SELECT * FROM umbracoNode n
INNER JOIN umbracoContent c ON n.nodeId = c.nodeId
INNER JOIN umbracoContentVersion cv ON n.nodeId = cv.nodeId AND cv.Current = 1
INNER JOIN cmsDocument d ON n.nodeId = d.nodeId
LEFT JOIN cmsContentType ctype ON c.contentTypeId = ctype.nodeId
LEFT JOIN umbracoContentVersion pcv ON n.nodeId = pcv.nodeId  -- published version
LEFT JOIN ContentVersionCultureVariationDto ccv ...
WHERE n.nodeObjectType = @documentType
ORDER BY n.path
OFFSET @offset ROWS FETCH NEXT @pageSize ROWS ONLY

-- Then SEPARATE query for properties using WHERE IN (expensive):
SELECT * FROM cmsPropertyData 
WHERE versionId IN (@v1, @v2, @v3, ...)  -- INDEX SCAN!

The WHERE versionId IN (...) pattern causes expensive index scans on the cmsPropertyData table.

---

NEW APPROACH: Direct SQL Queries

1. Dependency Removal (DatabaseCacheRepository.cs lines 24-40)

// OLD - Three repository dependencies:
private readonly IDocumentRepository _documentRepository;
private readonly IMediaRepository _mediaRepository;
private readonly IMemberRepository _memberRepository;

// NEW - No repository dependencies, direct SQL queries only

2. Lightweight Key Queries (lines 232-304)
Two new methods added for cache refresh that fetch ONLY keys:

// GetDocumentKeysWithPublishedStatus - Only fetches key + published flag
SELECT n.uniqueId, nu.published
FROM cmsContentNu nu
INNER JOIN umbracoNode n ON nu.nodeId = n.nodeId
INNER JOIN umbracoContent c ON n.nodeId = c.nodeId
INNER JOIN umbracoNode ct ON c.contentTypeId = ct.nodeId
WHERE n.nodeObjectType = @document
AND ct.uniqueId IN (@contentTypeKeys)

// GetMediaKeysByContentTypeKeys - Only fetches keys
SELECT n.uniqueId FROM cmsContentNu nu ...

3. Optimized Rebuild Pattern (lines 363-518)

Pre-fetches lookup data once:

Dictionary<int, byte> contentTypeVariations = GetContentTypeVariations(contentTypeIds);
Dictionary<short, string> languageMap = GetLanguageMap();
Dictionary<int, List<PropertyTypeInfo>> propertyInfoByContentType = GetPropertyInfoByContentType(contentTypeIds);

Paged ID query:

SELECT nodeId FROM umbracoNode n
INNER JOIN umbracoContent c ON n.nodeId = c.nodeId
WHERE n.nodeObjectType = @document
ORDER BY n.path
-- With paging

4. Efficient JOIN-Based Property Query (lines 453-468)

-- NEW: JOIN through nodeId → versionId path
SELECT pd.versionId, pd.languageId, pd.segment, pd.intValue, pd.decimalValue,
       pd.dateValue, pd.varcharValue, pd.textValue, pt.alias AS PropertyAlias
FROM cmsPropertyData pd
INNER JOIN cmsPropertyType pt ON pd.propertyTypeId = pt.id
INNER JOIN umbracoContentVersion cv ON pd.versionId = cv.id
WHERE cv.nodeId IN (@n1, @n2, @n3, ...)  -- More efficient query plan

The key difference: WHERE cv.nodeId IN (...) with a JOIN through ContentVersionDto produces better SQL Server query plans than WHERE versionId IN (...).

5. Parallel Serialization (lines 386-399)

var items = contentDtos
    .AsParallel()
    .WithDegreeOfParallelism(Environment.ProcessorCount)
    .SelectMany(content => BuildCacheDtosForDocument(...))
    .ToList();

---

Summary of Speed Improvements

Optimization	Old Approach	New Approach	Improvement
Entity hydration	Full IContent creation with change tracking	Direct DTO building, no entity	~50% faster
Property queries	WHERE versionId IN (...) - index scan	JOIN through nodeId - seek	Better query plans
Cache refresh	Loaded all serialized data	Only keys + status flags	99.3% faster
Serialization	Sequential	Parallel (Environment.ProcessorCount)	5-10% faster
Lookups	Per-item queries	Pre-fetched once	Reduced DB roundtrips

Safe Simplifications Made

1. Removed repository dependencies - The cache rebuild only needs raw data for serialization, not full domain objects with business logic
2. No content type resolution - Pre-fetches variations once instead of resolving full IContentType per item
3. No template loading - Cache doesn't need template references
4. No property collection creation - Builds property data dictionary directly without Property object overhead
5. No change tracking - Cache nodes are immutable; no need for dirty tracking infrastructure

[Migaroez]

I had a concern about the parralelism, but after digging deeper it seems like all the functions that are called are pure functions.

I am wondering whether we should add a comment to BuildCacheDtosForDocument that it is being used in a parallel path so that if any changes are being made to it, people don't forget to keep it pure.