Resolve TimeProvider via keyed DI per subsystem (fixes FakeTimeProvider timer test CI hang) - #10271
Merged
ReubenBond merged 9 commits intoJul 16, 2026
Conversation
TimerOrleansTest_Parallel and sibling timer tests intermittently hung for 10 minutes in CI (hitting the blame-hang timeout) and failed builds. The fixtures install a FakeTimeProvider as the silo's global TimeProvider so grain timers can be advanced deterministically. FakeTimeProvider.Advance invokes every due timer callback synchronously and inline on the advancing thread. The silo's own infrastructure maintenance loops (membership, gateway and directory maintenance, activation working-set monitoring, incoming-request monitoring, reminders, health checks) are driven by IAsyncTimer instances that repeatedly await Task.Delay against the ambient TimeProvider. When those loops run on the fake clock, a single Advance call resumes them inline; each observes its deadline has passed, re-arms a fresh Task.Delay and runs again, all inline on the advancing thread, producing a CPU-bound spin that never returns from Advance. Isolate the two concerns: keep grain timers on the fake clock (so tests retain control), but point the silo's IAsyncTimerFactory at TimeProvider.System so infrastructure timers are never fired inline by Advance. A shared FakeTimeSilo helper applies this to both the TimerOrleansTest and AsyncEnumerableGrainCall fixtures, and wraps Advance with a watchdog that fails fast (instead of hanging for minutes) should infrastructure timers ever be driven by the fake clock again. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5031128-9805-4f96-8fb2-c442bac6f736
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Introduce per-area keyed TimeProvider resolution so individual subsystems can be driven by different clocks. Each consumer resolves a keyed TimeProvider (see TimeProviderNames); a single KeyedService.AnyKey fallback registered by the silo/client resolves to the unkeyed default, so production behavior is unchanged. This replaces the earlier test-only IAsyncTimerFactory override used to stop the FakeTimeProvider silo timer tests from hanging in CI. Tests install a FakeTimeProvider as the default TimeProvider and pin background areas to TimeProvider.System via UseTimeProviderForBackgroundAreas (Orleans.TestingHost), so advancing the fake clock only fires the grain timers under test and never resumes background maintenance loops inline. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5031128-9805-4f96-8fb2-c442bac6f736
ReubenBond
changed the title
Move the extension-owned TimeProvider service keys out of the central Orleans.Core TimeProviderNames class and into per-package key classes (ReminderTimeProviderNames, StreamingTimeProviderNames, TransactionTimeProviderNames, DurableJobTimeProviderNames, JournalingTimeProviderNames). The core class now holds only the runtime-owned areas. Key string values drop the redundant 'TimeProvider' segment. UseTimeProviderForBackgroundAreas no longer enumerates a BackgroundAreas list; it pins all keyed areas via KeyedService.AnyKey and re-pins the grain-facing key to the default provider, so it no longer depends on extension-owned keys. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5031128-9805-4f96-8fb2-c442bac6f736
Inline the single catch-all keyed TimeProvider registration into DefaultSiloServices and DefaultClientServices and remove the TimeProviderHostingExtensions helper class. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5031128-9805-4f96-8fb2-c442bac6f736
IAsyncTimerFactory.Create now takes a required TimeProvider parameter instead of AsyncTimerFactory holding a single keyed provider. Each callsite injects and passes the TimeProvider appropriate to its subsystem: silo background/system timers pass the SystemTimers-keyed provider, and reminder timers pass the reminder subsystem's provider so they are consistent with the reminder due-time calculations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5031128-9805-4f96-8fb2-c442bac6f736
Cluster membership loops (membership agent, membership table manager/cleanup, and silo health monitoring) now resolve a dedicated Membership-keyed TimeProvider rather than sharing the blanket SystemTimers key, so membership timers can be controlled independently. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5031128-9805-4f96-8fb2-c442bac6f736
GrainDirectoryCacheFactory resolves the GrainDirectory-keyed TimeProvider, so the unit test must register its FakeTimeProvider under that key. Registering it only as the unkeyed default caused the keyed lookup to fall back to real time, so advancing the fake clock never expired the cache entry and the test timed out. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5031128-9805-4f96-8fb2-c442bac6f736
The Caching key had no consumer: the shared serialization LRU codec caches are static and use TimeProvider.System directly, and the grain directory cache resolves the GrainDirectory key. Remove the orphaned key rather than leave a name that never resolves to a distinct clock. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5031128-9805-4f96-8fb2-c442bac6f736
LocalDurableJobManagerTests.CreateJournaledServices builds a DI container that resolves JournaledStateManagerShared, which now injects the Journaling-keyed TimeProvider. Register the KeyedService.AnyKey catch-all (as the other durable job and journaling test fixtures do) so the keyed lookup resolves the registered FakeTimeProvider instead of failing to activate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5031128-9805-4f96-8fb2-c442bac6f736
This was referenced Jul 22, 2026
This was referenced Jul 29, 2026
This was referenced Jul 29, 2026
Merged
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
TimerTests.TimerOrleansTest_Paralleland sibling timer tests intermittently hung for ~10 minutes in CI, failing builds.Root cause: the timer test fixtures install a
FakeTimeProvideras the silo's globalTimeProvider.FakeTimeProvider.Advanceinvokes every due timer callback synchronously and inline on the advancing thread. The silo's background maintenance loops (cluster membership, gateway/grain-directory maintenance, activation working-set monitoring, request monitoring, reminders, health checks) also read that same ambient clock. Advancing the fake clock resumed those loops inline; each observed its deadline had passed, immediately re-armed a freshTask.Delayand ran again — a CPU-bound spin that never returned fromAdvance, hanging the test until CI's blame-hang timeout.Solution
Introduce per-area keyed
TimeProviderresolution so individual subsystems can be driven by different clocks:TimeProviderNames(new, public) defines one key per area (grains, reminders, messaging, system timers, activation management, grain directory, streaming, transactions, durable jobs, journaling, caching).[FromKeyedServices(TimeProviderNames.X)](orGetKeyedService).KeyedService.AnyKeyfallback that resolves to the unkeyed defaultTimeProvider, so every key defaults to the default provider and production behavior is unchanged.This replaces the earlier test-only
IAsyncTimerFactoryoverride. Tests install aFakeTimeProvideras the default provider (grain timers + grain-side delays) and pin all background areas toTimeProvider.SystemviaUseTimeProviderForBackgroundAreas(inOrleans.TestingHost). Advancing the fake clock then only ever fires the grain timers under test and never resumes background loops inline. A fail-fast watchdog aroundAdvanceconverts any future regression into an immediate, diagnosable failure instead of a multi-minute hang.Validation
TimerOrleansTest(15) +AsyncEnumerableGrainCallTests(50): pass; 20-iteration stress loop of the timer tests: 0 hangs.ReminderTestClockcascade), journaling (331), durable jobs (27), streaming BVT (64), and transaction overload detector tests: pass.Orleans.CoreandOrleans.TestingHost.