Skip to content
Merged
14 changes: 10 additions & 4 deletions docs/site/src/content/docs/grains/timers.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,13 @@ Register timers with <xref:Orleans.GrainBaseExtensions.RegisterGrainTimer*>. <xr

### Callback scheduling

A timer callback never overlaps itself. Orleans waits for the callback task to complete and then measures the period before scheduling the next callback. Callback duration therefore adds to the interval between callback starts.
Orleans completes each callback task before scheduling the next callback for that timer. It then measures the period, so callback duration adds to the interval between callback starts.

Timer callbacks are local-only messages addressed to their activation. They participate in normal turn scheduling and stay on the activation which registered them.
Timer callbacks are local-only messages addressed to their activation. They participate in normal turn scheduling and start a fresh call chain. Each callback receives a fresh request context, while the state passed at registration remains available to the callback.

A zero due time queues a callback directly on the activation. A zero period queues the next callback after the current callback completes. Both use the same activation scheduling, interleaving, and idle-lifetime rules as delayed ticks.

Orleans coalesces repeated immediate changes into one pending callback and reuses the timer's invocation and cancellation scope. It creates a physical timer when a delayed arm is needed and reuses it for later delayed arms. If a physical tick arrives early, Orleans re-arms the remaining delay rounded up to milliseconds, preserving the pending callback.

### Interleaving

Expand All @@ -50,9 +54,11 @@ With <xref:Orleans.Runtime.GrainTimerCreationOptions.KeepAlive> set to `true`, e

## Change or stop a timer

Call <xref:Orleans.Runtime.IGrainTimer.Change*> to replace the due time and period. The new due time schedules the next callback, and the new period applies after that callback completes. A change made inside a running callback takes effect after the callback completes.
Call <xref:Orleans.Runtime.IGrainTimer.Change*> to replace the due time and period. The new due time schedules the next callback, and the new period applies after that callback completes. Changes during a running callback take effect after it completes, with the latest change determining the next schedule.

A change to a delayed or infinite due time invalidates a queued tick. The queued message drains according to activation scheduling, and the replacement schedule determines the next callback.
Comment thread
ReubenBond marked this conversation as resolved.
Resolved

Dispose <xref:Orleans.Runtime.IGrainTimer> to cancel its callback token and stop future callbacks. Orleans also cancels the token and disposes the timer when the activation begins deactivating.
Dispose <xref:Orleans.Runtime.IGrainTimer> to invalidate queued ticks, cancel the token of an admitted callback, and stop further scheduling. Queued messages drain through activation scheduling. Orleans also cancels the token and disposes the timer when the activation begins deactivating.

## Handle callback failures

Expand Down
194 changes: 148 additions & 46 deletions src/Orleans.Runtime/Timers/GrainTimer.cs
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
using System.Diagnostics;
using System.Diagnostics.CodeAnalysis;
using System.Reflection;
using Microsoft.Extensions.Logging;
Expand All @@ -15,14 +16,20 @@ internal abstract partial class GrainTimer : IGrainTimer
protected static readonly TimerCallback TimerCallback = (state) => ((GrainTimer)state!).ScheduleTickOnActivation();
protected static readonly MethodInfo InvokableMethodInfo = typeof(IGrainTimerInvoker).GetMethod(nameof(IGrainTimerInvoker.InvokeCallbackAsync), BindingFlags.Instance | BindingFlags.Public)!;
private readonly CancellationTokenSource _cts = new();
private readonly ITimer _timer;
private ITimer? _timer;
private readonly IGrainContext _grainContext;
private readonly TimerRegistry _shared;
private readonly bool _interleave;
private readonly bool _keepAlive;
private readonly TimerTickInvoker _invoker;
private bool _changed;
private bool _firing;
private bool _queued;
private bool _pendingTick;
private bool _scheduled;
private bool _disposed;
private long _scheduledAt;
private TimeSpan _scheduledDueTime;
private TimeSpan _dueTime;
private TimeSpan _period;

Expand All @@ -38,17 +45,44 @@ public GrainTimer(TimerRegistry shared, IGrainContext grainContext, bool interle
_dueTime = Timeout.InfiniteTimeSpan;
_period = Timeout.InfiniteTimeSpan;
_invoker = new(this);
}

protected IGrainContext GrainContext => _grainContext;

// Avoid capturing async locals.
using (new ExecutionContextSuppressor())
// Called with _cts locked. There is at most one queued invocation and one active callback.
private bool ChangeTimer(TimeSpan dueTime)
{
_pendingTick = dueTime == TimeSpan.Zero;
_scheduled = dueTime != TimeSpan.Zero && dueTime != Timeout.InfiniteTimeSpan;
if (_scheduled)
{
_timer = shared.TimeProvider.CreateTimer(TimerCallback, this, Timeout.InfiniteTimeSpan, Timeout.InfiniteTimeSpan);
_scheduledAt = _shared.TimeProvider.GetTimestamp();
// Match the millisecond resolution used by System.Threading.Timer.
_scheduledDueTime = TimeSpan.FromMilliseconds((long)dueTime.TotalMilliseconds);
if (_timer is null)
{
using (new ExecutionContextSuppressor())
{
_timer = _shared.TimeProvider.CreateTimer(TimerCallback, this, Timeout.InfiniteTimeSpan, Timeout.InfiniteTimeSpan);
}
}

_timer.Change(dueTime, Timeout.InfiniteTimeSpan);
}
else
{
// Disarm an existing delayed timer, but never create one for an immediate or infinite arm.
_timer?.Change(Timeout.InfiniteTimeSpan, Timeout.InfiniteTimeSpan);
if (_pendingTick && !_queued)
{
_queued = true;
return true;
}
}

return false;
}

protected IGrainContext GrainContext => _grainContext;

private ILogger Logger => _shared.TimerLogger;

[DoesNotReturn]
Expand All @@ -65,13 +99,50 @@ private static void ThrowInvalidSchedulingContext()

protected void ScheduleTickOnActivation()
{
try
lock (_cts)
{
// Indicate that the timer is firing so that the effect of the next change call is deferred until after the tick completes.
_firing = true;
// A callback from a previous physical arm can arrive after Change or Dispose. Only the
// current elapsed deadline is eligible, and changing to zero/infinite invalidates it.
if (!_scheduled)
{
return;
}

// Note: this does not execute on the activation's execution context.
// An armed physical deadline belongs to a live timer awaiting callback admission.
Debug.Assert(!_disposed && !_firing && _timer is not null);
var elapsed = _shared.TimeProvider.GetElapsedTime(_scheduledAt);
if (elapsed < _scheduledDueTime)
{
// Physical timers can fire before this higher-resolution deadline. Preserve the
// one-shot arm, rounding up so a sub-millisecond remainder stays asynchronous.
var remaining = _scheduledDueTime - elapsed;
_timer!.Change(TimeSpan.FromMilliseconds(Math.Ceiling(remaining.TotalMilliseconds)), Timeout.InfiniteTimeSpan);
return;
}

_scheduled = false;
_pendingTick = true;
if (_queued)
{
return;
}

_queued = true;
}

QueueTickOnActivation();
}

// The admission is reserved under _cts, but delivered outside it: activation shutdown disposes
// timers under its own lock. Taking that lock while holding _cts would invert the lock order.
// Changes/disposal between reservation and delivery coalesce or invalidate this tick.
private void QueueTickOnActivation()
{
try
{
var msg = _shared.MessageFactory.CreateMessage(body: _invoker, options: InvokeMethodOptions.OneWay);
// Timer requests start a new call chain, including ticks queued immediately during registration.
msg.RequestContextData = null;
Comment thread
ReubenBond marked this conversation as resolved.
Resolved
msg.SetInfiniteTimeToLive();
msg.SendingGrain = _grainContext.GrainId;
msg.TargetGrain = _grainContext.GrainId;
Expand All @@ -88,14 +159,19 @@ protected void ScheduleTickOnActivation()
}
catch (Exception exception)
{
lock (_cts)
{
_queued = false;
_pendingTick = false;
}

try
{
LogErrorScheduleTickOnActivation(Logger, exception, this);
}
catch
{
// Ignore.
// Allowing an exception to escape here would crash the process.
// Allowing an exception to escape a physical timer callback would crash the process.
}
}
}
Expand All @@ -104,11 +180,25 @@ protected void ScheduleTickOnActivation()

private ValueTask<Response> InvokeGrainTimerCallbackAsync()
{
lock (_cts)
{
_queued = false;
if (!_pendingTick)
{
// A changed schedule or disposal invalidated this tick. Drain the activation
// message while preserving the replacement schedule and callback diagnostics.
return new(Response.Completed);
}

_pendingTick = false;
_firing = true;
_changed = false;
}

try
{
LogTraceBeforeCallback(Logger, this);

_changed = false;
GrainTimerEvents.EmitTickStart(GrainContext, this);
var task = InvokeCallbackAsync(_cts.Token);

Expand Down Expand Up @@ -138,32 +228,23 @@ private ValueTask<Response> InvokeGrainTimerCallbackAsync()

private void OnTickCompleted()
{
// Schedule the next tick.
try
bool queueTick;
lock (_cts)
{
if (_cts.IsCancellationRequested)
// Release the active callback before admitting another immediate tick. A Change while
// the callback was running is coalesced, with the last change winning.
_firing = false;
if (_disposed)
{
// The instance has been disposed. No further ticks should be fired.
return;
}

if (!_changed)
{
// If the timer was not modified during the tick, schedule the next tick based on the period.
_timer.Change(_period, Timeout.InfiniteTimeSpan);
}
else
{
// If the timer was modified during the tick, schedule the next tick based on the new due time.
_timer.Change(_dueTime, Timeout.InfiniteTimeSpan);
}
}
catch (ObjectDisposedException)
{
queueTick = ChangeTimer(_changed ? _dueTime : _period);
}
finally

if (queueTick)
{
_firing = false;
QueueTickOnActivation();
}
}

Expand Down Expand Up @@ -201,24 +282,30 @@ public void Change(TimeSpan dueTime, TimeSpan period)
{
ValidateArguments(dueTime, period);

_changed = true;
_dueTime = dueTime;
_period = period;

// If the timer is currently firing, the change will be deferred until after the tick completes.
// Otherwise, perform the change now.
if (!_firing)
var queueTick = false;
lock (_cts)
{
try
if (_disposed)
{
// This method resets the timer, so the next tick will be scheduled at the new due time and subsequent
// ticks will be scheduled after the specified period.
_timer.Change(dueTime, Timeout.InfiniteTimeSpan);
return;
}
catch (ObjectDisposedException)

_changed = true;
_dueTime = dueTime;
_period = period;

// Changes during an active callback take effect after completion. A queued message can
// instead be coalesced or invalidated now, without admitting a second invocation.
if (!_firing)
{
queueTick = ChangeTimer(dueTime);
}
}

if (queueTick)
{
QueueTickOnActivation();
}
}

private static void ValidateArguments(TimeSpan dueTime, TimeSpan period)
Expand All @@ -238,17 +325,32 @@ private static void ValidateArguments(TimeSpan dueTime, TimeSpan period)

public void Dispose()
{
lock (_cts)
{
if (_disposed)
{
return;
}

// Publish disposal before cancellation, whose registrations can reenter Change/Dispose.
_disposed = true;
Comment thread
ReubenBond marked this conversation as resolved.
Resolved
_scheduled = false;
// Admission shares this lock, so queued ticks drain even while cancellation is pending.
_pendingTick = false;
_timer?.Dispose();
_timer = null;
}

try
{
// Do not run cancellation registrations under the timer's state lock.
_cts.Cancel();
}
catch (Exception exception)
{
LogErrorCancellingCallback(Logger, exception);
}

_timer.Dispose();

GrainTimerEvents.EmitDisposed(GrainContext, this);

var timerRegistry = _grainContext.GetComponent<IGrainTimerRegistry>();
Expand Down
Loading
Loading