Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/site/src/content/docs/host/client.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,3 +97,11 @@ Use [grain observers](../grains/observers.md) for best-effort, one-way callbacks
## Dependency injection

Let the Generic Host own client startup and shutdown. Don't create a client per request, cache a second client in a static field, or dispose the dependency-injected singleton. When the host receives a termination signal, it closes the client as part of normal shutdown.

## Observer invocations during shutdown

Graceful client shutdown closes admission for new observer invocations and waits for admitted work to finish before shutting down transport. The drain includes queued invocations, running invocations, interleaved invocations, and response handling. Queued work keeps its per-observer order. Cancellation-control calls remain available while admitted application work drains, and then Orleans closes and drains that control path. New request-response calls receive a <xref:Orleans.Runtime.SiloUnavailableException>; rejected one-way calls are logged.

External clients complete outstanding outbound calls with a shutdown exception before waiting for observer invocations, allowing observers awaiting those calls to finish. Co-hosted clients first drain their incoming dispatch queue, then drain observer execution. Incoming responses and callback timeouts continue completing their outbound calls during that drain. Their owned dependency-injection scope is released after execution actually finishes.

The host shutdown token bounds the shutdown wait. When that token is canceled, the host advances its shutdown sequence and owns disposal of root and singleton services. Orleans continues accounting for admitted observer work and defers disposal of the co-hosted client's owned scope until that work exits. Size the [shutdown budget](configuration-guide/shutting-down-orleans.md#configure-a-shutdown-budget) for observer completion, and keep observer operations and cancellation callbacks bounded.
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ Set the orchestrator's termination grace period longer than the host shutdown ti
- Load balancers and readiness probes to stop sending new traffic.
- Gateway and membership changes to propagate.
- Grain deactivation callbacks and state writes.
- Admitted client-observer invocations and their response handling.
- Stream, reminder, storage, and telemetry providers to flush and stop.

If the external grace period expires first, the process is killed and graceful shutdown can't complete.
Expand Down
29 changes: 22 additions & 7 deletions src/Orleans.Core/Async/AdmissionGate.cs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@ namespace Orleans.Internal;
/// <remarks>
/// All gate methods support concurrent callers. Store each <see cref="TryEnter"/> result in a <c>using</c>
/// local and check <see cref="Admission.Entered"/> before starting work. Each admitted token has a single
/// owner which disposes it exactly once. The gate supports up to <see cref="int.MaxValue"/> outstanding entry attempts.
/// owner which disposes it exactly once. For transferred ownership, pair each successful
/// <see cref="TryEnterUnscoped"/> with exactly one <see cref="Exit"/> call.
/// The gate supports up to <see cref="int.MaxValue"/> outstanding entry attempts.
/// </remarks>
internal sealed class AdmissionGate
{
Expand All @@ -28,25 +30,38 @@ internal sealed class AdmissionGate
/// A token whose <see cref="Admission.Entered"/> property indicates whether the operation was admitted.
/// Dispose the token when the operation finishes, preferably with a <c>using</c> declaration.
/// </returns>
public Admission TryEnter()
public Admission TryEnter() => TryEnterUnscoped() ? new(this) : default;

/// <summary>
/// Attempts to admit an operation whose owner will explicitly call <see cref="Exit"/> when it finishes.
/// </summary>
/// <returns>
/// <see langword="true"/> when admitted, ordered atomically against <see cref="CloseAsync"/>.
/// Each successful admission must be paired with exactly one <see cref="Exit"/>, including exception and cancellation paths.
/// A rejected attempt owns no admission.
/// </returns>
public bool TryEnterUnscoped()
{
// Keep new rejected attempts from prolonging the drain after closure.
if (Volatile.Read(ref _state) < 0)
{
return default;
return false;
}

if (Interlocked.Increment(ref _state) > 0)
{
return new(this);
return true;
}

// A rejected attempt still owns a count and may be the last one to release it.
Exit();
return default;
return false;
}

private void Exit()
/// <summary>
/// Releases one successful admission. Its owner calls this exactly once, after the admitted operation finishes.
/// </summary>
public void Exit()
{
if (Interlocked.Decrement(ref _state) == Closed)
{
Expand All @@ -55,7 +70,7 @@ private void Exit()
}

/// <summary>
/// Permanently closes admission before returning, and completes the returned task once all admitted tokens have been disposed.
/// Permanently closes admission before returning, and completes the returned task once all admitted operations have exited.
/// </summary>
/// <returns>
/// The shared drain task, also returned by subsequent calls. Pending continuations run asynchronously.
Expand Down
5 changes: 4 additions & 1 deletion src/Orleans.Core/Core/DefaultClientServices.cs
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,10 @@ public static void AddDefaultServices(IClientBuilder builder)
services.TryAddSingleton<ConnectionCommon>();
services.TryAddSingleton<ConnectionManager>();
services.TryAddSingleton<ConnectionPreambleHelper>();
services.AddSingleton<ILifecycleParticipant<IClusterClientLifecycle>, ConnectionManagerLifecycleAdapter<IClusterClientLifecycle>>();
services.AddSingleton<ILifecycleParticipant<IClusterClientLifecycle>>(sp =>
new ConnectionManagerLifecycleAdapter<IClusterClientLifecycle>(
sp.GetRequiredService<ConnectionManager>(),
ct => sp.GetRequiredService<OutsideRuntimeClient>().StopObserverInvocationsAsync().WaitAsync(ct)));

services.AddKeyedSingleton<IConnectionFactory>(
ClientOutboundConnectionFactory.ServicesKey,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
using System;
using System.Threading;
using System.Threading.Tasks;

Expand All @@ -7,17 +8,29 @@ internal class ConnectionManagerLifecycleAdapter<TLifecycle>
: ILifecycleParticipant<TLifecycle>, ILifecycleObserver where TLifecycle : ILifecycleObservable
{
private readonly ConnectionManager connectionManager;
private readonly Func<CancellationToken, Task>? _beforeClose;

public ConnectionManagerLifecycleAdapter(ConnectionManager connectionManager)
public ConnectionManagerLifecycleAdapter(ConnectionManager connectionManager, Func<CancellationToken, Task>? beforeClose = null)
{
this.connectionManager = connectionManager;
_beforeClose = beforeClose;
}

public Task OnStart(CancellationToken ct) => Task.CompletedTask;

public async Task OnStop(CancellationToken ct)
{
await Task.Run(() => this.connectionManager.Close(ct), CancellationToken.None);
try
{
if (_beforeClose is { } beforeClose)
{
await beforeClose(ct).ConfigureAwait(false);
}
}
finally
{
await Task.Run(() => this.connectionManager.Close(ct), CancellationToken.None).ConfigureAwait(false);
}
}

public void Participate(TLifecycle lifecycle)
Expand Down
91 changes: 81 additions & 10 deletions src/Orleans.Core/Runtime/InvokableObjectManager.cs
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
using System.Threading.Tasks;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Orleans.Internal;
using Orleans.Runtime;
using Orleans.Serialization;
using Orleans.Serialization.Invocation;
Expand All @@ -24,6 +25,8 @@ internal sealed partial class InvokableObjectManager
private readonly DeepCopier deepCopier;
private readonly DeepCopier<Response> _responseCopier;
private readonly MessagingTrace messagingTrace;
private readonly AdmissionGate _invocations = new();
private readonly AdmissionGate _cancellations = new();
private List<IIncomingGrainCallFilter>? _grainCallFilters;

private List<IIncomingGrainCallFilter> GrainCallFilters
Expand Down Expand Up @@ -63,6 +66,32 @@ public bool TryDeregister(ObserverGrainId objectId)
return true;
}

/// <summary>
/// Closes observer admission and drains invocation processing, including cancellation controls and response handling.
/// </summary>
public async Task StopAsync()
{
// Cancellation controls can release admitted work while application admission is closed.
await _invocations.CloseAsync().ConfigureAwait(false);
await _cancellations.CloseAsync().ConfigureAwait(false);
Comment thread
ReubenBond marked this conversation as resolved.
Resolved
}

internal static bool IsCancellationRequest(Message message) =>
message.BodyObject is IInvokable request && request.GetInterfaceType() == typeof(IGrainCallCancellationExtension);

private AdmissionGate GetAdmissionGate(Message message) =>
IsCancellationRequest(message) ? _cancellations : _invocations;

internal void RejectMessage(Message message)
{
LogRejectingMessageDuringShutdown(logger, message);
if (message.Direction == Message.Directions.Request)
{
runtimeClient.SendResponse(message, Response.FromException(
new SiloUnavailableException("The local Orleans host is shutting down and can no longer accept observer invocations.")));
}
}

public void Dispatch(Message message)
{
if (!ObserverGrainId.TryParse(message.TargetGrain, out var observerId))
Expand Down Expand Up @@ -153,6 +182,30 @@ void IGrainContext.SetComponent<TComponent>(TComponent? value) where TComponent
public void ReceiveMessage(object msg)
{
var message = (Message)msg;
var gate = _manager.GetAdmissionGate(message);
if (!gate.TryEnterUnscoped())
{
_manager.RejectMessage(message);
return;
}

var ownsAdmission = true;
try
{
// Queue publication transfers admission ownership to the pump or interleaved invocation.
ReceiveMessage(message, ref ownsAdmission);
}
finally
{
if (ownsAdmission)
{
gate.Exit();
}
}
}

private void ReceiveMessage(Message message, ref bool ownsAdmission)
{
var obj = this.LocalObject.Target;
if (obj is null)
{
Expand Down Expand Up @@ -184,6 +237,7 @@ public void ReceiveMessage(object msg)
CancellationToken.None,
TaskCreationOptions.DenyChildAttach,
TaskScheduler.Default).Unwrap();
ownsAdmission = false;
_runningRequests.Add(message, task);
}
}
Expand All @@ -208,6 +262,7 @@ public void ReceiveMessage(object msg)
else
{
this.Messages.Enqueue(message);
ownsAdmission = false;
start = !this.Running;
this.Running = true;
}
Expand Down Expand Up @@ -247,7 +302,7 @@ public void ReceiveMessage(object msg)
this,
CancellationToken.None,
TaskCreationOptions.DenyChildAttach,
TaskScheduler.Default);
TaskScheduler.Default).Unwrap();
}
}

Expand Down Expand Up @@ -344,20 +399,22 @@ private async Task ProcessMessageAsync(Message message)
{
this.ReportException(message, exc);
}
finally
{
// Clear the running request when done.
lock (Messages)
{
_runningRequests.Remove(message);
}
}
}
catch (Exception outerException)
{
// Ignore and keep looping.
LogErrorProcessingMessage(_manager.logger, outerException, message);
}
finally
{
lock (Messages)
{
_runningRequests.Remove(message);
}

// Generated interface metadata is stable throughout the request's lifetime.
_manager.GetAdmissionGate(message).Exit();
}
}

internal void OnDeregistered()
Expand Down Expand Up @@ -535,7 +592,15 @@ void TryCancelRequest()
// If the message did begin executing, wait for it to observe the cancellation token and respond itself.
if (wasWaiting)
{
SendCanceledResponse(message);
try
{
SendCanceledResponse(message);
}
finally
{
_manager.GetAdmissionGate(message).Exit();
}

didCancel = true;
}
else if (message.BodyObject is IInvokable invokableRequest)
Expand Down Expand Up @@ -601,6 +666,12 @@ private readonly struct ObserverGrainIdLogValue(ObserverGrainId observerId)
public override string ToString() => observerId.ToString();
}

[LoggerMessage(
Level = LogLevel.Debug,
Message = "Rejecting observer invocation during shutdown: '{Message}'."
)]
private static partial void LogRejectingMessageDuringShutdown(ILogger logger, Message message);

[LoggerMessage(
EventId = (int)ErrorCode.ProxyClient_OGC_TargetNotFound_2,
Level = LogLevel.Error,
Expand Down
45 changes: 34 additions & 11 deletions src/Orleans.Core/Runtime/OutsideRuntimeClient.cs
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ internal partial class OutsideRuntimeClient : IRuntimeClient, IDisposable, IClus
private readonly ConcurrentDictionary<CorrelationId, CallbackData> callbacks;
private InvokableObjectManager? localObjects;
private int _isStopping;
private bool disposing;
private int _disposeRequested;
private bool disposed;

private readonly MessagingTrace messagingTrace;
Expand Down Expand Up @@ -158,14 +158,21 @@ public async Task StartAsync(CancellationToken cancellationToken)
LogStartedClient(logger, CurrentActivationAddress, _localClientDetails.ClientId);
}

public async Task StopAsync(CancellationToken cancellationToken)
internal Task StopObserverInvocationsAsync()
{
Volatile.Write(ref _isStopping, 1);
var drain = this.localObjects?.StopAsync() ?? Task.CompletedTask;
this.callbackTimer.Dispose();

// Fault callbacks before any cancellation-sensitive waits. Completing them can resume code
// which issues follow-up calls, so request admission must already be closed.
BreakOutstandingMessages();
Comment thread
ReubenBond marked this conversation as resolved.
Resolved
return drain;
}

public async Task StopAsync(CancellationToken cancellationToken)
{
await StopObserverInvocationsAsync().WaitAsync(cancellationToken).ConfigureAwait(false);

if (this.callbackTimerTask is { } task)
{
Expand Down Expand Up @@ -437,17 +444,27 @@ public void DeleteObjectReference(IAddressable obj)

public void Dispose()
{
if (this.disposing) return;
this.disposing = true;
Volatile.Write(ref _isStopping, 1);

this.callbackTimer.Dispose();
BreakOutstandingMessages();

Utils.SafeExecute(() => MessageCenter?.Dispose());
if (Interlocked.Exchange(ref _disposeRequested, 1) != 0) return;
_ = DisposeAfterDrainAsync(StopObserverInvocationsAsync());

GC.SuppressFinalize(this);
disposed = true;
}

private async Task DisposeAfterDrainAsync(Task drain)
{
try
{
await drain.ConfigureAwait(false);
MessageCenter?.Dispose();
}
catch (Exception exception)
{
LogErrorDisposing(logger, exception);
}
finally
{
disposed = true;
}
}

public void BreakOutstandingMessagesToSilo(SiloAddress deadSilo)
Expand Down Expand Up @@ -608,6 +625,12 @@ private void ThrowIfDisposed()
)]
private static partial void LogErrorWhileProcessingCallbackExpiry(ILogger logger, Exception ex);

[LoggerMessage(
Level = LogLevel.Error,
Message = "Error disposing the client message center after draining observer invocations."
)]
private static partial void LogErrorDisposing(ILogger logger, Exception exception);

[LoggerMessage(
Level = LogLevel.Debug,
EventId = (int)ErrorCode.Runtime_Error_100011,
Expand Down
Loading
Loading