Skip to content

[FIX] Dispose StatelessWorkerGrainContext when it contains no more workers #9636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ReubenBond merged 3 commits into from
Aug 28, 2025
Merged

[FIX] Dispose StatelessWorkerGrainContext when it contains no more workers #9636

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2be6566
ensure disposal is called when no workers are alive
ledjon-behluli Aug 4, 2025
a078007
hook continuation in order to log any erros that might occur during d…
ledjon-behluli Aug 4, 2025
8ae48d5
fully align SW context behavior based on if idle worker removal shoul…
ledjon-behluli Aug 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 24 additions & 12 deletions src/Orleans.Runtime/Catalog/StatelessWorkerGrainContext.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ internal partial class StatelessWorkerGrainContext : IGrainContext, IAsyncDispos
private double _integralTerm = 0d;
private int _detectedIdleCyclesCount = 0;
private readonly int _minIdleCyclesBeforeRemoval;
private readonly bool _isIdleWorkerRemovalStrategy;

public StatelessWorkerGrainContext(
GrainAddress address,
Expand All @@ -52,7 +53,8 @@ public StatelessWorkerGrainContext(
var strategy = (StatelessWorkerPlacement)_shared.PlacementStrategy;
var options = _shared.StatelessWorkerOptions;

if (strategy.RemoveIdleWorkers && options.RemoveIdleWorkers)
_isIdleWorkerRemovalStrategy = strategy.RemoveIdleWorkers && options.RemoveIdleWorkers;
if (_isIdleWorkerRemovalStrategy)
{
_minIdleCyclesBeforeRemoval = options.MinIdleCyclesBeforeRemoval > 0 ? options.MinIdleCyclesBeforeRemoval : 1;
_inspectionTimer = new Timer(
Expand Down Expand Up @@ -172,9 +174,23 @@ private async Task RunMessageLoop()

if (_workers.Count == 0)

@DeagleGross DeagleGross Aug 5, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can there be a scenario where _workers.Count reached 0, but in the queue there is an item to process WorkItemType.Message which will then create a new worker? In that case a new worker will be created right after.

@ledjon-behluli ledjon-behluli Aug 5, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very unlikely since the destruction of the workers comes after the fact of a grain disposing (which means it wont receive more messages), but possible! If that happens, the dispose command will be processed after that message.

@ledjon-behluli ledjon-behluli Aug 5, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The alternative would be to stop the disposal if a worker exists within the SW context, we should not do that, we should respect the call for disposal from outside!

{
// When the last worker is destroyed, we can consider the stateless worker grain
// activation to be destroyed as well
// When the last worker is destroyed, we can consider the stateless worker grain activation to be destroyed as well
_shared.InternalRuntime.Catalog.UnregisterMessageTarget(this);

if (_isIdleWorkerRemovalStrategy)
{
var completion = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
EnqueueWorkItem(WorkItemType.DisposeAsync, new DisposeAsyncWorkItemState(completion));

// DO NOT await this as it would deadlock the work loop!
_ = completion.Task.ContinueWith(t =>
{
if (t.Exception is { } ex)
{
LogErrorInMessageLoop(_shared.Logger, ex);
}
}, CancellationToken.None, TaskContinuationOptions.OnlyOnFaulted, TaskScheduler.Current);
}
}
break;
}
Expand Down Expand Up @@ -343,16 +359,12 @@ private async Task DeactivatedTaskInternal(TaskCompletionSource completion)

private async Task DisposeAsyncInternal(TaskCompletionSource completion)
{
try
{
if (_inspectionTimer != null)
{
await _inspectionTimer.DisposeAsync();
_inspectionTimer = null;
}
}
catch
if (_inspectionTimer != null)
{
await _inspectionTimer.DisposeAsync().AsTask()

@DeagleGross DeagleGross Aug 5, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is the change here needed? what is the benefit of AsTask().ConfigureAwait(...) against former DisposeAsync() -> only to suppress exceptions?

@ledjon-behluli ledjon-behluli Aug 5, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only to suppress exceptions

Yes, this is an "just in case" thing so the rest of the method continues

.ConfigureAwait(ConfigureAwaitOptions.SuppressThrowing | ConfigureAwaitOptions.ContinueOnCapturedContext);

_inspectionTimer = null;
}

try
Expand Down
Loading