Skip to content

Conversation

@mnahkies
Copy link
Contributor

@mnahkies mnahkies commented Nov 21, 2025

What was changed

call the workflow interface dispose method, same as in github.com/temporalio/sdk-typescript/blob/main/packages/worker/src/workflow/vm.ts#L127-L130

see also: #1432

Why?

otherwise memory leaks occur when AsyncLocalStorage is used, as interceptors like

export class OpenTelemetryInternalsInterceptor implements WorkflowInternalsInterceptor {
async dispose(input: DisposeInput, next: Next<WorkflowInternalsInterceptor, 'dispose'>): Promise<void> {
if (contextManager !== undefined) {
contextManager.disable();
}
next(input);
}
won't be run.

from my testing we re-evaluate modules on every workflow run, even when in the ReusableVMWorkflow is used. meaning eg: module level const AsyncLocalStorage instances are re-created for each workflow, and so it's important they are disposed of.

Checklist

  1. How was this tested:
    Tested manually with a minimal example of our real workload. Pending testing with our production workflow code.

call the workflow interface dispose method, same as in
github.com/temporalio/sdk-typescript/blob/main/packages/worker/src/workflow/vm.ts#L127-L130

otherwise memory leaks occur when `AsyncLocalStorage` is used.

from my testing we re-evaluate modules on every workflow run, even
when in the `ReusableVMWorkflow`

see also: temporalio#1432
@mnahkies mnahkies requested a review from a team as a code owner November 21, 2025 18:59
@CLAassistant
Copy link

CLAassistant commented Nov 21, 2025

CLA assistant check
All committers have signed the CLA.

@THardy98 THardy98 force-pushed the mn/fix/run-dispose-interceptor branch from 99b413e to 3c20e67 Compare November 25, 2025 19:02
@mjameswh
Copy link
Contributor

Yeah, that definitely seems right. I'm totally baffled this bug didn’t get exposed earlier.

Thanks a lot @mnahkies for your work in investigating this.

@THardy98 THardy98 merged commit f4ec862 into temporalio:main Nov 25, 2025
72 of 77 checks passed
@mnahkies mnahkies deleted the mn/fix/run-dispose-interceptor branch December 1, 2025 09:25
@mnahkies
Copy link
Contributor Author

mnahkies commented Dec 1, 2025

Thanks for adding the test @THardy98 and getting this merged 🎉

Thought I'd just drop some runtime metrics showing the impact of this change for one of our workers that is under a roughly constant load of workflows, tldr; it is much more stable

  • CPU now flat, rather than increasing with memory consumption
  • Event loop delay now flat
  • GC pause / count now flat
image

Previously it was OOMing frequently and restarting, with performance degrading up until the point it restarted.

This has also decreased the end-to-end duration of the workflows processed by this worker, roughly 2-3x improvement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants