Skip to content

Conversation

@davidmrdavid
Copy link
Collaborator

@davidmrdavid davidmrdavid commented Sep 7, 2023

Due to the limitations of the out-of-process replay protocol used in the Python SDK, the SDK needs to jump through some hoops when trying creating composite tasks (WhenAll/WhenAny) out of sub-tasks that were already scheduled.

For example, consider the following orchestrator:

def generator_function_reuse_task_in_whenany(context):
    task1 = context.call_activity("Hello", "Tokyo")
    task2 = context.call_activity("Hello", "Seattle")
    pending_tasks = [task1, task2]

    #  Yield until first task is completed
    finished_task1 = yield context.task_any(pending_tasks)
    # both task 1 and task 2 have been scheduled already

    #  Remove completed task from pending tasks
    pending_tasks.remove(finished_task1)

    task3 = context.call_activity("Hello", "London")
    tasks = pending_tasks + [task3]

    #  We now create a composite task (WhenAny) out of one already scheduled task (either task1 or task2),
    # and a brand new one (task3)
    yield context.task_any(tasks)
    return ""

In our out-of-process replay protocol, we have to return a list of all actions taken in the replay.

Assuming that the "hello tokyo activity" completes first:* a naive way to represent the final replay of the above orchestrator in this protocol, would be as follows:

[
WhenAny(
  [CallActivity("Hello", "Tokyo"), CallActivity("Hello``, "Seattle")],
WhenAny(
  [CallActivity("Hello", "Seattle"), CallActivity("Hello", "London")]
]

However, it is assumed that each action in the actions array corresponds to a new task, and above we're scheduling the "Hello Seattle" activity twice, which isn't right. Instead, we need to be clever and exclude the "Hello Seattle" activity from the 2nd WhenAny task, resulting in the following actions array:

[
WhenAny(
  [CallActivity("Hello", "Tokyo"), CallActivity("Hello``, "Seattle")],
WhenAny(
  [ CallActivity("Hello", "London")]
]

This "fix" isn't perfect, but it's the best we can do given today's protocol. This fix was recently implemented but not correctly, we forgot to also avoid re-registering the "Hello Seattle" activity in the "open tasks" list of the "TaskOrchestrationExecutor" if we know it's been alreayd scheduled. This PR does just that.

@davidmrdavid davidmrdavid merged commit b27dd33 into dev Sep 7, 2023
@davidmrdavid davidmrdavid deleted the dajusto/prevent-already-scheduled-tasks-from-erroring branch September 7, 2023 22:58
greenie-msft pushed a commit to greenie-msft/azure-functions-durable-python that referenced this pull request Sep 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants