"Non-Deterministic workflow detected: TaskScheduledEvent: 0 TaskScheduled" ERROR when activity task failed

[Josephjxu]

Hello,

I am running into issues with orchestration durable function. The function is written in Python (listed below) but my questions refer to durable functions and related tables in general.

def orchestrator_function(context: df.DurableOrchestrationContext):
    request = context.get_input()

    retry_options = df.RetryOptions(first_retry_interval_in_milliseconds=5000, max_number_of_attempts=3)
    parallel_tasks = []

    my_request = request.copy()
    context.set_custom_status("Creating Tasks")
    task_messages = yield context.call_activity_with_retry("task_a_activity", retry_options=retry_options, input_=my_request)

    context.set_custom_status("Preparing Tasks")
    for task_request in task_messages:
        parallel_tasks.append(context.call_activity_with_retry("task_b_activity", retry_options=retry_options, input_=task_request))

    context.set_custom_status(f"Running on {len(task_messages)} tasks")
    outputs = yield context.task_all(parallel_tasks)

    context.set_custom_status("Completed")
    complete_request = request.copy()
    yield context.call_activity("task_c_activity", complete_request)
    return f"Orchestration Completed"

main = df.Orchestrator.create(orchestrator_function)

The orchestration request we have could generate hundreds activity tasks (IO and CPU computing) that each run a few minutes. There could be occasionally run time error on task_b_activity which causes taskFailed error (e.g., Python exit unexpectedly). However, the orchestration sometimes show the following error

**Non-Deterministic workflow detected: TaskScheduledEvent: 0 TaskScheduled task_a_activity**

My questions are:

1. What is exactly the error Non-Deterministic workflow detected: TaskScheduledEvent: 0 TaskScheduled XYZ?
2. Is there anything wrong with the above code?
3. The error occurred on task_b, why orchestration error on task_a?
4. When orchestration failed, I notice that the orchestration history table no longer get updated but all activity functions in work queue are still being picked up. Is this expected? Is there a way to update the orchestration history table when activity functions are still running?
5. The function is running App Service with vnet. Is there way to verify if we are using the latest durable extension?

Your assistance is greatly appreciated.

Best regards.

[cgillum]

Hey @Josephjxu - thanks for trying out our Durable Functions for Python preview! I suspect you may be running into an issue that @davidmrdavid knows about, but let me see if I can answer your questions anyways.

1. The non-deterministic workflow error means that the code path taken during a replay doesn't match the code path taken during a previous execution of this instance. This most often happens when the orchestrator function has non-deterministic code in it, like random numbers, direct I/O operations, etc. More information here.
2. I don't see anything wrong with your code. That's why I suspect this is a bug in the Python SDK related to retries.
3. I would need to look at the orchestration history and understand the bug to know for sure, but I suspect that the history thinks that another call to task_a was expected when it reached the call to task_b.
4. This is expected behavior. Activity functions execute completely independently of the parent orchestrator and there is no check to see if the orchestrator is still in a running state when starting to run activity functions. The result of the activity functions will be discarded once the return messages make it back to the failed orchestrator.
5. If you're using Application Insights for capturing logs, you should be able to see the Durable extension version in the trace collection (look for customDimensions.prop__extensionVersion). More info on Durable traces for Application Insights here.

Also, I'll go ahead and transfer this to the Python repo since I think the issue you're running into is likely specific to the Python SDK.

[davidmrdavid]

Hi @Josephjxu,

Thank you for reaching out. I see @cgillum has already answered a few of your questions, but please let us know if those do not clarify your concerns. In the meantime, I'll respond to question (2) as it may pertain to a known bug in the SDK.

Currently, the released version of the Python Durable SDK contains a regression in our retry-handling logic. This causes us to sometimes read our internal history array in the wrong order, causing the exception you reported above. There's currently a PR to fix this here (https://github.com/Azure/azure-functions-durable-python/pull/186/files) but it's missing a few tests before merge. I'll look into adding those today so we can release the fix as soon as possible. Thank you for your patience during the beta stage of Python Durable :)

Let me know if you need anything else!

⚡ ⚡

[davidmrdavid]

@Josephjxu
Two short questions:

(1) I notice that your code uses .copy() on the orchestrator input in a few places. Why is that and what datatype is the input to the orchestrator?
(2) You say that ""task_b_activity" sometimes causes a taskfailed error. What do you mean by this? Do you mean that it throws an exception?When that happens, is that expected on your end or is that another bug on our end?
Thanks!