Providing suborchestrations with an instanceID yields a non-determinism error

[davidmrdavid]

Related to: https://github.com/MicrosoftDocs/azure-docs/issues/77186

It appears that calling sub-orchestrations with a provided instanceID is failing with a non-determinism error. This appears like a regression and needs to be investigated asap.

[cgillum]

@davidmrdavid can you provide some example code that shows how to reproduce this problem? I'm curious to understand if the issue is as broad as the title makes it out to be or whether it's specific to the way in which sub-orchestration instance IDs are generated.

[mec-sa]

We can also reproduce this using nodejs durable functions
Sample code we were using:

        const prValidationMonitorParallelTasks = [];
        const mainPrValidationMonitorOrchestrator = context.df.callSubOrchestrator(
            'prValidationMonitorOrchestrator',
            {
                pullRequestPayload: mainContentPullRequestPayload,
                ...prValidationMonitorSharedParams,
            },
            `${context.df.instanceId}-prValidationMonitorOrchestrator`  // <-----  Deterministic InstanceId1
        );
        prValidationMonitorParallelTasks.push(mainPrValidationMonitorOrchestrator);

        let placeholderPrValidationMonitorOrchestrator;
        if (placeholderContentPullRequestPayload) {
            placeholderPrValidationMonitorOrchestrator = context.df.callSubOrchestrator(
                'prValidationMonitorOrchestrator',
                {
                    pullRequestPayload: placeholderContentPullRequestPayload,
                    ...prValidationMonitorSharedParams,
                },
                `${context.df.instanceId}-prValidationMonitorOrchestrator-placeholder`  // <-----  Deterministic InstanceId2
            );
            prValidationMonitorParallelTasks.push(placeholderPrValidationMonitorOrchestrator);
        }

        yield context.df.Task.all(prValidationMonitorParallelTasks);   // <------------ ERROR 

Error:

[2021-07-13T05:46:54.499Z] Executed 'Functions.publishOrchestrator' (Failed, Id=aa3f8762-9eca-4f31-b746-a34bd095d548, Duration=103ms)
[2021-07-13T05:46:54.499Z] System.Private.CoreLib: Exception while executing function: Functions.publishOrchestrator. Microsoft.Azure.WebJobs.Extensions.DurableTask: Orchestrator function 'publishOrchestrator' failed: The sub-orchestration call (n = 1) should be executed with an instance id of cef4837f06ec422db7768137528613c6-prValidationMonitorOrchestrator-placeholder instead of the provided instance id of cef4837f06ec422db7768137528613c6-prValidationMonitorOrchestrator. Check your code for non-deterministic behavior.
[2021-07-13T05:46:54.500Z] cef4837f06ec422db7768137528613c6: Orchestration 'publishOrchestrator' awaited and scheduled 1 durable operation(s).
[2021-07-13T05:46:54.500Z] cef4837f06ec422db7768137528613c6: Orchestration completed with a 'Failed' status and 218 bytes of output. Details: Unhandled exception while executing orchestration: DurableTask.Core.Exceptions.NonDeterministicOrchestrationException: Non-Deterministic workflow detected: SubOrchestrationInstanceCreatedEvent: 4 SubOrchestrationInstanceCreated prValidationMonitorOrchestrator cef4837f06ec422db7768137528613c6-prValidationMonitorOrchestrator-placeholder
[2021-07-13T05:46:54.501Z]

[davidmrdavid]

This is being fixed, in Python, with this PR: #307

Hi @mec-sa, thank you for reaching out! Could you please report this issue in the JS SDK repo here: https://github.com/Azure/azure-functions-durable-js/issues . In the issue report, please provide or attach a minimal reproducer as I was unable to reproduce this issue in nodeJS. Thanks!

[davidmrdavid]

@cgillum: The issue was as broad as the title statement made it out to be, as instanceId fields were getting serialized into a JSON field titled instance_id, which the durable-extension would fail to parse (since it expects camelCase) and therefore ignore.

As simple reproducer is as follows:

Sub orchestration

import logging
import json

import azure.functions as func
import azure.durable_functions as df


def orchestrator_function(context: df.DurableOrchestrationContext):
    return "Done!"

main = df.Orchestrator.create(orchestrator_function)

Orchestrator

import logging
import json

import azure.functions as func
import azure.durable_functions as df


def orchestrator_function(context: df.DurableOrchestrationContext):
    yield context.call_sub_orchestrator('DurableFunctionsSubOrchestrator', "", instance_id="0")
    return ""


main = df.Orchestrator.create(orchestrator_function)

This is fixed in the PR I linked above. In the PR, I also write a bit about how to prevent these surface-level bugs in the future :)

[davidmrdavid]

Closing as this has been fixed