AzOps Pull fails with exit code null received from tool '/usr/bin/pwsh'

[jsandquist]

Describe the bug

This is more of an observation with a proposed work-around rather than an issue as such.

With default value "Core.ThrottleLimit": 10 in settings.json the AzOps pull pipeline has failed consistently for me for a couple of days. Changing the value from 10 to 1 solved the problem and it runs to completion in some 20 mins.

With a throttle limit set to 10 the pipeline stopped with an error like:

[21:55:48][Get-AzOpsResourceDefinition] Processing detail: Policy Definitions for [/subscriptions/749188b6-...]
[21:55:48][Get-AzOpsPolicyDefinition] Retrieving custom policy definitions for Subscription [contoso-landingzone-abcd-001] (749188b6-...)
##[debug]Exit code null received from tool '/usr/bin/pwsh'
##[debug]STDIO streams have closed for tool '/usr/bin/pwsh'

I've tested with both 1.6.4 or 1.7.2 and they behave the same. Using Azure Pipelines with Microsoft-hosted build agents in these tests. Interestingly, the pull pipeline also ran for much longer, between 1h30m to 2h30 min before crashing instead of the 20 mins when it previously finished OK.

Steps to reproduce

1. Unfortunately no idea
2. But I changed the "Core.SubscriptionsToIncludeResourceGroups" to match various subsets of subscriptions to no avail

Screenshots/Files

settings.json

{
    "Version": 1,
    "DynamicTree": {
        "AzOps": {
            "Core.AutoGeneratedTemplateFolderPath": ".",
            "Core.AutoInitialize": true,
            "Core.DefaultDeploymentRegion": "northeurope",
            "Core.EnrollmentAccountPrincipalName": "",
            "Core.ExcludedSubOffer": [
                "AzurePass_2014-09-01",
                "FreeTrial_2014-09-01",
                "AAD_2015-09-01"
            ],
            "Core.ExcludedSubState": [
                "Disabled",
                "Deleted",
                "Warned",
                "Expired"
            ],
            "Core.ExportRawTemplate": false,
            "Core.IgnoreContextCheck": false,
            "Core.InvalidateCache": true,
            "Core.OfferType": "MS-AZR-0017P",
            "Core.PartialMgDiscoveryRoot": [
                "contoso"
            ],
            "Core.SkipPolicy": false,
            "Core.SkipResource": true,
            "Core.SkipChildResource": true,
            "Core.SkipResourceGroup": false,
            "Core.SkipResourceType": [
                "microsoft.alertsmanagement/smartDetectorAlertRules",
                "Microsoft.PowerPlatform/accounts",
                "Microsoft.PowerPlatform/enterprisePolicies",
                "microsoft.visualstudio/account",
                "Microsoft.VSOnline/plans"
            ],
            "Core.SkipRole": true,
            "Core.SubscriptionsToIncludeResourceGroups": [
                "contoso-platform-*",
                "contoso-landingzone-*"
            ],
            "Core.TemplateParameterFileSuffix": ".json",
            "Core.ThrottleLimit": 10,
            "Core.WhatifExcludedChangeTypes": [
                "NoChange",
                "Ignore"
            ]
        },
        "|AzureDevOps": {
            "!Condition": "EnvAzDevPipeline",
            "AzOps.Core.State": "%PIPELINE_WORKSPACE%/root"
        },
        "|GitHub": {
            "!Condition": "EnvGithubAction",
            "AzOps.Core.State": "%GITHUB_WORKSPACE%/root"
        },
        "PSFramework": {
            "Message.Info.Maximum": 9
        }
    }
}

The verbose log output

[21:55:48][New-AzOpsScope] Starting creation of new scope object
[21:55:48][New-AzOpsScope] Execution Confirmed: Creating new AzOpsScope object using scope [/subscriptions/749188b6-...]
[21:55:48][AzOpsScope] Calling Constructor with value /subscriptions/749188b6-...
[21:55:48][AzOpsScope] Calling InitializeMemberVariablesFromDirectory with value /subscriptions/749188b6-...
[21:55:48][InitializeMemberVariables] Calling InitializeMemberVariables with scope /subscriptions/749188b6-...
[21:55:48][AzOpsScope] SubscriptionId found in Azure: /subscriptions/749188b6-...
[21:55:48][AzOpsScope] Subscription DisplayName found in Azure: contoso-landingzone-abcd-001
[21:55:48][InitializeMemberVariables] Calling InitializeMemberVariables with scope /subscriptions/749188b6-...
[21:55:48][New-AzOpsScope] Execution Successful: Creating new AzOpsScope object using scope [/subscriptions/749188b6-...]
[21:55:48][ConvertTo-AzOpsState] Appending child resource name: contoso-landingzones/749188b6-...
[21:55:48][ConvertTo-AzOpsState] Exporting AzOpsState to /home/vsts/work/1/s/root/contoso (contoso)/contoso-landingzones (contoso-landingzones)/contoso-landingzone-abcd-001 (749188b6-...)/./microsoft.subscription_subscriptions-749188b6-....json
[21:55:48][Get-AzOpsResourceDefinition] Processing detail: Policy Definitions for [/subscriptions/749188b6-...]
[21:55:48][Get-AzOpsPolicyDefinition] Retrieving custom policy definitions for Subscription [contoso-landingzone-abcd-001] (749188b6-...)
##[debug]Exit code null received from tool '/usr/bin/pwsh'
##[debug]STDIO streams have closed for tool '/usr/bin/pwsh'
##[debug]task result: Failed
##[error]PowerShell exited with code 'null'.
##[debug]Processed: ##vso[task.issue type=error;]PowerShell exited with code 'null'.
##[debug]Processed: ##vso[task.complete result=Failed;]PowerShell exited with code 'null'.
Finishing: Initialize

[daltondhcp]

Thank you for reporting!
Unfortunately, this is not something that I have seen before nor something that I have observed in my environments. 😞

How many subscriptions do you approximately have in your environment?

[jsandquist]

Thanks @daltondhcp - good to know. 100+ subscriptions. I have so far failed to reproduce it locally following the instructions but that is due to issues on my end. Will hopefully give it some more time soon.

[daltondhcp]

@jsandquist - have you made any progress around this issue?

[jsandquist]

Unfortunately not. There is some infrastructure getting in the way that stops me still. I can start Invoke-AzOpsPull -Rebuild and it runs for quite a while with no issues but then fails due to intermittent issues on my end so never runs to completion.

We could possibly close this issue for now - if I manage to reproduce the issue and get some useful information out of it we'll revisit this.

[jsandquist]

I fail to reproduce this with AzOps 1.7.3 so closing this now.