An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load file or assembly 'System.IO, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

[northtyphoon]

Description

The test output includes "System.IO 4.1.2.0" and the following binding redirect in TestAssembly.dll.config

<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
  <dependentAssembly>
    <assemblyIdentity name="System.IO" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
    <bindingRedirect oldVersion="0.0.0.0-4.1.2.0" newVersion="4.1.2.0" />
  </dependentAssembly>
</assemblyBinding>

The test passed in Visual Studio. However, the test failed in the build machine as the following error.

An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load file or assembly 'System.IO, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies

I also notice if I manually removed TestAssembly.dll.config from the output folder, the test started to pass in the build machine.

Steps to reproduce

In VSTS build, run the latest Visual Studio Test task.

Expected behavior

Test pass.

Actual behavior

2017-10-19T23:47:12.3630779Z Starting test execution, please wait...
2017-10-19T23:47:19.4800462Z Error: An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load file or assembly 'System.IO, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
2017-10-19T23:47:19.4800462Z

Environment

Operating system : WS2016
VS2017 15.4.1
Visual Studio test runner 2.1.15

Package version of MSTest:
"MSTest.TestAdapter" Version="1.2.0"
"MSTest.TestFramework" Version="1.2.0"

[kant2002]

If this is just some binding redirect issues, it will greatly benefit from increased logging with information which test assembly could not be loaded, since that alone greatly speedup troubleshooting. I hit similar issue in the past where binding redirects was required by the test.

[Izzmo]

@kant2002

it will greatly benefit from increased logging with information which test assembly could not be loaded

Okay... you might want to enlighten us how one would do this :)

[AbhitejJohn]

To get more logs from the CI pipeline, could you set System.Debug in the Build definition to true and share the build logs with us please?

If you could repro this locally using vstest.console, then it would also help to enable vstest.executionengine*.exe.config logging. Here is how you can enable that.

I think we need to add a troubleshooting section in testfx-docs to help get more logging.

[codito]

I think we need to add a troubleshooting section in testfx-docs to help get more logging.

Added docs for collecting traces in CI task here: https://github.com/Microsoft/vstest-docs/blob/master/docs/diagnose.md#collect-trace-in-vsts-ci-vstest-task

[northtyphoon]

Thanks for the troubleshooting guide to collect detail trace. I will try to get more logs and post back later.

[northtyphoon]

I enabled system.debug.

2017-10-31T21:44:42.7837155Z Microsoft (R) Test Execution Command Line Tool Version 15.0.26929.2
2017-10-31T21:44:42.7837155Z Copyright (c) Microsoft Corporation. All rights reserved.
2017-10-31T21:44:42.7837155Z
2017-10-31T21:44:43.2496697Z Starting test execution, please wait...
2017-10-31T21:44:47.4338786Z Error: An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load file or assembly 'System.IO, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
2017-10-31T21:44:47.4338786Z
2017-10-31T21:44:47.8549250Z
2017-10-31T21:44:47.9249367Z Information: Additionally, you can try specifying '/UseVsixExtensions' command if the test discoverer & executor is installed on the machine as vsix extensions and your installation supports vsix extensions. Example: vstest.console.exe myTests.dll /UseVsixExtensions:true
2017-10-31T21:44:47.9249367Z
2017-10-31T21:44:47.9489567Z ##[debug]rc:1
2017-10-31T21:44:47.9489567Z ##[debug]success:false
2017-10-31T21:44:47.9509367Z ##[debug]rm -rf E:\A_work_temp\b3938700-be84-11e7-91e2-b52a751f8768.runsettings
2017-10-31T21:44:47.9509367Z ##[debug]removing file
2017-10-31T21:44:47.9509367Z ##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
2017-10-31T21:44:47.9509367Z ##[debug]Processed: ##vso[task.issue type=warning;]Vstest failed with error. Check logs for failures. There might be failed tests.
2017-10-31T21:44:47.9509367Z ##[debug]Release.ReleaseUri=undefined
2017-10-31T21:44:47.9509367Z ##[debug]Release.ReleaseId=undefined
2017-10-31T21:44:47.9509367Z ##[debug]Build.BuildUri=vstfs:///Build/Build/1101004
2017-10-31T21:44:47.9509367Z ##[debug]Build.Buildid=1101004
2017-10-31T21:44:47.9509367Z ##[debug]Agent.Version=2.120.2
2017-10-31T21:44:47.9509367Z ##[debug]telemetry area: TestExecution feature: TestExecutionTask data: {"builduri":"vstfs:///Build/Build/1101004","buildid":"1101004","areacode":"ExecuteVsTest","result":"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe failed with return code: 1","tracepoint":1005,"isusererror":true}
2017-10-31T21:44:48.4491707Z ##[debug]Processed: ##vso[telemetry.publish area=TestExecution;feature=TestExecutionTask;]{"builduri":"vstfs:///Build/Build/1101004","buildid":"1101004","areacode":"ExecuteVsTest","result":"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe failed with return code: 1","tracepoint":1005,"isusererror":true}
2017-10-31T21:44:48.4491707Z ##[error]Error: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe failed with return code: 1

[jayaranigarg]

@northtyphoon : Can you check which version of System.IO.dll is present in TestRun folder in both machine(one with VS installed and build machine) and let us know.

Also, instead of hard-coding the assembly binding redirects in app.config, can you add this in your *.csproj

<PropertyGroup>
    <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
    <GenerateBindingRedirectsOutputType>true</GenerateBindingRedirectsOutputType>
</PropertyGroup>

This will generate a *.dll.config which will have the appropriate assembly binding redirects. See if things work for you after this change.

[Mertsch]

@JayJanarthanan I have two different Test-Projects both with automatic binding redirect.
Both do copy the same System.IO.dll to their output directory.
Both have the following binding redirect

<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
  <dependentAssembly>
    <assemblyIdentity name="System.IO" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
    <bindingRedirect oldVersion="0.0.0.0-4.1.2.0" newVersion="4.1.2.0" />
  </dependentAssembly>
</assemblyBinding>

One works fine, one has the error mentioned above

2017.11.01 14:40:20.779   ERROR An item with the same key has already been added.
2017.11.01 14:40:20.832   ERROR An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load file or assembly 'System.IO, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

[jayaranigarg]

@Mertsch : Can you please share repro projects with us?

[Mertsch]

@jayaranigarg No I can not, but I found the root cause for this issue.
There is a difference between the normal project output and DeploymentItem output.

I have a fairly complex solution with a lot of dependencies on other projects and frameworks, like the .NET Standard 2.0 library and EF Core (from .NET Framework 4.6.1 projects) and the <project>\bin\Debug folder of set projects contain all required .dlls from all dependencies.

But whenever I specify any DeploymentItem, my <SolutionDir>\TestResults\Deploy_User 2017-11-02 18_24_42\Out contains only a limited set of dependencies.

The System.IO is simply missing from the \Out dir.

[Mertsch]

@jayaranigarg I was now able to nail down the issue
Demo repo is here https://github.com/Mertsch/testfxIssues295

So what is the issue?

In my demo I have referenced a .NET Standard 2.0 library https://www.nuget.org/packages/fm.Extensions.Common/
this causes the project output (\bin\Debug) to contain System.IO.dll and the appropriate binding redirects. Note these references are added implicitly! There is no direct reference to System.IO and they are only added to the project output because a .NET Standard 2.0 library is referenced.

If you now specify any DeploymentItem, the \TestResults\Deploy\Out directory contains an incomplete set of assemblies, but still retains the binding redirects.

Which leads to NormalTests to run fine on their own

But together with DeploymentItem, they all fail