Skip to content
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
148 changes: 129 additions & 19 deletions docs/Overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,10 @@
- [Test Logger](#test-logger)
- [Runtime Provider](#runtime-provider)
- [TranslationLayer extension points](#translationlayer-extension-points)
- [Public surface of the TranslationLayer project](#public-surface-of-the-translationlayer-project)
- [Wrapper operations (IVsTestConsoleWrapper)](#wrapper-operations-ivstestconsolewrapper)
- [Callback interfaces you implement (the plug-in points)](#callback-interfaces-you-implement-the-plug-in-points)
- [Configuring the runner process (ConsoleParameters)](#configuring-the-runner-process-consoleparameters)
- [.NET Implementation](#net-implementation)
- [Architecture](#architecture)

Expand Down Expand Up @@ -89,7 +93,7 @@ While the tests execute, the results are reported back to the runner, aggregated

The client processes the results and shows them in their UI, for example as TestExplorer does it here:

<TODO gif>
Comment thread
azat-msft marked this conversation as resolved.
Resolved
![Visual Studio Test Explorer showing the results of a completed test run](resources/test-explorer-example.gif)

A simplified flow describing the whole process is as follows:

Expand Down Expand Up @@ -139,8 +143,6 @@ The Run workflow described above is very common in command line tools, and proba

### Base Protocol

TODO: fill in more details.

Data are passed as JSON serialized strings over TCP. The messages are serialized using binary format that delimits messages by a length prefix. The size prefix is written as 7 bit encoded int. (The basics of encoding that number are summarized here: <https://stackoverflow.com/a/49780224/3065397>).

In .NET the data are written using BinaryWriter and BinaryReader which do all the needed conversions automatically when writing and reading the string (<https://learn.microsoft.com/en-us/dotnet/api/system.io.binarywriter.write?view=net-6.0#system-io-binarywriter-write(system-string>)).
Expand Down Expand Up @@ -228,10 +230,10 @@ There is no header in the message itself. There is header only in the binary mes

TestPlatform protocol defines a set of JSON request, response and notification messages, that are exchanged using the above base protocol. This section starts by describing the basic JSON structures used in the protocol. The description uses C# classes, and types, with nullability enabled. Meaning that every type is non-nullable by default, and nullability is denoted by `?` following the type name.

<TODO example?>

The protocol assumes that one server serves one tool. There is no support in the protocol to share one server between different tools.

See [Message documentation](#message-documentation) for the format used to describe every message, including a worked [ProtocolVersion request](#protocolversion-request) example.

#### Capabilities

The client, runner + datacollector, and testhost are shipped separately. Each of those components can have a different version and hence a different set of functionality they support. A single number (protocol version) is used to represent the whole set of capabilities that a given component supports. Each newer version includes complete functionality of the previous version. There are no granular capabilities.
Expand All @@ -257,9 +259,31 @@ All notifications are sent before a response is sent.
TestPlatformProtocol is defined by a set of requests, responses and notifications. Each of those are described using the following format:

- a header describing the request
Comment thread
azat-msft marked this conversation as resolved.
Resolved
- a request section describing the format of the
- a request section describing the request payload format
Comment thread
azat-msft marked this conversation as resolved.
Resolved
- a response section describing the response payload format, when the message has a response
- examples of the JSON sent on the wire

For example, the [ProtocolVersion request](#protocolversion-request) below follows this format. Because it is exchanged during negotiation, it uses the version 0 message envelope and therefore omits the `Version` property in the JSON examples.
Its header describes the message, the *Request* section documents the payload both as a C# type and as
the JSON sent on the wire, and the *Response* section documents the reply the same way:
Comment thread
azat-msft marked this conversation as resolved.
Resolved
Comment thread
azat-msft marked this conversation as resolved.
Resolved

*Request:*

<TODO>
```json
{
"MessageType": "ProtocolVersion",
"Payload": 7
}
```

*Response:*

```json
{
"MessageType": "ProtocolVersion",
"Payload": 7
}
```

### Basic structures

Expand Down Expand Up @@ -302,7 +326,7 @@ Versions:
- 2: Changed serialization from a generic bag that described each property and its type, to explicit properties that are serialized without additional type info.
- 3: Added AttachDebugger message.
- 4: Added because version 3 did not update the serialization to use, and it will use v1 serialization (bag) rather than explicit properties. Right side should avoid negotiating 3 and downgrade to 2.
- 5: Unknown. (TODO)
- 5: Unknown in the core `ProtocolVersioning` table (the source still marks this version as `// 5: ???`). The TranslationLayer defines a private `MinimumProtocolVersionWithTestSessionSupport = 5` in `VsTestConsoleRequestSender`, but the constant is currently unused, so the exact change associated with v5 is unclear from the current source.
Comment thread
azat-msft marked this conversation as resolved.
Resolved
- 6: Added Abort and Cancel with handlers that report the status.
- 7: Added SkippedDiscoveredSources.

Expand Down Expand Up @@ -616,7 +640,7 @@ public class DiscoveryCompletePayload
public IList<string>? NotDiscoveredSources { get; set; } = new List<string>();

// Gets or sets the collection of discovered extensions.
// TODO: since?
// Not gated by a protocol version; always serialized (as an empty object when there is nothing to report).
public Dictionary<string, HashSet<string>>? DiscoveredExtensions { get; set; } = new();
Comment thread
azat-msft marked this conversation as resolved.
Resolved
Comment thread
azat-msft marked this conversation as resolved.
Resolved
}
```
Expand Down Expand Up @@ -707,7 +731,7 @@ Contains full paths to one or more test sources, and settings to use for the dis
```csharp
public class DiscoveryCriteria
{
// Gets the test Containers (e.g. .appx, .appxrecipie) TODO what???
// Gets the test container package path (for example, an appx package).
public string? Package { get; set; }


Expand Down Expand Up @@ -791,7 +815,7 @@ public class DiscoveryCompletePayload
public IList<string>? NotDiscoveredSources { get; set; } = new List<string>();

// Gets or sets the collection of discovered extensions.
// TODO: since?
// Not gated by a protocol version; always serialized (as an empty object when there is nothing to report).
public Dictionary<string, HashSet<string>>? DiscoveredExtensions { get; set; } = new();
Comment thread
azat-msft marked this conversation as resolved.
Resolved
Comment thread
azat-msft marked this conversation as resolved.
Resolved
}
```
Expand Down Expand Up @@ -1105,8 +1129,7 @@ public class TestRunCompleteEventArgs
// Error encountered in the run that is not linked to any test.
public Exception? Error { get; private set; }

// Gets the attachment sets associated with the test run.
// TODO: HOW is this different from RunAttachments above?
// Gets the attachment sets associated with the test run (for example data-collector output). These are distinct from RunAttachments on the enclosing TestRunCompletePayload, which carries the run-context AttachmentSets (runContextAttachments) sent alongside ExecutionComplete; note that the TRX report itself is produced by the TRX logger on the runner side, not carried here.
public Collection<AttachmentSet> AttachmentSets { get; private set; }
Comment thread
azat-msft marked this conversation as resolved.
Resolved
Comment thread
azat-msft marked this conversation as resolved.
Resolved

// Gets the invoked data collectors for the test session.
Expand Down Expand Up @@ -1249,7 +1272,9 @@ public class TestExecutionContext
// Gets or sets a value indicating whether testhost process should be kept running after test run completion.
public bool KeepAlive { get; set; }

// Gets or sets a value indicating whether test case level events need to be sent or not. TODO: what is it? Is there since first commit, no usages on grep.app.
// Gets or sets a value indicating whether test case level events (TestCaseStart / TestCaseEnd)
// are required. The cross-platform execution managers currently always pass false here
// (see ProxyExecutionManager and InProcessProxyexecutionManager).
Comment thread
azat-msft marked this conversation as resolved.
Resolved
public bool AreTestCaseLevelEventsRequired { get; set; }

// Gets or sets a value indicating whether execution is in debug mode.
Expand Down Expand Up @@ -1355,8 +1380,7 @@ public class TestRunCompleteEventArgs
// Error encountered in the run that is not linked to any test.
public Exception? Error { get; private set; }

// Gets the attachment sets associated with the test run.
// TODO: HOW is this different from RunAttachments above?
// Gets the attachment sets associated with the test run (for example data-collector output). These are distinct from RunAttachments on the enclosing TestRunCompletePayload, which carries the run-context AttachmentSets (runContextAttachments) sent alongside ExecutionComplete; note that the TRX report itself is produced by the TRX logger on the runner side, not carried here.
public Collection<AttachmentSet> AttachmentSets { get; private set; }

// Gets the invoked data collectors for the test session.
Expand Down Expand Up @@ -1856,10 +1880,96 @@ Additional example of a toy test framework and adapter can be found in <https://

### TranslationLayer extension points
Comment thread
azat-msft marked this conversation as resolved.
Resolved

TODO
The TranslationLayer (assembly `Microsoft.TestPlatform.VsTestConsole.TranslationLayer`,
NuGet package `Microsoft.TestPlatform.TranslationLayer`) is the client-facing library that
tools such as Visual Studio, the Azure DevOps test task, and custom runners use to drive
TestPlatform programmatically. Instead of shelling out to `vstest.console.exe` and parsing
its output, a consumer references the library, points it at a `vstest.console` binary, and
receives strongly-typed callbacks. Internally the wrapper starts the `vstest.console`
process, opens the socket connection to it, and exchanges the JSON protocol messages
described earlier in this document; all of that is hidden behind the interfaces below.

There are two categories of extension point:

1. The **wrapper API** you call to drive an operation (`IVsTestConsoleWrapper` and the
public types shipped alongside it).
2. The **callback interfaces you implement** and pass in, so TestPlatform can call back into
your tool as discovery/execution progresses (the handler and launcher interfaces from
`Microsoft.VisualStudio.TestPlatform.ObjectModel.Client`).

#### Public surface of the TranslationLayer project

| Type | Kind | Purpose |
| --- | --- | --- |
| `IVsTestConsoleWrapper` | interface | Synchronous entry point. Controller for every test operation on the wrapped runner. Derives from `IVsTestConsoleWrapperAsync`. |
| `IVsTestConsoleWrapperAsync` | interface | Asynchronous counterpart. All of its `*Async` operations are marked `[Obsolete("The async APIs don't work, use the sync API instead.")]`; only `ProcessTestRunAttachmentsAsync` is a supported async-only method. |
| `VsTestConsoleWrapper` | class | The default implementation. Constructed with the path to `vstest.console` and, optionally, a `ConsoleParameters`. |
| `ConsoleParameters` | class | Configures how the `vstest.console` process is launched (see below). |
| `DiscoveryEventsHandleConverter` | class | Adapter that lets an older `ITestDiscoveryEventsHandler` be used where an `ITestDiscoveryEventsHandler2` is required. |
| `TransationLayerException` | class | Exception type thrown for TranslationLayer-specific failures. |

The `ITranslationLayerRequestSender` / `IProcessManager` interfaces in the project are
`internal` plumbing (socket request sender and process manager) and are not part of the
public extension surface.

#### Wrapper operations (`IVsTestConsoleWrapper`)

- `StartSession()` - starts the `vstest.console` process and readies it for requests.
- `EndSession()` - ends the session and stops the runner process. Also `CancelDiscovery()`,
`CancelTestRun()`, and `AbortTestRun()` to interrupt in-flight operations.
- `InitializeExtensions(IEnumerable<string> pathToAdditionalExtensions)` - registers extra
extension DLLs (adapters, loggers, data collectors) by full path before discovery/execution.
- `DiscoverTests(...)` - discovers tests in the given sources. Overloads accept an optional
`TestPlatformOptions` and `TestSessionInfo`, and report results through an
`ITestDiscoveryEventsHandler` (legacy) or `ITestDiscoveryEventsHandler2`.
- `RunTests(...)` - runs tests, selected either by `sources` (assemblies) or by an explicit
list of `TestCase` objects. Overloads accept `TestPlatformOptions`, a `TestSessionInfo`
(to run against a pre-warmed test session), an `ITestRunEventsHandler`, and optionally an
`ITelemetryEventsHandler`.
- `RunTestsWithCustomTestHost(...)` - same as `RunTests`, but the testhost process is
launched by a caller-supplied `ITestHostLauncher`. This is how IDEs attach a debugger to
the testhost or otherwise control how it is started.
- `ProcessTestRunAttachmentsAsync(...)` (async only) - post-processes the `AttachmentSet`s
produced by a previous run, for example merging code-coverage files. Progress and
completion are reported through an `ITestRunAttachmentsProcessingEventsHandler`, and the
call honours a `CancellationToken`.

#### Callback interfaces you implement (the plug-in points)

These live in `Microsoft.VisualStudio.TestPlatform.ObjectModel.Client(.Interfaces)`. Your
tool implements the relevant one and passes an instance to the wrapper operation; TestPlatform
invokes it as the operation progresses. They all derive from `ITestMessageEventHandler`, which
provides `HandleRawMessage(string)` and `HandleLogMessage(TestMessageLevel, string?)`.

| Interface | Passed to | Key callbacks |
| --- | --- | --- |
| `ITestDiscoveryEventsHandler` | `DiscoverTests` (legacy overload) | `HandleDiscoveredTests`, `HandleDiscoveryComplete(long totalTests, ..., bool isAborted)` |
| `ITestDiscoveryEventsHandler2` | `DiscoverTests` | `HandleDiscoveredTests`, `HandleDiscoveryComplete(DiscoveryCompleteEventArgs, ...)` (richer completion args) |
| `ITestRunEventsHandler` | `RunTests`, `RunTestsWithCustomTestHost` | `HandleTestRunStatsChange`, `HandleTestRunComplete`, `LaunchProcessWithDebuggerAttached` |
| `ITelemetryEventsHandler` | `RunTests` (telemetry overloads) | `HandleTelemetryEvent(TelemetryEvent)` |
| `ITestRunAttachmentsProcessingEventsHandler` | `ProcessTestRunAttachmentsAsync` | `HandleProcessedAttachmentsChunk`, `HandleTestRunAttachmentsProcessingProgress`, `HandleTestRunAttachmentsProcessingComplete` |
| `ITestHostLauncher` | `RunTestsWithCustomTestHost` | `IsDebug`, `LaunchTestHost(TestProcessStartInfo, ...)` |
| `ITestHostLauncher2` : `ITestHostLauncher` | `RunTestsWithCustomTestHost` | adds `AttachDebuggerToProcess(int pid, ...)` for attaching to an already-running testhost |
| `ITestHostLauncher3` : `ITestHostLauncher2` | `RunTestsWithCustomTestHost` | adds `AttachDebuggerToProcess(AttachDebuggerInfo, CancellationToken)` (carries the target framework so the correct debugger engine is used) |

The launcher hierarchy is versioned rather than modified so that older consumers keep working:
the wrapper upcasts the `ITestHostLauncher` it is given to `ITestHostLauncher2` / `3` when it
needs the newer debugger-attach capabilities, and falls back gracefully when they are not
implemented.

#### Configuring the runner process (`ConsoleParameters`)

`ConsoleParameters` controls how `vstest.console` is started and is another extensibility seam:

- `EnvironmentVariables` / `InheritEnvironmentVariables` - environment for the runner process.
By default the entries are merged onto the inherited environment; set
`InheritEnvironmentVariables` to `false` to supply the full environment yourself.
- `TraceLevel` and `LogFilePath` - diagnostic logging for the runner. The setter for
`LogFilePath` creates the target directory if needed and quotes the path.
- The constructor also accepts an `IFileHelper`, allowing the file-system interactions to be
substituted (primarily for testing). `PortNumber` and `ParentProcessId` exist but are
`internal` and set by the wrapper, so they are not part of the public contract.

### .NET Implementation

#### Architecture


4 changes: 2 additions & 2 deletions docs/configure.md
Original file line number Diff line number Diff line change
Expand Up @@ -458,8 +458,8 @@ available to tests during runtime.
An adapter may provide a section in runsettings for users. See [mstest config][]
and [nunit config][] for more details.

[mstest config]: TODO
[nunit config]: TODO
[mstest config]: https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-configure
[nunit config]: https://docs.nunit.org/articles/vs-test-adapter/Adapter-Settings.html

# Execution thread apartment state

Expand Down
Binary file added docs/resources/test-explorer-example.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.