Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 30 additions & 20 deletions docs/Overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ While the tests execute, the results are reported back to the runner, aggregated

The client processes the results and shows them in their UI, for example as TestExplorer does it here:

<TODO gif>
Comment thread
azat-msft marked this conversation as resolved.
Resolved
![test-explorer-example.gif](resources/test-explorer-example.gif)
Comment thread
azat-msft marked this conversation as resolved.
Resolved

A simplified flow describing the whole process is as follows:

Expand Down Expand Up @@ -139,8 +139,6 @@ The Run workflow described above is very common in command line tools, and proba

### Base Protocol

TODO: fill in more details.

Data are passed as JSON serialized strings over TCP. The messages are serialized using binary format that delimits messages by a length prefix. The size prefix is written as 7 bit encoded int. (The basics of encoding that number are summarized here: <https://stackoverflow.com/a/49780224/3065397>).

In .NET the data are written using BinaryWriter and BinaryReader which do all the needed conversions automatically when writing and reading the string (<https://learn.microsoft.com/en-us/dotnet/api/system.io.binarywriter.write?view=net-6.0#system-io-binarywriter-write(system-string>)).
Expand Down Expand Up @@ -228,8 +226,6 @@ There is no header in the message itself. There is header only in the binary mes

TestPlatform protocol defines a set of JSON request, response and notification messages, that are exchanged using the above base protocol. This section starts by describing the basic JSON structures used in the protocol. The description uses C# classes, and types, with nullability enabled. Meaning that every type is non-nullable by default, and nullability is denoted by `?` following the type name.

<TODO example?>

The protocol assumes that one server serves one tool. There is no support in the protocol to share one server between different tools.

#### Capabilities
Expand Down Expand Up @@ -257,9 +253,9 @@ All notifications are sent before a response is sent.
TestPlatformProtocol is defined by a set of requests, responses and notifications. Each of those are described using the following format:

- a header describing the request
Comment thread
azat-msft marked this conversation as resolved.
Resolved
- a request section describing the format of the

<TODO>
- a request section describing the request payload format
Comment thread
azat-msft marked this conversation as resolved.
Resolved
- a response section describing the response payload format, when the message has a response
- examples of the JSON sent on the wire

### Basic structures

Expand Down Expand Up @@ -302,7 +298,7 @@ Versions:
- 2: Changed serialization from a generic bag that described each property and its type, to explicit properties that are serialized without additional type info.
- 3: Added AttachDebugger message.
- 4: Added because version 3 did not update the serialization to use, and it will use v1 serialization (bag) rather than explicit properties. Right side should avoid negotiating 3 and downgrade to 2.
- 5: Unknown. (TODO)
- 5: Added test session support. The TranslationLayer requires at least this version to negotiate test sessions (`MinimumProtocolVersionWithTestSessionSupport = 5` in `VsTestConsoleRequestSender`); the base `ProtocolVersioning` table in the source still marks it as `// 5: ???`.
Comment thread
azat-msft marked this conversation as resolved.
Resolved
- 6: Added Abort and Cancel with handlers that report the status.
- 7: Added SkippedDiscoveredSources.

Expand Down Expand Up @@ -616,7 +612,7 @@ public class DiscoveryCompletePayload
public IList<string>? NotDiscoveredSources { get; set; } = new List<string>();

// Gets or sets the collection of discovered extensions.
// TODO: since?
// Added in 17.2.0 (not gated by a protocol version; always serialized when present).
Comment thread
azat-msft marked this conversation as resolved.
Resolved
public Dictionary<string, HashSet<string>>? DiscoveredExtensions { get; set; } = new();
Comment thread
azat-msft marked this conversation as resolved.
Resolved
Comment thread
azat-msft marked this conversation as resolved.
Resolved
}
```
Expand Down Expand Up @@ -707,7 +703,7 @@ Contains full paths to one or more test sources, and settings to use for the dis
```csharp
public class DiscoveryCriteria
{
// Gets the test Containers (e.g. .appx, .appxrecipie) TODO what???
// Gets the test container package path (for example, an appx package).
public string? Package { get; set; }


Expand Down Expand Up @@ -791,7 +787,7 @@ public class DiscoveryCompletePayload
public IList<string>? NotDiscoveredSources { get; set; } = new List<string>();

// Gets or sets the collection of discovered extensions.
// TODO: since?
// Added in 17.2.0 (not gated by a protocol version; always serialized when present).
Comment thread
azat-msft marked this conversation as resolved.
Resolved
public Dictionary<string, HashSet<string>>? DiscoveredExtensions { get; set; } = new();
Comment thread
azat-msft marked this conversation as resolved.
Resolved
Comment thread
azat-msft marked this conversation as resolved.
Resolved
}
```
Expand Down Expand Up @@ -1105,8 +1101,7 @@ public class TestRunCompleteEventArgs
// Error encountered in the run that is not linked to any test.
public Exception? Error { get; private set; }

// Gets the attachment sets associated with the test run.
// TODO: HOW is this different from RunAttachments above?
// Gets the attachment sets associated with the test run. Unlike RunAttachments on the enclosing payload (run-level files such as TRX reports), these are the per-run attachment sets collected during execution.
Comment thread
nohwnd marked this conversation as resolved.
Resolved
Comment thread
azat-msft marked this conversation as resolved.
Resolved
Comment thread
azat-msft marked this conversation as resolved.
Resolved
public Collection<AttachmentSet> AttachmentSets { get; private set; }

// Gets the invoked data collectors for the test session.
Expand Down Expand Up @@ -1249,7 +1244,11 @@ public class TestExecutionContext
// Gets or sets a value indicating whether testhost process should be kept running after test run completion.
public bool KeepAlive { get; set; }

// Gets or sets a value indicating whether test case level events need to be sent or not. TODO: what is it? Is there since first commit, no usages on grep.app.
// Gets or sets a value indicating whether test case level events (TestCaseStart / TestCaseEnd)
// need to be raised to data collectors during the run. It is set based on whether any registered
// data collector subscribed to per-test-case events; when true a dedicated socket port is opened
// for those events (see DataCollectionRequestHandler). The runner-side execution managers currently
// always pass false here.
Comment thread
azat-msft marked this conversation as resolved.
Resolved
Comment thread
azat-msft marked this conversation as resolved.
Resolved
Comment thread
azat-msft marked this conversation as resolved.
Resolved
public bool AreTestCaseLevelEventsRequired { get; set; }

// Gets or sets a value indicating whether execution is in debug mode.
Expand Down Expand Up @@ -1355,8 +1354,7 @@ public class TestRunCompleteEventArgs
// Error encountered in the run that is not linked to any test.
public Exception? Error { get; private set; }

// Gets the attachment sets associated with the test run.
// TODO: HOW is this different from RunAttachments above?
// Gets the attachment sets associated with the test run. Unlike RunAttachments on the enclosing payload (run-level files such as TRX reports), these are the per-run attachment sets collected during execution.
Comment thread
nohwnd marked this conversation as resolved.
Resolved
Comment thread
azat-msft marked this conversation as resolved.
Resolved
public Collection<AttachmentSet> AttachmentSets { get; private set; }

// Gets the invoked data collectors for the test session.
Expand Down Expand Up @@ -1856,10 +1854,22 @@ Additional example of a toy test framework and adapter can be found in <https://

### TranslationLayer extension points
Comment thread
azat-msft marked this conversation as resolved.
Resolved

TODO
The TranslationLayer exposes client-facing APIs for tools that want to drive discovery,
Comment thread
azat-msft marked this conversation as resolved.
Resolved
execution, session management, and attachment processing through TestPlatform without
shelling out to `vstest.console.exe`. The entry point is `IVsTestConsoleWrapper` (and its
async counterpart `IVsTestConsoleWrapperAsync`), which wraps a `vstest.console` process and
sends protocol messages to it. The main extension points are:

- `StartSession` / `EndSession` - start and tear down the wrapped `vstest.console` process.
- `InitializeExtensions` - register additional adapter/data-collector extension paths.
- `DiscoverTests` - discover tests and receive results through an `ITestDiscoveryEventsHandler`.
- `RunTests` / `RunTestsWithCustomTestHost` - run tests, optionally hosting the testhost through a
custom `ITestHostLauncher` (used to attach a debugger or control the testhost process).
- `StartTestSession` / `StopTestSession` - pre-start testhosts for a set of sources so subsequent
runs are faster (requires protocol version 5 or higher).
Comment thread
azat-msft marked this conversation as resolved.
Resolved
- `ProcessTestRunAttachmentsAsync` (on `IVsTestConsoleWrapperAsync`) - post-process attachments

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is still very sparse and does not talk at all about vstest console wrapper extension points. make it go through all public interfaces in the project there are extensions for discovery, run, etc.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please take a look at the updated version

produced by a previous run, e.g. merge code coverage files.

### .NET Implementation

#### Architecture


4 changes: 2 additions & 2 deletions docs/configure.md
Original file line number Diff line number Diff line change
Expand Up @@ -458,8 +458,8 @@ available to tests during runtime.
An adapter may provide a section in runsettings for users. See [mstest config][]
and [nunit config][] for more details.

[mstest config]: TODO
[nunit config]: TODO
[mstest config]: https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-configure
[nunit config]: https://docs.nunit.org/articles/vs-test-adapter/Tips-And-Tricks.html
Comment thread
azat-msft marked this conversation as resolved.
Resolved

# Execution thread apartment state

Expand Down
Binary file added docs/resources/test-explorer-example.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.