How to programmatically force a test to fail?

[aradalvand]

I've tried this:

public ValueTask OnTestStart(TestContext context)
{
	CancellationTokenSource cts = new();
	context.AddLinkedCancellationToken(cts.Token);
	cts.Cancel();

	context.OverrideResult(TestState.Failed, "arbitrary reason");

	return ValueTask.CompletedTask;
}

But neither AddLinkedCancellationToken (with a canceled token), nor context.OverrideResult seem to have any effect whatsoever.

What am I missing? Is there a way to programmatically short-circuit a test and have it fail immediately using its associated TestContext instance?

[aradalvand]

@thomhurst Can you please answer this?

[thomhurst]

Sorry for the slow reply!

To stop the test before its method runs and mark it as failed, call Fail.Test from OnTestStart:

public ValueTask OnTestStart(TestContext context)
{
    Fail.Test("arbitrary reason");
    return ValueTask.CompletedTask;
}

AddLinkedCancellationToken only links another token to the test's cancellation token. Cancellation is cooperative, so cancelling it does not forcibly stop the test lifecycle. OverrideResult changes the final reported result; it does not short-circuit execution, and is best used from OnTestEnd or an After(Test) hook.

One important detail: when this question was originally posted, TUnit accidentally swallowed exceptions thrown by ITestStartEventReceiver. That was fixed in #3131 and released in 0.60.1, so please upgrade if you are using an older version.

[thomhurst]

Yes, a custom ITestExecutor can bridge this. Here is a complete example.

First, store a per-test failure signal in the TestContext and expose an extension method that the background task can call:

using TUnit.Assertions.Exceptions;

internal sealed class BackgroundFailureSignal
{
    private readonly TaskCompletionSource<Exception> _source =
        new(TaskCreationOptions.RunContinuationsAsynchronously);

    public Task<Exception> Task => _source.Task;

    public bool TrySet(Exception exception) =>
        _source.TrySetResult(exception);
}

public static class BackgroundFailureExtensions
{
    private const string StateKey = "BackgroundFailure.Signal";

    internal static BackgroundFailureSignal GetBackgroundFailureSignal(
        this TestContext context) =>
        context.StateBag.GetOrAdd(StateKey, _ => new BackgroundFailureSignal());

    public static bool FailFromBackground(
        this TestContext context,
        string reason,
        Exception? innerException = null)
    {
        var exception = innerException is null
            ? new AssertionException(reason)
            : new AssertionException(reason, innerException);

        return context.GetBackgroundFailureSignal().TrySet(exception);
    }
}

Then race the test body against that signal in a custom executor:

using TUnit.Core.Interfaces;

public sealed class BackgroundFailureExecutor : ITestExecutor
{
    public async ValueTask ExecuteTest(
        TestContext context,
        Func<ValueTask> action)
    {
        var failureTask = context.GetBackgroundFailureSignal().Task;

        using var cancellationTokenSource = new CancellationTokenSource();
        context.Execution.AddLinkedCancellationToken(
            cancellationTokenSource.Token);

        var testBodyTask = action().AsTask();
        var completedTask = await Task.WhenAny(testBodyTask, failureTask);

        if (completedTask == testBodyTask && !failureTask.IsCompleted)
        {
            await testBodyTask;
            return;
        }

        var failure = await failureTask;
        cancellationTokenSource.Cancel();

        // Give cooperative cancellation a short chance to stop the body.
        try
        {
            await testBodyTask.WaitAsync(TimeSpan.FromSeconds(1));
        }
        catch
        {
            // The background failure is the result we want to report.
        }

        throw failure;
    }
}

Apply the executor to the relevant tests:

[TestExecutor<BackgroundFailureExecutor>]
public class MyTests
{
    // Tests...
}

Then your existing background code can signal the failure through its captured context:

_ = Task.Run(async () =>
{
    try
    {
        await MonitorAsync();
    }
    catch (Exception exception)
    {
        context.FailFromBackground(
            "The background monitor failed",
            exception);
    }
});

CancellationToken cannot forcibly terminate arbitrary .NET code, so the test body still needs to observe cancellation. The executor can nevertheless report the background failure immediately after the grace period.

I also found a framework issue while verifying this: the test action captured the original cancellation token before the custom executor could link its token. I opened #6565 to fix that, so an injected test CancellationToken will observe the executor's cancellation as intended.

[thomhurst]

Actually I think being able to cancel from somewhere else could prove quite useful. I might put a PR out to make this functionality easier 😄

[aradalvand]

Actually I think being able to cancel from somewhere else could prove quite useful. I might put a PR out to make this functionality easier 😄

@thomhurst Very much so, something like a TestContext.Execution.Cancel() would be perfect.

By the way, shouldn't being able to cancel at least from a TestStart hook already be achievable by ITestExecution.AddLinkedCancellationToken? I tried to use hooks with AddLinkedCancellationToken (coupled with cooperative cancellation in the test body) to accomplish what I intended here but apparently it doesn't work. Just filed an issue at #6567
Update: Does #6565 fix that?

[thomhurst]

@aradalvand Try the latest version

[aradalvand]

Brilliant. Thank you @thomhurst. You're great.