Skip to content

Avoid flaky tests#211

Open
dmytro-pryvedeniuk wants to merge 3 commits intoJasperFx:masterfrom
dmytro-pryvedeniuk:avoid-flaky-tests
Open

Avoid flaky tests#211
dmytro-pryvedeniuk wants to merge 3 commits intoJasperFx:masterfrom
dmytro-pryvedeniuk:avoid-flaky-tests

Conversation

@dmytro-pryvedeniuk
Copy link

This PR includes fixes to avoid flaky tests when using Alba.

  1. AlbaHost can create IHost using two ways: either owning it directly ("direct" host) or via IAlbaWebApplicationFactory ("factory" host). Ideally both types should behave the same, but it's not the case. E.g. when AlbaHost.StopAsync is called only "direct" host is stopped

    return _host == null ? Task.CompletedTask : _host.StopAsync(cancellationToken);

    I make StopAsync to behave the same for both types. It can make some workarounds redundant (see Fix flaky Wolverine.Http.Tests wolverine#2218 (comment)).

  2. AlbaHost itself have flaky tests. Set of the tests if running together can return unexpected 401 status codes. The problem is caused by code from Microsoft.Identity.Web. There is a static singleton - LogHelper.Logger - configured by the library (see https://github.com/AzureAD/microsoft-identity-web/blob/716239cb42aa167b5ae2a221e8b3bf1681afbd5b/src/Microsoft.Identity.Web.TokenAcquisition/MicrosoftIdentityBaseAuthenticationBuilder.cs#L69). One scenario completes, the host is disposed, but the static singleton keeps the disposed services (e.g. EventLogInternal). When another scenario is running token validation tries to log and fails with ObjectDisposedException. As the result a valid token returns unexpected 401 status code.
    I disable logging for Microsoft.Identity.Web to avoid using the static singleton.

Two selected tests (or the whole set from the screenshot) when running together fail:
AlbaFlakyFriends

This is how token validation fails:
AlbaHostInvalidToken

@dmytro-pryvedeniuk
Copy link
Author

I want to include one more change to this PR. Converting to draft for now.

@dmytro-pryvedeniuk dmytro-pryvedeniuk marked this pull request as draft February 25, 2026 18:30
@dmytro-pryvedeniuk
Copy link
Author

dmytro-pryvedeniuk commented Feb 28, 2026

TLTR: I disable Windows EventLog for AlbaHost.

The issue is not specific for LogHelper.Logger singleton mentioned above (though less singletons is better for integration testing).

In host_stop_usage_repeated tests I tried to replicate how Alba is used for Wolverine tests. There AlbaHost is created 10 times using a hosted service similar to WolverineRuntime: it receives ILoggerFactory, creates ILogger and logs a message on StopAsync. There are different flavors: with explicit stop, without stop, for minimal api (read "factory" host), for mvc ("direct" host).

Used master branch to run host_stop_usage_repeated without other changes:

  1. The tests for minimal api always fail regardless of whether StopAsync is called explicitly or implicitly as a part of host disposal. The tests for mvc are green.
  2. Runtime1 is a simplified version of WolverineRuntime. According to the output StopAsync can be called from multiple threads so the current protection (simple _hasStopped check) is not enough to protect from simultaneous executions. Runtime2 version uses a lock, can be an improvement for Wolverine.
    Both versions fail for minimal api with varying number of the failed iterations.

Output for Runtime1. Note different threads calling StopAsync simultaneously.

Iteration #0: started, Thread: 12
Stopping 51492383, Thread: 12
Stopping 51492383, Thread: 7
Stopped 51492383, Thread: 12
Stopped 51492383, Thread: 7
Iteration #0: stopped, Thread: 12

Iteration #1: started, Thread: 12
Stopping 44741746, Thread: 12
Stopping 44741746, Thread: 18
Stopped 44741746, Thread: 18
Stopped 44741746, Thread: 12
Iteration #1: stopped, Thread: 12

Iteration #2: started, Thread: 12
Stopping 51772423, Thread: 12
Failed with System.ObjectDisposedException 51772423, Thread: 12
Iteration #2: stopped, Thread: 12

Iteration #3: started, Thread: 12
Stopping 59583888, Thread: 12
Stopping 59583888, Thread: 13
Stopped 59583888, Thread: 13
Stopped 59583888, Thread: 12
Iteration #3: stopped, Thread: 12

Iteration #4: started, Thread: 12
Stopping 3829499, Thread: 12
Failed with System.ObjectDisposedException 3829499, Thread: 12
Iteration #4: stopped, Thread: 12

Iteration #5: started, Thread: 12
Stopping 1636903, Thread: 12
Stopping 1636903, Thread: 18
Stopped 1636903, Thread: 12
Stopped 1636903, Thread: 18
Iteration #5: stopped, Thread: 12

Iteration #6: started, Thread: 12
Stopping 34657907, Thread: 12
Failed with System.ObjectDisposedException 34657907, Thread: 12
Iteration #6: stopped, Thread: 12

Iteration #7: started, Thread: 12
Stopping 24929770, Thread: 12
Failed with System.ObjectDisposedException 24929770, Thread: 12
Iteration #7: stopped, Thread: 12

Iteration #8: started, Thread: 12
Stopping 46209412, Thread: 12
Stopping 46209412, Thread: 18
Stopped 46209412, Thread: 12
Stopped 46209412, Thread: 18
Iteration #8: stopped, Thread: 12

Iteration #9: started, Thread: 12
Stopping 13695991, Thread: 12
Stopping 13695991, Thread: 18
Stopped 13695991, Thread: 12
Stopped 13695991, Thread: 18
Iteration #9: stopped, Thread: 12

Output for Runtime2. Version with a lock, single thread calls StopAsync, but still there are exceptions.

Iteration #0: started, Thread: 12
Stopping 48634068, Thread: 12
Stopped 48634068, Thread: 12
Iteration #0: stopped, Thread: 12

Iteration #1: started, Thread: 12
Stopping 46559211, Thread: 12
Failed System.ObjectDisposedException 46559211, Thread: 12
Iteration #1: stopped, Thread: 12

Iteration #2: started, Thread: 12
Stopping 51719118, Thread: 12
Failed System.ObjectDisposedException 51719118, Thread: 12
Iteration #2: stopped, Thread: 12

Iteration #3: started, Thread: 12
Stopping 56758734, Thread: 12
Failed System.ObjectDisposedException 56758734, Thread: 12
Iteration #3: stopped, Thread: 12

Iteration #4: started, Thread: 12
Stopping 55422917, Thread: 12
Failed System.ObjectDisposedException 55422917, Thread: 12
Iteration #4: stopped, Thread: 12

Iteration #5: started, Thread: 12
Stopping 51733474, Thread: 12
Failed System.ObjectDisposedException 51733474, Thread: 12
Iteration #5: stopped, Thread: 12

Iteration #6: started, Thread: 12
Stopping 57519598, Thread: 12
Failed System.ObjectDisposedException 57519598, Thread: 12
Iteration #6: stopped, Thread: 12

Iteration #7: started, Thread: 12
Stopping 28639838, Thread: 12
Failed System.ObjectDisposedException 28639838, Thread: 12
Iteration #7: stopped, Thread: 12

Iteration #8: started, Thread: 12
Stopping 41516407, Thread: 12
Failed System.ObjectDisposedException 41516407, Thread: 12
Iteration #8: stopped, Thread: 12

Iteration #9: started, Thread: 12
Stopping 52885958, Thread: 12
Failed System.ObjectDisposedException 52885958, Thread: 12
Iteration #9: stopped, Thread: 12
  1. When a test fails it's ObjectDisposedException (sometimes NullReferenceException) related to EventLogInternal. It's the same as on the screenshot above.
An error occurred while writing to logger(s). (Cannot access a disposed object.
Object name: 'EventLogInternal'.)
  1. As the used GitHub Actions are based on Linux it's impossible to catch such issues when running on GitHub.

@dmytro-pryvedeniuk dmytro-pryvedeniuk marked this pull request as ready for review February 28, 2026 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant