Skip to content

Fix flaky Wolverine.Http.Tests#2218

Merged
jeremydmiller merged 3 commits intoJasperFx:mainfrom
dmytro-pryvedeniuk:fix-flaky-http-tests
Feb 24, 2026
Merged

Fix flaky Wolverine.Http.Tests#2218
jeremydmiller merged 3 commits intoJasperFx:mainfrom
dmytro-pryvedeniuk:fix-flaky-http-tests

Conversation

@dmytro-pryvedeniuk
Copy link
Contributor

@dmytro-pryvedeniuk dmytro-pryvedeniuk commented Feb 20, 2026

This PR fixes an issue with Wolverine.Http tests. Note that the issue is not reproducible when the tests run on GitHub.

On my machine (Windows 11) Wolverine.Http tests always fail, both when running via cmd or from VS. There are different tests and failures: NullReferenceException, ObjectDisposedException, unexpected NotFound errors in AlbaHost.Scenario. See the screenshots below.

To reproduce it's enough to run 2 tests in VS, e.g. multi_tenanted_session_factory_without_wolverine.can_do_the_tenancy_detection and soft_deleted_attribute_usage.try_it_out.
If they run separately they succeed as expected even when running multiple times, but when they run together try_it_out fails with NullReferenceException.

Reasons:

  1. When a test from "integration" collection (e.g. soft_deleted_attribute_usage.try) runs together with a test that creates its own host (e.g. multi_tenanted_session_factory_without_wolverine.can_do_the_tenancy_detection) the latter manages to set WolverineOptions.RememberedApplicationAssembly BEFORE Host creation in AppFixture.InitializeAsync. As the result the endpoints from WolverineWebApi can not be discovered in "integration" collection tests.
  2. IntegrationContext class in Scenario method tries to reset host when 404 status code is received. It's not nice as 404 can be a valid response. Also, it does not help and causes NullReferenceException for soft_deleted_attribute_usage.try_it_out.

As a possible solution I set WolverineOptions.RememberedApplicationAssembly in IntegrationContext and remove dangerous ResetHost.
Also, can_not_compile_with_manual_discovery_by_default test is "fixed" to make the outcome the same when running on GitHub (the scenario returns the expected error as 500 response) and locally (the scenario throws the expected exception directly). Have not managed to find the reason of such different behavior.

Failure1 Failure2 Failure3 Failure4 Failure5

@dmytro-pryvedeniuk dmytro-pryvedeniuk marked this pull request as ready for review February 20, 2026 19:29
@dmytro-pryvedeniuk dmytro-pryvedeniuk marked this pull request as draft February 22, 2026 09:14
@jeremydmiller
Copy link
Member

Good catch. I'm 100% Rider and then GitHub actions, and there is some variability with how different test harnesses will run tests in different order that can mask issues like you hit. Thanks for taking the time to work through that!

@jeremydmiller jeremydmiller marked this pull request as ready for review February 22, 2026 15:51
@jeremydmiller
Copy link
Member

@dmytro-pryvedeniuk Do you want this in now, or do you have other things you want to push later? I'm not in a hurry for a Wolverine release, but there's a new bug to mess with I guess.

@dmytro-pryvedeniuk
Copy link
Contributor Author

Just noticed another flaky http test (see the stack below). AlbaHost does not call StopAsync for the underlying IHost created via web application factory. When stop is called as a part of AlbaHost disposal sometimes it results in ObjectDisposedException. "Sometimes" means it can fail on iteration 43 or 49 (used Run Until Failure in VS Test Runner for use_cascaded_messages_with_http test class).

 Iteration 43)   Wolverine.Http.Tests.use_cascaded_messages_with_http.no_content_chains_should_use_cascading_messages_for_create_variables 
  Duration: 1 ms

  Message: 
	[Test Collection Cleanup Failure (integration)]: System.AggregateException : An error occurred while writing to logger(s). (Cannot access a disposed object.
	Object name: 'EventLogInternal'.)
	---- System.ObjectDisposedException : Cannot access a disposed object.
	Object name: 'EventLogInternal'.

  Stack Trace: 
	Logger.ThrowLoggingError(List`1 exceptions)
	Logger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)
	LoggerExtensions.Log(ILogger logger, LogLevel logLevel, EventId eventId, Exception exception, String message, Object[] args)
	LoggerExtensions.LogError(ILogger logger, Exception exception, String message, Object[] args)
	EndpointCollection.DrainAsync() line 408
	WolverineRuntime.StopAsync(CancellationToken cancellationToken) line 219
	Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
	Host.StopAsync(CancellationToken cancellationToken)
	WebApplicationFactory`1.DisposeAsync()
	AlbaHost.DisposeAsync()
	AppFixture.DisposeAsync() line 59
	----- Inner Stack Trace -----
	EventLogInternal.OpenForWrite(String currentMachineName)
	EventLogInternal.InternalWriteEvent(UInt32 eventID, UInt16 category, EventLogEntryType type, String[] strings, Byte[] rawData, String currentMachineName)
	EventLogInternal.WriteEvent(EventInstance instance, Byte[] data, Object[] values)
	WindowsEventLog.WriteEntry(String message, EventLogEntryType type, Int32 eventID, Int16 category)
	EventLogLogger.WriteMessage(String message, EventLogEntryType eventLogEntryType, Int32 eventId)
	EventLogLogger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)
	Logger.<Log>g__LoggerLog|14_0[TState](LogLevel logLevel, EventId eventId, ILogger logger, Exception exception, Func`3 formatter, List`1& exceptions, TState& state)

With the fix (manual stop for Wolverine runtime) managed to reach 658 successful iterations (then stopped manually).

Success659

@dmytro-pryvedeniuk
Copy link
Contributor Author

I see there are other tests failing on GitHub, but I assume they are flaky themselves. My changes are related to Wolverine.Http.Tests only.

@dmytro-pryvedeniuk
Copy link
Contributor Author

dmytro-pryvedeniuk commented Feb 22, 2026

@jeremydmiller the PR is ready for review.

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants