Skip to content

Make every readiness gate fatal, and pin the images that still rolled (GH-3763) #3813

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jeremydmiller merged 2 commits into from
Aug 4, 2026
Merged

Make every readiness gate fatal, and pin the images that still rolled (GH-3763) #3813

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
87ed165
Make every readiness gate fatal, and pin the images that still rolled…
jeremydmiller Aug 4, 2026
fdb2a59
Fix the SQL Server digest: resolve it with the tool that does the pul…
jeremydmiller Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
278 changes: 135 additions & 143 deletions build/CITargets.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,154 +133,162 @@ void AwaitDockerServices(params string[] services)
WaitForPubsubEmulatorToBeReady();
}

void WaitForPubsubEmulatorToBeReady()
/// <summary>
/// Polls until a service reports itself ready, and <b>throws</b> when the budget runs out.
///
/// <para>Every gate in this file used to log a warning and carry on. That is how
/// <c>CIAzureServiceBus</c> spent 22 of its 25-retry budget on four consecutive <i>green</i> main
/// runs: the gate declared the emulator ready in under a second, provisioning ran against an api
/// still answering 503, a class fixture threw, and 22 tests failed together — as flaky tests, not
/// as infrastructure that never came up. It was found by accident. GH-3783 made that one gate
/// fatal; this makes the rest of them fatal through a single shared path, so a gate added later
/// cannot quietly reintroduce the warn-and-continue shape.</para>
///
/// <para>A container that never starts should fail its job in seconds with a message naming the
/// service, not hand the whole suite to a broker that cannot serve it.</para>
/// </summary>
/// <param name="probe">
/// Returns null when the service is ready, or a short reason why it is not. Exceptions are
/// treated as "not ready" and their message becomes the reason, so the final error carries the
/// last real failure rather than a generic timeout.
/// </param>
void awaitService(string name, TimeSpan budget, Func<string> probe)
{
var attempt = 0;
while (attempt < 30)
var deadline = DateTime.UtcNow.Add(budget);
var lastReason = "no attempt completed";
var attempts = 0;

while (true)
{
attempts++;

try
{
using var tcpClient = new System.Net.Sockets.TcpClient();
tcpClient.Connect("localhost", 8085);
Log.Information("GCP Pub/Sub emulator is up and ready!");
return;
var reason = probe();
if (reason is null)
{
Log.Information("{Service} is up and ready (attempt {Attempts})", name, attempts);
return;
}

lastReason = reason;
}
catch (Exception)
catch (Exception e)
{
// ignore connection errors
lastReason = e.Message;
}

if (DateTime.UtcNow >= deadline) break;

Thread.Sleep(2000);
attempt++;
}

Log.Warning("GCP Pub/Sub emulator did not become ready after 60 seconds");
throw new InvalidOperationException(
$"{name} was not ready after {budget.TotalSeconds:n0}s ({attempts} attempts). " +
$"Last attempt: {lastReason}");
}

void WaitForKafkaToBeReady()
/// <summary>
/// Opens a TCP connection, returning null when it succeeds. The weakest probe shape available —
/// a listening socket is not the same claim as a broker that will serve a request — so prefer a
/// real protocol call wherever the client library is already referenced here.
/// </summary>
static string tcpProbe(string host, int port)
{
using var tcpClient = new System.Net.Sockets.TcpClient();
tcpClient.Connect(host, port);
return null;
}

void WaitForPubsubEmulatorToBeReady()
{
var attempt = 0;
while (attempt < 30)
// The emulator answers HTTP on its main port, so this asks it a question rather than only
// checking that something is listening — the distinction the ASB gate was built on.
using var http = new System.Net.Http.HttpClient { Timeout = TimeSpan.FromSeconds(5) };

awaitService("GCP Pub/Sub emulator", TimeSpan.FromSeconds(60), () =>
{
try
{
using var tcpClient = new System.Net.Sockets.TcpClient();
tcpClient.Connect("localhost", 9092);
Log.Information("Kafka is up and ready!");
return;
}
catch (Exception)
{
// ignore connection errors
}
var response = http.GetAsync("http://localhost:8085/").GetAwaiter().GetResult();

Thread.Sleep(2000);
attempt++;
}
// Any HTTP answer means the emulator is serving. Deliberately not asserting 200: the
// lesson from GH-3783 is that pinning a gate to one status code is how it hangs when the
// service legitimately answers a different one.
return response is null ? "no response" : null;
});
}

Log.Warning("Kafka did not become ready after 60 seconds");
void WaitForKafkaToBeReady()
{
// Still TCP-only, and knowingly the weakest gate here: a Kafka broker accepts connections
// before it will serve metadata or allow topic creation, so this can pass while the broker
// is not yet usable. Fixing it properly means a metadata request through an AdminClient, and
// Confluent.Kafka is not referenced by this build project (nor centrally versioned), so that
// is a change with its own risk rather than a line here. Tracked separately; making the gate
// fatal is the part that matters today.
awaitService("Kafka", TimeSpan.FromSeconds(60), () => tcpProbe("localhost", 9092));
}

void WaitForSqlServerToBeReady()
{
var attempt = 0;
while (attempt < 30)
{
try
{
using var conn = new Microsoft.Data.SqlClient.SqlConnection("Server=localhost,1434;User Id=sa;Password=P@55w0rd;Timeout=5;Encrypt=False");
conn.Open();
var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT 1";
cmd.ExecuteNonQuery();
Log.Information("SQL Server is up and ready!");
return;
}
catch (Exception)
{
Thread.Sleep(2000);
attempt++;
}
}

Log.Warning("SQL Server did not become ready after 60 seconds");
// Budget is now wall-clock rather than an attempt count. The old "60 seconds" in the warning
// was never true: each failed attempt could burn the connection's own 5s timeout before the
// 2s sleep, so 30 attempts was anywhere from 60s to 210s depending on how the failure came
// back. A deadline says what it means.
awaitService("SQL Server", TimeSpan.FromMinutes(2), () =>
{
using var conn = new Microsoft.Data.SqlClient.SqlConnection(
"Server=localhost,1434;User Id=sa;Password=P@55w0rd;Timeout=5;Encrypt=False");
conn.Open();
var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT 1";
cmd.ExecuteNonQuery();
return null;
});
}

void WaitForMySqlToBeReady()
{
var attempt = 0;
while (attempt < 30)
{
try
{
using var conn = new MySqlConnector.MySqlConnection("Server=localhost;Port=3306;Database=wolverine;User=root;Password=P@55w0rd;");
conn.Open();
var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT 1";
cmd.ExecuteNonQuery();
Log.Information("MySQL is up and ready!");
return;
}
catch (Exception)
{
Thread.Sleep(2000);
attempt++;
}
}

Log.Warning("MySQL did not become ready after 60 seconds");
awaitService("MySQL", TimeSpan.FromMinutes(2), () =>
{
using var conn = new MySqlConnector.MySqlConnection(
"Server=localhost;Port=3306;Database=wolverine;User=root;Password=P@55w0rd;");
conn.Open();
var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT 1";
cmd.ExecuteNonQuery();
return null;
});
}

void WaitForOracleToBeReady()
{
var attempt = 0;
while (attempt < 60)
{
try
{
using var conn = new Oracle.ManagedDataAccess.Client.OracleConnection("User Id=wolverine;Password=wolverine;Data Source=localhost:1521/FREEPDB1");
conn.Open();
var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT 1 FROM DUAL";
cmd.ExecuteNonQuery();
Log.Information("Oracle is up and ready!");
return;
}
catch (Exception)
{
Thread.Sleep(2000);
attempt++;
}
}

Log.Warning("Oracle did not become ready after 120 seconds");
// Oracle is the slowest image in the compose file to reach a usable state, hence the roomiest
// budget. Connecting as the application user against FREEPDB1 (rather than pinging the
// listener) is deliberate: the listener answers well before the pluggable database will
// accept an application login.
awaitService("Oracle", TimeSpan.FromMinutes(3), () =>
{
using var conn = new Oracle.ManagedDataAccess.Client.OracleConnection(
"User Id=wolverine;Password=wolverine;Data Source=localhost:1521/FREEPDB1");
conn.Open();
var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT 1 FROM DUAL";
cmd.ExecuteNonQuery();
return null;
});
}

void WaitForLocalStackToBeReady()
{
var attempt = 0;
using var httpClient = new System.Net.Http.HttpClient { Timeout = TimeSpan.FromSeconds(5) };
while (attempt < 30)
{
try
{
var response = httpClient.GetAsync("http://localhost:4566/_localstack/health").GetAwaiter().GetResult();
if (response.IsSuccessStatusCode)
{
Log.Information("LocalStack is up and ready!");
return;
}
}
catch (Exception)
{
// ignore connection errors
}

Thread.Sleep(2000);
attempt++;
}
awaitService("LocalStack", TimeSpan.FromMinutes(2), () =>
{
var response = httpClient.GetAsync("http://localhost:4566/_localstack/health")
.GetAwaiter().GetResult();

Log.Warning("LocalStack did not become ready after 60 seconds");
return response.IsSuccessStatusCode ? null : $"health endpoint answered {(int)response.StatusCode}";
});
}

/// <summary>
Expand Down Expand Up @@ -311,42 +319,26 @@ void WaitForAzureServiceBusEmulatorToBeReady()
// depend on getting the ATOM api-version right here.
const string managementProbe = "http://localhost:5300/$Resources/topics";

var deadline = DateTime.UtcNow.AddMinutes(3);
var lastReason = "no attempt completed";

while (DateTime.UtcNow < deadline)
// Deliberately fatal, via the same shared path as every other gate. This used to log a warning
// and carry on, so an emulator that never came up at all still fed the entire suite into a
// broker that could not serve it — and the resulting failures read as flaky tests rather than
// as infrastructure that never started.
awaitService("Azure Service Bus emulator", TimeSpan.FromMinutes(3), () =>
{
try
{
using (var tcpClient = new System.Net.Sockets.TcpClient())
{
tcpClient.Connect("localhost", 5673);
}

var response = http.GetAsync(managementProbe).GetAwaiter().GetResult();
if (response.StatusCode != System.Net.HttpStatusCode.ServiceUnavailable)
{
Log.Information(
"Azure Service Bus emulator is up and ready for provisioning (management api answered {StatusCode})",
(int)response.StatusCode);
return;
}

lastReason = "the management api on 5300 is still warming up (503)";
}
catch (Exception e)
using (var tcpClient = new System.Net.Sockets.TcpClient())
{
lastReason = e.Message;
tcpClient.Connect("localhost", 5673);
}

Thread.Sleep(2000);
}
var response = http.GetAsync(managementProbe).GetAwaiter().GetResult();

// Deliberately fatal. This used to log a warning and carry on, so an emulator that never came up at
// all still fed the entire suite into a broker that could not serve it — and the resulting failures
// read as flaky tests rather than as infrastructure that never started.
throw new InvalidOperationException(
$"The Azure Service Bus emulator was not ready for provisioning after 3 minutes. Last attempt: {lastReason}");
// Keyed on "not 503", NOT on "== 200". CI answers 400 here and a developer machine
// answers 200; asserting 200 would hang for the whole budget and then fail the job. This
// distinction cost a day to find — do not tighten it.
return response.StatusCode == System.Net.HttpStatusCode.ServiceUnavailable
? "the management api on 5300 is still warming up (503)"
: null;
});
}

/// <summary>
Expand Down
39 changes: 19 additions & 20 deletions build/build.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -405,30 +405,29 @@ bool IsDotNetToolInstalled(string toolName)
return output.Any(line => line.Contains(toolName, StringComparison.OrdinalIgnoreCase));
}

/// <summary>
/// Postgres readiness. This was the thinnest gate in the build by a wide margin: ten attempts
/// separated by 250ms is a <b>2.5 second</b> budget for a container start, after which it logged
/// an error and let the suite run anyway.
///
/// <para>It was already running out of room in production. In CIMQTT5 on main run 30847233633 it
/// spent four of its ten attempts before Postgres answered — roughly 1.1s of a 2.5s allowance —
/// so a slower runner would have sailed past the end and started the tests against a database
/// that was not up. Every failure after that would have looked like a test problem.</para>
/// </summary>
private void WaitForDatabaseToBeReady()
{
var attempt = 0;
while (attempt < 10)
try
{
using var conn = new Npgsql.NpgsqlConnection(PostgresConnectionString + ";Pooling=false");
conn.Open();

var cmd = conn.CreateCommand();
cmd.CommandText = "select 1";
cmd.ExecuteNonQuery();
awaitService("PostgreSQL", TimeSpan.FromMinutes(2), () =>
{
using var conn = new Npgsql.NpgsqlConnection(PostgresConnectionString + ";Pooling=false");
conn.Open();

Log.Information("Postgresql is up and ready!");
return;
}
catch (Exception ex)
{
Log.Information("Database is not ready ({Error})", ex.Message);
Thread.Sleep(250);
attempt++;
}
var cmd = conn.CreateCommand();
cmd.CommandText = "select 1";
cmd.ExecuteNonQuery();

Log.Error("Database is not ready after all attempts.");
return null;
});
}

private Dictionary<string, string[]> ReferencedProjects = new()
Expand Down
Loading
Loading