Kafka readiness gate is TCP-only: a broker accepts connections before it will serve metadata

[jeremydmiller]

Split out from #3813, which made every readiness gate fatal but left this one's probe unchanged.

WaitForKafkaToBeReady opens a TCP connection to localhost:9092 and calls that ready. A Kafka broker accepts connections well before it will serve metadata or allow topic creation, so the gate can pass while the broker is not yet usable — and the tests that then fail (Leader not available, Not enough replicas, topic-creation errors) look like flaky tests rather than a broker that was not up.

This is the same class of defect as the ASB emulator gate in #3783: a probe that answers a weaker question than the one the suite depends on. That one cost 22 retries per run across four green main runs before anyone noticed.

Why it was not fixed in #3813

A real probe means a metadata request through a Confluent.Kafka AdminClient. That package is not referenced by build/build.csproj and is not in Directory.Packages.props, so it is a dependency addition to the build project rather than a line change — enough risk to be worth its own PR.

Suggested fix

Add Confluent.Kafka to the build project and probe with something like:

using var admin = new AdminClientBuilder(new AdminClientConfig { BootstrapServers = "localhost:9092" }).Build();
admin.GetMetadata(TimeSpan.FromSeconds(5));   // throws until the broker will actually serve

then pass that through the existing awaitService helper, which already handles budget, retry and the fatal ending.

Note

CIKafka is currently green and at 0 retries, so this is latent, not active. It is worth doing because the failure mode is invisible when it does happen, not because it is failing today.