Pulsar tests start one heavy broker container per worker process, on an unpinned :latest, and hang silently when Docker runs out of memory

[jeremydmiller]

Follow-up to #3763 / #3798, where all three Pulsar compliance classes were restored to CI. Working on that suite surfaced three separate problems with how the tests get a broker. None of them is the transport's fault; all of them cost real time.

1. One container per worker process

PulsarContainerFixture starts a Pulsar container from a [ModuleInitializer]:

[ModuleInitializer]
internal static void Initialize()
{
    ...
    _container = new PulsarBuilder().WithImage("apachepulsar/pulsar:latest")...Build();
    _container.StartAsync().GetAwaiter().GetResult();
}

A [ModuleInitializer] runs once per process. The supervisor partitions a project across worker processes — I saw runs at 2, 3, 6 and 24 workers depending on machine load — and the retry harness spawns another fresh process per retried test. So a single CIPulsar run can ask Docker for many concurrent Pulsar brokers, and Pulsar standalone is not a small image.

The fixture's own comment already anticipates the leak half of this ("a single run can strand several containers that then live forever"), but the cost while the run is live is the bigger one.

2. Docker starvation produces a silent hang, not an error

With nine unrelated containers holding 4.7 GiB of a 7.7 GiB Docker VM, CIPulsar sat at:

[INF]   Filter kept 224 of 224 discovered test(s)
[INF]   224 test(s): 224 batched, 0 isolated

…and never progressed. No Pulsar container was running (docker ps showed none), no timeout fired, nothing in the output pointed at Docker. A compose-started Pulsar in the same conditions was OOM-killed with exit 137. Stopping the unrelated containers fixed it immediately.

Worth a startup precheck, or at minimum a bounded wait with a message naming Docker.

3. The image tag is unpinned, and disagrees with docker-compose

Where	Image
PulsarContainerFixture	apachepulsar/pulsar:**latest**
docker-compose.yml	apachepulsar/pulsar:**4.0.3**

We have already been bitten by exactly this once: an unpinned :latest on the Azure Service Bus emulator moved 2.0.0 → 2.0.1 underneath us and made a CI-only failure look like a code regression (#3783). A pinned tag is also what makes a bisect trustworthy.

A cheap fix for all three

docker-compose.yml already defines a pulsar service, pinned at 4.0.3 — and nothing uses it. CIPulsar starts only postgresql:

StartDockerServices("postgresql");

Meanwhile the fixture already has the escape hatch: set WOLVERINE_PULSAR (and optionally WOLVERINE_PULSAR_HTTP) and it skips the container start entirely — "An already-running Pulsar wins", same shape as Servers.cs. That env var is currently set nowhere.

So the obvious direction is to start the pinned compose Pulsar once per job and point every worker process at it:

StartDockerServices("pulsar", "postgresql");
// + export WOLVERINE_PULSAR for the test processes

One broker per job instead of one per process, a pinned version, and the same code path developers already use for Postgres and SQL Server. The Testcontainers path stays as the fallback for anyone without compose.

Worth confirming first that a shared broker doesn't reintroduce cross-process topic collisions — the RabbitMQ pass (#3795) found exactly that shape where test resource names were not unique per process.

🤖 Generated with Claude Code

https://claude.ai/code/session_0116vfBcKwcjWn8msM4ZjkuA

[jeremydmiller]

The image-pinning half of this is addressed in #3813, and it turned out to be live rather than theoretical.

apachepulsar/pulsar:latest moved to 4.2.4 on 2026-08-03 — the same day — while docker-compose.yml pinned 4.0.3 (Feb 2025). Since the tests run through Testcontainers (PulsarContainerFixture), CI and anyone running docker compose up were on different Pulsar versions, and the one CI used could move without anything in the repo recording it.

Digests at the time of pinning:

apachepulsar/pulsar:latest -> sha256:cd5d4a64...451fb4   (updated 2026-08-03)
apachepulsar/pulsar:4.2.4  -> sha256:cd5d4a64...451fb4   <- identical
apachepulsar/pulsar:4.0.3  -> sha256:50af6732...c07f62   (Feb 2025)

#3813 pins both the fixture and docker-compose.yml to 4.2.4 — byte-for-byte what CI was already running, so no behaviour change, and there is now one Pulsar version in the repository instead of two. The literal is a single shared PulsarContainerFixture.PulsarImage constant, since there was a second unpinned site in PulsarPerTenantConnectionTests.

Still open here: the heavier half — one Pulsar broker container per worker process (the fixture starts it from a [ModuleInitializer], so it is per-process, and CIPulsar runs 3 workers), and the silent hang when Docker runs out of memory. That is unchanged.