Skip to content

Share one pinned Pulsar broker per job instead of one container per process (GH-3799) - #3845

Merged
jeremydmiller merged 1 commit into
mainfrom
gh-3799/pulsar-shared-broker
Aug 5, 2026
Merged

Share one pinned Pulsar broker per job instead of one container per process (GH-3799)#3845
jeremydmiller merged 1 commit into
mainfrom
gh-3799/pulsar-shared-broker

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Closes #3799.

PulsarContainerFixture starts a Pulsar container from a [ModuleInitializer], which runs once per process. The supervisor partitions this project across worker processes (2, 3, 6 and 24 observed depending on machine load) and the retry harness spawns another fresh process per retried test — so one CIPulsar run could ask Docker for that many concurrent Pulsar standalone brokers. That is not a small image.

docker-compose.yml already defines a pinned pulsar service that nothing used, and the fixture already has the escape hatch — WOLVERINE_PULSAR wins over starting a container, "same shape as Servers.cs". That variable was set nowhere. So the fix is mostly wiring: start the compose broker once per job and point every worker process at it.

The readiness gate is the half that matters most

With Docker starved, the old arrangement did not fail. It sat at

[INF]   Filter kept 224 of 224 discovered test(s)
[INF]   224 test(s): 224 batched, 0 isolated

forever — no Pulsar container running, no timeout, nothing in the output naming Docker. A silent hang is the worst failure shape we have, because it looks identical to slow. A gate that throws turns it into a named failure in the job that owns it.

The probe asks /admin/v2/brokers/health rather than opening a socket. Pulsar standalone accepts on 6650 well before it will serve, and #3814 measured exactly what a TCP-only gate is worth on Kafka: it passed at 0.0s every time while the broker was still ~3 seconds from ready. Budget is 3 minutes, since a cold Pulsar standalone genuinely takes tens of seconds.

On the third item in the issue

The image pin is already done — both PulsarContainerFixture.PulsarImage and docker-compose.yml are on apachepulsar/pulsar:4.2.4, and the fixture carries the note about :latest having moved that same day. Nothing to do there.

Verification

Ran the CIPulsar target locally:

pulsar containers BEFORE: 1
Starting docker services: pulsar postgresql
Pulsar is up and ready (attempt 1)
=== Wolverine.Pulsar.Tests: 224 passed (0 retries, 2 worker processes) ===

224 passed, and exactly one Pulsar container for the whole run — two worker processes that would previously have started two of their own. docker ps -a --filter ancestor=apachepulsar/pulsar:4.2.4 shows only wolverine-pulsar-1 from compose, so no Testcontainers broker was created at any point.

The probe itself was checked against a real container before wiring it in: GET /admin/v2/brokers/health200 ok.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m

…rocess (GH-3799)

PulsarContainerFixture starts a Pulsar container from a [ModuleInitializer], which runs once
per PROCESS. The supervisor partitions this project across worker processes (2, 3, 6 and 24
observed depending on machine load) and the retry harness spawns another fresh process per
retried test, so a single CIPulsar run could ask Docker for that many concurrent Pulsar
standalone brokers -- not a small image.

docker-compose.yml already defines a pinned pulsar service that nothing used, and the fixture
already has the escape hatch: WOLVERINE_PULSAR wins over starting a container, "same shape as
Servers.cs". That variable was set nowhere. So CIPulsar now starts the compose broker and
points every worker process at it, and no worker starts a container at all.

Adds a readiness gate, which is the half that matters most. With Docker starved, the old
arrangement did not fail -- it sat at "224 test(s): 224 batched, 0 isolated" forever, with no
Pulsar container running, no timeout, and nothing in the output naming Docker. A gate that
throws turns that into a named failure in the job that owns it.

The probe asks /admin/v2/brokers/health rather than opening a socket. Pulsar standalone accepts
on 6650 well before it will serve, and GH-3814 measured exactly what a TCP-only gate is worth
on Kafka: it passed at 0.0s every time while the broker was 3 seconds from ready.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WHAuhdWS3XeAk16swV9G8m
@jeremydmiller
jeremydmiller merged commit a4a3629 into main Aug 5, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pulsar tests start one heavy broker container per worker process, on an unpinned :latest, and hang silently when Docker runs out of memory

1 participant