fix(OMN-12533): retry broker metadata startup errors

[jonahgabriel]

Summary

• Treat aiokafka BrokerNotAvailableError as a transient consumer-start metadata convergence error.
• Reuse the existing bounded metadata retry path used for UnknownTopicOrPartitionError and InvalidPartitionsError.
• Cap local Docker topic provisioning to one partition by default and make ensure_topic_exists() honor the same cap.
• Add focused coverage for BrokerNotAvailableError retry and single-topic cap enforcement.

Runtime evidence

• .201 runtime-effects has ONEX_STATE_DIR=/app/data/.onex_state but remains in runtime_pending while startup emits repeated BrokerNotAvailableError metadata fetches for omnimarket command topics.
• Redpanda is healthy and reachable from the effects container.
• Redpanda is saturated: rpk reports topic_partitions_per_shard=7000 and current topics=1184 / partitions=6994.
• Creating a missing topic with 6 partitions fails: INVALID_PARTITIONS / hardware constraints.
• Creating a missing topic with 1 partition also currently fails on the hot broker: BROKER_NOT_AVAILABLE / no eligible allocation nodes, meaning the live .201 broker needs capacity/cleanup in addition to this code fix.

Verification

• uv run pytest tests/integration/event_bus/test_kafka_invalid_partitions_retry_integration.py tests/unit/event_bus/test_service_topic_manager.py -q
• uv run ruff check src/omnibase_infra/event_bus/event_bus_kafka.py src/omnibase_infra/event_bus/service_topic_manager.py tests/integration/event_bus/test_kafka_invalid_partitions_retry_integration.py tests/unit/event_bus/test_service_topic_manager.py
• git diff --check

Evidence / OCC

• OCC receipt work is still required before this can leave draft / pass Receipt Gate. No receipt is claimed in this PR yet.

Evidence-Source: OCC#2005
Evidence-Ticket: OMN-12533

[coderabbitai]

Warning

Review limit reached

@jonahgabriel, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 36 minutes and 11 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 806e315f-448d-4736-aeb3-1dd8cc5ed7ff

📥 Commits

Reviewing files that changed from the base of the PR and between ae6c8b5 and 6c92f4b.

📒 Files selected for processing (6)

• docker/Dockerfile.runtime
• docker/docker-compose.infra.yml
• src/omnibase_infra/event_bus/event_bus_kafka.py
• src/omnibase_infra/event_bus/service_topic_manager.py
• tests/integration/event_bus/test_kafka_invalid_partitions_retry_integration.py
• tests/unit/event_bus/test_service_topic_manager.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch jonah/omn-12533-runtime-effects-broker-metadata-retry

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[jonahgabriel]

Receipt Gate blocker update (2026-05-31T16:48Z)

The gate failure moved from missing Evidence-Source: to invalid evidence source format. The gate only accepts OCC#<number> or an OCC SHA, so I created central evidence instead of using prose in the PR body.

Central OCC evidence PR: OmniNode-ai/onex_change_control#2004
PR body now has: Evidence-Source: OCC#2004

Local OCC validation passed:

• uv run validate-yaml contracts/OMN-12533.yaml
• uv run pre-commit run --files contracts/OMN-12533.yaml drift/dod_receipts/OMN-12533/dod-infra-pr-1813/command.yaml drift/dod_receipts/OMN-12533/dod-local-validation/command.yaml drift/dod_receipts/OMN-12533/dod-runtime-evidence/command.yaml drift/dod_receipts/OMN-12533/dod-occ-pr-self/command.yaml

Rerunning Receipt Gate contexts now.

[jonahgabriel]

Receipt Gate follow-up (2026-05-31T16:52Z)

OCC PR #2004 merged quickly, but its self-receipt had a malformed commit_sha field. The current #1813 Receipt Gate failures are now from that central OCC receipt validation, not from #1813 source.

I branched a follow-up OCC fix from current onex_change_control/dev to correct drift/dod_receipts/OMN-12533/dod-occ-pr-self/command.yaml, and I am validating/opening that now. After the follow-up OCC evidence PR is available, I will repoint this PR to the new Evidence-Source and rerun Receipt Gate again.

[jonahgabriel]

Receipt Gate evidence update (2026-05-31T16:53Z)

Opened follow-up OCC evidence PR for the malformed self-receipt from #2004: OmniNode-ai/onex_change_control#2005

Updated this PR body to Evidence-Source: OCC#2005 and reran Receipt Gate contexts. #2005 is based on current onex_change_control/dev, touches only drift/dod_receipts/OMN-12533/dod-occ-pr-self/command.yaml, and local validation passed:

• uv run validate-yaml contracts/OMN-12533.yaml
• uv run pre-commit run --files drift/dod_receipts/OMN-12533/dod-occ-pr-self/command.yaml

[jonahgabriel]

Receipt Gate schema update (2026-05-31T16:56Z)

OCC evidence is now clean and merged through onex_change_control#2005. The next gate failure required the PR body to include an Evidence-Ticket line whenever Evidence-Source is present.

PR body now contains:

• Evidence-Source: OCC#2005
• Evidence-Ticket: OMN-12533

Rerunning Receipt Gate contexts again.

[jonahgabriel]

Receipt Gate cleared from current rollup (2026-05-31T16:57Z)

After adding Evidence-Ticket: OMN-12533, the current PR rollup no longer shows Receipt Gate as failed. The remaining visible failed required context is stale Docker Integration Tests from Docker Build run 26704731266, whose parent workflow had previously completed success while leaving that job context stale.

Reran Docker Build 26704731266 to refresh the required Docker Integration context.

[jonahgabriel]

Foreground action: reject-skip run 26718529500 is terminal cancelled, so I reran failed jobs. The Topic Enum Drift cancellation in CI parent 26704731327 is active-parent-blocked because that CI parent still has Migration Freeze, Lint, and ONEX Validators in progress plus queued jobs. No source action indicated; continuing to poll active parents.

[jonahgabriel]

Foreground action: Env Parity run 26704731272 is terminal cancelled, so I reran failed jobs. The Topic Enum Drift cancellation in CI parent 26704731327 remains active-parent-blocked because Migration Freeze/Lint/ONEX Validators are in progress and other CI jobs are queued.

[jonahgabriel]

Foreground tick update for #1813:

• Standards run 26704731260 attempt 2 is still an active/queued parent: PEP 604 Type Union Check (UP007) remains queued.
• The visible Type Safety Validation failure (78741018087) is setup/transport, not source: uv sync --no-cache --all-extras failed downloading/extracting cryptography==46.0.7 with Failed to download distribution due to network timeout. Try increasing UV_HTTP_TIMEOUT (current value: 30s).
• The CI cancellation (Topic Enum Drift Check, run 26704731327) is also active-parent-blocked while that CI parent remains queued.

No source patch is indicated from these logs. I am not rerunning the Standards or CI parent while they are still active/queued; next action is to rerun only after the relevant parent reaches a terminal state, unless a real source failure appears.

[jonahgabriel]

Foreground tick update for #1813:

• Security Scan run 26704731311 attempt 2 is terminal failure on CodeQL / CodeQL Analysis (python) job 78741019109.
• Raw job logs return GitHub Actions storage BlobNotFound; no source-visible CodeQL assertion is available from the logs.
• Reran failed jobs for terminal Security Scan parent 26704731311.

I left Standards run 26704731260 and CI run 26704731327 alone because both are still active/queued parents with queued children; rerun there remains unsafe until parent settlement.

[jonahgabriel]

Foreground tick update for #1813:

• CI run 26704731327 attempt 2 is still active/queued. Current active jobs include Lint, Version Pin Compliance, and Kafka Boundary Compat, with additional queued children.
• New visible ONEX Validators failure (78741018579) is setup/transport, not source: uv sync failed after retries on git dependency fetches for omnibase_spi, onex_change_control, and omnibase_core with Empty reply from server, GnuTLS recv error, early EOF, and invalid index-pack output.
• Existing Topic Enum Drift Check cancellation remains active-parent-blocked inside the same CI parent.

No source patch is indicated from these logs. I am not rerunning CI until parent 26704731327 reaches terminal state.

[jonahgabriel]

Foreground tick update for #1813:

• CI parent 26704731327 remains active/queued with Topic Drift, Version Pin Compliance, Kafka Boundary Compat, and Kafka Schema Handshake in progress, plus queued children.
• New cancelled Lint job 78741018576 joins the existing cancelled Topic Enum Drift Check; ONEX Validators remains setup-transport from git fetch failures, already classified.

No rerun while parent 26704731327 is still active/queued.

[jonahgabriel]

Codex foreground tick: CI parent 26704731327 remains non-terminal with Contract Compliance, Kafka Boundary Compat, and Contract Compliance Check still active plus queued children. The new Compose Required-Env Coverage / Kafka Schema Handshake failures and cancelled jobs are still active-parent-blocked; leaving rerun until the parent settles.

[jonahgabriel]

Codex foreground tick: CI parent 26704731327 remains non-terminal with Kafka Boundary Compat and Contract Compliance Check still active plus queued children. The new Contract Compliance cancellation is active-parent-blocked with the existing failed/cancelled fanout; no rerun until the parent reaches terminal state.