[jonahgabriel]

OMN-18012 phase B — the dev lane's broker now refuses a plaintext client

Phase A (#3258) created the lane's SCRAM principal and deliberately changed
nothing else: the listener still accepted PLAINTEXT. This is the half that makes
the refusal exist.

Escape 1 of 2026-09-06: omnimarket's credential_publisher and
generation_publisher built their event bus without
apply_environment_overrides, produced a PLAINTEXT client against the IAM-only
MSK listener, and every BYOK registration 503'd in front of a customer. No local
check could catch it, because every local proof lane runs a no-auth broker and a
no-auth broker accepts that client.

What lands

redpanda-sasl-enable	dev-lane one-shot: grants the principal superuser, sets enable_sasl=true, then asserts an unauthenticated rpk fails and an authenticated one succeeds
x-dev-lane-broker-auth-env	binds the four names ModelKafkaEventBusConfig.apply_environment_overrides() reads, on all 15 Kafka-speaking dev-lane services
DEV_LANE_EXTRA_BROKER_CLIENTS	deploy-runtime.sh now recreates context-audit-consumer on the dev lane
refresh_dev_lane.sh	redpanda-sasl-enable added to REFRESH_BUILD_SERVICES so a governed refresh re-asserts the flip

Measured, not assumed

Read live from this lane's own admin API cluster_config/schema: enable_sasl
needs_restart=False, superusers needs_restart=False. That is what makes a
flip on a running lane safe rather than a broker recreate.
kafka_enable_authorization is null — in Redpanda that means "follow
enable_sasl" — so the superuser grant must precede the flip. Reversed, every
client authenticates and is then authorized for nothing, which reads identically
in a client log.

A cluster-config one-shot and not --set on the broker command: --set is a
node-startup flag, this lane's redpanda_data volume already exists (cluster
config version 2), so a --set would render, deploy, log nothing and leave the
listener open. It is also the only safe form — redpanda is declared in the BASE
file that four other lanes merge.

Two defects the new gates caught, both fixed here rather than deferred

1. context-audit-consumer is a dev-lane Kafka client that no deploy path
   restarts — absent from RUNTIME_SERVICES and from
   DEV_LANE_ONLY_RUNTIME_SERVICES. Up 2 weeks across every intervening deploy.
   Binding a credential on a container nobody recreates is inert, so it would
   have kept speaking PLAINTEXT to a listener that now refuses it — escape 1's
   own shape, reintroduced by the change meant to prevent it.
2. redpanda-sasl-enable was outside refresh_dev_lane.sh's build scope, so
   a governed refresh would never re-assert the flip.

The four observability consumers looked like an exception and are not: they
resolve their bootstrap address from a service-prefixed variable and their
auth from the unprefixed names, through the same
apply_environment_overrides(). Read in each consumer module, not inferred from
the prefix — that asymmetry is why this was not a search-and-replace.

RED / GREEN

The 10 new assertions fail on the parent (origin/dev a868bbd91) and pass
here. The restart-coverage gate isolates the defect exactly: with the compose
half applied and the deploy half not, the uncovered set is precisely
['context-audit-consumer'].

Blast radius, by real docker compose config render

lane	services	authenticated clients	one-shots
dev (before → after)	29 → 30	0 → 15	1 → 2
prod	12	0	0
stability-test	19	0	0

Zero occurrences of any of the five needles in the base file or in the prod,
stability-test, judge and lakshman overlays. Positive control: 28 occurrences in
the dev-lane overlay. The stability-test / judge / lakshman render errors seen
during this work reproduce identically at the parent commit and are unrelated to
this change.

Credentials are fail-closed compose references resolved from the operator env
file, never literals. Not a credential rotation (Operating Rule 22): this
binds the synthetic principal phase A provisioned; nothing is rotated, re-issued
or revoked, and none is authorized.

Residual

This proves PLAINTEXT-vs-authenticated, not IAM. Staging and prod are
AWS_MSK_IAM over SASL_SSL; the mechanism differs, the pinned failure mode — a
client built with no credentials at all — does not. The IAM leg stays a staging
concern.

Ticket: OMN-18012

Evidence-Ticket: OMN-18012
Evidence-Source: OCC#8555

[github-actions]

✅ Hostile Reviewer — REVIEWED

Critical findings: 2
Major findings: 2
Total findings: 14
Models succeeded: qwen3-review,qwen3-review-b,glm-review

Action required: findings were posted as review threads. Address or reject each one, then resolve its thread — the Hostile Review Thread Gate fails while hostile-reviewer threads are unresolved (OMN-17492).

---

Semantics (OMN-17492 — the model finds, thread resolution gates)

Surface	Meaning	Blocks merge?
Review threads	Per-finding, posted by the reviewer	No (informational)
Hostile Review Thread Gate	Deterministic: unresolved hostile-reviewer threads exist	Fails until resolved (not yet a required context)
degraded verdict	Fewer than 2 models succeeded (infra)	No

Powered by omniintelligence.review_pairing.cli_review — multi-model adversarial review: qwen3-review, qwen3-review-b, glm-review (OMN-8468/OMN-8524/OMN-17492)

[github-actions]

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: glm-review
Models failed: codex
New finding threads: 2
Deduped (already posted on this PR): 0
Nit-level findings suppressed: 1

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Findings not anchored to a changed file

• [CRITICAL] hostile-reviewer (glm-review)

  Flip one-shot replaces the entire superusers list | rpk cluster config set superusers "['$USER']" is a full-value assignment on a list-typed cluster property. Any pre-existing superusers on this cluster are silently removed, and any future operator-added superuser will be removed the next time the idempotent re-run executes (the comment explicitly advertises re-running as safe). Idempotence here is destructive idempotence: the script re-asserts a singleton list, not membership. | Evidence: /usr/bin/rpk cl

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MAJOR] hostile-reviewer (glm-review)

  No retry around the unauthenticated-refusal check after a dynamic config set | enable_sasl was confirmed needs_restart=False, but a dynamically set cluster property still propagates through the controller log to each broker. The script flips enable_sasl and immediately probes the Kafka listener with an unauthenticated client. If propagation has not completed, the probe succeeds, the one-shot exits 1, and the deploy fails spuriously. The readback for enable_sasl reads the controller's view, not the listene

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  Credential passed on the rpk command line | The AUTH string expands the password into argv for every authenticated rpk invocation. Inside a single-purpose container the exposure is limited, but the password is readable from /proc//cmdline for the process lifetime and may surface in any future exec/logging instrumentation. The username and password are already exported in the service environment; argv is strictly worse. | Evidence: AUTH="-X sasl.mechanism=SCRAM-SHA-256 -X user=$$DEV_KAFKA_SASL_USERNAME

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  Substring grep on the superusers readback can false-positive | grep -q "$DEV_KAFKA_SASL_USERNAME" matches if the username appears as a substring of any element of the serialized list (e.g. principal svc_kafka matches a list containing svc_kafka_v2). The assertion that the grant landed is therefore weaker than the comment claims. | Evidence: /usr/bin/rpk cluster config get superusers $$ADMIN | grep -q "$$DEV_KAFKA_SASL_USERNAME" | Fix: Match with delimiters appropriate to rpk's output format (grep -E "

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  enable_sasl readback asserts on an unspecified output format | grep -qx true assumes rpk cluster config get enable_sasl prints the bare token true. rpk's default YAML-flavored config output may quote or format boolean values; a format change in the pinned image line or a future image bump breaks the readback as a false failure, and the assertion's coupling to presentation is undocumented. | Evidence: /usr/bin/rpk cluster config get enable_sasl $$ADMIN | grep -qx true | Fix: Use `rpk cluster config get

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

Findings demoted from threads (anchor rejected)

• [MAJOR] hostile-reviewer (glm-review)

  redpanda-sasl-enable is absent from the deploy-runtime restart path | The change adds DEV_LANE_EXTRA_BROKER_CLIENTS to deploy-runtime.sh and adds the flip to refresh_dev_lane.sh, but the flip one-shot appears in neither deploy-runtime restart array. The ticket's central invariant (the refusal exists and is verified) is asserted only by a service that compose runs on up. If the broker volume is rebuilt through the sanctioned deploy path, or the one-shot's prior run predates a controller-log change, the dep

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  Branch-scoping test depends on incidental file ordering | test_the_extra_array_is_appended_only_on_the_dev_lane_branch slices the script with text.index("lane_only_services=(") and text.index("docker-compose.stability-test.yml)"). This assumes the dev branch is the first lane_only_services assignment and precedes the stability-test case label. Reordering cases or adding a lane above the dev branch silently turns the slice into the wrong region, and the test then asserts against an irrelevant span (and can p

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

[github-actions]

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: qwen3-review, qwen3-review-b, glm-review
Models failed: none
New finding threads: 3
Deduped (already posted on this PR): 0
Nit-level findings suppressed: 3

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Findings not anchored to a changed file

• [CRITICAL] hostile-reviewer (qwen3-review)

  Insecure credential handling in rpk command arguments | The redpanda-sasl-enable service passes SASL credentials directly as command-line arguments via rpk cluster config set. In Linux environments, command-line arguments are visible to all users via ps aux or /proc/<pid>/cmdline. This exposes the SCRAM-SHA-256 password in plaintext to any local user or process with read access to the process table. While the container is ephemeral, this violates standard security practices for credential handling.

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [CRITICAL] hostile-reviewer (qwen3-review-b)

  Credentials exposed via container inspection and logs | The redpanda-sasl-enable service runs in the same Docker network as the broker and other clients. It receives DEV_KAFKA_SASL_USERNAME and DEV_KAFKA_SASL_PASSWORD as environment variables. Docker containers expose their environment variables via docker inspect and /proc/self/environ. Any compromised container or user with docker inspect access can retrieve these plaintext credentials. Furthermore, the script uses echo statements that may l

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MAJOR] hostile-reviewer (glm-review)

  Unauthenticated admin API can silently undo the SASL flip | The flip's entire purpose is to make the refusal exist, yet the same one-shot acknowledges the admin API stays open with admin_api_require_auth=false. Any client that can reach redpanda:9644 on the omnibase-infra-network can issue cluster config set enable_sasl false or rewrite superusers, silently reverting the gate. Nothing in the flip service, the refresh script, or the test suite detects this reversion; the readback runs once at flip time a

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MAJOR] hostile-reviewer (glm-review)

  SCRAM password exposed in process arguments and container metadata | The password is interpolated into rpk -X pass=$$DEV_KAFKA_SASL_PASSWORD as a command-line argument, readable via ps inside the container and via process auditing. It is additionally stored in plaintext in the flip container's environment, visible to anyone with docker inspect on the host. The compose comments treat reference-vs-literal as the credential boundary, but that boundary only covers the YAML file; the runtime exposure is un

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  Superuser readback grep is a substring match and treats the username as a regex | grep -q "$DEV_KAFKA_SASL_USERNAME" against the superusers list will match if the username appears as a substring of any other principal, and treats regex metacharacters in the username as patterns. A username such as svc would falsely confirm when svc-admin is a superuser and the intended principal's grant silently failed. The flip would then enable SASL with no authorized principal, producing exactly the lockout the ord

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  Unquoted shell expansions break on spaces or special characters in credentials | $$ADMIN, $$BROKER, and $$AUTH are expanded unquoted, and AUTH embeds the username and password directly. Any whitespace or glob character in the operator-supplied password splits into multiple argv entries, producing an authentication failure whose diagnostics will not indicate the cause. set -u does not catch this class. | Evidence: AUTH="-X sasl.mechanism=SCRAM-SHA-256 -X user=$$DEV_KAFKA_SASL_USERNAME -X pass=$$DEV_K

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  SASL_PLAINTEXT with SCRAM leaves the lane vulnerable to MITM credential harvesting | SCRAM-SHA-256 over a plaintext listener protects the raw password but is subject to man-in-the-middle downgrade and offline dictionary attack on the SCRAM exchange. The comments acknowledge the mechanism differs from staging (SASL_SSL) and argue the failure mode transfers, but the credential-transfer property does not: a network observer on the dev lane captures auth material staging never exposes. The tradeoff may be accep

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  Flip ordering relative to client container recreation is unverified | The flip one-shot runs during compose up, while the deploy script restarts runtime services from its arrays. If any client container starts (or is recreated) before the flip completes, it attempts a plaintext connection against a listener that may already require SASL, producing a transient failure window. Nothing in depends_on or the deploy script sequences runtime restarts after redpanda-sasl-enable exits successfully; the change relies

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

Findings demoted from threads (anchor rejected)

• [MINOR] hostile-reviewer (glm-review)

  Grant-precedes-flip test asserts on comment text, not executable script | script.index("cluster config set superusers") searches the full command text including the extensive comment block. If a future comment mentions the string earlier than the actual command, the ordering assertion passes while the lockout window it guards is open. The same brittleness applies to the refusal/positive-control string assertions, which test echo literals rather than rpk invocations. | Evidence: grant_at = script.index("cl

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  Bash array regex is fragile against inline closing parens | _bash_array matches ^readonly NAME=\((.*?)^\) non-greedily to the first line starting with ). Any array whose body contains a line beginning with ) (for example a wrapped one-element array or a comment terminating in a paren placed at line start) truncates the parse. The positive control checks counts >= 15, which mitigates but does not eliminate this: a truncation that still leaves 15+ entries yields a silently wrong set and a vacuous cove

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (glm-review)

  Test derives client set from the overlay only, missing base-declared clients on the dev lane | _authenticated_dev_lane_clients enumerates services declared in docker-compose.dev-lane.yml. Any Kafka client declared solely in the base infra file and inheriting credentials through a shared anchor (or added to the dev lane later via x-common-env paths) escapes the guard. The test's docstring claims it 'derives the client set from the compose overlay' as a virtue, but the merge of base and overlay is the actua

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).