Deploy Redis in k3s and pin the orchestrator to the Redis message store (#2662) - #3153
Conversation
The k3s manifests deployed no Redis and set no message-store env, so the orchestrator's auto backend selection always fell back to the in-memory MessageStore — RedisMessageStore had zero live end-to-end coverage and messages did not survive orchestrator restarts (#3076). Since #3077 slice-6, every deployed boot also logged the fail-loud fallback marker and reported a degraded message_store health component. The gap had a third layer the issue didn't name: the redis client package was not installed in the orchestrator image at all, so even with Redis deployed the import inside _create_message_store() would have failed before the backend was probed. - egg-redis image (config/redis/Dockerfile): stock Redis pinned at 8.6.4-alpine, built and published through the local loopback-registry supply chain like the other egg images (no public image references in the manifests). - k8s/base/redis-deployment.yaml + redis-service.yaml: single replica, Recreate, 1Gi PVC with appendonly, maxmemory 256mb + noeviction so memory pressure fails writes loudly instead of dropping messages. Agents cannot reach it (egg-agents is default-deny egress). - orchestrator-deployment.yaml: EGG_MESSAGE_STORE_BACKEND=redis (explicit, not auto — an outage fails loudly; the lazily created store singleton retries until Redis is reachable) + REDIS_HOST/PORT. - redis>=5.0,<7.0 added to orchestrator/requirements.txt and pyproject runtime deps. - integration_tests/test_message_store_backend.py: pins the deployed backend choice, asserts /api/v1/health carries no fallback marker, and round-trips a message through the live Redis from inside the orchestrator pod via the production selection path. - docs: concurrent-execution.md and coordination-state.md now match deployed behavior.
This comment has been minimized.
This comment has been minimized.
Autofix tracking{"Test/Integration Tests / Integration Tests": 2} |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Caught live by the first canary pipeline on the deployed Redis backend: the connection pool sets socket_timeout=5, and redis-py enforces that timeout on the blocked read itself — so every agent long-poll (wait=25..60s) issued a single XREAD BLOCK that died at the 5s mark with "Timeout reading from socket", surfacing as an ERROR log and a 500 from /messages/wait. Agents only progressed because their wait CLI retries; event-driven waiting was effectively degraded to error-spam polling. fakeredis has no sockets, so the unit tier structurally could not catch this. Fix: blocking reads are sliced into XREAD BLOCK chunks of at most _MAX_BLOCK_MS (4s, safely under the 5s socket timeout), looping on the caller's wait deadline. XREAD returns immediately when data arrives, so chunking costs one extra round-trip per idle slice, not delivery latency. Fast-path semantics are preserved (wait ends at the first batch of rows, filtered or not). Belt-and-braces: a TimeoutError that still fires on a blocking slice degrades to an idle slice instead of killing the whole wait; non-blocking reads keep raising. Tests pin the slice cap (including its < socket_timeout relationship), both blocking branches, the idle-slice degradation, and that non-blocking timeouts still propagate.
The integration test workflow built and deployed the new egg-redis image but never imported it into k3s/containerd, so 'make deploy' rewrote the redis manifest to egg-redis:<sha> (imagePullPolicy: IfNotPresent) and the redis pod failed with ErrImagePull -> 'egg-system pods cannot pull image tag'. Import both egg-redis:latest and egg-redis:$EGG_IMAGE_TAG alongside the other egg images, and include egg-redis in the post-import Docker reclaim grep.
|
Live-canary finding, fixed in 2b2dc9f. Deployed this branch to the local cluster and submitted a throwaway pipeline as the first real traffic through the Redis backend. Writes and non-blocking reads worked immediately (stream keys created, heartbeats flowing), but every agent long-poll errored at exactly 5s: Root cause: the connection pool sets Fix: blocking reads are sliced into Re-running the canary (including a mid-phase orchestrator restart to exercise the #3076 durability story) after redeploying with this fix. |
This comment has been minimized.
This comment has been minimized.
|
Round-2 canary + restart drill: PASS (cluster redeployed with 2b2dc9f + 887b3ef).
On the old in-memory backend this restart would have dropped the entire transcript and de-synced the consensus tracker. End-to-end validation of both halves of the issue (live Redis coverage + restart durability) is done. |
There was a problem hiding this comment.
No agent-mode design concerns.
This is infrastructure-only — Redis k3s manifests, the egg-redis Dockerfile, dependency pins, integration tests, and doc updates. It has no agent-prompt surface: no prompts, no model identifiers, no structured-output-for-humans, no pre-fetching, and no post-processing pipelines. The lone requests.get in the new integration test targets the orchestrator's own /api/v1/health endpoint, not the Anthropic API, and httpx is only a dependency pin — neither implicates the EGG200 direct-API-call boundary.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
|
Full live-validation record for this branch (consolidating the thread):
Follow-ups spawned by this validation, both out of scope here:
|
There was a problem hiding this comment.
Review: Deploy Redis in k3s + pin orchestrator to Redis message store (#2662)
I traced the full path from the deployed redis-server args through the orchestrator's connection logic and the blocking-read code. The deployment scaffolding (Makefile/CI/overlay/await-egg-deploy.sh wiring, NetworkPolicy isolation, securityContext, dep declaration + lockfile) is correct and the lazy-retry / no-startup-ordering claim checks out. But there is one blocking issue that makes the feature non-functional end-to-end, and a second real correctness bug in the blocking-read change that ships alongside it.
🔴 BLOCKING 1 — Redis rejects the orchestrator's connection: protected mode + no password
k8s/base/redis-deployment.yaml runs:
redis-server --appendonly yes --dir /data --maxmemory 256mb --maxmemory-policy noeviction
No --requirepass, no --protected-mode no. Stock Redis defaults to protected-mode yes with a no-password default user. In Redis 8 the protected-mode gate is (verified in the 8.0 source):
src/networking.c(~L1366): rejects the connection whenserver.protected_mode && (DefaultUser->flags & USER_FLAG_NOPASS)and the peer is not loopback. There is no bind-address exemption in this check anymore — binding to all interfaces (the compiled-in default{"*","-::*"},src/server.h:119) does not relax it.src/socket.c:331connSocketIsLocal()treats only127.*/::1as local.
The orchestrator connects from a different pod via redis.egg-system.svc.cluster.local (orchestrator-deployment.yaml), i.e. a non-loopback ClusterIP source. So get_redis_message_store()'s client.ping() (orchestrator/redis_message_store.py:649) receives DENIED Redis is running in protected mode…, raises ConnectionError, and because the backend is pinned to explicit redis, _create_message_store() re-raises (message_store.py:722-723). Every message operation then 500s — inter-agent messaging, the exact thing this PR is enabling, does not work.
This is the canonical Docker/Kubernetes protected-mode failure mode (it's why Bitnami's chart forces either a password or ALLOW_EMPTY_PASSWORD=yes). The orchestrator client passes no password (get_redis_message_store has no password param, redis_message_store.py:627-646), so authentication cannot satisfy the gate either.
Why CI didn't catch it: your own integration test #3 (test_redis_streams_roundtrip_from_orchestrator_pod) is precisely the probe that would fail here — but the PR description states the integration tests have only been collected, not run live ("they exercise the live cluster on the next make redeploy / make test-integration"). The single most important validation for this PR hasn't run yet.
Fix (minimal, consistent with the current no-auth client): add --protected-mode no to the container args. The isolation protected mode would provide is already enforced by network-policies.yaml (egg-agents is default-deny egress with no Redis allow rule; only orchestrator in egg-system, which has no NetworkPolicy, can reach it). Alternatively set --requirepass <secret> — but that additionally requires threading a password=/REDIS_PASSWORD through get_redis_message_store() and the pool, plus a Secret, since the client sends no password today. Please redeploy and let test #3 actually run green before merge.
🔴 BLOCKING 2 — Chunked blocking read re-resolves $ per slice → message-drop window (diverges from the in-memory backend)
The new chunking in redis_message_store.py re-issues XREAD BLOCK in ≤_MAX_BLOCK_MS slices. On the from_tip path start_id is "$", and it is never advanced across idle slices:
- Fast path (
L420-434):start_idstays"$"; the loop discardslast_sid(messages, _ = _read_once(start_id, …)). wait_for_typespath (L484-487):current_start = last_sidonly fires when a row was read; on a pure-idle slicelast_sid is None, socurrent_startstays"$".
$ is resolved server-side to the current tip on every re-issue. So in the gap between one XREAD BLOCK returning empty and the next being issued (~one RTT), any XADD advances the tip, and the next $ block starts after it — that message is never delivered by this call. The route then makes the loss permanent: on timeout it sets cursor = get_latest_id() (routes/messages.py:782), which now points at the missed message, and the next wait-loop call passes it as since_id and reads strictly after it (messages.py:767, from_tip disabled once since_id is set).
Contrast the in-memory backend, which snapshots the tip once at call entry under the lock (message_store.py:437-438, start_idx = len(initial_msgs)) and re-filters from that fixed index — it cannot drop a message that arrives mid-wait. The comment at message_store.py:375-377 calls this out explicitly ("so from_tip semantics are race-free against concurrent add_message calls"). The Redis path now violates that invariant, so the two backends give different results for the same sequence of events — exactly the cross-path inconsistency this kind of consensus infra can't afford (a dropped CONSENSUS_CONFIRMED/wake-up can stall a pipeline).
It is low-probability (the blind window is ~RTT per slice, and it closes as soon as any traffic flows, since a non-matching row replaces $ with a concrete id), and it's strictly better than the pre-PR state where every long-poll 500'd at the 5 s socket timeout. But it's a real, hard-to-debug loss in the critical path and the fix is cheap: resolve the tip to a concrete stream id once before the chunked loop (e.g. XREVRANGE key + 1, or "0-0" fallback when empty) and advance from that concrete id across slices, mirroring the in-memory once-at-entry snapshot. Then $ is never re-resolved mid-wait.
🟡 Non-blocking
-
test_health_reports_message_store_okis vacuous in explicit-redis mode (integration_tests/test_message_store_backend.py:214-223). The health route derivesmessage_storepurely fromis_memory_fallback_degraded()(routes/health.py:137-144), and that flag is only ever set on the auto→memory fallback path (message_store.py:730). In explicitredismode the flag is structurally alwaysFalse, so the component is always{"status":"ok"}— even if Redis is completely unreachable (in which case ops just 500, the health flag never flips). The docstring's claim that assertingEGG_MESSAGE_STORE_BACKEND=redis"makes the health check below non-vacuous" is backwards: pinningredisis what makes this assertion a tautology. No harm (the round-trip test #3 carries the real coverage), but the test and its docstring overstate what they verify — consider trimming it or pointing it at the round-trip's backend assertion instead. -
_MAX_BLOCK_MSvssocket_timeoutare two independent hardcoded constants (redis_message_store.py:50= 4000;:643socket_timeout=5). The guard testtest_cap_stays_below_pool_socket_timeoutasserts_MAX_BLOCK_MS < 5000against a literal 5000, not the actual poolsocket_timeout. Lowersocket_timeoutto 3 and the test still passes while production regresses. Derive the bound from the real value (or makesocket_timeouta named constant the test references). Mitigated by the newTimeoutError→idle-slice degradation, which keeps a cap/timeout mismatch from 500ing, but the test gives false confidence as written. -
PR-description scope mismatch. The body says this PR "only changes deployment config, deps, tests, and docs" and "selection-precedence semantics are unchanged," but it also makes a substantive change to production blocking-read logic in
redis_message_store.py(the chunking +TimeoutErrorhandling). That's the change carrying BLOCKING 2 above; it deserves a "Changes" bullet so reviewers focus on it rather than treating the PR as config-only. -
uv.lockdowngradesredis7.3.0 → 6.4.0 — expected and an improvement (redis was previously an unconstrained transitive of thefakeredisdev-dep; it's now a pinned>=5.0,<7.0runtime dep, so dev and prod converge). Just calling it out since it's a behavior-affecting lockfile change, not noise.
Verdict: request changes. BLOCKING 1 means the feature does not function in the cluster as written — the orchestrator cannot connect to this Redis. Add --protected-mode no (or a password + client wiring), then run make test-integration so test #3 proves the path live. BLOCKING 2 should be fixed in the same pass since the PR is already in this code and the divergence from the in-memory backend is a genuine message-loss path.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Thanks for the thorough trace — both blocking issues were real. All items addressed in commit d23d242. 🔴 BLOCKING 1 — protected mode rejects the orchestrator connectionAgreed. 🔴 BLOCKING 2 — chunked blocking read re-resolves
|
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
No agent-mode design concerns.
The delta since my last review (commit d23d242) is confined to the Redis message-store transport and its k8s deployment: the from_tip concrete-tip snapshot fix, the _SOCKET_TIMEOUT_SEC-derived block cap, --protected-mode no on the Redis pod, and test/docstring corrections. None of these touch agent-mode design surface — no prompt pre-fetching, structured-output-for-humans, post-processing pipelines, or out-of-sandbox LLM calls. If anything, the from_tip fix strengthens the BRC consensus messaging path by closing a silent message-drop window, which is aligned with reliable multi-agent coordination.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Re-review: Deploy Redis in k3s + pin orchestrator to Redis message store (#2662)
Re-reviewed the delta 887b3ef..d23d242 (4 files). Both blocking issues from my prior review are properly fixed — not superficially patched. I traced the data flow end-to-end again and verified the security claim against the actual manifests, and ran the targeted unit tests.
🔴 BLOCKING 1 — protected mode → RESOLVED
--protected-mode no is added to the redis-server args. I verified the accompanying security justification rather than taking it on faith:
k8s/base/network-policies.yaml:egg-agentscarries bothdefault-deny-ingressanddefault-deny-egress(podSelector: {}). The only egress allow-rules are to the gateway (9848/3129), orchestrator (9849), and kube-dns (53) — no rule for Redis 6379. So untrusted agent pods structurally cannot reach the service.egg-systemhas no NetworkPolicy (confirmed by grep), so the orchestrator's password-less ClusterIP connection succeeds.
The no-auth posture is acceptable because the isolation is enforced at the network layer against the actual threat (agents), consistent with the existing egg-system service posture. AOF lives on the PVC in plaintext, but only trusted control-plane pods can reach it. ✓
🔴 BLOCKING 2 — $-per-slice drop window → RESOLVED
_resolve_tip_stream_id() snapshots the tip to a concrete id once (XREVRANGE … COUNT 1, "0-0" fallback) before the chunked loop. I traced both consumers of start_id:
- Fast path (
L443-451):start_idis the concrete tip and is never advanced (last_siddiscarded). Each idle slice re-blocksXREADfrom the same concrete id, so a mid-waitXADD(> tip) is caught on the next slice. ✓ - wait_for_types path (
L484-507):current_startadvances tolast_sidonly when rows were actually read (raw last id, set pre-filter — so filtered-out rows are correctly consumed, not re-read); on a pure-idle slicelast_sid is Noneandcurrent_startstays the concrete tip, re-scanning the gap. ✓
This restores parity with the in-memory backend's once-at-entry snapshot (message_store.py:383-403). The three new tests pin it (concrete id, empty→0-0, and never-$-to-XREAD via the production get_messages path). All 6 from_tip tests + the cap test pass locally. ✓
🟡 Non-blocking (no change required to merge)
-
Silent
RedisErrordegradation in_resolve_tip_stream_id(redis_message_store.py:535) —except redis.RedisError: return "0-0"swallows the error with no log. Two notes: (a) it's inconsistent with the analogoussince_idtransient degradation a few lines up (L314-319), which logs awarningbefore degrading to full history — this path should too, for debuggability; (b) on a non-empty from_tip stream, a transientxrevrangeblip (that doesn't also fail the subsequentXREAD) makes the call return all pre-existing history as if new, rather than only-new. That's the safe direction (at-least-once / re-deliver vs. drop) and the window is narrow, so not blocking — but alogger.warning(...)here would make the rare event visible. The docstring already reasons about the fallback; just emit a log alongside it. -
Stale test comment (
test_redis_message_store.py:852-855) —test_pre_existing_match_ignored_with_from_tipstill says "fakeredis's XREAD with$is a no-op on streams with data". Production no longer passes$; the test now passes becauseXREADfrom the concrete tip (exclusive) returns empty. The assertion is still correct and non-vacuous (regressing_resolve_tip_stream_idto always-0-0would surface the pre-existing match and fail it), but the comment's explanation is now inaccurate.
The prior non-blocking items (vacuous-health docstring, _SOCKET_TIMEOUT_SEC extraction with the test deriving from it, scope-mismatch acknowledgement, lockfile callout) are all handled as described.
Verdict: approve. Both blocking issues are correctly and durably fixed, the security reasoning checks out against the real network policies, and the new logic is sound across both read paths. The two items above are minor polish, not merge gates — assuming CI's live integration run (test #3, the orchestrator→Redis round-trip) is green, which the workflow gates this review on.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Thanks for the re-review and the approve. Both non-blocking polish items addressed in commit ea27a1a. 🟡 Non-blocking 1 — silent
|
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Re-review (agent-mode design): no concerns.
The delta since my last review (d23d242..ea27a1a, commit "Address review: log from_tip degradation, fix stale test comment") is confined to two non-blocking polish items in the Redis message-store transport:
orchestrator/redis_message_store.py:535-546—_resolve_tip_stream_idnow emits alogger.warning(...)before degrading to"0-0"on a transientRedisError, mirroring thesince_iddegradation path. Pure observability.orchestrator/tests/test_redis_message_store.py:852-856— stale test comment intest_pre_existing_match_ignored_with_from_tiprewritten to describe the concrete-tip resolution path. Comment-only.
Neither change touches agent-mode design surface — no prompt pre-fetching, no structured-output-for-humans, no post-processing pipelines, no out-of-sandbox LLM calls, no hardcoded model identifiers. As before, the underlying work strengthens the BRC consensus messaging path (reliable multi-agent coordination), which is aligned.
No agent-mode design concerns.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Re-review: delta d23d242..ea27a1a (2 files)
The delta is confined to the two non-blocking polish items from my prior review. Both are properly fixed, not superficially patched.
🟡 Non-blocking 1 (silent RedisError degradation) → RESOLVED
orchestrator/redis_message_store.py:535-546 — _resolve_tip_stream_id now emits logger.warning("from_tip tip resolution failed transiently; degrading to 0-0", pipeline_id=..., error=str(exc)) before return "0-0". This mirrors the since_id transient-degradation path at L314-319 exactly — same structlog kwargs shape (pipeline_id, error=str(exc)). The added comment correctly characterizes the non-empty-stream case as at-least-once re-delivery (the safe direction vs. dropping a message), bounded by the caller's deadline loop. The rare blip is now observable. ✓
🟡 Non-blocking 2 (stale test comment) → RESOLVED
orchestrator/tests/test_redis_message_store.py:852-856 — the inaccurate "fakeredis's XREAD with $ is a no-op on streams with data" comment is replaced with an accurate description: production resolves the tip to a concrete id via _resolve_tip_stream_id, so XREAD starts from that id exclusively and the pre-existing match is never delivered. The test still drives the real production path through store.get_messages(...) and remains non-vacuous — regressing the resolver to always-"0-0" would surface the pre-existing match (it matches wait_for_types) and fail assert messages == []. The comment now documents exactly that. ✓
No new issues and no regressions introduced. The earlier blocking fixes (protected-mode network isolation, _resolve_tip_stream_id concrete-tip snapshot) and the other non-blocking items from prior rounds remain in place and unaffected by this delta.
Verdict: approve. Both polish items correctly and durably addressed; the delta touches only observability and a test comment, with no behavioral change to the read paths. Assuming CI's live integration run (orchestrator→Redis round-trip) is green, which this review is gated on.
— Authored by egg
|
egg review completed. View run logs 13 previous review(s) hidden. |
Redis Streams (RedisMessageStore) is now the only message-store backend. Follow-up to #2662 / PR #3153, which pinned every deployment to explicit redis mode and made the in-memory backend reachable only by accident — re-introducing the #3076 mid-phase-restart message-loss risk. Production: - message_store.py shrinks to the shared types (Message, MessageType, GetMessagesMeta, HEARTBEAT_STATES, coerce_deprecated_message_type) and the singleton accessor. The ~370-line in-memory MessageStore (incl. the blocking-Condition machinery) is gone. - Backend selection collapses: EGG_MESSAGE_STORE_BACKEND unset or "redis" selects Redis; the removed multi-backend-era values ("memory"/"auto") and unknown values raise at creation, as does an unreachable Redis — no fallback. - The #3077 slice-6 fail-loud scaffolding that existed only because the auto→memory fallback was possible is removed: MEMORY_FALLBACK_MARKER, is_memory_fallback_degraded(), _reset_memory_fallback_state_for_test(), and the degraded components.message_store entry in /api/v1/health. Tests: - orchestrator/tests/conftest.py installs a session-scoped fakeredis-backed creator for get_message_store(), replacing the silent auto→memory fallback unit tests used to land on. - test_message_store.py is rewritten around the type surface and the redis-only fail-loud creation semantics; the in-memory behavioral suite is dropped where test_redis_message_store.py already pins the same contracts, and the plural from_roles / slice filter matrix — previously pinned only against the in-memory store — is ported to the redis unit file. - Direct MessageStore() constructions migrate to fakeredis-backed RedisMessageStore; MagicMock(spec=MessageStore) sweeps to spec=RedisMessageStore; the dual-backend parametrizations in test_pipelines_status_wait_route.py and the #2640 regression tier drop their in-memory arm. - integration_tests/test_message_store_backend.py now pins that /api/v1/health carries NO message_store component. Docs/manifests: concurrent-execution.md, coordination-state.md, STRUCTURE.md, and the k8s manifest comments describe the redis-only world; the explicit EGG_MESSAGE_STORE_BACKEND=redis pin stays as deployed documentation. Closes #3159
Fixes #2662.
Problem
The k3s manifests deployed no Redis and passed no
REDIS_HOST/EGG_MESSAGE_STORE_BACKENDto the orchestrator, soautobackend selection always fell back to the in-memoryMessageStore:RedisMessageStorehad zero live end-to-end coverage (unit tier only, via fakeredis).MESSAGE_STORE_AUTO_FALLBACK_TO_MEMORYmarker and/api/v1/healthreported a permanently degradedmessage_storecomponent.The gap had a third layer the issue didn't name: the
redisclient package was not installed in the orchestrator image at all (absent fromorchestrator/requirements.txtand pyproject runtime deps — onlyfakeredisexisted, in dev deps). Deploying Redis + env alone would have changed nothing: theimport redisinside_create_message_store()fails before the backend is ever probed.Changes
egg-redis image (
config/redis/Dockerfile): stock Redis pinned at8.6.4-alpine, built and published through the local loopback-registry supply chain like every other egg image — no public image references in the manifests. Wired intoEGG_ALL_IMAGES,make build, the deploy tag rewrite, the local overlayimageslist, andawait-egg-deploy.sh.Manifests (
k8s/base/redis-deployment.yaml,redis-service.yaml): single replica,Recreate(never deadlocks on the RWO volume), 1Gi PVC withappendonly yesso messages also survive Redis pod restarts,maxmemory 256mb+noevictionso memory pressure fails writes loudly instead of silently dropping stream entries, hardened securityContext matching the existing deployments,redis-cli pingprobes. Agent pods cannot reach it —egg-agentsis default-deny egress with no Redis allow rule.Orchestrator wiring:
EGG_MESSAGE_STORE_BACKEND=redis— explicit, notauto. The silent fallback is exactly what the slice-6 degraded flag exists to catch; in explicit mode a Redis outage raises, and since the store singleton is created lazily and a failed creation leaves it unset, the orchestrator naturally retries until Redis is reachable (no startup-ordering dependency). PlusREDIS_HOST/REDIS_PORT, andredis>=5.0,<7.0inorchestrator/requirements.txt+ pyproject runtime deps.End-to-end coverage (
integration_tests/test_message_store_backend.py), running against the clustermake test-integrationdeploys:EGG_MESSAGE_STORE_BACKEND=redis(makes the health check non-vacuous — explicit mode cannot silently fall back);/api/v1/healthcarries no fallback marker;_create_message_store()— exercising the env wiring, the in-image client, Service DNS, and the live Redis, and asserting the selected backend class isRedisMessageStore.Docs:
concurrent-execution.md's claim that production stores messages in Redis Streams is now true;coordination-state.md's wipe-semantics section notes the accidental-loss row can no longer occur in-cluster.Out of scope
Message-routing test coverage on top of the store (#2640 / #2661). Selection-precedence semantics are unchanged (frozen by #3077 HITL Q3) — this PR only changes deployment config, deps, tests, and docs.
Validation
make lintclean;kubectl kustomize k8s/overlays/local/renders with the image substitution and env in place.orchestrator/tests/test_redis_message_store.py+test_message_store.py: 123 passed.make redeploy/make test-integration.