From 8fa76e08ab1bb7a34a4b80521eafd93f3693835f Mon Sep 17 00:00:00 2001 From: POWERFULMOVES Date: Tue, 16 Jun 2026 12:09:53 -0400 Subject: [PATCH] fix(nats): multi-home nats on pmoves_external so published ports actually bind MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit nats declares `ports:` (4222/9223) and the service comment says the 0.0.0.0 bind is "intentional to allow Tailscale mesh peers to connect" — but its only network, pmoves_bus, is `internal: true`. Docker installs NO published-port DNAT rules for a container on an internal-only network, so the bindings are recorded (docker inspect shows 0.0.0.0:4222) yet never plumbed: 4222 is connection-refused over the tailnet. This is almost certainly why NATS has been unreachable fleet-wide — in-stack service-to-service works over the internal bus, but no cross-node peer can reach the hub. Fix: multi-home nats onto the non-internal pmoves_external (declared external in core.yml, internal:false in base.yml) in BOTH the monolith and the core overlay, keeping pmoves_bus for isolated in-stack comms. With a routable interface present, Docker plumbs the published ports and tailnet peers can reach 4222/9223. Verified on kvm4-2: nats container healthy but `iptables -t nat -L DOCKER` had only the gateway-agent DNAT rule (on the normal bridge) and none for nats on pmoves_bus (172.30.3.x); 4222 connection-refused from a peer node. Follow-ups (separate): (1) VPS should set NATS_BIND to the node tailnet IP so 4222 is mesh-only, not public; (2) consider a dedicated non-internal published-bus network for nats/minio/etc. rather than reusing pmoves_external. Co-Authored-By: Claude Opus 4.8 --- pmoves/docker-compose.core.yml | 6 ++++++ pmoves/docker-compose.yml | 6 ++++++ 2 files changed, 12 insertions(+) diff --git a/pmoves/docker-compose.core.yml b/pmoves/docker-compose.core.yml index 8fab2d02d0..a53078e7f9 100644 --- a/pmoves/docker-compose.core.yml +++ b/pmoves/docker-compose.core.yml @@ -41,7 +41,13 @@ services: - ${NATS_BIND:-0.0.0.0}:${NATS_PORT:-4222}:4222 - ${NATS_BIND:-0.0.0.0}:${NATS_MONITORING_PORT:-9223}:8222 networks: + # pmoves_bus is internal:true (isolated in-stack mesh bus). A container on an + # internal-only network gets NO published-port DNAT rules from Docker, so the + # ports above are recorded but never plumbed — Tailscale mesh peers can't reach + # 4222/9223 cross-node. Multi-home nats onto the non-internal pmoves_external so + # its published ports actually bind, while keeping pmoves_bus for in-stack comms. - pmoves_bus + - pmoves_external environment: - DOCKED_MODE=${DOCKED_MODE:-true} - TOPOLOGY_MODE=${TOPOLOGY_MODE:-docked} diff --git a/pmoves/docker-compose.yml b/pmoves/docker-compose.yml index 7ddc5944ee..4258a75e3b 100644 --- a/pmoves/docker-compose.yml +++ b/pmoves/docker-compose.yml @@ -2361,7 +2361,13 @@ services: - ${NATS_BIND:-0.0.0.0}:${NATS_PORT:-4222}:4222 - ${NATS_BIND:-0.0.0.0}:${NATS_MONITORING_PORT:-9223}:8222 networks: + # pmoves_bus is internal:true (isolated in-stack mesh bus). A container on an + # internal-only network gets NO published-port DNAT rules from Docker, so the + # ports above are recorded but never plumbed — Tailscale mesh peers can't reach + # 4222/9223 cross-node. Multi-home nats onto the non-internal pmoves_external so + # its published ports actually bind, while keeping pmoves_bus for in-stack comms. - pmoves_bus + - pmoves_external environment: - DOCKED_MODE=${DOCKED_MODE:-true} - TOPOLOGY_MODE=${TOPOLOGY_MODE:-docked}