fix(nats): multi-home nats on pmoves_external so published ports bind (fleet-bus unblock) - #1824
Conversation
…ally bind nats declares `ports:` (4222/9223) and the service comment says the 0.0.0.0 bind is "intentional to allow Tailscale mesh peers to connect" — but its only network, pmoves_bus, is `internal: true`. Docker installs NO published-port DNAT rules for a container on an internal-only network, so the bindings are recorded (docker inspect shows 0.0.0.0:4222) yet never plumbed: 4222 is connection-refused over the tailnet. This is almost certainly why NATS has been unreachable fleet-wide — in-stack service-to-service works over the internal bus, but no cross-node peer can reach the hub. Fix: multi-home nats onto the non-internal pmoves_external (declared external in core.yml, internal:false in base.yml) in BOTH the monolith and the core overlay, keeping pmoves_bus for isolated in-stack comms. With a routable interface present, Docker plumbs the published ports and tailnet peers can reach 4222/9223. Verified on kvm4-2: nats container healthy but `iptables -t nat -L DOCKER` had only the gateway-agent DNAT rule (on the normal bridge) and none for nats on pmoves_bus (172.30.3.x); 4222 connection-refused from a peer node. Follow-ups (separate): (1) VPS should set NATS_BIND to the node tailnet IP so 4222 is mesh-only, not public; (2) consider a dedicated non-internal published-bus network for nats/minio/etc. rather than reusing pmoves_external. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Docker Hardening ValidationHardening Validation ReportValidated: Tue Jun 16 16:14:40 UTC 2026Services CheckedPMOVES.AI Docker Hardening Validation[INFO] Checking: pmoves/docker-compose.hardened.yml [INFO] Validating: hi-rag-gateway-v2 [INFO] Validating: extract-worker [INFO] Validating: langextract [INFO] Validating: presign [INFO] Validating: render-webhook [INFO] Validating: retrieval-eval [INFO] Validating: pdf-ingest [INFO] Validating: jellyfin-bridge [INFO] Validating: invidious-companion-proxy [INFO] Validating: ffmpeg-whisper [INFO] Validating: media-video [INFO] Validating: media-audio [INFO] Validating: hi-rag-gateway-v2-gpu [INFO] Validating: hi-rag-gateway-gpu [INFO] Validating: deepresearch [INFO] Validating: supaserch [INFO] Validating: publisher-discord [INFO] Validating: mesh-agent [INFO] Validating: nats-echo-req [INFO] Validating: nats-echo-res [INFO] Validating: publisher [INFO] Validating: analysis-echo [INFO] Validating: graph-linker [INFO] Validating: comfy-watcher [INFO] Validating: grayjay-plugin-host [INFO] Validating: agent-zero [INFO] Validating: archon [INFO] Validating: channel-monitor [INFO] Validating: pmoves-yt [INFO] Validating: notebook-sync [INFO] Validating: supabase_service_role_key [INFO] Validating: supabase_jwt_secret ====================================== |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8fa76e08ab
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # 4222/9223 cross-node. Multi-home nats onto the non-internal pmoves_external so | ||
| # its published ports actually bind, while keeping pmoves_bus for in-stack comms. | ||
| - pmoves_bus | ||
| - pmoves_external |
There was a problem hiding this comment.
When this service is attached to pmoves_external, Docker starts plumbing both published ports above, including ${NATS_MONITORING_PORT:-9223}:8222. The NATS HTTP monitoring server enabled by -m 8222 is not protected by the --user/--pass client auth, so any host that uses the default NATS_BIND=0.0.0.0 now exposes /varz, /connz, /subsz, and JetStream metadata to the tailnet/public interface; the same issue is duplicated in the monolithic compose file. Please either stop publishing the monitoring port or bind it separately to localhost/a monitoring-only network while only making 4222 mesh-reachable.
Useful? React with 👍 / 👎.
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughTwo Docker Compose files ( ChangesNATS Cross-Node Network Access
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Poem
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
… check-gate) (#1826) Adds a manual deploy job that brings up / restarts the NATS event bus on the canonical fleet hub (kvm4-2) without interactive SSH — sidestepping the Tailscale check-mode re-auth gate (#1820) that now fronts all fleet SSH. - Targets the kvm4-2 runner via the node-unique `kvm4-2` label just added to it (both VPS runners share `kvm4`, so that label alone can't pin the node). - Operates on the durable /opt/PMOVES.AI checkout, reusing the env chain + pmoves_external network already provisioned there; NATS creds are static `:-` defaults so no secrets are needed. - Idempotently ensures the external-declared pmoves_external network exists, then runs `make overlay-up-bus` (the guard-preserving target from #1813) and verifies varz over the node tailnet IP (NATS_BIND is mesh-only, not localhost). - Uses only `run:` steps (no `uses:`) → exempt from the sha-pinning gate. Companion to #1813 (overlay-up-bus) and #1824 (multi-home nats on pmoves_external). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…lueprint (#1831) * docs(architecture): Tailscale-native fleet access + NATS hub design blueprint Maps Tailscale's official AI-infrastructure access pattern (tagged infra + deny-by-default grants + per-tenant tags + service sidecars) onto PMOVES, as a staged blueprint. Keystone = tag the fleet as infrastructure, which fixes the owner-SSH-check issue idiomatically (tagged nodes leave autogroup:self), ends server re-auth churn, and is the prerequisite for real multi-tenant isolation. Captures the 2026-06-16 decision: NATS is a DEDICATED HUB (separate concern), kept on its working host-publish + tailnet-bind path; net-new services (JuiceFS, inference) use Tailscale sidecars + tags (hybrid), which sidesteps the internal-network/DNAT bug class that #1824 worked around. pmoves_bus_pub (the earlier dedicated-published-network idea) is dropped in favor of the hub + sidecars. No live changes — execution is staged (SSH stopgap #1828 → tag fleet → acls→grants → sidecar JuiceFS/inference → onboard tenants). Companion to #1828 and the NATS restoration PRs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(architecture): correct §8 — gitops-apply exists (#1832), not console-only The "no gitops-apply / apply manually" caveat was wrong. PR #1832 wires tailscale/gitops-acl-action (test-on-PR diff, apply-on-merge); the policy can also be applied via the Tailscale API. Allowlist done; TS_TAILNET secret pending. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(architecture): define new role tags in tagOwners first + kernel networking for sidecars Codex P2 x2: - The rollout references tag:hub/tag:storage/tag:inference but those are not in tagOwners; Tailscale rejects an auth key advertising an undefined tag, so the staged re-enroll would block. Add a note to define them in tailscale-acl-policy.json tagOwners first. - The sidecar example left TS_USERSPACE at its Docker default (userspace), which only handles inbound port-forwards; outbound MagicDNS/100.x from the shared namespace needs kernel networking. Add TS_USERSPACE=false + /dev/net/tun. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ack) (#1873) The VPS gateway-agent deploy failed: network pmoves_external was found but has incorrect label com.docker.compose.network set to "" (expected: pmoves_external) Root cause: pmoves_external is shared by TWO compose stacks — the overlay/NATS stack (which multi-homes nats onto it, PR #1824) and the monolith. The overlay declares it external:true (so core.yml parses standalone) and provisions it via docker network create (UNLABELED). The monolith declared it compose-MANAGED, so when it ran it found an existing network without compose's ownership label and refused to adopt it. A compose-managed network can't be shared across stacks. Fix: declare pmoves_external external:true in docker-compose.yml to match core.yml — compose then ADOPTS the externally-created network without label-checking. Teach clean_networks.py to ENSURE the external network exists (create-if-missing, never remove it — external nets carry no label by design and removing it would break the stack holding it open), so the make-target up paths provision it on fresh nodes. CI deploy nodes already have it. base.yml left as-is (overridden by core.yml in overlay paths). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 participant
Root cause: nats's only network pmoves_bus is internal:true, so Docker installs no published-port DNAT rules and 4222 is connection-refused over the tailnet (recorded 0.0.0.0:4222 bindings never plumbed). This is why NATS has been unreachable fleet-wide — in-stack comms work over the internal bus but no cross-node peer reaches the hub. Fix: multi-home nats onto non-internal pmoves_external in both the monolith and core overlay, keeping pmoves_bus for in-stack comms. Evidence on kvm4-2: iptables nat DOCKER chain had only gateway-agent's rule, none for nats; 4222 refused from z890. Follow-ups: NATS_BIND to node tailnet IP (mesh-only, not public); dedicated published-bus network later. Generated with Claude Code.
Summary by CodeRabbit
Release Notes