feat(edge): add up-edge — Traefik + sso-auth had overlays but no make target - #2411
Conversation
… target docker-compose.traefik.yml and docker-compose.sso.yml have existed since #2221 with NO make target to start them, so the edge was never brought up through the pipeline. The only reference to the traefik overlay anywhere in the Makefile is a comment. Observed on z890 2026-08-05: - no pmoves-traefik container has ever been created on this node - so media.pmoves.ai and auth.pmoves.ai resolve to nothing, and every traefik.* label in the fleet is inert - pmoves-sso-auth is Up 2 days (healthy) and protecting nothing: sweeping all running containers, ZERO carry a traefik.http.routers.*.middlewares label, so the pmoves-forward-auth middleware defined in config/traefik/dynamic.yml is referenced by no router - services therefore fall back to their own auth. Open Notebook's RemoteUserAuthMiddleware is fail-closed and needs the Traefik-injected X-Forward-Auth-Secret header, so a direct :8503 hit gets the password prompt exactly as designed Adds up-edge / down-edge / edge-health following the existing EXTERNAL_DC pattern, so the edge starts through COMPOSE_ENV_FILES like everything else rather than by raw compose. Both overlays go up together on purpose: the auth.pmoves.ai router is declared on the traefik container but points at `sso-auth@docker`, so Traefik without sso-auth is an edge whose own login route 404s. up-edge preflights before starting, because Traefik publishes 80/443 directly and a bind failure otherwise scrolls past in compose output: - creates pmoves_external if absent (same as up-external) - refuses to start if 80 or 443 is already LISTENING - refuses to start if config/traefik/dynamic.yml is missing, since the forward-auth middleware would silently not load The port check tests the OUTPUT, not the exit code. A pipeline ending in `head` exits 0 on no matches; writing it the obvious way reports every port as occupied. That bug produced a false "80/443 are in use" reading while diagnosing this, which is why the guard is written the awkward way. edge-health checks each component separately rather than with one combined docker ps filter — a combined filter passes when only ONE of the two is up, which is precisely the misleading state this fixes (sso-auth healthy behind a Traefik that does not exist). It also prints which routers use forward-auth, and flags that an empty [] binding list means the publish silently no-oped (the same-subnet ghost-adapter pattern on Windows). Verified on z890: - all three targets resolve; merged compose config validates - pmoves_external is internal=false so it can publish; 80/443 confirmed free; host port binding demonstrably works (grafana bound on 0.0.0.0:3002) - the substrate probe does flag a same-subnet ghost adapter on this host, a latent risk for Docker port binds — not currently manifesting, hence the note in edge-health rather than a hard block - edge-health output before bring-up correctly reports traefik NOT RUNNING, sso-auth healthy, and no routers on forward-auth NOT in this PR: attaching pmoves-forward-auth to any router. That is a security behaviour change on protected compose files, needs a compose: Known Road, and needs a decision per service — notably whether Jellyfin uses gateway forward-auth or the in-app OIDC plugin, since running both double-prompts. Prerequisites the operator must confirm (secrets manifest is zero-access, correctly): CLOUDFLARE_DNS_API_TOKEN for the ACME DNS-01 resolver, and SSO_FORWARD_AUTH_SECRET without which header trust can never engage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 43 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe Makefile adds a dedicated edge Compose project for Traefik and ChangesEdge management
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Operator
participant Makefile
participant DockerCompose
participant Traefik
participant SSOAuth
Operator->>Makefile: run up-edge
Makefile->>Makefile: validate ports and dynamic configuration
Makefile->>DockerCompose: start edge Compose overlays
DockerCompose->>Traefik: start container
DockerCompose->>SSOAuth: start container
Operator->>Makefile: run edge-health
Makefile->>DockerCompose: inspect containers and ports
Makefile->>Traefik: inspect forward-auth router references
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2d2bc295e6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| .PHONY: up-edge | ||
| up-edge: ensure-env-shared ## Start the Traefik edge + sso-auth (media./auth.pmoves.ai, forward-auth) | ||
| @echo "→ Starting edge (Traefik + sso-auth)..." | ||
| @docker network inspect pmoves_external >/dev/null 2>&1 || docker network create --driver bridge pmoves_external |
There was a problem hiding this comment.
On a fresh edge host this only materializes pmoves_external, but the SSO overlay also declares pmoves_app and pmoves_api as external networks, so docker compose up will fail before starting Traefik/SSO unless the core stack already happened to create them. Docker’s Compose networking docs state that an external network must exist before docker compose up or Compose fails with a network-not-found error: https://docs.docker.com/compose/how-tos/networking/.
Useful? React with 👍 / 👎.
| busy="$$(netstat -ano 2>/dev/null | grep -E "^ *TCP +[0-9.]+:$$p +.*LISTENING" | head -2)"; \ | ||
| if [ -n "$$busy" ]; then echo "ERROR: port $$p already in use — Traefik cannot bind:"; echo "$$busy"; exit 1; fi; \ |
There was a problem hiding this comment.
When make up-edge is run a second time on an edge host where the existing pmoves-traefik container is already listening on 80/443, this preflight treats that expected listener as a conflict and exits before Compose can apply an idempotent up -d or pick up config/env changes. The port check should ignore the current Traefik container/listener or run through Compose for that case.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pmoves/Makefile`:
- Around line 4200-4248: Update the canonical operations documentation to
describe the operator interface introduced by the Makefile targets up-edge,
down-edge, and edge-health. Document required secrets, the pmoves_external
network prerequisite, ownership of ports 80 and 443, startup and health-check
steps, and rollback using down-edge; also add the validation commands actually
run and their output to the PR Testing section.
- Around line 4241-4243: Update the router-detection loop in edge-health to use
Docker formatted inspect output for Traefik router middleware labels instead of
grepping unrestricted docker inspect output. Identify labels matching
traefik.http.routers.<router>.middlewares that include pmoves-forward-auth, then
print each matching container together with the router’s middleware label entry
under the existing “Routers attached to pmoves-forward-auth” output.
- Around line 4208-4211: Update the port-check loop in the Makefile to use an
OS-aware listener probe that detects both Windows LISTENING and Linux/BSD LISTEN
states for ports 80 and 443. Select an available platform-appropriate command,
and fail explicitly when no supported probing tool is present before starting
Traefik.
- Line 4213: Update the deployment configuration invoked by the Makefile target
containing EDGE_DC up -d so the pmoves-forward-auth to pmoves-sso-auth
verification hop uses encrypted transport and an explicit authRequestHeaders
allowlist that excludes credential-bearing session cookies such as
pmoves_session; preserve only the headers required for verification, or
otherwise enforce equivalent network isolation.
- Around line 4201-4203: Update the up-edge target to ensure all external Docker
networks declared by docker-compose.sso.yml—pmoves_app, pmoves_external, and
pmoves_api—exist before starting the stack. Prefer adding the existing
ensure-networks target as a dependency, or create the missing pmoves_app and
pmoves_api networks alongside pmoves_external, while preserving the current
startup command.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 5ee46655-169a-4fed-b0c4-d61150d3c407
📒 Files selected for processing (1)
pmoves/Makefile
| .PHONY: up-edge | ||
| up-edge: ensure-env-shared ## Start the Traefik edge + sso-auth (media./auth.pmoves.ai, forward-auth) | ||
| @echo "→ Starting edge (Traefik + sso-auth)..." | ||
| @docker network inspect pmoves_external >/dev/null 2>&1 || docker network create --driver bridge pmoves_external | ||
| @# Traefik publishes 80/443 directly. Fail loudly here rather than let compose | ||
| @# emit a bind error that scrolls past — and check the OUTPUT, not the exit code: | ||
| @# a pipeline ending in `head` exits 0 on no matches and would report every port | ||
| @# as occupied. | ||
| @for p in 80 443; do \ | ||
| busy="$$(netstat -ano 2>/dev/null | grep -E "^ *TCP +[0-9.]+:$$p +.*LISTENING" | head -2)"; \ | ||
| if [ -n "$$busy" ]; then echo "ERROR: port $$p already in use — Traefik cannot bind:"; echo "$$busy"; exit 1; fi; \ | ||
| done | ||
| @test -f config/traefik/dynamic.yml || { echo "ERROR: config/traefik/dynamic.yml missing — forward-auth middleware would not load"; exit 1; } | ||
| @$(EDGE_DC) up -d | ||
| @echo "✔ Edge up — verify with 'make edge-health'" | ||
|
|
||
| .PHONY: down-edge | ||
| down-edge: ## Stop the Traefik edge + sso-auth | ||
| @$(EDGE_DC) down | ||
| @echo "✔ Edge down" | ||
|
|
||
| .PHONY: edge-health | ||
| edge-health: ## Show edge status: containers, published ports, and which routers use forward-auth | ||
| @# Check each component separately. A combined filter passes when only ONE of | ||
| @# the two is up, which is the exact state that looks fine and is not: sso-auth | ||
| @# healthy behind a Traefik that does not exist. | ||
| @echo "=== Edge containers ===" | ||
| @for svc in pmoves-traefik pmoves-sso-auth; do \ | ||
| st="$$(docker ps --filter name=$$svc --format '{{.Status}}' 2>/dev/null | head -1)"; \ | ||
| if [ -n "$$st" ]; then printf ' %-18s %s\n' "$$svc" "$$st"; \ | ||
| else printf ' %-18s NOT RUNNING\n' "$$svc"; fi; \ | ||
| done | ||
| @echo "" | ||
| @echo "=== Traefik published ports ===" | ||
| @p="$$(docker inspect pmoves-traefik --format '{{json .NetworkSettings.Ports}}' 2>/dev/null)"; \ | ||
| if [ -n "$$p" ]; then echo " $$p"; else echo " (no traefik container)"; fi | ||
| @echo " NOTE: an EMPTY [] binding list means the publish silently no-oped." | ||
| @echo " On Windows that is the same-subnet ghost-adapter pattern -" | ||
| @echo " see docs/operations/SAME_SUBNET_GHOST_PATTERN.md" | ||
| @echo "" | ||
| @echo "=== Routers attached to pmoves-forward-auth ===" | ||
| @found=0; for c in $$(docker ps --format '{{.Names}}' 2>/dev/null); do \ | ||
| m="$$(docker inspect $$c 2>/dev/null | grep -o 'pmoves-forward-auth' | head -1)"; \ | ||
| if [ -n "$$m" ]; then echo " $$c"; found=1; fi; \ | ||
| done; \ | ||
| if [ "$$found" != 1 ]; then \ | ||
| echo " NONE - the middleware is defined in config/traefik/dynamic.yml but"; \ | ||
| echo " no router references it, so sso-auth protects nothing yet."; \ | ||
| fi |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Document the new operator interface.
Document up-edge, down-edge, and edge-health in the canonical operations documentation. Include required secrets, network prerequisites, port ownership, run steps, and rollback steps. Add the executed validation commands and output to the PR Testing section.
As per coding guidelines, “Update documentation and schemas when interfaces change” and “Keep changes atomic and include relevant testing evidence in pull requests.”
🧰 Tools
🪛 checkmake (0.3.2)
[warning] 4201-4201: Target body for "up-edge" exceeds allowed length of 5 lines (13).
(maxbodylength)
[warning] 4222-4222: Target body for "edge-health" exceeds allowed length of 5 lines (26).
(maxbodylength)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pmoves/Makefile` around lines 4200 - 4248, Update the canonical operations
documentation to describe the operator interface introduced by the Makefile
targets up-edge, down-edge, and edge-health. Document required secrets, the
pmoves_external network prerequisite, ownership of ports 80 and 443, startup and
health-check steps, and rollback using down-edge; also add the validation
commands actually run and their output to the PR Testing section.
Source: Coding guidelines
| if [ -n "$$busy" ]; then echo "ERROR: port $$p already in use — Traefik cannot bind:"; echo "$$busy"; exit 1; fi; \ | ||
| done | ||
| @test -f config/traefik/dynamic.yml || { echo "ERROR: config/traefik/dynamic.yml missing — forward-auth middleware would not load"; exit 1; } | ||
| @$(EDGE_DC) up -d |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C4 'forwardAuth:|address:|authRequestHeaders' pmoves/config/traefik/dynamic.yml
rg -n -C4 'Cookie|Authorization|auth/verify|SSO_' pmoves/services/sso-auth
rg -n -C3 'pmoves_external' pmoves/docker-compose.traefik.yml pmoves/docker-compose.sso.ymlRepository: POWERFULMOVES/PMOVES.AI
Length of output: 16038
🌐 Web query:
Traefik HTTP forwardAuth authRequestHeaders documentation
💡 Result:
In Traefik, the authRequestHeaders option is a configuration setting for the ForwardAuth middleware [1][2]. It defines a list of headers from the original incoming request that should be forwarded to the external authentication server [1][2]. Key details regarding authRequestHeaders: - Purpose: It allows you to filter which headers are passed to the authentication server [1][2]. This is useful for security or privacy, ensuring that sensitive information is not sent to the authentication service unnecessarily. - Default Behavior: If the authRequestHeaders option is not set or is left empty, Traefik will forward all headers from the original request to the authentication server by default [1][2]. - Configuration: You provide a list of header names that should be explicitly copied from the incoming request to the request made to the auth service [1][2]. In addition to the headers you specify via authRequestHeaders, Traefik automatically provides several X-Forwarded- headers to the authentication server, including X-Forwarded-Method, X-Forwarded-Proto, X-Forwarded-Host, X-Forwarded-Uri, and X-Forwarded-For [1].
Citations:
- 1: https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/forwardauth/
- 2: https://doc.traefik.io/traefik/v3.4/middlewares/http/forwardauth/
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '--- composer files mentioning pmoves_external ---\n'
rg -n -C3 'networks:|pmoves_external|entrypoints=websecure|pmoves-forward-auth|pmoves-sso-auth' pmoves/docker-compose*.yml pmoves/config pmoves/services/sso-auth/app.py pmoves/services/sso-auth/config.py
printf '\n--- exact forwardAuth block ---\n'
sed -n '1,35p' pmoves/config/traefik/dynamic.yml
printf '\n--- app verify implementation ---\n'
sed -n '1,80p' pmoves/services/sso-auth/app.py
printf '\n--- compose target around line 4213 ---\n'
sed -n '4205,4218p' pmoves/MakefileRepository: POWERFULMOVES/PMOVES.AI
Length of output: 252
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- composer files mentioning pmoves_external ---'
rg -n -C3 'networks:|pmoves_external|entrypoints=websecure|pmoves-forward-auth|pmoves-sso-auth' pmoves/docker-compose*.yml pmoves/config pmoves/services/sso-auth/app.py pmoves/services/sso-auth/config.py || true
printf '%s\n' ''
printf '%s\n' '--- exact forwardAuth block ---'
sed -n '1,35p' pmoves/config/traefik/dynamic.yml
printf '%s\n' ''
printf '%s\n' '--- app verify implementation ---'
sed -n '1,80p' pmoves/services/sso-auth/app.py
printf '%s\n' ''
printf '%s\n' '--- compose target around line 4213 ---'
sed -n '4205,4218p' pmoves/MakefileRepository: POWERFULMOVES/PMOVES.AI
Length of output: 50380
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- restricted network declarations ---'
rg -n 'networks:[[:space:]]*$|^ $|external: true|internal: true|name:' pmoves/docker-compose.base.yml pmoves/docker-compose.sso.yml pmoves/docker-compose.traefik.yml pmoves/docker-compose.external.yml | head -200
printf '%s\n' ''
printf '%s\n' '--- exact app verify implementation ---'
sed -n '1,95p' pmoves/services/sso-auth/app.py
printf '%s\n' ''
printf '%s\n' '--- target around line 4213 ---'
sed -n '4205,4218p' pmoves/MakefileRepository: POWERFULMOVES/PMOVES.AI
Length of output: 7769
Sensitive Data Exposure (CWE-319): Cleartext Transmission of Sensitive Information
Reachability: External
Encrypt the ForwardAuth hop to the SSO verifier.
pmoves-forward-auth calls http://pmoves-sso-auth:8080/auth/verify, and Traefik forwards all incoming headers when authRequestHeaders is unset. /auth/verify reads pmoves_session, while pmoves_external has no internal: true or host-port isolation, so authenticated session cookies can cross an unencrypted network. Add an explicit allowlist and transport encryption for this hop, or prove no credential-bearing request headers are exposed.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pmoves/Makefile` at line 4213, Update the deployment configuration invoked by
the Makefile target containing EDGE_DC up -d so the pmoves-forward-auth to
pmoves-sso-auth verification hop uses encrypted transport and an explicit
authRequestHeaders allowlist that excludes credential-bearing session cookies
such as pmoves_session; preserve only the headers required for verification, or
otherwise enforce equivalent network isolation.
…ent rerun P2 #1 (Makefile:4203) — up-edge only created pmoves_external, but the SSO overlay also declares pmoves_app and pmoves_api as external, so compose fails with network-not-found before Traefik ever starts on a host where the core stack has not run. Correct. Implemented, but NOT by creating them. Both are `internal: true` in the core stack (docker-compose.yml:5364,5373 — confirmed at runtime, internal=true). A plain `docker network create --driver bridge` would materialize them NON-internal, and the core stack would later attach to a network that no longer blocks egress. That turns a loud missing-network error into a quiet security regression, which is worse than the bug. up-edge now requires them to pre-exist and points at the core stack instead. pmoves_external is still created here — it is ours, non-internal by design, and up-external already does the same. P2 #2 (Makefile:4210) — the port preflight treated our own running Traefik as a conflict, so a second `make up-edge` to pick up config/env changes was blocked by its own listener. The target was effectively one-shot. Correct. Now skipped entirely when a pmoves-traefik container is already running; compose handles the recreate and port handoff. First bring-up (no traefik) still gets the full check. Verified: all three targets resolve; the missing-network branch fires on an absent network; with no traefik running the preflight still executes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Both P2s were correct and are fixed in P2 #1 — missing external networks. Confirmed: the SSO overlay declares Fixed, but not by creating them. Both are P2 #2 — non-idempotent rerun. Confirmed and a real one-shot bug: the preflight treated our own Traefik as a conflict, so a rerun to pick up config changes was blocked by its own listener. Now skipped when a Verified: all three targets resolve; the missing-network branch fires on an absent network; with no Traefik running the preflight still executes. |
…er, runbook Three findings addressed, one deferred with reasoning. FIXED — OS-aware port probe (Minor, but a real bug I introduced). The preflight matched `LISTENING`, which is Windows netstat's wording; Linux and BSD print `LISTEN`. So on the Linux hosts where the edge actually runs, the probe matched nothing, treated 80/443 as free, and protected nothing — the failure it exists to prevent. Now prefers `ss`, falls back to `netstat` matching BOTH spellings, and warns-and-continues if neither tool exists. Deviation from the review, which asked to fail when no probe is found: a missing probe is not evidence of a conflict, and hard-failing would block a legitimate bring-up on a minimal host. Compose still surfaces a bind error there. FIXED — router-label reader in edge-health (Minor). It grepped the whole `docker inspect` blob for the literal string, so any env var or comment containing "pmoves-forward-auth" counted, and it printed only the container name — useless on a container serving several routers. Now parses `traefik.http.routers.<name>.middlewares` labels and prints container + router + middleware chain. Positive-controlled against the same pipeline matching `.rule`, which correctly extracts `router=media` from pmoves-jellyfin. FIXED — operator documentation (Major). New docs/operations/EDGE_TRAEFIK_SSO_RUNBOOK.md: targets, required secrets and what breaks without each, network prerequisites and why app/api are not auto-created, port ownership and idempotent-rerun behaviour, run steps, rollback (including the acme volume and the rate-limit reason not to delete it), troubleshooting table, and an explicit Known Gaps section — stale containers missing labels, services with no edge presence, the double-prompt rule, why Jellyfin is deliberately excluded, and sso-auth's single-tenant limit. Also splits the preflight into its own `edge-preflight` target. It was flagged by checkmake for body length, and a prerequisite check that can be run without starting anything is more useful standalone. DEFERRED — CWE-319 on the ForwardAuth hop (Major). The finding is fair: the hop is `http://` and `authRequestHeaders` is unset, so Traefik forwards every incoming header to the verifier. Two reasons not to change it in this PR: 1. It is pre-existing configuration in config/traefik/dynamic.yml from #2221, not introduced here. This PR adds a make target; fixing middleware security in it couples an unrelated behaviour change to the bring-up. 2. Getting the allowlist wrong breaks login, and the happy path has never run even once — Traefik has never started on this fleet. Tightening auth transport before we can observe a working login means any breakage is indistinguishable from the many other things not yet wired. Right sequence is: land the edge, prove one login end to end, then tighten the hop with a working baseline to diff against. Recorded in the runbook's Known Gaps. Verified: all four targets resolve; edge-preflight passes on z890 (networks present, 80/443 free, dynamic.yml present); edge-health correctly reports no protected routers and now explains that external.yml does set them but the running containers predate it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Review addressed. Three fixed, one deferred with reasoning. ✅ OS-aware port probe (Minor — but a real bug I introduced). One deliberate deviation: the review asked to fail when no probe is found. I warn and continue instead — a missing probe isn't evidence of a conflict, and hard-failing would block a legitimate bring-up on a minimal host. Compose still surfaces a bind error there. ✅ Router-label reader in ✅ Operator documentation (Major). New Also split the preflight into a standalone ⏸️ Deferred — CWE-319 on the ForwardAuth hop (Major). The finding is fair: the hop is
Right sequence: land the edge → prove one login end-to-end → then tighten the hop against a working baseline. Recorded in the runbook's Known Gaps so it isn't lost. Verified: all four targets resolve; |
…no labels (#2453) Two independent faults. Either alone makes the edge inert; together they presented as one symptom, which is why the first look misread it. 1. `up-edge` could never run alongside the stack it fronts. EDGE_DC used `-p $(PROJECT)-edge`, a SEPARATE compose project, but `sso-auth` is already a service of the main `pmoves` project (its config_files label reads docker-compose.yml,docker-compose.sso.yml). So `make up-edge` on a live node dies with: Conflict. The container name "/pmoves-sso-auth" is already in use The name clash is the visible half. The dangerous half is what the edge project had already created before hitting it: `pmoves-edge_sso-oidc-key`, a SHADOW of the real `pmoves_sso-oidc-key`. Had the run completed, sso-auth would have come back holding a freshly generated OIDC signing key and invalidated every issued token. Same for traefik-acme, which would have re-requested certificates against a fresh store. Fixed by joining the main project: `-p $(PROJECT)`. Traefik and sso-auth are then ordinary members of the stack, sharing its volumes. Compose prints an orphan-containers notice because only two of the overlays are passed; that is informational and no service is touched. 2. Traefik v3.3's Docker provider cannot talk to Docker Engine 29. Once it started, Traefik logged this once a minute, forever: ERR Failed to retrieve information of the docker client and server host error="Error response from daemon: " providerName=docker Empty message, no other signal, and EVERY route 404s — including auth.pmoves.ai, which is declared on the Traefik container itself, so the 404 meant no label was being read at all. The socket was never the problem: mounted, visible, root-owned, and another container reads /version and /info through the identical `:ro` mount with a 200. The daemon is Engine 29.6.2 with MinAPIVersion 1.40; Traefik v3.3 pins Docker API v1.24. Probed directly: /v1.24/version -> 400 {"Version":"","ApiVersion":"","Os":"","Arch":""} /v1.40/version -> 200 Version 29.6.2 /v1.44/version -> 200 The 400 has an empty body, which is precisely how it reaches the log with nothing after the colon. `DOCKER_API_VERSION=1.44` does NOT fix it — measured, 7 provider errors in 12s; Traefik pins its client version and ignores the env var. Only a bump works. Measured, identical flags and socket, 10s each: traefik:v3.4 -> 7 provider errors traefik:v3.5 -> 6 traefik:v3.6 -> 0 So v3.6, digest-pinned, with a comment marking it a FLOOR rather than a preference so nobody downgrades it back into silence. Why it stayed hidden: the edge had no make target until #2411, so Traefik had never been started on this node. Every traefik.* label in the fleet has been inert since it was written. `edge-health` reported "no router uses forward-auth", which was read as "the app containers predate the labels" — true, and still true — but it masked the fact that Traefik could not have seen any label regardless. Verified on z890 after both fixes: docker logs pmoves-traefik | grep -c "Provider error" -> 0 curl -k --resolve auth.pmoves.ai:443:127.0.0.1 \ https://auth.pmoves.ai/login -> 200, PMOVES Sign in curl -k --resolve media.pmoves.ai:443:127.0.0.1 \ https://media.pmoves.ai/ -> 302 (Jellyfin) ports 0.0.0.0:80 / 0.0.0.0:443, no empty-[] ghost-adapter bind pmoves_sso-oidc-key reused; no pmoves-edge_* volume remains Note the --resolve: a plain `-H 'Host: ...'` against https://localhost sends SNI=localhost and 404s even when the router is correct. That cost a detour. Full brief: pmoves/docs/handoffs/traefik-docker-provider-api-version-2026-08-06.md Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 participant
The gap
docker-compose.traefik.ymlanddocker-compose.sso.ymlhave existed since #2221 with no make target to start them. The only reference to the traefik overlay anywhere in the Makefile is a comment. So the edge was never brought up through the pipeline.Observed on z890:
pmoves-traefikcontainer has ever been created on this node.media.pmoves.aiandauth.pmoves.airesolve to nothing, and everytraefik.*label in the fleet is inert.pmoves-sso-authisUp 2 days (healthy)and protecting nothing. Sweeping every running container, zero carry atraefik.http.routers.*.middlewareslabel — so thepmoves-forward-authmiddleware defined inconfig/traefik/dynamic.ymlis referenced by no router.RemoteUserAuthMiddlewareis fail-closed and requires the Traefik-injectedX-Forward-Auth-Secretheader, so a direct:8503hit gets the password prompt exactly as designed.That last point is worth stating plainly: the password prompt people keep hitting is not a bug in the app. It's the absence of an edge.
What lands
up-edge/down-edge/edge-health, following the existingEXTERNAL_DCpattern so the edge starts throughCOMPOSE_ENV_FILESlike everything else rather than by raw compose.Both overlays go up together on purpose: the
auth.pmoves.airouter is declared on the traefik container but points atsso-auth@docker, so Traefik without sso-auth is an edge whose own login route 404s.up-edgepreflights before starting, because Traefik publishes 80/443 directly and a bind failure otherwise just scrolls past in compose output:pmoves_externalif absent (same asup-external)LISTENINGconfig/traefik/dynamic.ymlis missing — the forward-auth middleware would silently not loadThe port check tests the output, not the exit code. A pipeline ending in
headexits 0 on no matches, so the obvious formulation reports every port as occupied. That bug produced a false "80/443 are in use" reading while diagnosing this, which is why the guard is written the awkward way.edge-healthchecks each component separately rather than with one combineddocker psfilter — a combined filter passes when only one of the two is up, which is precisely the misleading state this fixes. Current output, pre-bring-up:Verified on z890
pmoves_externalisinternal=falseso it can publish (unlikepmoves_data/app/api/bus, allinternal=true— which is the real reason so many containers show empty port maps, and is not a bind failure)0.0.0.0:3002edge-healthreports the current broken state accuratelyCaveat surfaced by the substrate probe: z890 has a same-subnet ghost adapter (two active NICs on one /24). That's the documented cause of Docker Desktop silently failing port binds — see
SAME_SUBNET_GHOST_PATTERN.md. It is not currently manifesting (grafana binds fine), soedge-healthnotes it rather than hard-blocking. A plainnetstatcheck would never have surfaced this.Not in this PR
Attaching
pmoves-forward-authto any router. That's a security-behaviour change on protected compose files, needs acompose:Known Road, and needs a per-service decision — notably whether Jellyfin uses gateway forward-auth or the in-app OIDC plugin from #2407, since running both double-prompts.Operator prerequisites
The secrets manifest is zero-access (correctly), so I could not verify these — please confirm before
up-edge:CLOUDFLARE_DNS_API_TOKEN— the ACME DNS-01 resolver (certresolver=cf) needs it or cert issuance failsSSO_FORWARD_AUTH_SECRET— without it, header trust can never engage and forward-auth stays inert even once wiredRelated: #2407 (Jellyfin in-app OIDC), #2221/#2229 (the gateway this finally starts).
🤖 Generated with Claude Code
Summary by CodeRabbit