From 9080b8e9e1978774ecaf69d7262f11dbe63db34f Mon Sep 17 00:00:00 2001 From: Sakib Sadman Shajib Date: Tue, 18 Aug 2026 17:26:20 -0400 Subject: [PATCH 1/9] feat: make the self-hosted supabase profile bootable and able to authenticate The enterprise profile had never been run end to end. Standing it up from scratch surfaced five separate reasons it could not have worked, each fixed here, plus the token-validation gap that would have rejected every request. GoTrue was configured with a symmetric secret only. Its JWKS endpoint excludes symmetric keys by design, so it published an empty key set, and edge-api validates with jwt.WithKeySet against exactly that endpoint. Every token would have failed, and the initial refresh would have stopped edge-api from booting at all. GOTRUE_JWT_KEYS now carries an EC P-256 signing key, generated by scripts/generate-enterprise-jwt-keys.py, and PostgREST and Storage take the matching verification set so the existing HS256 anon and service_role keys keep working alongside the new ES256 user tokens. edge-api refuses a plain http JWKS URL, which is the right call and is not relaxed here. The new caddy-supabase service terminates real TLS with Caddy's own local certificate authority, and edge-api trusts that one authority through the new SUPABASE_JWKS_CA_FILE. Chain and hostname verification still apply; a named CA file that is missing or holds no certificate is fatal rather than a silent downgrade to the system pool. That same service replaces Kong. It fronts GoTrue, PostgREST and Storage on one origin with the hosted /auth/v1, /rest/v1 and /storage/v1 prefixes, so a single SUPABASE_URL stays correct for control-plane, the Supabase client libraries and the seeding scripts. The GoTrue pin moves from v2.170.0 to v2.189.0. The old tag predates the OAuth 2.1 authorization server that Open WebUI signs in through, so self-hosting on it would have silently lost chat login. Three bring-up bugs, none of which could have been seen without running it: PostgREST's healthcheck could never pass because that image has no shell, and Storage waited on it forever; Storage's own healthcheck probed localhost, which resolves to IPv6 first while the server binds IPv4; and bucket creation failed because the storage tables are created by the Storage API after the init script runs, so no grant existed for them. Verified against a stack brought up from scratch: all services healthy, both buckets created, all 88 migrations applied, and a password-grant token from the self-hosted GoTrue accepted by edge-api on a real request (HTTP 200 from /v1/models), with a tampered token rejected 401. Co-Authored-By: Claude Opus 5 --- .env.example | 58 +++- Makefile | 1 + apps/edge-api/cmd/server/jwt_env_test.go | 56 ++++ apps/edge-api/cmd/server/main.go | 13 +- apps/edge-api/internal/auth/jwt_supabase.go | 57 +++- .../internal/auth/jwt_supabase_ca_test.go | 118 ++++++++ deploy/docker/Caddyfile.supabase | 67 +++++ deploy/docker/docker-compose.enterprise.yml | 158 +++++++++-- deploy/supabase/init/00-extensions.sql | 34 +++ scripts/generate-enterprise-jwt-keys.py | 263 ++++++++++++++++++ 10 files changed, 794 insertions(+), 31 deletions(-) create mode 100644 apps/edge-api/cmd/server/jwt_env_test.go create mode 100644 apps/edge-api/internal/auth/jwt_supabase_ca_test.go create mode 100644 deploy/docker/Caddyfile.supabase create mode 100755 scripts/generate-enterprise-jwt-keys.py diff --git a/.env.example b/.env.example index cd3e25524..cf535c9b1 100644 --- a/.env.example +++ b/.env.example @@ -592,6 +592,22 @@ ENTERPRISE_JWT_SECRET= ENTERPRISE_JWT_ISSUER=http://supabase-auth:9999 ENTERPRISE_JWT_EXP=3600 +# Asymmetric signing material. NOT optional: with the symmetric secret alone, +# GoTrue signs HS256 and publishes an EMPTY JWKS (it excludes symmetric keys +# from that endpoint by design), and edge-api validates against the JWKS, so +# every token is rejected and the gateway does not come up. +# +# Generate both lines at once, using the secret above: +# ENTERPRISE_JWT_SECRET="" python3 scripts/generate-enterprise-jwt-keys.py +# +# ENTERPRISE_JWT_KEYS is the private signing set and goes to GoTrue only. +# ENTERPRISE_JWT_JWKS is the verification set (EC public key plus the legacy +# symmetric key) and goes to PostgREST and Storage, so both keep accepting the +# existing anon and service_role keys while gaining the new ES256 user tokens. +# Both are secret material. Never commit either. +ENTERPRISE_JWT_KEYS= +ENTERPRISE_JWT_JWKS= + # Supabase API keys derived from ENTERPRISE_JWT_SECRET (see helper above). ENTERPRISE_ANON_KEY= ENTERPRISE_SERVICE_ROLE_KEY= @@ -666,25 +682,45 @@ ENTERPRISE_SSO_MICROSOFT_TENANT_URL=https://login.microsoftonline.com/ # SUPABASE_SERVICE_ROLE_KEY= # SUPABASE_DB_URL=postgres://postgres:@supabase-db:5432/postgres # SUPABASE_JWT_ISSUER=http://supabase-auth:9999 # -# JWKS NOTE: edge-api/cmd/server/main.go rejects http:// JWKS URLs as insecure. -# For a production enterprise box, terminate TLS in Caddy and set: -# SUPABASE_JWKS_URL=https:///auth/v1/.well-known/jwks.json -# For a LAN-only or air-gapped dev box where TLS is not available, set -# SUPABASE_JWKS_URL to the internal http URL only after confirming the edge-api -# HTTPS guard is relaxed in config or the service is behind an internal TLS proxy. -# SUPABASE_JWKS_URL=http://supabase-auth:9999/.well-known/jwks.json +# JWKS NOTE: edge-api/cmd/server/main.go rejects http:// JWKS URLs as insecure, +# and that guard is not relaxed for self-hosting. An attacker on the path +# between edge-api and an http JWKS can swap the key set and mint tokens this +# gateway would accept, so there is no LAN-only or air-gapped exception. +# +# The enterprise profile answers this with the in-stack caddy-supabase gateway, +# which terminates TLS using Caddy's local certificate authority. Both +# variables below already default to that, so nothing needs to be set here for +# the enterprise profile to work: +# SUPABASE_JWKS_URL=https://caddy-supabase/auth/v1/.well-known/jwks.json +# SUPABASE_JWKS_CA_FILE=/etc/hive/supabase-ca/caddy/pki/authorities/local/root.crt +# SUPABASE_JWKS_CA_FILE names an EXTRA authority to trust for that one fetch. +# It does not disable verification: the chain and the hostname are still +# checked, and an unreadable or certificate-free file is a boot failure. +# +# Standalone GoTrue serves its routes at the root, with no /auth/v1 prefix. +# The gateway restores the hosted prefixes, so the URL above keeps the shape +# every consumer already expects. +# +# SUPABASE_DOMAIN is the browser-facing hostname for the gateway, served over +# plain HTTP for an external TLS terminator (Cloudflare Tunnel on the demo +# box) to front. Only needed once browsers talk to the self-hosted stack. +# SUPABASE_DOMAIN=supabase-hive.example.com # # Storage: Supabase Storage API with local filesystem backend (no external S3). # The S3-compatible endpoint is at /storage/v1/s3 (matches hosted Supabase path). -# S3_ENDPOINT=http://supabase-storage:5000/storage/v1/s3 +# S3_ENDPOINT=http://caddy-supabase/storage/v1/s3 # S3_ACCESS_KEY= # S3_SECRET_KEY= # S3_REGION=local diff --git a/Makefile b/Makefile index 0aeb74f1b..f824c4bba 100644 --- a/Makefile +++ b/Makefile @@ -28,3 +28,4 @@ test-scripts: python3 scripts/test_owui_ui_surfaces.py python3 scripts/test_caddy_owui_blocklist.py python3 scripts/test_owui_model_picker_filter.py + python3 scripts/generate-enterprise-jwt-keys.py --self-check diff --git a/apps/edge-api/cmd/server/jwt_env_test.go b/apps/edge-api/cmd/server/jwt_env_test.go new file mode 100644 index 000000000..17d53db04 --- /dev/null +++ b/apps/edge-api/cmd/server/jwt_env_test.go @@ -0,0 +1,56 @@ +package main + +import ( + "testing" +) + +// The https-only rule on SUPABASE_JWKS_URL is the reason the self-hosted +// profile needs a TLS front for GoTrue at all. It is load bearing: an http +// JWKS URL lets anything on the path swap the key set and mint tokens this +// service would accept. These tests exist so a future "just make enterprise +// work" change cannot quietly relax it. + +func TestLoadJWTAuthEnv_RejectsPlainHTTPJWKS(t *testing.T) { + t.Setenv("SUPABASE_JWT_ISSUER", "http://supabase-auth:9999") + t.Setenv("SUPABASE_JWT_AUDIENCE", "authenticated") + t.Setenv("SUPABASE_JWKS_URL", "http://supabase-auth:9999/.well-known/jwks.json") + + if _, err := loadJWTAuthEnv(); err == nil { + t.Fatal("expected a plain http JWKS URL to be rejected") + } +} + +func TestLoadJWTAuthEnv_CAFileIsOptionalAndCarried(t *testing.T) { + t.Setenv("SUPABASE_JWT_ISSUER", "https://auth.example/auth/v1") + t.Setenv("SUPABASE_JWT_AUDIENCE", "authenticated") + t.Setenv("SUPABASE_JWKS_URL", "https://auth.example/auth/v1/.well-known/jwks.json") + t.Setenv("SUPABASE_JWKS_CA_FILE", "") + + cfg, err := loadJWTAuthEnv() + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if cfg.CAFile != "" { + t.Fatalf("expected no CA file, got %q", cfg.CAFile) + } + + t.Setenv("SUPABASE_JWKS_CA_FILE", " /etc/hive/jwks-ca.pem ") + cfg, err = loadJWTAuthEnv() + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if cfg.CAFile != "/etc/hive/jwks-ca.pem" { + t.Fatalf("expected the trimmed CA path, got %q", cfg.CAFile) + } +} + +func TestLoadJWTAuthEnv_CAFileDoesNotExcuseHTTP(t *testing.T) { + t.Setenv("SUPABASE_JWT_ISSUER", "http://supabase-auth:9999") + t.Setenv("SUPABASE_JWT_AUDIENCE", "authenticated") + t.Setenv("SUPABASE_JWKS_URL", "http://supabase-auth:9999/.well-known/jwks.json") + t.Setenv("SUPABASE_JWKS_CA_FILE", "/etc/hive/jwks-ca.pem") + + if _, err := loadJWTAuthEnv(); err == nil { + t.Fatal("a CA file must not make a plain http JWKS URL acceptable") + } +} diff --git a/apps/edge-api/cmd/server/main.go b/apps/edge-api/cmd/server/main.go index 2ee06abdc..23b00cbc1 100644 --- a/apps/edge-api/cmd/server/main.go +++ b/apps/edge-api/cmd/server/main.go @@ -51,6 +51,11 @@ type jwtAuthEnv struct { Issuer string Audience string JWKSURL string + // CAFile is optional and names a PEM certificate authority to trust + // for the JWKS fetch, on top of the system roots. The self-hosted + // (enterprise) profile serves its JWKS through an in-stack TLS + // terminator using a private CA, which no public root vouches for. + CAFile string } type storageConfig struct { @@ -486,6 +491,7 @@ func main() { Issuer: jwtCfg.Issuer, JWKSURL: jwtCfg.JWKSURL, JWTAudience: jwtCfg.Audience, + CAFile: jwtCfg.CAFile, }) if err != nil { log.Fatalf("failed to initialize Supabase JWT validator: %v", err) @@ -989,7 +995,12 @@ func loadJWTAuthEnv() (jwtAuthEnv, error) { return jwtAuthEnv{}, fmt.Errorf("SUPABASE_JWKS_URL must be https (got %q)", jwksURL) } - return jwtAuthEnv{Issuer: issuer, Audience: audience, JWKSURL: jwksURL}, nil + // Optional, and additive only: the https requirement above still + // holds, chain and hostname verification still happen. This names an + // extra authority, it does not turn verification off. + caFile := strings.TrimSpace(os.Getenv("SUPABASE_JWKS_CA_FILE")) + + return jwtAuthEnv{Issuer: issuer, Audience: audience, JWKSURL: jwksURL, CAFile: caFile}, nil } // jwtAuditLogger returns the audit hook handed to the JWT middleware. For diff --git a/apps/edge-api/internal/auth/jwt_supabase.go b/apps/edge-api/internal/auth/jwt_supabase.go index dd52e630e..de1a46e0b 100644 --- a/apps/edge-api/internal/auth/jwt_supabase.go +++ b/apps/edge-api/internal/auth/jwt_supabase.go @@ -6,8 +6,12 @@ package auth import ( "context" + "crypto/tls" + "crypto/x509" "errors" "fmt" + "net/http" + "os" "strings" "time" @@ -37,6 +41,18 @@ type SupabaseJWTConfig struct { // ClockSkew tolerates small clock drift between this process and the // token issuer. Defaults to 30s when zero. ClockSkew time.Duration + // CAFile optionally names a PEM file holding an extra certificate + // authority to trust when fetching JWKSURL, on top of the system + // roots. It exists for the self-hosted (enterprise) deployment, where + // the JWKS is served by an in-stack TLS terminator holding a private + // CA's certificate rather than a publicly trusted one. Leave empty on + // deployments whose JWKS host presents a public certificate. + // + // This is deliberately additive to the https-only rule enforced at + // the caller, not a way around it: the transport still requires TLS + // and still verifies the chain and hostname. What changes is which + // authority may vouch for the certificate. + CAFile string } // Claims holds the subset of token claims the edge-api consumes downstream. @@ -77,7 +93,15 @@ func NewSupabaseJWTValidator(ctx context.Context, cfg SupabaseJWTConfig) (*Supab cfg.ClockSkew = 30 * time.Second } cache := jwk.NewCache(ctx) - if err := cache.Register(cfg.JWKSURL, jwk.WithRefreshInterval(cfg.JWKSTTL)); err != nil { + registerOpts := []jwk.RegisterOption{jwk.WithRefreshInterval(cfg.JWKSTTL)} + if cfg.CAFile != "" { + client, err := httpClientTrusting(cfg.CAFile) + if err != nil { + return nil, err + } + registerOpts = append(registerOpts, jwk.WithHTTPClient(client)) + } + if err := cache.Register(cfg.JWKSURL, registerOpts...); err != nil { return nil, fmt.Errorf("auth: jwks register: %w", err) } if _, err := cache.Refresh(ctx, cfg.JWKSURL); err != nil { @@ -86,6 +110,37 @@ func NewSupabaseJWTValidator(ctx context.Context, cfg SupabaseJWTConfig) (*Supab return &SupabaseJWTValidator{cfg: cfg, cache: cache}, nil } +// httpClientTrusting returns an HTTP client that trusts the system roots plus +// whatever certificate authority the PEM file at path holds. An unreadable +// file, or one carrying no certificate, is fatal rather than a silent +// downgrade to the system pool: a deployment that asked for a private CA and +// silently did not get one would fail later, at the first token, with a much +// worse error. +func httpClientTrusting(path string) (*http.Client, error) { + pemBytes, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("auth: read jwks ca file: %w", err) + } + pool, err := x509.SystemCertPool() + if err != nil || pool == nil { + // A platform without a system pool still gets the private CA, + // which is the only authority this client actually needs. + pool = x509.NewCertPool() + } + if !pool.AppendCertsFromPEM(pemBytes) { + return nil, fmt.Errorf("auth: jwks ca file %q holds no certificate", path) + } + return &http.Client{ + Timeout: 15 * time.Second, + Transport: &http.Transport{ + TLSClientConfig: &tls.Config{ + RootCAs: pool, + MinVersion: tls.VersionTLS12, + }, + }, + }, nil +} + // Parse validates the token signature, issuer, audience, and expiration, // then extracts edge-api claims into a Claims struct. func (v *SupabaseJWTValidator) Parse(ctx context.Context, raw string) (Claims, error) { diff --git a/apps/edge-api/internal/auth/jwt_supabase_ca_test.go b/apps/edge-api/internal/auth/jwt_supabase_ca_test.go new file mode 100644 index 000000000..b97f79197 --- /dev/null +++ b/apps/edge-api/internal/auth/jwt_supabase_ca_test.go @@ -0,0 +1,118 @@ +package auth_test + +import ( + "context" + "encoding/pem" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "testing" + + "github.com/google/uuid" + "github.com/stretchr/testify/require" + + "github.com/sakibsadmanshajib/hive/apps/edge-api/internal/auth" +) + +// The self-hosted (enterprise) profile serves its JWKS through an in-stack +// Caddy holding a private CA's certificate, because edge-api refuses a plain +// http JWKS URL and no public authority can vouch for a compose service name. +// SUPABASE_JWKS_CA_FILE names that authority. These tests pin the two halves +// of that behaviour: the fetch stays verified (an unknown authority is still +// rejected), and a named authority is actually honoured. + +const caTestIssuer = "https://selfhosted.test/auth/v1" + +func privateCAJWKSServer(t *testing.T, jwksJSON []byte) *httptest.Server { + t.Helper() + mux := http.NewServeMux() + mux.HandleFunc("/jwks", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write(jwksJSON) + }) + srv := httptest.NewTLSServer(mux) + t.Cleanup(srv.Close) + return srv +} + +func writeServerCA(t *testing.T, srv *httptest.Server) string { + t.Helper() + path := filepath.Join(t.TempDir(), "ca.pem") + encoded := pem.EncodeToMemory(&pem.Block{ + Type: "CERTIFICATE", + Bytes: srv.Certificate().Raw, + }) + require.NotNil(t, encoded) + require.NoError(t, os.WriteFile(path, encoded, 0o600)) + return path +} + +func TestJWTValidator_UnknownAuthority_RejectedWithoutCAFile(t *testing.T) { + _, _, jwksJSON := newTestKey(t) + srv := privateCAJWKSServer(t, jwksJSON) + + // No CA file: the JWKS host presents a certificate no system root + // signed, so the initial refresh must fail and the process must not + // come up believing it can validate tokens. + _, err := auth.NewSupabaseJWTValidator(context.Background(), auth.SupabaseJWTConfig{ + Issuer: caTestIssuer, + JWKSURL: srv.URL + "/jwks", + }) + require.Error(t, err) +} + +func TestJWTValidator_PrivateCAFile_ValidatesToken(t *testing.T) { + priv, _, jwksJSON := newTestKey(t) + srv := privateCAJWKSServer(t, jwksJSON) + caPath := writeServerCA(t, srv) + + uid := uuid.New() + token := signToken(t, priv, caTestIssuer, map[string]any{ + "sub": uid.String(), + "email": "ada@office.example", + "aud": "authenticated", + "role": "authenticated", + }) + + v, err := auth.NewSupabaseJWTValidator(context.Background(), auth.SupabaseJWTConfig{ + Issuer: caTestIssuer, + JWKSURL: srv.URL + "/jwks", + CAFile: caPath, + }) + require.NoError(t, err) + + claims, err := v.Parse(context.Background(), token) + require.NoError(t, err) + require.Equal(t, uid, claims.Sub) +} + +func TestJWTValidator_CAFileMissing_FailsClosed(t *testing.T) { + _, _, jwksJSON := newTestKey(t) + srv := privateCAJWKSServer(t, jwksJSON) + + // A named-but-absent CA file must be fatal. Falling back to the system + // pool would turn a deployment mistake into an outage at the first + // token instead of at boot. + _, err := auth.NewSupabaseJWTValidator(context.Background(), auth.SupabaseJWTConfig{ + Issuer: caTestIssuer, + JWKSURL: srv.URL + "/jwks", + CAFile: filepath.Join(t.TempDir(), "absent.pem"), + }) + require.ErrorContains(t, err, "read jwks ca file") +} + +func TestJWTValidator_CAFileWithoutCertificate_FailsClosed(t *testing.T) { + _, _, jwksJSON := newTestKey(t) + srv := privateCAJWKSServer(t, jwksJSON) + + path := filepath.Join(t.TempDir(), "not-a-cert.pem") + require.NoError(t, os.WriteFile(path, []byte("this is not a certificate\n"), 0o600)) + + _, err := auth.NewSupabaseJWTValidator(context.Background(), auth.SupabaseJWTConfig{ + Issuer: caTestIssuer, + JWKSURL: srv.URL + "/jwks", + CAFile: path, + }) + require.ErrorContains(t, err, "holds no certificate") +} diff --git a/deploy/docker/Caddyfile.supabase b/deploy/docker/Caddyfile.supabase new file mode 100644 index 000000000..5ab0fbadc --- /dev/null +++ b/deploy/docker/Caddyfile.supabase @@ -0,0 +1,67 @@ +# Supabase API gateway for the self-hosted (enterprise) profile. +# +# Hosted Supabase puts Kong in front of GoTrue, PostgREST and Storage, which +# is what lets every consumer hold ONE base URL and append the familiar +# /auth/v1, /rest/v1 and /storage/v1 prefixes. The enterprise compose had no +# equivalent, so those consumers would each have to learn a different host and +# a different path shape: +# +# apps/control-plane/internal/auth/client.go builds supabaseURL + /auth/v1/user +# @supabase/ssr appends /auth/v1 and /rest/v1 to its base +# the seeding scripts derive both from a single base URL +# +# Standalone GoTrue serves /user with no prefix, and PostgREST is a different +# host entirely, so a bare SUPABASE_URL=http://supabase-auth:9999 is wrong for +# all of them. This file is the Kong replacement: same prefixes, no second +# proxy technology, expressed in the reverse proxy this repo already runs for +# three other origins. +# +# handle_path strips the matched prefix, which is exactly Kong's behaviour +# here. Storage keeps its own root-level routes (/s3, /object, /bucket, +# /health, verified in the pinned image), so /storage/v1/s3 lands on /s3 the +# way the hosted S3-compatible endpoint does. +# +# TLS, and why the https listener exists at all: edge-api refuses a plain http +# SUPABASE_JWKS_URL (apps/edge-api/cmd/server/main.go), because anything on +# the path to an http JWKS can swap the key set and mint tokens this gateway +# would accept. That guard is NOT relaxed for self-hosting. `tls internal` +# terminates real TLS using Caddy's own local certificate authority, and +# edge-api is pointed at that authority through SUPABASE_JWKS_CA_FILE, so the +# chain and the hostname are still verified. No public DNS name and no ACME +# account are involved, so it behaves identically on a laptop, in CI and on an +# air-gapped box. +# +# The plain-http listener on the same names is for the in-network consumers +# that do not carry the CA (control-plane, the seeding scripts) and for an +# external TLS terminator in front (Cloudflare Tunnel on the demo box). The +# explicit http:// scheme is the same reasoning as Caddyfile.owui and +# Caddyfile.artifacts: without it Caddy turns on automatic HTTPS and redirects +# plain HTTP back to itself, which loops forever behind a terminator. + +{ + auto_https disable_redirects +} + +(supabase_routes) { + handle_path /auth/v1/* { + reverse_proxy supabase-auth:9999 + } + handle_path /rest/v1/* { + reverse_proxy supabase-rest:3000 + } + handle_path /storage/v1/* { + reverse_proxy supabase-storage:5000 + } + handle { + respond "hive self-hosted supabase gateway" 200 + } +} + +https://caddy-supabase { + tls internal + import supabase_routes +} + +http://caddy-supabase, http://{$SUPABASE_DOMAIN:supabase.localhost} { + import supabase_routes +} diff --git a/deploy/docker/docker-compose.enterprise.yml b/deploy/docker/docker-compose.enterprise.yml index 16c91b86b..b5a027fdc 100644 --- a/deploy/docker/docker-compose.enterprise.yml +++ b/deploy/docker/docker-compose.enterprise.yml @@ -21,6 +21,8 @@ # ENTERPRISE_JWT_SECRET openssl rand -base64 48 (min 32 chars) # ENTERPRISE_ANON_KEY sign {"role":"anon"} JWT with ENTERPRISE_JWT_SECRET # ENTERPRISE_SERVICE_ROLE_KEY sign {"role":"service_role"} JWT with ENTERPRISE_JWT_SECRET +# ENTERPRISE_JWT_KEYS scripts/generate-enterprise-jwt-keys.py +# ENTERPRISE_JWT_JWKS same command, printed on the second line # # See .env.example for the full ENTERPRISE_* block and generation instructions. @@ -55,12 +57,21 @@ services: restart: unless-stopped # ── supabase-auth (GoTrue) ───────────────────────────────────────────────── - # Local JWT issuer. edge-api validates tokens against this service via - # SUPABASE_JWKS_URL=http://supabase-auth:9999/.well-known/jwks.json. + # Local JWT issuer. edge-api validates tokens against its JWKS, reached + # through the caddy-supabase gateway over TLS, never directly over http: + # SUPABASE_JWKS_URL=https://caddy-supabase/auth/v1/.well-known/jwks.json. # Set ENTERPRISE_MAILER_AUTOCONFIRM=true for air-gapped installs without SMTP. supabase-auth: - # Pinned to immutable digest. Refresh: docker pull supabase/gotrue:v2.170.0 && docker inspect --format='{{index .RepoDigests 0}}' supabase/gotrue:v2.170.0 - image: supabase/gotrue:v2.170.0@sha256:0cfa54e096dcae4ed30162811fa8794f7c443708c41973329e07f7d44c1cc3e7 + # Pinned to immutable digest. Refresh: docker pull supabase/gotrue:v2.189.0 && docker inspect --format='{{index .RepoDigests 0}}' supabase/gotrue:v2.189.0 + # + # Bumped from v2.170.0 (2025-03-06), which predates the OAuth 2.1 + # authorization server entirely: on that tag /oauth/authorize and both + # well-known discovery documents answer 404, verified by running the image + # rather than by reading notes. Open WebUI signs in through that server + # (OPENID_PROVIDER_URL in docker-compose.yml), so self-hosting on the old + # pin would have silently lost chat login. The server first ships in + # v2.180.0; v2.189.0 is what upstream's own self-host compose pins. + image: supabase/gotrue:v2.189.0@sha256:385184459f57569c54c25209f51f3b2be99ddd7c4ce9e3555b5d3eea8447b7cf profiles: - enterprise depends_on: @@ -69,8 +80,12 @@ services: environment: GOTRUE_API_HOST: "0.0.0.0" GOTRUE_API_PORT: "9999" - # Required by GoTrue v2: the public-facing base URL for auth links in emails. - API_EXTERNAL_URL: ${ENTERPRISE_SITE_URL:-http://localhost:9999} + # The AUTH origin, not the app origin. It is the base GoTrue puts in + # email links and, with the issuer below, in its discovery document, so + # it must be the externally reachable address of THIS service. Distinct + # from GOTRUE_SITE_URL below, which is the web-console origin: deriving + # both from one variable cannot be correct for either. + API_EXTERNAL_URL: ${ENTERPRISE_AUTH_EXTERNAL_URL:-http://caddy-supabase/auth/v1} GOTRUE_DB_DRIVER: postgres GOTRUE_DB_DATABASE_URL: "postgres://${ENTERPRISE_DB_USER:-postgres}:${ENTERPRISE_DB_PASSWORD:?set ENTERPRISE_DB_PASSWORD in .env}@supabase-db:5432/${ENTERPRISE_DB_NAME:-postgres}?search_path=auth" GOTRUE_SITE_URL: ${ENTERPRISE_SITE_URL:-http://localhost:3000} @@ -79,13 +94,31 @@ services: # Set ENTERPRISE_DISABLE_SIGNUP=false in .env to re-enable self-serve signup. GOTRUE_DISABLE_SIGNUP: ${ENTERPRISE_DISABLE_SIGNUP:-true} GOTRUE_JWT_SECRET: ${ENTERPRISE_JWT_SECRET:?set ENTERPRISE_JWT_SECRET in .env (openssl rand -base64 48)} + # Asymmetric signing key set, and the reason this profile can + # authenticate at all. With GOTRUE_JWT_SECRET alone GoTrue signs HS256 + # and its /.well-known/jwks.json publishes an EMPTY key set: the + # handler skips every symmetric key (internal/api/jwks.go, verified in + # this exact pinned image). edge-api validates with jwt.WithKeySet + # against that endpoint and fails its initial refresh, so every token + # is rejected and the gateway will not come up. Generate the value + # with scripts/generate-enterprise-jwt-keys.py; GoTrue accepts exactly + # one signing key, so rotation is a replace and forces a re-login. + # The legacy HS256 anon and service_role keys keep working: they carry + # no kid and fall back to GOTRUE_JWT_SECRET above. + GOTRUE_JWT_KEYS: ${ENTERPRISE_JWT_KEYS:?set ENTERPRISE_JWT_KEYS in .env (scripts/generate-enterprise-jwt-keys.py)} GOTRUE_JWT_EXP: ${ENTERPRISE_JWT_EXP:-3600} GOTRUE_JWT_DEFAULT_GROUP_NAME: authenticated GOTRUE_JWT_ADMIN_ROLES: service_role GOTRUE_JWT_AUD: authenticated # Must match what edge-api expects in SUPABASE_JWT_ISSUER / SUPABASE_JWKS_URL. # Default is the in-stack service name, not localhost. - GOTRUE_JWT_ISSUER: ${ENTERPRISE_JWT_ISSUER:-http://supabase-auth:9999} + # Drives the whole OpenID discovery document and the iss claim on every + # token, including the OAuth id_token. Left empty it yields relative + # endpoints and an id_token with no iss, so it is not optional. It must + # equal edge-api's SUPABASE_JWT_ISSUER, which defaults to the same value. + # For a deployment browsers reach, override BOTH with the public https + # auth origin. + GOTRUE_JWT_ISSUER: ${ENTERPRISE_JWT_ISSUER:-http://caddy-supabase/auth/v1} GOTRUE_EXTERNAL_EMAIL_ENABLED: "true" GOTRUE_MAILER_AUTOCONFIRM: ${ENTERPRISE_MAILER_AUTOCONFIRM:-true} GOTRUE_SMTP_HOST: ${ENTERPRISE_SMTP_HOST:-} @@ -100,7 +133,14 @@ services: # Enable the custom access token hook so JWTs include tenant_id, tenants, # and role claims. Migration 20260516_07 installs public.custom_access_token_hook; # without this flag GoTrue issues plain tokens and all RLS/authz middleware breaks. - GOTRUE_HOOK_CUSTOM_ACCESS_TOKEN_ENABLED: "true" + # Overridable for one specific case: a database that has not had the + # migrations applied yet has no public.custom_access_token_hook, and + # GoTrue fails every token issuance when the hook it is told to call + # does not exist. Standing the stack up before loading the schema is + # exactly that case. Default stays true, because a deployment serving + # traffic without the hook issues tokens with no tenant claim and every + # authz check downstream then behaves as if the user belongs nowhere. + GOTRUE_HOOK_CUSTOM_ACCESS_TOKEN_ENABLED: ${ENTERPRISE_CUSTOM_ACCESS_TOKEN_HOOK_ENABLED:-true} GOTRUE_HOOK_CUSTOM_ACCESS_TOKEN_URI: "pg-functions://postgres/public/custom_access_token_hook" # Auth rate limits and password policy (OSFI B-10 / PIPEDA baseline). # Units: requests per hour (GoTrue float64 rate). GoTrue source defaults: @@ -168,17 +208,24 @@ services: PGRST_DB_URI: "postgres://${ENTERPRISE_DB_USER:-postgres}:${ENTERPRISE_DB_PASSWORD:?set ENTERPRISE_DB_PASSWORD in .env}@supabase-db:5432/${ENTERPRISE_DB_NAME:-postgres}" PGRST_DB_SCHEMAS: "public,storage,graphql_public" PGRST_DB_ANON_ROLE: anon - PGRST_JWT_SECRET: ${ENTERPRISE_JWT_SECRET:?set ENTERPRISE_JWT_SECRET in .env} + # A JWKS, not a raw secret: GoTrue now signs user tokens with ES256, so + # a symmetric-only secret here would reject every one of them. The + # generated set carries the EC public key AND the legacy symmetric key, + # so the HS256 anon and service_role keys keep validating too. + PGRST_JWT_SECRET: ${ENTERPRISE_JWT_JWKS:?set ENTERPRISE_JWT_JWKS in .env (scripts/generate-enterprise-jwt-keys.py)} PGRST_DB_USE_LEGACY_GUCS: "false" PGRST_APP_SETTINGS_JWT_SECRET: ${ENTERPRISE_JWT_SECRET:?set ENTERPRISE_JWT_SECRET in .env} PGRST_APP_SETTINGS_JWT_EXP: ${ENTERPRISE_JWT_EXP:-3600} - healthcheck: - # ponytail: postgrest image has no wget/curl; /dev/tcp is a bash builtin TCP probe. - test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/localhost/3000' 2>/dev/null || exit 1"] - interval: 5s - timeout: 3s - retries: 10 - start_period: 15s + # No healthcheck, deliberately. The postgrest image ships the binary and + # nothing else: it has no /bin/sh, so every CMD-SHELL probe answers + # "exec: /bin/sh: no such file or directory" and the container is marked + # unhealthy forever. The previous probe here did exactly that, and because + # supabase-storage waited on service_healthy, the whole enterprise profile + # could never finish starting. A check that can never pass is worse than + # none: it blocks every dependent instead of reporting anything. + # ponytail: dependents wait on service_started; add a real check if the + # image ever grows a shell, or by enabling PGRST_ADMIN_SERVER_PORT and + # probing /ready from a container that has an HTTP client. restart: unless-stopped # ── supabase-storage ─────────────────────────────────────────────────────── @@ -195,12 +242,18 @@ services: supabase-db: condition: service_healthy supabase-rest: - condition: service_healthy + # service_started, not service_healthy: see the note on supabase-rest. + condition: service_started environment: ANON_KEY: ${ENTERPRISE_ANON_KEY:?set ENTERPRISE_ANON_KEY in .env (sign anon JWT with ENTERPRISE_JWT_SECRET)} SERVICE_KEY: ${ENTERPRISE_SERVICE_ROLE_KEY:?set ENTERPRISE_SERVICE_ROLE_KEY in .env (sign service_role JWT)} POSTGREST_URL: http://supabase-rest:3000 + # Storage takes the raw secret here (it hands this to jsonwebtoken as an + # HMAC key, so a JWKS document in this variable would break the anon and + # service_role keys it is given) and takes the asymmetric set separately + # through JWT_JWKS, which is the variable its own config reads for that. PGRST_JWT_SECRET: ${ENTERPRISE_JWT_SECRET:?set ENTERPRISE_JWT_SECRET in .env} + JWT_JWKS: ${ENTERPRISE_JWT_JWKS:?set ENTERPRISE_JWT_JWKS in .env (scripts/generate-enterprise-jwt-keys.py)} DATABASE_URL: "postgres://${ENTERPRISE_DB_USER:-postgres}:${ENTERPRISE_DB_PASSWORD:?set ENTERPRISE_DB_PASSWORD in .env}@supabase-db:5432/${ENTERPRISE_DB_NAME:-postgres}" FILE_SIZE_LIMIT: ${ENTERPRISE_STORAGE_FILE_SIZE_LIMIT:-52428800} STORAGE_BACKEND: file @@ -213,7 +266,10 @@ services: volumes: - supabase-storage-data:/var/lib/storage healthcheck: - test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:5000/status || exit 1"] + # 127.0.0.1, not localhost: the storage image resolves localhost to ::1 + # first, the server binds IPv4 only, and the probe then fails with + # "connection refused" forever while the service is in fact serving. + test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://127.0.0.1:5000/status || exit 1"] interval: 5s timeout: 3s retries: 10 @@ -257,6 +313,54 @@ services: echo "Bucket init complete." restart: "no" + # ── caddy-supabase (the Kong replacement) ────────────────────────────────── + # One gateway in front of GoTrue, PostgREST and Storage, so a single + # SUPABASE_URL with the familiar /auth/v1, /rest/v1 and /storage/v1 prefixes + # keeps working exactly as it does against hosted Supabase. Without it every + # consumer would need a different host and a different path shape, and + # apps/control-plane/internal/auth/client.go, @supabase/ssr and the seeding + # scripts would each break in a different way. See Caddyfile.supabase. + # + # It also carries the TLS listener edge-api's JWKS fetch requires. That + # guard (https only) is not relaxed for self-hosting: Caddy terminates real + # TLS with its own local certificate authority, and edge-api trusts exactly + # that authority through SUPABASE_JWKS_CA_FILE. + # + # Same image and volume pattern as caddy-owui, caddy-console and + # caddy-artifacts, with its own named volumes so certificate state never + # mixes between origins. + # + # The healthcheck is an HTTPS request to this service's own name, routed + # through to GoTrue's /health: it proves the TLS listener answers AND that + # the prefix routing reaches an upstream. It also gives edge-api a + # service_healthy condition to wait on, so the certificate authority file + # exists on the shared volume before the first JWKS fetch. + caddy-supabase: + image: caddy:2-alpine@sha256:86deaf5e3d3408a6ccec08fbb79989783dd26e206ae10bcf78a801dc8c9ab794 + profiles: + - enterprise + depends_on: + supabase-auth: + condition: service_healthy + environment: + SUPABASE_DOMAIN: ${SUPABASE_DOMAIN:-supabase.localhost} + volumes: + - ./Caddyfile.supabase:/etc/caddy/Caddyfile:ro + - caddy-supabase-data:/data + - caddy-supabase-config:/config + healthcheck: + test: ["CMD-SHELL", "wget --no-check-certificate --no-verbose --tries=1 -O /dev/null https://caddy-supabase/auth/v1/health || exit 1"] + interval: 5s + timeout: 3s + retries: 10 + start_period: 15s + logging: + driver: json-file + options: + max-size: "10m" + max-file: "3" + restart: unless-stopped + # ── depends_on overrides for edge-api and control-plane ─────────────────── # The base docker-compose.yml already declares required:false stubs for # supabase-auth and supabase-storage. This override adds supabase-init so @@ -266,6 +370,20 @@ services: depends_on: supabase-init: condition: service_completed_successfully + caddy-supabase: + condition: service_healthy + environment: + # Self-hosted GoTrue serves its routes at the root, not under /auth/v1 + # the way hosted Supabase does, so the JWKS path has no prefix. The + # issuer is a string comparison against the token claim, never fetched, + # so it stays the in-network name GoTrue actually stamps. + SUPABASE_JWT_ISSUER: ${SUPABASE_JWT_ISSUER:-${ENTERPRISE_JWT_ISSUER:-http://caddy-supabase/auth/v1}} + SUPABASE_JWKS_URL: ${SUPABASE_JWKS_URL:-https://caddy-supabase/auth/v1/.well-known/jwks.json} + SUPABASE_JWKS_CA_FILE: ${SUPABASE_JWKS_CA_FILE:-/etc/hive/supabase-ca/caddy/pki/authorities/local/root.crt} + volumes: + # Read-only view of caddy-supabase's state, for one file: the local + # authority's root certificate. Nothing else in edge-api reads it. + - caddy-supabase-data:/etc/hive/supabase-ca:ro control-plane: depends_on: @@ -279,3 +397,7 @@ volumes: # Object files written by the Storage API local filesystem backend. # Holds hive-files and hive-images bucket contents on the box filesystem. supabase-storage-data: + # caddy-supabase's certificate and configuration state. The local authority's + # root certificate lives here and is mounted read-only into edge-api. + caddy-supabase-data: + caddy-supabase-config: diff --git a/deploy/supabase/init/00-extensions.sql b/deploy/supabase/init/00-extensions.sql index 16983f0d7..b489d03e1 100644 --- a/deploy/supabase/init/00-extensions.sql +++ b/deploy/supabase/init/00-extensions.sql @@ -36,6 +36,17 @@ BEGIN IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'supabase_admin') THEN CREATE ROLE supabase_admin NOLOGIN NOINHERIT BYPASSRLS; END IF; + -- Owners of the auth and storage schemas on hosted Supabase. Nothing here + -- logs in as either: GoTrue and the Storage API connect as postgres. They + -- exist so a pg_restore of a hosted dump, whose objects are owned by these + -- roles, does not fail on an unknown owner. Cheaper to create now than to + -- discover mid-restore. + IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'supabase_auth_admin') THEN + CREATE ROLE supabase_auth_admin NOLOGIN NOINHERIT; + END IF; + IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'supabase_storage_admin') THEN + CREATE ROLE supabase_storage_admin NOLOGIN NOINHERIT; + END IF; -- hive_app is the application role used by control-plane and edge-api. -- RLS policies in supabase/migrations/20260529_01_rls_tenant_tables.sql -- grant full access to this role (NOLOGIN, non-BYPASSRLS so RLS still applies). @@ -53,3 +64,26 @@ GRANT USAGE ON SCHEMA storage TO hive_app; GRANT USAGE ON SCHEMA public TO anon, authenticated, service_role; GRANT USAGE ON SCHEMA storage TO anon, authenticated, service_role; GRANT USAGE ON SCHEMA extensions TO anon, authenticated, service_role; + +-- Table privileges in the storage schema, for tables that do not exist yet. +-- +-- The Storage API creates storage.buckets, storage.objects and friends from +-- its own migrations, which run after this file. A plain GRANT here would +-- therefore grant on nothing, and the service_role connection would answer +-- every bucket call with SQLSTATE 42501. The Storage API reports that code as +-- "new row violates row-level security policy" whatever the actual cause, so +-- it reads as an RLS problem and is in fact a missing GRANT. That is exactly +-- what broke bucket creation on the first enterprise-profile boot. +-- +-- Default privileges apply to whatever postgres creates later, which is what +-- the Storage API connects as. +-- +-- service_role only, deliberately. Nothing in this product talks to Storage +-- from a browser: edge-api and control-plane hold the service key and mediate +-- every file operation. Granting anon and authenticated as hosted Supabase +-- does would only be safe alongside real RLS policies on these tables, and +-- there are none here. Add both together if a browser-direct path ever lands. +ALTER DEFAULT PRIVILEGES FOR ROLE postgres IN SCHEMA storage + GRANT ALL ON TABLES TO service_role; +ALTER DEFAULT PRIVILEGES FOR ROLE postgres IN SCHEMA storage + GRANT ALL ON SEQUENCES TO service_role; diff --git a/scripts/generate-enterprise-jwt-keys.py b/scripts/generate-enterprise-jwt-keys.py new file mode 100755 index 000000000..f65b80bdd --- /dev/null +++ b/scripts/generate-enterprise-jwt-keys.py @@ -0,0 +1,263 @@ +#!/usr/bin/env python3 +"""Generate the asymmetric JWT signing material the enterprise profile needs. + +Why this exists +--------------- +docker-compose.enterprise.yml configured GoTrue with a symmetric +GOTRUE_JWT_SECRET and nothing else. GoTrue's JWKS endpoint deliberately +excludes symmetric keys: internal/api/jwks.go skips every key whose public +half is nil or of type jwa.OctetSeq, so /.well-known/jwks.json answered with +an empty key set. apps/edge-api validates tokens with jwt.WithKeySet against +exactly that endpoint and refuses to boot when the initial refresh yields no +usable key (apps/edge-api/internal/auth/jwt_supabase.go), so a self-hosted +stack configured that way could not authenticate a single request. + +The fix is upstream-supported and config-only: hand GoTrue an EC P-256 +signing key through GOTRUE_JWT_KEYS. GoTrue then signs access tokens with +ES256, publishes the matching public key on its JWKS endpoint, and keeps +accepting the legacy HS256 anon and service_role keys through the +GOTRUE_JWT_SECRET fallback in conf.FindPublicKeyByKid. + +This script prints two values: + + ENTERPRISE_JWT_KEYS the private signing set, for GoTrue only. Contains the + EC private scalar. Secret. + ENTERPRISE_JWT_JWKS the verification set, for PostgREST. Carries the EC + public key plus the legacy symmetric key so PostgREST + can verify both the new ES256 user tokens and the + existing HS256 anon and service_role keys. The + symmetric entry is the HMAC secret in another + encoding, so this value is secret too. + +Both go in .env, which is never committed. Neither is ever logged. + +Usage: run with ENTERPRISE_JWT_SECRET set in the environment to your existing +enterprise JWT secret. Add --self-check to run the offline guards instead of +generating anything. + +Rotation: GoTrue accepts exactly one signing key (conf.JwtKeysDecoder.Validate +rejects zero or more than one key carrying the "sign" operation), so rotation +is a replace, not an append. Tokens signed by the old key stop validating once +the old public key leaves the JWKS, which is a forced re-login, not data loss. +""" + +import argparse +import base64 +import json +import os +import subprocess +import sys +import uuid + +CURVE = "prime256v1" # NIST P-256, the curve behind ES256. +COORD_BYTES = 32 # P-256 field element width. + + +def b64url(raw): + """base64url without padding, per RFC 7515 section 2.""" + return base64.urlsafe_b64encode(raw).rstrip(b"=").decode("ascii") + + +def run_openssl(args, stdin=None): + """Run openssl and return stdout, raising a readable message on failure.""" + try: + proc = subprocess.run( + ["openssl"] + list(args), + input=stdin, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=True, + ) + except FileNotFoundError: + raise SystemExit("openssl not found on PATH; install it and re-run") from None + except subprocess.CalledProcessError as exc: + detail = exc.stderr.decode("utf-8", "replace").strip() + raise SystemExit("openssl failed: " + detail) from None + return proc.stdout + + +def parse_openssl_ec_text(text): + """Pull the private scalar and the uncompressed public point out of + `openssl ec -text -noout` output. + + The output labels two hex blocks, priv and pub, each continued over + indented lines. The private scalar may carry a leading zero byte because + openssl prints it as a signed integer, and it may be shorter than the + field width when its top bytes are zero, so it is normalised to exactly + 32 bytes rather than trusted as printed. + """ + section = None + chunks = {"priv": [], "pub": []} + for line in text.splitlines(): + stripped = line.strip() + if stripped.startswith("priv:"): + section = "priv" + stripped = stripped[len("priv:"):].strip() + elif stripped.startswith("pub:"): + section = "pub" + stripped = stripped[len("pub:"):].strip() + elif not line.startswith(" ") or ":" not in stripped: + # A non-indented line, or an "ASN1 OID: prime256v1" style label, + # ends the current hex block. + section = None + continue + if section and stripped: + chunks[section].append(stripped) + + def to_bytes(name): + joined = "".join(chunks[name]).replace(":", "").replace(" ", "") + if not joined: + raise SystemExit("could not read the " + name + " block from openssl") + return bytes.fromhex(joined) + + priv = to_bytes("priv").lstrip(b"\x00").rjust(COORD_BYTES, b"\x00") + pub = to_bytes("pub") + if len(priv) != COORD_BYTES: + raise SystemExit("private scalar has the wrong width") + if len(pub) != 1 + 2 * COORD_BYTES or pub[0] != 0x04: + raise SystemExit("public point is not an uncompressed P-256 point") + return priv, pub + + +def generate_ec_jwk(kid): + """Return (private JWK, public JWK) for a fresh P-256 keypair.""" + pem = run_openssl(["ecparam", "-name", CURVE, "-genkey", "-noout"]) + text = run_openssl(["ec", "-text", "-noout"], stdin=pem).decode("utf-8", "replace") + priv, pub = parse_openssl_ec_text(text) + x = pub[1 : 1 + COORD_BYTES] + y = pub[1 + COORD_BYTES :] + + public_jwk = { + "kty": "EC", + "crv": "P-256", + "x": b64url(x), + "y": b64url(y), + "alg": "ES256", + "kid": kid, + "use": "sig", + "key_ops": ["verify"], + } + # GoTrue picks its signing key by looking for the "sign" operation + # (conf.GetSigningJwk) and picks the algorithm off the "alg" field + # (conf.GetSigningAlg), which falls back to HS256 when "alg" is absent. + # Both fields are load bearing, not decoration. + private_jwk = { + "kty": "EC", + "crv": "P-256", + "x": public_jwk["x"], + "y": public_jwk["y"], + "d": b64url(priv), + "alg": "ES256", + "kid": kid, + "key_ops": ["sign", "verify"], + } + return private_jwk, public_jwk + + +def legacy_symmetric_jwk(secret): + """The existing GOTRUE_JWT_SECRET expressed as an oct JWK. + + GoTrue keeps validating the HS256 anon and service_role keys through its + plain secret, but PostgREST is handed a JWKS instead of a raw secret and + would otherwise lose the ability to verify them. No kid is set: those + tokens carry no kid header, and a keyed entry would not match them. + """ + return { + "kty": "oct", + "k": b64url(secret.encode("utf-8")), + "alg": "HS256", + "key_ops": ["verify"], + } + + +def build(secret): + kid = str(uuid.uuid4()) + private_jwk, public_jwk = generate_ec_jwk(kid) + keys = json.dumps([private_jwk], separators=(",", ":")) + jwks = json.dumps( + {"keys": [public_jwk, legacy_symmetric_jwk(secret)]}, separators=(",", ":") + ) + return keys, jwks + + +def self_check(): + """Offline guards. No network, no framework, no files written.""" + secret = "self-check-secret-that-is-long-enough-for-hs256" + keys_raw, jwks_raw = build(secret) + keys = json.loads(keys_raw) + jwks = json.loads(jwks_raw) + + assert isinstance(keys, list) and len(keys) == 1, "JWT_KEYS must be a one-key array" + private_jwk = keys[0] + + # GoTrue rejects a key set with no signing key or with more than one. + signing = [k for k in keys if "sign" in k.get("key_ops", [])] + assert len(signing) == 1, "expected exactly one signing key" + + assert private_jwk["alg"] == "ES256", "alg must be ES256 or GoTrue falls back to HS256" + assert private_jwk["kid"], "GoTrue indexes keys by kid; an empty kid collides" + for field in ("d", "x", "y"): + raw = base64.urlsafe_b64decode(private_jwk[field] + "==") + assert len(raw) == COORD_BYTES, field + " has the wrong width" + + # The verification set must never carry private material. + ec_public = [k for k in jwks["keys"] if k["kty"] == "EC"] + assert len(ec_public) == 1, "expected exactly one EC entry in the verification set" + assert "d" not in ec_public[0], "private scalar leaked into the verification set" + assert ec_public[0]["kid"] == private_jwk["kid"], "public and private kid must match" + assert ec_public[0]["x"] == private_jwk["x"] and ec_public[0]["y"] == private_jwk["y"] + + # The legacy symmetric entry has to round-trip the existing secret, or + # every HS256 anon and service_role key stops validating at PostgREST. + oct_keys = [k for k in jwks["keys"] if k["kty"] == "oct"] + assert len(oct_keys) == 1, "expected exactly one symmetric entry" + assert base64.urlsafe_b64decode(oct_keys[0]["k"] + "==").decode("utf-8") == secret + assert "kid" not in oct_keys[0], "legacy tokens carry no kid; a keyed entry cannot match" + + # Two runs must not produce the same key. + other_keys, _ = build(secret) + assert json.loads(other_keys)[0]["d"] != private_jwk["d"], "keypair is not fresh" + + # A short private scalar (leading zero bytes) must still normalise to 32. + priv, pub = parse_openssl_ec_text( + "Private-Key: (256 bit)\npriv:\n 00:01:02\npub:\n 04:" + + ":".join(["11"] * 64) + + "\nASN1 OID: prime256v1\n" + ) + assert priv == b"\x00" * 30 + b"\x01\x02", "short private scalar was not left-padded" + assert len(pub) == 65 and pub[0] == 0x04 + + print("generate-enterprise-jwt-keys self-check: OK") + return 0 + + +def main(): + parser = argparse.ArgumentParser(description="Generate enterprise JWT key material") + parser.add_argument( + "--self-check", + action="store_true", + help="run the offline guards and exit; generates nothing for use", + ) + args = parser.parse_args() + if args.self_check: + return self_check() + + secret = os.environ.get("ENTERPRISE_JWT_SECRET", "").strip() + if len(secret) < 32: + print( + "ENTERPRISE_JWT_SECRET must be set to your existing enterprise JWT " + "secret (at least 32 characters) so the legacy anon and service_role " + "keys keep validating. Generate one with: openssl rand -base64 48", + file=sys.stderr, + ) + return 2 + + keys, jwks = build(secret) + print("# Paste both lines into .env. Secret material: never commit, never log.") + print("ENTERPRISE_JWT_KEYS=" + keys) + print("ENTERPRISE_JWT_JWKS=" + jwks) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 5a83ef19c1466b64038e9b97e760b8d3d1b00e2d Mon Sep 17 00:00:00 2001 From: Sakib Sadman Shajib Date: Tue, 18 Aug 2026 17:37:23 -0400 Subject: [PATCH 2/9] fix: narrow the jwks trust set and the gateway's public surface Review hardening on the auth path, plus the OAuth client registration the self-hosted stack needs to build itself. SUPABASE_JWKS_CA_FILE now REPLACES the system roots rather than adding to them. Naming a CA file is a statement that a specific operator-controlled authority serves the JWKS, which here is an in-stack terminator on a compose service name no public authority could ever issue for. Leaving the public roots in the pool alongside it left every public CA able to vouch for that fetch for no benefit. A deployment whose JWKS host has a public certificate leaves the variable unset and gets the system pool, unchanged. The gateway now serves a smaller route set to browsers than in network. PostgREST and Storage are no longer exposed on the public hostname: nothing in this product reaches either from a browser, so a public /rest/v1 would put the whole public schema one anon key away from the internet, governed only by whatever grants happen to exist. GoTrue's admin API is refused at the public edge for the same reason, so a leaked service key cannot be used from outside the box. Unmatched paths answer 404 instead of a 200 banner. scripts/register-owui-oauth-client.py registers the Open WebUI OAuth client through GoTrue's admin API, which is dashboard-only state today: nothing in the repo creates it, so it disappears with the cloud project leaving no diff and no error. Idempotent, and the redirect URI match is exact string, which is what its self-check guards. GOTRUE_OAUTH_SERVER_ENABLED and the consent path are wired now that the pin supports them, with dynamic registration left off. Co-Authored-By: Claude Opus 5 --- Makefile | 1 + apps/edge-api/internal/auth/jwt_supabase.go | 52 +++-- .../internal/auth/jwt_supabase_ca_test.go | 55 +++++ deploy/docker/Caddyfile.supabase | 60 +++++- deploy/docker/docker-compose.enterprise.yml | 23 ++ scripts/register-owui-oauth-client.py | 197 ++++++++++++++++++ 6 files changed, 361 insertions(+), 27 deletions(-) create mode 100755 scripts/register-owui-oauth-client.py diff --git a/Makefile b/Makefile index f824c4bba..8cbe9b3c6 100644 --- a/Makefile +++ b/Makefile @@ -29,3 +29,4 @@ test-scripts: python3 scripts/test_caddy_owui_blocklist.py python3 scripts/test_owui_model_picker_filter.py python3 scripts/generate-enterprise-jwt-keys.py --self-check + python3 scripts/register-owui-oauth-client.py --self-check diff --git a/apps/edge-api/internal/auth/jwt_supabase.go b/apps/edge-api/internal/auth/jwt_supabase.go index de1a46e0b..2e6673ec7 100644 --- a/apps/edge-api/internal/auth/jwt_supabase.go +++ b/apps/edge-api/internal/auth/jwt_supabase.go @@ -41,17 +41,19 @@ type SupabaseJWTConfig struct { // ClockSkew tolerates small clock drift between this process and the // token issuer. Defaults to 30s when zero. ClockSkew time.Duration - // CAFile optionally names a PEM file holding an extra certificate - // authority to trust when fetching JWKSURL, on top of the system - // roots. It exists for the self-hosted (enterprise) deployment, where - // the JWKS is served by an in-stack TLS terminator holding a private - // CA's certificate rather than a publicly trusted one. Leave empty on - // deployments whose JWKS host presents a public certificate. + // CAFile optionally names a PEM file holding the certificate authority + // to trust when fetching JWKSURL. When set it REPLACES the system + // roots for that one fetch, so the named authority is the only one + // that can vouch for the JWKS host. It exists for the self-hosted + // (enterprise) deployment, where the JWKS is served by an in-stack TLS + // terminator on a compose service name, holding a private CA's + // certificate that no public authority could issue. Leave it empty on + // deployments whose JWKS host presents a publicly trusted certificate. // - // This is deliberately additive to the https-only rule enforced at - // the caller, not a way around it: the transport still requires TLS - // and still verifies the chain and hostname. What changes is which - // authority may vouch for the certificate. + // This never weakens the https-only rule enforced at the caller: the + // transport still requires TLS and still verifies the chain and the + // hostname. It narrows which authority is acceptable, it does not skip + // verification. CAFile string } @@ -110,23 +112,29 @@ func NewSupabaseJWTValidator(ctx context.Context, cfg SupabaseJWTConfig) (*Supab return &SupabaseJWTValidator{cfg: cfg, cache: cache}, nil } -// httpClientTrusting returns an HTTP client that trusts the system roots plus -// whatever certificate authority the PEM file at path holds. An unreadable -// file, or one carrying no certificate, is fatal rather than a silent -// downgrade to the system pool: a deployment that asked for a private CA and -// silently did not get one would fail later, at the first token, with a much -// worse error. +// httpClientTrusting returns an HTTP client that trusts exactly one +// certificate authority: whatever the PEM file at path holds. The system +// roots are deliberately NOT included. +// +// Naming a CA file is a statement that the JWKS is served by a specific, +// operator-controlled authority, which for this deployment is an in-stack TLS +// terminator on a compose service name that no public authority could ever +// issue for. Keeping the public roots in the pool alongside it would leave +// every public CA able to vouch for that fetch for no benefit, so the file +// replaces the trust set rather than extending it. A deployment whose JWKS +// host presents a publicly trusted certificate simply leaves the variable +// unset and gets the system pool, which is the default path. +// +// An unreadable file, or one carrying no certificate, is fatal rather than a +// silent fall back to the system pool: a deployment that asked for a private +// CA and quietly did not get one would fail later, at the first token, with a +// far worse error. func httpClientTrusting(path string) (*http.Client, error) { pemBytes, err := os.ReadFile(path) if err != nil { return nil, fmt.Errorf("auth: read jwks ca file: %w", err) } - pool, err := x509.SystemCertPool() - if err != nil || pool == nil { - // A platform without a system pool still gets the private CA, - // which is the only authority this client actually needs. - pool = x509.NewCertPool() - } + pool := x509.NewCertPool() if !pool.AppendCertsFromPEM(pemBytes) { return nil, fmt.Errorf("auth: jwks ca file %q holds no certificate", path) } diff --git a/apps/edge-api/internal/auth/jwt_supabase_ca_test.go b/apps/edge-api/internal/auth/jwt_supabase_ca_test.go index b97f79197..841dfe434 100644 --- a/apps/edge-api/internal/auth/jwt_supabase_ca_test.go +++ b/apps/edge-api/internal/auth/jwt_supabase_ca_test.go @@ -2,12 +2,19 @@ package auth_test import ( "context" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/x509" + "crypto/x509/pkix" "encoding/pem" + "math/big" "net/http" "net/http/httptest" "os" "path/filepath" "testing" + "time" "github.com/google/uuid" "github.com/stretchr/testify/require" @@ -116,3 +123,51 @@ func TestJWTValidator_CAFileWithoutCertificate_FailsClosed(t *testing.T) { }) require.ErrorContains(t, err, "holds no certificate") } + +func TestJWTValidator_CAFileReplacesSystemRoots(t *testing.T) { + _, _, jwksJSON := newTestKey(t) + srv := privateCAJWKSServer(t, jwksJSON) + + // A CA file naming a DIFFERENT authority must not reach this server. That + // is what proves the file is the trust set rather than an addition to it: + // if the system roots were still in the pool, or if the file were merely + // advisory, this would connect. + // + // The other authority has to be genuinely unrelated. httptest hands every + // TLS server the same built-in certificate, so a second httptest server + // would not be a different issuer at all. + unrelated := unrelatedCAFile(t) + + _, err := auth.NewSupabaseJWTValidator(context.Background(), auth.SupabaseJWTConfig{ + Issuer: caTestIssuer, + JWKSURL: srv.URL + "/jwks", + CAFile: unrelated, + }) + require.Error(t, err) +} + +// unrelatedCAFile writes a self-signed certificate that signed nothing in this +// test, and returns its path. +func unrelatedCAFile(t *testing.T) string { + t.Helper() + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + require.NoError(t, err) + tmpl := &x509.Certificate{ + SerialNumber: big.NewInt(1), + Subject: pkix.Name{CommonName: "unrelated-authority"}, + NotBefore: time.Now().Add(-time.Hour), + NotAfter: time.Now().Add(time.Hour), + IsCA: true, + KeyUsage: x509.KeyUsageCertSign, + BasicConstraintsValid: true, + } + der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key) + require.NoError(t, err) + + path := filepath.Join(t.TempDir(), "unrelated-ca.pem") + require.NoError(t, os.WriteFile(path, pem.EncodeToMemory(&pem.Block{ + Type: "CERTIFICATE", + Bytes: der, + }), 0o600)) + return path +} diff --git a/deploy/docker/Caddyfile.supabase b/deploy/docker/Caddyfile.supabase index 5ab0fbadc..af8e4d482 100644 --- a/deploy/docker/Caddyfile.supabase +++ b/deploy/docker/Caddyfile.supabase @@ -21,6 +21,14 @@ # /health, verified in the pinned image), so /storage/v1/s3 lands on /s3 the # way the hosted S3-compatible endpoint does. # +# The gateway is not an authorization boundary and must not be mistaken for +# one: GoTrue, PostgREST and Storage each authenticate every request they +# receive, and each of the three prefixes below is meant to reach its backend. +# There is therefore nothing here for a path-confusion trick to unlock, since +# no route is protected BY this file. What the file does control is which +# backends are reachable from which listener, which is why the public listener +# below carries a smaller route set than the internal one. +# # TLS, and why the https listener exists at all: edge-api refuses a plain http # SUPABASE_JWKS_URL (apps/edge-api/cmd/server/main.go), because anything on # the path to an http JWKS can swap the key set and mint tokens this gateway @@ -42,7 +50,8 @@ auto_https disable_redirects } -(supabase_routes) { +# Internal routes: everything, for consumers inside the compose network. +(supabase_internal) { handle_path /auth/v1/* { reverse_proxy supabase-auth:9999 } @@ -53,15 +62,56 @@ reverse_proxy supabase-storage:5000 } handle { - respond "hive self-hosted supabase gateway" 200 + respond 404 + } +} + +# Browser-facing routes: /auth/v1 only, deliberately. +# +# PostgREST and Storage are NOT exposed on the public hostname. Hosted +# Supabase does expose them, because serving browsers directly is its product; +# here nothing does. control-plane and edge-api reach the database directly and +# mediate every file operation with the service key, so a public /rest/v1 would +# put the whole public schema one anon key away from the internet, governed +# only by whatever grants happen to exist, and a public /storage/v1 would do +# the same for objects. Neither buys anything. Sign-in and the OAuth flow are +# the only parts a browser needs, and those are all under /auth/v1. +# +# Add a prefix here only together with the RLS policies and grants that make it +# safe to serve anonymously. +(supabase_public) { + # GoTrue's admin API is not browser surface. It is guarded by the + # service_role key, and that key is only ever used server side and in + # network, so refusing it at the edge costs nothing and means a leaked + # service key cannot be used from outside the box at all. Same posture as + # the Caddyfile.owui mutation blocks. Source order matters here: handle + # blocks are mutually exclusive and evaluated in the order written, so + # this has to precede the catch-all proxy below. + @admin path /auth/v1/admin /auth/v1/admin/* + handle @admin { + respond 404 + } + handle_path /auth/v1/* { + reverse_proxy supabase-auth:9999 + } + handle { + respond 404 } } https://caddy-supabase { tls internal - import supabase_routes + import supabase_internal +} + +http://caddy-supabase { + import supabase_internal } -http://caddy-supabase, http://{$SUPABASE_DOMAIN:supabase.localhost} { - import supabase_routes +# No host port is published for this service. Until cutover the gateway is +# reachable only from inside the compose network, which is what keeps a +# stack standing "alongside" from carrying any traffic. Publishing a port, or +# pointing a tunnel at this listener, is a cutover step and not a bring-up one. +http://{$SUPABASE_DOMAIN:supabase.localhost} { + import supabase_public } diff --git a/deploy/docker/docker-compose.enterprise.yml b/deploy/docker/docker-compose.enterprise.yml index b5a027fdc..6bcd8bbe3 100644 --- a/deploy/docker/docker-compose.enterprise.yml +++ b/deploy/docker/docker-compose.enterprise.yml @@ -119,6 +119,19 @@ services: # For a deployment browsers reach, override BOTH with the public https # auth origin. GOTRUE_JWT_ISSUER: ${ENTERPRISE_JWT_ISSUER:-http://caddy-supabase/auth/v1} + # OAuth 2.1 authorization server. Open WebUI signs in through it, so + # without this the chat front end has no login path at all on a + # self-hosted stack. Needs v2.180.0 or newer, hence the pin above. + # + # AUTHORIZATION_PATH is a path on GOTRUE_SITE_URL (the web-console + # origin), where the consent page already lives. GoTrue has no built-in + # consent UI and redirects there instead. + # + # Dynamic registration stays OFF and unset: it is an unauthenticated + # client-creation endpoint, and this deployment needs exactly one client, + # registered by scripts/register-owui-oauth-client.py. + GOTRUE_OAUTH_SERVER_ENABLED: ${ENTERPRISE_OAUTH_SERVER_ENABLED:-true} + GOTRUE_OAUTH_SERVER_AUTHORIZATION_PATH: ${ENTERPRISE_OAUTH_CONSENT_PATH:-/oauth/consent} GOTRUE_EXTERNAL_EMAIL_ENABLED: "true" GOTRUE_MAILER_AUTOCONFIRM: ${ENTERPRISE_MAILER_AUTOCONFIRM:-true} GOTRUE_SMTP_HOST: ${ENTERPRISE_SMTP_HOST:-} @@ -383,6 +396,16 @@ services: volumes: # Read-only view of caddy-supabase's state, for one file: the local # authority's root certificate. Nothing else in edge-api reads it. + # + # Stated plainly because it is not obvious: this volume also holds that + # authority's PRIVATE key, and Docker cannot mount a single file out of + # a named volume. The exposure is real but empty: this authority is + # trusted by exactly one process, edge-api itself, so holding its key + # buys an attacker only the ability to forge a JWKS for the service he + # has already compromised. Splitting the certificate into its own volume + # would need a copy container between Caddy and edge-api, which is more + # moving parts than the risk justifies. Revisit if anything else ever + # trusts this authority. - caddy-supabase-data:/etc/hive/supabase-ca:ro control-plane: diff --git a/scripts/register-owui-oauth-client.py b/scripts/register-owui-oauth-client.py new file mode 100755 index 000000000..ab398d48e --- /dev/null +++ b/scripts/register-owui-oauth-client.py @@ -0,0 +1,197 @@ +#!/usr/bin/env python3 +"""Register the Open WebUI OAuth client on a self-hosted GoTrue. + +Why this exists +--------------- +Open WebUI signs in through Supabase's OAuth 2.1 authorization server +(OPENID_PROVIDER_URL in deploy/docker/docker-compose.yml), using a client id +and secret. On hosted Supabase that client is dashboard state: nothing in this +repository creates it, nothing tracks it, and it leaves no diff. Delete the +hosted project and it is simply gone, with no error to read, and chat login +stops working for a reason nothing in the tree explains. + +Self-hosting is the chance to fix that. GoTrue exposes the same registration +over its admin API, so the client becomes a bring-up step a fresh environment +can run for itself. + +Requires GoTrue v2.180.0 or newer. The old enterprise pin (v2.170.0) has no +OAuth server at all, and answers 404 on every route this script uses. + +Usage +----- + GOTRUE_ADMIN_URL=http://caddy-supabase/auth/v1 \\ + SERVICE_ROLE_KEY=... \\ + OWUI_REDIRECT_URI=https://chat.example.com/oauth/oidc/callback \\ + python3 scripts/register-owui-oauth-client.py + +Prints the two .env lines on success. The secret is shown exactly once, at +creation, because GoTrue stores only its hash. Idempotent: a client already +registered with the same name and the same redirect URIs is reported and left +alone, since re-registering would mint a second client and a second secret. + +Add --self-check to run the offline guards instead of talking to anything. +""" + +import argparse +import json +import os +import sys +import urllib.error +import urllib.request + +# GoTrue matches redirect_uris by exact string (isValidRedirectURI upstream), +# so a trailing slash, a different scheme, or a different host is a different +# URI and the authorize call fails with invalid redirect_uri. Open WebUI posts +# back to /oauth/oidc/callback on its own origin. +CALLBACK_PATH = "/oauth/oidc/callback" + +CLIENT_NAME = "Hive Chat" + + +def build_payload(redirect_uris, client_uri, client_name=CLIENT_NAME): + """The registration body. Confidential client, authorization code with + refresh, secret sent with HTTP Basic, which is what Open WebUI does.""" + return { + "client_name": client_name, + "redirect_uris": list(redirect_uris), + "grant_types": ["authorization_code", "refresh_token"], + "response_types": ["code"], + "token_endpoint_auth_method": "client_secret_basic", + "client_uri": client_uri, + } + + +def find_existing(clients, payload): + """Return the already-registered client matching this registration, or None. + + Matched on name AND the exact redirect URI set, so a client registered for + a different environment's callback is correctly treated as a different + client rather than silently reused with the wrong redirect. + """ + want = sorted(payload["redirect_uris"]) + for c in clients: + if c.get("client_name") != payload["client_name"]: + continue + if sorted(c.get("redirect_uris") or []) == want: + return c + return None + + +def api(base, path, token, method="GET", body=None): + req = urllib.request.Request( + base.rstrip("/") + path, + method=method, + data=json.dumps(body).encode() if body is not None else None, + headers={ + "Authorization": "Bearer " + token, + "Content-Type": "application/json", + "Accept": "application/json", + }, + ) + try: + with urllib.request.urlopen(req, timeout=30) as resp: + raw = resp.read().decode("utf-8", "replace") + return resp.status, (json.loads(raw) if raw else {}) + except urllib.error.HTTPError as exc: + raw = exc.read().decode("utf-8", "replace") + if exc.code == 404: + raise SystemExit( + "GoTrue answered 404 for " + path + ". That endpoint exists only " + "when the OAuth server is enabled (GOTRUE_OAUTH_SERVER_ENABLED) " + "and only on v2.180.0 or newer. Check the image tag first." + ) from None + if exc.code in (401, 403): + raise SystemExit( + "GoTrue rejected the credential (HTTP " + str(exc.code) + "). This " + "needs the service_role key, not the anon key." + ) from None + raise SystemExit("GoTrue error HTTP " + str(exc.code) + ": " + raw[:300]) from None + except urllib.error.URLError as exc: + raise SystemExit("could not reach GoTrue at " + base + ": " + str(exc.reason)) from None + + +def self_check(): + """Offline guards. No network, no framework.""" + payload = build_payload(["https://chat.example.com" + CALLBACK_PATH], "https://chat.example.com") + assert payload["token_endpoint_auth_method"] == "client_secret_basic" + assert payload["response_types"] == ["code"] + assert "refresh_token" in payload["grant_types"], "OWUI needs refresh to keep a session" + assert payload["redirect_uris"] == ["https://chat.example.com/oauth/oidc/callback"] + + # Exact-string matching is the whole risk here, so the guards are about + # near-misses, not about the happy path. + assert find_existing([], payload) is None + same = {"client_name": CLIENT_NAME, "redirect_uris": list(payload["redirect_uris"]), "client_id": "abc"} + assert find_existing([same], payload) is same, "an identical registration must be reused" + + trailing = {"client_name": CLIENT_NAME, "redirect_uris": ["https://chat.example.com/oauth/oidc/callback/"]} + assert find_existing([trailing], payload) is None, "a trailing slash is a different URI" + + other_host = {"client_name": CLIENT_NAME, "redirect_uris": ["https://other.example.com" + CALLBACK_PATH]} + assert find_existing([other_host], payload) is None, "another host is a different client" + + other_name = {"client_name": "Something Else", "redirect_uris": list(payload["redirect_uris"])} + assert find_existing([other_name], payload) is None + + # Order must not matter when a client carries several callbacks. + multi = build_payload(["https://b.example/x", "https://a.example/y"], "https://a.example") + reversed_order = {"client_name": CLIENT_NAME, "redirect_uris": ["https://a.example/y", "https://b.example/x"]} + assert find_existing([reversed_order], multi) is reversed_order + + print("register-owui-oauth-client self-check: OK") + return 0 + + +def main(): + parser = argparse.ArgumentParser(description="Register the Open WebUI OAuth client") + parser.add_argument("--self-check", action="store_true", help="run offline guards and exit") + args = parser.parse_args() + if args.self_check: + return self_check() + + base = os.environ.get("GOTRUE_ADMIN_URL", "http://caddy-supabase/auth/v1").strip() + token = os.environ.get("SERVICE_ROLE_KEY", "").strip() + chat_url = os.environ.get("HIVE_CHAT_URL", "").strip().rstrip("/") + redirect = os.environ.get("OWUI_REDIRECT_URI", "").strip() + + if not token: + print("SERVICE_ROLE_KEY is required (the service_role JWT, not the anon key)", file=sys.stderr) + return 2 + if not redirect: + if not chat_url: + print( + "Set OWUI_REDIRECT_URI to Open WebUI's callback, or HIVE_CHAT_URL to its " + "origin. GoTrue matches this by exact string, so it must be byte identical " + "to what Open WebUI sends, trailing slash included.", + file=sys.stderr, + ) + return 2 + redirect = chat_url + CALLBACK_PATH + client_uri = chat_url or redirect.split(CALLBACK_PATH)[0] + + payload = build_payload([redirect], client_uri) + + _, listing = api(base, "/admin/oauth/clients", token) + clients = listing.get("clients", listing) if isinstance(listing, dict) else listing + existing = find_existing(clients or [], payload) + if existing: + print("# Client already registered; leaving it alone. GoTrue stores only a hash") + print("# of the secret, so the secret cannot be printed again. Rotate through") + print("# GoTrue if it has been lost.") + print("SUPABASE_OAUTH_CLIENT_ID=" + str(existing.get("client_id", ""))) + return 0 + + status, created = api(base, "/admin/oauth/clients", token, method="POST", body=payload) + if status not in (200, 201) or not created.get("client_id"): + print("unexpected registration response: HTTP " + str(status), file=sys.stderr) + return 1 + + print("# Paste into .env. The secret is shown once and never again.") + print("SUPABASE_OAUTH_CLIENT_ID=" + created["client_id"]) + print("SUPABASE_OAUTH_CLIENT_SECRET=" + created.get("client_secret", "")) + print("# redirect_uri registered: " + redirect, file=sys.stderr) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 02692b031152a8013ca333853b5819bed6094ae4 Mon Sep 17 00:00:00 2001 From: Sakib Sadman Shajib Date: Tue, 18 Aug 2026 17:43:22 -0400 Subject: [PATCH 3/9] fix: carry the legacy symmetric key inside GOTRUE_JWT_KEYS Found by standing the stack up on the demo box and calling the admin API with the service_role key: GoTrue answered "signing method HS256 is invalid", HTTP 403. Once GoTrue is handed a key set it validates incoming tokens against that set alone, so configuring only the EC key silently revoked every HS256 anon and service_role key at GoTrue's own door. Storage and PostgREST were unaffected, since they validate independently, which is exactly why this was invisible until an admin call was made. The generated key set now carries the legacy symmetric key alongside the EC signing key, verify only so it can never become a second signing key, and with no kid so it matches the kid-less legacy tokens. The compose comment claiming those keys fell back to GOTRUE_JWT_SECRET is corrected: on this pin they do not. Co-Authored-By: Claude Opus 5 --- deploy/docker/docker-compose.enterprise.yml | 11 ++++-- scripts/generate-enterprise-jwt-keys.py | 39 +++++++++++++++------ 2 files changed, 37 insertions(+), 13 deletions(-) diff --git a/deploy/docker/docker-compose.enterprise.yml b/deploy/docker/docker-compose.enterprise.yml index 6bcd8bbe3..bfb355aae 100644 --- a/deploy/docker/docker-compose.enterprise.yml +++ b/deploy/docker/docker-compose.enterprise.yml @@ -103,8 +103,15 @@ services: # is rejected and the gateway will not come up. Generate the value # with scripts/generate-enterprise-jwt-keys.py; GoTrue accepts exactly # one signing key, so rotation is a replace and forces a re-login. - # The legacy HS256 anon and service_role keys keep working: they carry - # no kid and fall back to GOTRUE_JWT_SECRET above. + # + # That generated value also carries the legacy symmetric key, verify + # only, and it has to. Once GoTrue is given a key set it validates + # incoming tokens against that set alone: observed on this pin, an admin + # call made with the HS256 service_role key is refused with "signing + # method HS256 is invalid" (HTTP 403) when the EC key is configured by + # itself, which would break the seeding scripts and the invite flow. + # GOTRUE_JWT_SECRET below stays set regardless, since Storage validates + # against the raw secret. GOTRUE_JWT_KEYS: ${ENTERPRISE_JWT_KEYS:?set ENTERPRISE_JWT_KEYS in .env (scripts/generate-enterprise-jwt-keys.py)} GOTRUE_JWT_EXP: ${ENTERPRISE_JWT_EXP:-3600} GOTRUE_JWT_DEFAULT_GROUP_NAME: authenticated diff --git a/scripts/generate-enterprise-jwt-keys.py b/scripts/generate-enterprise-jwt-keys.py index f65b80bdd..44b9533ac 100755 --- a/scripts/generate-enterprise-jwt-keys.py +++ b/scripts/generate-enterprise-jwt-keys.py @@ -20,8 +20,8 @@ This script prints two values: - ENTERPRISE_JWT_KEYS the private signing set, for GoTrue only. Contains the - EC private scalar. Secret. + ENTERPRISE_JWT_KEYS the signing set, for GoTrue only. Carries the EC + private scalar and the legacy symmetric key. Secret. ENTERPRISE_JWT_JWKS the verification set, for PostgREST. Carries the EC public key plus the legacy symmetric key so PostgREST can verify both the new ES256 user tokens and the @@ -157,10 +157,21 @@ def generate_ec_jwk(kid): def legacy_symmetric_jwk(secret): """The existing GOTRUE_JWT_SECRET expressed as an oct JWK. - GoTrue keeps validating the HS256 anon and service_role keys through its - plain secret, but PostgREST is handed a JWKS instead of a raw secret and - would otherwise lose the ability to verify them. No kid is set: those - tokens carry no kid header, and a keyed entry would not match them. + This goes in BOTH printed values, for two different reasons. + + In GOTRUE_JWT_KEYS, because once GoTrue is given a key set it validates + incoming tokens against that set alone. Observed directly on v2.189.0: with + only the EC key configured, GoTrue answers every admin call made with the + HS256 service_role key "signing method HS256 is invalid" (HTTP 403), which + breaks the seeding scripts, the invite flow and every other admin path. + Carrying the legacy key here, verify-only, restores them. + + In the verification set, because PostgREST is handed a JWKS instead of a + raw secret and would otherwise lose the ability to check those same tokens. + + No kid is set: the legacy tokens carry no kid header, and a keyed entry + could not match them. key_ops is verify only, never sign, so it cannot + become a second signing key, which GoTrue rejects outright. """ return { "kty": "oct", @@ -173,10 +184,9 @@ def legacy_symmetric_jwk(secret): def build(secret): kid = str(uuid.uuid4()) private_jwk, public_jwk = generate_ec_jwk(kid) - keys = json.dumps([private_jwk], separators=(",", ":")) - jwks = json.dumps( - {"keys": [public_jwk, legacy_symmetric_jwk(secret)]}, separators=(",", ":") - ) + legacy = legacy_symmetric_jwk(secret) + keys = json.dumps([private_jwk, legacy], separators=(",", ":")) + jwks = json.dumps({"keys": [public_jwk, legacy]}, separators=(",", ":")) return keys, jwks @@ -187,9 +197,16 @@ def self_check(): keys = json.loads(keys_raw) jwks = json.loads(jwks_raw) - assert isinstance(keys, list) and len(keys) == 1, "JWT_KEYS must be a one-key array" + assert isinstance(keys, list) and len(keys) == 2, "JWT_KEYS carries the EC key and the legacy key" private_jwk = keys[0] + # The legacy key has to travel with the signing key or GoTrue stops + # accepting the HS256 anon and service_role keys entirely. + legacy_in_keys = [k for k in keys if k["kty"] == "oct"] + assert len(legacy_in_keys) == 1, "the legacy symmetric key must be in JWT_KEYS" + assert legacy_in_keys[0]["key_ops"] == ["verify"], "the legacy key must never sign" + assert base64.urlsafe_b64decode(legacy_in_keys[0]["k"] + "==").decode("utf-8") == secret + # GoTrue rejects a key set with no signing key or with more than one. signing = [k for k in keys if "sign" in k.get("key_ops", [])] assert len(signing) == 1, "expected exactly one signing key" From 3b3c9c4af7a67b16a5a3f19ad3762e3537e43309 Mon Sep 17 00:00:00 2001 From: Sakib Sadman Shajib Date: Tue, 18 Aug 2026 17:49:00 -0400 Subject: [PATCH 4/9] fix: one auth-origin setting, and a usage recipe that actually runs Both from CodeRabbit on the final diff. The auth origin was spread across three settings that had to agree: API_EXTERNAL_URL, GoTrue's issuer, and the issuer edge-api checks. Two of them defaulted to the in-network gateway name, which no browser can reach, and keeping them consistent was left to whoever edited .env. They now all derive from ENTERPRISE_AUTH_EXTERNAL_URL, so a deployment browsers reach is one edit and the three cannot drift. Rendered both ways to confirm: unset gives the in-network default to all three, set gives the public origin to all three. The registration script's usage block showed a host-shell invocation against caddy-supabase, which does not resolve from a host shell because the gateway publishes no port. Replaced with the container-on-the-compose-network form, which is what was actually run on the demo box. Co-Authored-By: Claude Opus 5 --- .env.example | 15 ++++++++++++--- deploy/docker/docker-compose.enterprise.yml | 20 ++++++++++++-------- scripts/register-owui-oauth-client.py | 18 ++++++++++++++---- 3 files changed, 38 insertions(+), 15 deletions(-) diff --git a/.env.example b/.env.example index cf535c9b1..e0ad5641e 100644 --- a/.env.example +++ b/.env.example @@ -586,10 +586,19 @@ ENTERPRISE_DB_NAME=postgres # JWT configuration for GoTrue (supabase-auth) and PostgREST (supabase-rest). # ENTERPRISE_JWT_SECRET is the HS256 signing secret. Minimum 32 characters. -# ENTERPRISE_JWT_ISSUER must match what GoTrue uses to issue tokens so that -# apps/edge-api/internal/auth/jwt_supabase.go validates them correctly. +# ENTERPRISE_AUTH_EXTERNAL_URL below must match what GoTrue stamps as the +# issuer, which apps/edge-api/internal/auth/jwt_supabase.go checks on every +# token. Both sides derive from it, so setting it once is enough. ENTERPRISE_JWT_SECRET= -ENTERPRISE_JWT_ISSUER=http://supabase-auth:9999 +# The auth origin: one setting for API_EXTERNAL_URL, GoTrue's issuer and +# edge-api's expected issuer, so those three cannot drift apart. Replaces the +# former ENTERPRISE_JWT_ISSUER, which named only one of the three. +# +# The default (the in-stack gateway) is correct only while no browser talks to +# this stack. Set this to the public https auth origin before any deployment a +# browser reaches, and remember that changing it invalidates tokens carrying +# the old issuer, so it is a forced re-login. +ENTERPRISE_AUTH_EXTERNAL_URL=http://caddy-supabase/auth/v1 ENTERPRISE_JWT_EXP=3600 # Asymmetric signing material. NOT optional: with the symmetric secret alone, diff --git a/deploy/docker/docker-compose.enterprise.yml b/deploy/docker/docker-compose.enterprise.yml index bfb355aae..c50856eae 100644 --- a/deploy/docker/docker-compose.enterprise.yml +++ b/deploy/docker/docker-compose.enterprise.yml @@ -119,13 +119,17 @@ services: GOTRUE_JWT_AUD: authenticated # Must match what edge-api expects in SUPABASE_JWT_ISSUER / SUPABASE_JWKS_URL. # Default is the in-stack service name, not localhost. - # Drives the whole OpenID discovery document and the iss claim on every - # token, including the OAuth id_token. Left empty it yields relative - # endpoints and an id_token with no iss, so it is not optional. It must - # equal edge-api's SUPABASE_JWT_ISSUER, which defaults to the same value. - # For a deployment browsers reach, override BOTH with the public https - # auth origin. - GOTRUE_JWT_ISSUER: ${ENTERPRISE_JWT_ISSUER:-http://caddy-supabase/auth/v1} + # Same variable as API_EXTERNAL_URL above, on purpose: the issuer and the + # external URL are the same origin, and one auth origin is one setting. + # It drives the whole OpenID discovery document and the iss claim on + # every token, including the OAuth id_token, and edge-api's + # SUPABASE_JWT_ISSUER derives from it too, so the pair cannot drift. + # + # The default is the in-network gateway name, which is correct only for + # a stack carrying no browser traffic. Any deployment a browser reaches + # sets ENTERPRISE_AUTH_EXTERNAL_URL to the public https auth origin, and + # that one edit moves all three. + GOTRUE_JWT_ISSUER: ${ENTERPRISE_AUTH_EXTERNAL_URL:-http://caddy-supabase/auth/v1} # OAuth 2.1 authorization server. Open WebUI signs in through it, so # without this the chat front end has no login path at all on a # self-hosted stack. Needs v2.180.0 or newer, hence the pin above. @@ -397,7 +401,7 @@ services: # the way hosted Supabase does, so the JWKS path has no prefix. The # issuer is a string comparison against the token claim, never fetched, # so it stays the in-network name GoTrue actually stamps. - SUPABASE_JWT_ISSUER: ${SUPABASE_JWT_ISSUER:-${ENTERPRISE_JWT_ISSUER:-http://caddy-supabase/auth/v1}} + SUPABASE_JWT_ISSUER: ${SUPABASE_JWT_ISSUER:-${ENTERPRISE_AUTH_EXTERNAL_URL:-http://caddy-supabase/auth/v1}} SUPABASE_JWKS_URL: ${SUPABASE_JWKS_URL:-https://caddy-supabase/auth/v1/.well-known/jwks.json} SUPABASE_JWKS_CA_FILE: ${SUPABASE_JWKS_CA_FILE:-/etc/hive/supabase-ca/caddy/pki/authorities/local/root.crt} volumes: diff --git a/scripts/register-owui-oauth-client.py b/scripts/register-owui-oauth-client.py index ab398d48e..284991e1f 100755 --- a/scripts/register-owui-oauth-client.py +++ b/scripts/register-owui-oauth-client.py @@ -19,10 +19,20 @@ Usage ----- - GOTRUE_ADMIN_URL=http://caddy-supabase/auth/v1 \\ - SERVICE_ROLE_KEY=... \\ - OWUI_REDIRECT_URI=https://chat.example.com/oauth/oidc/callback \\ - python3 scripts/register-owui-oauth-client.py +GoTrue is not published on a host port, so this runs on the compose network +rather than from a host shell, where "caddy-supabase" does not resolve: + + docker run --rm --network _default + -e GOTRUE_ADMIN_URL=http://caddy-supabase/auth/v1 + -e SERVICE_ROLE_KEY="$ENTERPRISE_SERVICE_ROLE_KEY" + -e OWUI_REDIRECT_URI=https://chat.example.com/oauth/oidc/callback + -e HIVE_CHAT_URL=https://chat.example.com + -v "$PWD/scripts:/s:ro" python:3.12-alpine python3 /s/register-owui-oauth-client.py + +The network name is the compose project name plus "_default"; a stack started +as "-p hive" gives "hive_default". Point GOTRUE_ADMIN_URL somewhere else +instead if a deployment does publish the gateway, for instance at the public +https auth origin. Prints the two .env lines on success. The secret is shown exactly once, at creation, because GoTrue stores only its hash. Idempotent: a client already From 3f918d0150abf923f0c816c479cc5b3e207f620f Mon Sep 17 00:00:00 2001 From: Sakib Sadman Shajib Date: Wed, 19 Aug 2026 11:03:57 -0400 Subject: [PATCH 5/9] fix: enforce the public route split at the listener, and let a non-root edge-api read the CA Independent security review of this PR. Nothing it found was exploitable on anything running today, but two items would have become real at cutover and one was silently doing nothing. The public and internal route sets shared port 80, so "the public listener carries a smaller route set" was enforced by the client choosing to send an honest Host header. A request with Host: caddy-supabase against the public port got the full internal set, admin API included. The public site now binds its own port, with a catch-all so an unmatched Host there gets 404 rather than Caddy's empty 200. Only 8080 is ever published or tunnelled; 80 and 443 stay in network. Probed live: spoofed Host on 8080 now answers 404 for the admin API, /rest/v1 and /storage/v1, where it previously reached each backend. scripts/test_caddy_supabase_routes.py pins all of it, in the shape of test_caddy_owui_blocklist.py. It was checked against three deliberate regressions rather than assumed: reordering the admin refusal after the proxy, returning the public site to the shared port, and exposing /rest/v1 publicly all turn it red, and it is green on the file as written. /auth/v1/invite joins the admin refusal. It is the one route upstream guards with requireAdminCredentials while leaving it outside the /admin group, so a leaked service key could still create users and send invitations from the internet. The CA mount only worked because edge-api ran as root. Caddy writes its pki directory 0700 root-owned, and Dockerfile.edge-api.prod, which is what buildx publishes, runs as uid 10001. A one-shot service now exports just the authority certificate into a volume of its own, world readable, and edge-api mounts only that. Proven by running the production image: it boots as uid 10001 and completes the JWKS fetch, while the same image against the old mount dies with "read jwks ca file: permission denied". That also ends the accepted risk about edge-api holding a private key, since the export contains one public certificate and nothing else that volume may come to hold. GoTrue's four rate limits were configured and never consulted: its limiter returns immediately when the header name is empty, and nothing set one. GOTRUE_RATE_LIMIT_HEADER is now set, and caddy-supabase overwrites that header inbound rather than appending, so a caller cannot hand itself a fresh bucket. Verified by hitting the limit: 35 requests to /auth/v1/otp answered 1 time 200 and 34 times 429. Smaller items from the same review: the JWKS cache now has an error sink, so a refresh that starts failing is logged instead of silently serving a stale key set until restart; the comments claiming the CA file is additive are corrected to say it replaces the system roots, which is what the code and its tests already did; ENTERPRISE_JWT_JWKS becomes ENTERPRISE_JWT_VERIFY_KEYS, since the old name reads like a publishable artifact while the value carries the HMAC secret; the three new ENTERPRISE_* toggles are documented in .env.example; and the registration script's usage passes the service key by env file rather than on a command line where ps and docker inspect would show it. Co-Authored-By: Claude Opus 5 --- .env.example | 31 ++++- Makefile | 1 + apps/edge-api/cmd/server/main.go | 18 ++- apps/edge-api/internal/auth/jwt_supabase.go | 19 ++- deploy/docker/Caddyfile.supabase | 59 ++++++-- deploy/docker/docker-compose.enterprise.yml | 89 +++++++++--- scripts/generate-enterprise-jwt-keys.py | 9 +- scripts/register-owui-oauth-client.py | 27 ++-- scripts/test_caddy_supabase_routes.py | 145 ++++++++++++++++++++ 9 files changed, 343 insertions(+), 55 deletions(-) create mode 100755 scripts/test_caddy_supabase_routes.py diff --git a/.env.example b/.env.example index e0ad5641e..fd1281fed 100644 --- a/.env.example +++ b/.env.example @@ -610,12 +610,12 @@ ENTERPRISE_JWT_EXP=3600 # ENTERPRISE_JWT_SECRET="" python3 scripts/generate-enterprise-jwt-keys.py # # ENTERPRISE_JWT_KEYS is the private signing set and goes to GoTrue only. -# ENTERPRISE_JWT_JWKS is the verification set (EC public key plus the legacy +# ENTERPRISE_JWT_VERIFY_KEYS is the verification set (EC public key plus the legacy # symmetric key) and goes to PostgREST and Storage, so both keep accepting the # existing anon and service_role keys while gaining the new ES256 user tokens. # Both are secret material. Never commit either. ENTERPRISE_JWT_KEYS= -ENTERPRISE_JWT_JWKS= +ENTERPRISE_JWT_VERIFY_KEYS= # Supabase API keys derived from ENTERPRISE_JWT_SECRET (see helper above). ENTERPRISE_ANON_KEY= @@ -652,6 +652,27 @@ ENTERPRISE_RATE_LIMIT_TOKEN_REFRESH=30 # Minimum password length. GoTrue default is 6; OSFI B-10 and PIPEDA require # at least 12 characters for regulated workloads. ENTERPRISE_PASSWORD_MIN_LENGTH=12 +# Header GoTrue keys its rate limits on. Without it none of the four limits +# above is ever consulted: GoTrue's limiter returns immediately when the header +# name is empty, so they read as configured while doing nothing. caddy-supabase +# overwrites this header inbound, so a caller cannot pick its own bucket. +ENTERPRISE_RATE_LIMIT_HEADER=X-Forwarded-For + +# ── Enterprise bring-up escape hatch, and two OAuth toggles ──────────────── +# +# ENTERPRISE_CUSTOM_ACCESS_TOKEN_HOOK_ENABLED: leave true. It is the source of +# the tenant_id, tenants and role claims that every authorisation check reads. +# Set false ONLY to stand a stack up before its schema is loaded, because +# GoTrue fails every token issuance when the hook function does not exist yet. +# A stack serving traffic with it false issues tenant-less tokens; edge-api +# refuses those (401, "missing principal claims"), so the cost is an outage +# rather than an isolation failure, but it is still not a setting to leave off. +ENTERPRISE_CUSTOM_ACCESS_TOKEN_HOOK_ENABLED=true +# OAuth 2.1 authorization server. Open WebUI signs in through it, so turning it +# off removes chat login. The consent path is a route on the web-console origin +# (GOTRUE_SITE_URL), where the consent page lives; GoTrue has no built-in one. +ENTERPRISE_OAUTH_SERVER_ENABLED=true +ENTERPRISE_OAUTH_CONSENT_PATH=/oauth/consent # ── EnterpriseEdge SSO: SAML 2.0 and OIDC providers (issue #237) ────────── # These variables are ONLY needed when running --profile enterprise with SSO. @@ -725,6 +746,12 @@ ENTERPRISE_SSO_MICROSOFT_TENANT_URL=https://login.microsoftonline.com/_default + --env-file /path/to/.env -e GOTRUE_ADMIN_URL=http://caddy-supabase/auth/v1 - -e SERVICE_ROLE_KEY="$ENTERPRISE_SERVICE_ROLE_KEY" -e OWUI_REDIRECT_URI=https://chat.example.com/oauth/oidc/callback -e HIVE_CHAT_URL=https://chat.example.com -v "$PWD/scripts:/s:ro" python:3.12-alpine python3 /s/register-owui-oauth-client.py -The network name is the compose project name plus "_default"; a stack started -as "-p hive" gives "hive_default". Point GOTRUE_ADMIN_URL somewhere else -instead if a deployment does publish the gateway, for instance at the public -https auth origin. +The env file needs SERVICE_ROLE_KEY; the enterprise .env already carries the +same value as ENTERPRISE_SERVICE_ROLE_KEY, so either add the alias or export it +from a wrapper. The network name is the compose project name plus "_default"; +a stack started as "-p hive" gives "hive_default". + +Keep GOTRUE_ADMIN_URL on the compose network. The service_role key travels in +an Authorization header, so a plain http URL that leaves the box puts an +omnipotent credential on the wire, and the public listener refuses +/auth/v1/admin/* by design anyway. Prints the two .env lines on success. The secret is shown exactly once, at creation, because GoTrue stores only its hash. Idempotent: a client already @@ -106,9 +114,10 @@ def api(base, path, token, method="GET", body=None): raw = exc.read().decode("utf-8", "replace") if exc.code == 404: raise SystemExit( - "GoTrue answered 404 for " + path + ". That endpoint exists only " - "when the OAuth server is enabled (GOTRUE_OAUTH_SERVER_ENABLED) " - "and only on v2.180.0 or newer. Check the image tag first." + "GoTrue answered 404 for " + path + ". Most likely this is pointed at " + "the PUBLIC gateway listener, which refuses /auth/v1/admin/* on purpose: " + "use the in-network address. Otherwise the OAuth server is off " + "(GOTRUE_OAUTH_SERVER_ENABLED) or the image predates v2.180.0." ) from None if exc.code in (401, 403): raise SystemExit( diff --git a/scripts/test_caddy_supabase_routes.py b/scripts/test_caddy_supabase_routes.py new file mode 100755 index 000000000..7c897b544 --- /dev/null +++ b/scripts/test_caddy_supabase_routes.py @@ -0,0 +1,145 @@ +#!/usr/bin/env python3 +"""Pin the Caddyfile.supabase route split against silent reopening. + +Three properties in this file are load bearing, and all three fail silently: +the config stays valid, Caddy starts, every legitimate request still works, +and the only difference is that something which should be refused is not. + + 1. The public site listens on its OWN port. While the public and internal + site blocks shared port 80, a request carrying "Host: caddy-supabase" + against the public port received the full internal route set: /rest/v1, + /storage/v1 and the admin API. Host matching is client supplied and case + insensitive, so it enforced nothing. A separate port does. + 2. The @admin refusal precedes the proxy inside the public snippet. handle + blocks are mutually exclusive and evaluated in written order, so swapping + those two blocks moves /auth/v1/admin/users from 404 to a proxied request + with nothing else observable changing, and no other test in this + repository fails. + 3. The public snippet exposes /auth/v1 alone. PostgREST connects to Postgres + as a superuser and a token's role claim selects a database role, so a + publicly reachable /rest/v1 is the whole schema behind whatever grants + happen to exist. Storage is the same argument for objects. + +Same shape as test_caddy_owui_blocklist.py: the file is parsed rather than +duplicated, so editing the Caddyfile is what this measures. No framework, no +Docker, no network. Run via `make test-scripts`. +""" + +import pathlib +import re +import sys + +CADDYFILE = pathlib.Path(__file__).resolve().parent.parent / "deploy" / "docker" / "Caddyfile.supabase" + +PUBLIC_SNIPPET = "supabase_public" +INTERNAL_SNIPPET = "supabase_internal" + +# Upstream guards /invite with requireAdminCredentials while leaving it outside +# the /admin group, so it needs naming separately. Re-read GoTrue's route list +# on every image bump: this list is tracking someone else's invariant. +REQUIRED_ADMIN_PATHS = ["/auth/v1/admin", "/auth/v1/admin/*", "/auth/v1/invite"] + +# Never reachable from the public listener without RLS policies and grants +# that do not exist yet. +FORBIDDEN_PUBLIC_PREFIXES = ["/rest/v1", "/storage/v1"] + +failures = [] + + +def fail(msg): + failures.append(msg) + + +def snippet_body(text, name): + """Return the body of a Caddy snippet definition, e.g. (name) { ... }.""" + start = text.index("(" + name + ") {") + depth = 0 + for i in range(start, len(text)): + if text[i] == "{": + depth += 1 + elif text[i] == "}": + depth -= 1 + if depth == 0: + return text[start:i + 1] + raise AssertionError("unbalanced braces in snippet " + name) + + +def main(): + text = CADDYFILE.read_text() + + public = snippet_body(text, PUBLIC_SNIPPET) + internal = snippet_body(text, INTERNAL_SNIPPET) + + # 1. The admin refusal has to come first, or it never runs. + admin_at = public.find("@admin") + proxy_at = public.find("handle_path /auth/v1/*") + if admin_at == -1: + fail("the public snippet has no @admin matcher; the admin API is reachable from outside") + if proxy_at == -1: + fail("the public snippet no longer proxies /auth/v1") + if admin_at != -1 and proxy_at != -1 and admin_at > proxy_at: + fail( + "the @admin handle now follows handle_path /auth/v1/*, so it never matches: " + "handle blocks are evaluated in written order and the proxy takes the request first" + ) + + # 2. Every admin-credentialed route upstream keeps outside /admin. + matcher = re.search(r"@admin\s+path\s+([^\n]+)", public) + if not matcher: + fail("could not read the @admin path matcher out of the public snippet") + else: + declared = matcher.group(1).split() + for want in REQUIRED_ADMIN_PATHS: + if want not in declared: + fail("the @admin matcher no longer covers " + want) + + # 3. Nothing but auth on the public listener. + for prefix in FORBIDDEN_PUBLIC_PREFIXES: + if prefix in public: + fail( + "the public snippet mentions " + prefix + ": that backend must not be reachable " + "from a browser without the RLS policies and grants to make it safe" + ) + for prefix in FORBIDDEN_PUBLIC_PREFIXES: + if prefix not in internal: + fail("the internal snippet no longer routes " + prefix + ", which in-stack callers need") + + # 4. The public site keeps its own listener port, and the internal sites + # keep theirs. Host matching alone is not enforcement. + public_site = re.search(r"http://\{\$SUPABASE_DOMAIN[^}]*\}(:\d+)?\s*\{", text) + if not public_site: + fail("could not find the public site block") + elif public_site.group(1) != ":8080": + fail( + "the public site no longer binds its own port (found " + + str(public_site.group(1)) + + "): sharing a port with the internal sites makes the route split depend on the " + "client sending an honest Host header" + ) + if not re.search(r"^:8080\s*\{", text, re.M): + fail("no catch-all on :8080; an unmatched Host there gets Caddy's empty 200") + for line in text.splitlines(): + stripped = line.strip() + if stripped.startswith(("http://caddy-supabase", "https://caddy-supabase")) and ":8080" in stripped: + fail("an internal site is bound to the public port 8080: " + stripped) + + # 5. The rate-limit key must not be caller-controlled. GoTrue keys its + # limits on this header, and Caddy appends to an inbound value unless + # told to replace it. + if "header_up X-Forwarded-For" not in public: + fail( + "the public proxy no longer overwrites X-Forwarded-For, so a caller can vary it and " + "hand itself a fresh GoTrue rate-limit bucket per request" + ) + + if failures: + print("Caddyfile.supabase route split: FAIL") + for f in failures: + print(" - " + f) + return 1 + print("Caddyfile.supabase route split: OK") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From 7286426fe6b310336f4f8e93dc2d0a07616ba237 Mon Sep 17 00:00:00 2001 From: Sakib Sadman Shajib Date: Wed, 19 Aug 2026 11:06:26 -0400 Subject: [PATCH 6/9] fix: refuse to register an OAuth client from a possibly truncated listing CodeRabbit, on the final diff. GET /admin/oauth/clients is paginated, so reading only the first page could miss an existing registration and mint a second client with a second secret, which is precisely what the idempotency check exists to prevent. Asks for a large page and refuses to register when the page comes back full, rather than looping. This deployment registers one client; failing closed beats guessing, and the comment names the loop as the upgrade if that changes. Co-Authored-By: Claude Opus 5 --- scripts/register-owui-oauth-client.py | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/scripts/register-owui-oauth-client.py b/scripts/register-owui-oauth-client.py index 13c88cddb..4bc68e3ea 100755 --- a/scripts/register-owui-oauth-client.py +++ b/scripts/register-owui-oauth-client.py @@ -190,9 +190,23 @@ def main(): payload = build_payload([redirect], client_uri) - _, listing = api(base, "/admin/oauth/clients", token) + # The listing is paginated. Reading only the first page and finding no + # match would register a SECOND client with a second secret, which is + # exactly what the idempotency check exists to prevent. + # ponytail: one large page plus a refusal, rather than a pagination loop. + # This deployment registers one client; if that ever stops being true the + # loop is the upgrade, and until then refusing beats guessing. + page_size = 1000 + _, listing = api(base, "/admin/oauth/clients?per_page=" + str(page_size), token) clients = listing.get("clients", listing) if isinstance(listing, dict) else listing - existing = find_existing(clients or [], payload) + clients = clients or [] + if len(clients) >= page_size: + raise SystemExit( + "GoTrue returned a full page of OAuth clients, so the listing may be " + "truncated and an existing registration could be missed. Refusing to " + "register rather than risk a duplicate client; check by hand." + ) + existing = find_existing(clients, payload) if existing: print("# Client already registered; leaving it alone. GoTrue stores only a hash") print("# of the secret, so the secret cannot be printed again. Rotate through") From 5aab5746c3de6fd836e611f5045b2268ba9d9bc6 Mon Sep 17 00:00:00 2001 From: Sakib Sadman Shajib Date: Wed, 19 Aug 2026 11:08:48 -0400 Subject: [PATCH 7/9] fix: make bucket init survive a second compose up Found by restarting the alongside stack on the demo box, which is the shape a cutover has. The Storage API answers a duplicate bucket with HTTP 400 carrying {"statusCode":"409","error":"Duplicate"} in the body, so the status-only check for 409 never matched and the init container exited 1 on every run after the first. edge-api and control-plane both wait on that container completing successfully, so a stack that came up cleanly the first time refused to start the second, with a bucket error that says nothing about restarts. The check now reads the response body as well as the status, treats a duplicate as success, and prints the body on a real failure so the next person is not guessing. Co-Authored-By: Claude Opus 5 --- deploy/docker/docker-compose.enterprise.yml | 24 ++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/deploy/docker/docker-compose.enterprise.yml b/deploy/docker/docker-compose.enterprise.yml index a5bebd0aa..59f528140 100644 --- a/deploy/docker/docker-compose.enterprise.yml +++ b/deploy/docker/docker-compose.enterprise.yml @@ -310,8 +310,15 @@ services: # ── supabase-init ────────────────────────────────────────────────────────── # One-shot container: creates the hive-files and hive-images buckets via the # Storage API. Runs once per compose up; restart: "no" prevents re-runs. - # Bucket creation is idempotent: a 409 from an existing bucket is treated as - # success (|| true) so re-runs after a container recreate do not fail. + # + # Re-running has to be a no-op, and getting that right needs the response + # BODY, not just the status. The Storage API answers a duplicate bucket with + # HTTP 400 carrying {"statusCode":"409","error":"Duplicate"}, so a status-only + # check for 409 never matches and the second `up` of any enterprise stack + # fails here. That is not cosmetic: edge-api and control-plane both wait on + # this container completing successfully, so a stack that came up cleanly the + # first time refuses to start the second, which is the restart shape a + # cutover has. supabase-init: # Pinned to immutable digest. Refresh: docker pull curlimages/curl:8.7.1 && docker inspect --format='{{index .RepoDigests 0}}' curlimages/curl:8.7.1 image: curlimages/curl:8.7.1@sha256:25d29daeb9b14b89e2fa8cc17c70e4b188bca1466086907c2d9a4b56b59d8e21 @@ -332,13 +339,20 @@ services: AUTH_HEADER="Authorization: Bearer $${SERVICE_KEY}" for BUCKET in hive-files hive-images; do echo "Creating bucket: $${BUCKET}" - HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$${STORAGE_URL}/bucket" \ + RESPONSE=$(curl -s -w '\n%{http_code}' -X POST "$${STORAGE_URL}/bucket" \ -H "Content-Type: application/json" \ -H "$${AUTH_HEADER}" \ -d "{\"id\":\"$${BUCKET}\",\"name\":\"$${BUCKET}\",\"public\":false}") + HTTP_STATUS=$(echo "$${RESPONSE}" | tail -n1) + BODY=$(echo "$${RESPONSE}" | sed '$$d') case "$${HTTP_STATUS}" in - 2*|409) echo " bucket $${BUCKET}: ok (status $${HTTP_STATUS})" ;; - *) echo " bucket $${BUCKET}: FAILED (status $${HTTP_STATUS})"; exit 1 ;; + 2*) echo " bucket $${BUCKET}: created" ;; + 400|409) + case "$${BODY}" in + *Duplicate*|*already*exists*) echo " bucket $${BUCKET}: already exists" ;; + *) echo " bucket $${BUCKET}: FAILED ($${HTTP_STATUS}) $${BODY}"; exit 1 ;; + esac ;; + *) echo " bucket $${BUCKET}: FAILED ($${HTTP_STATUS}) $${BODY}"; exit 1 ;; esac done echo "Bucket init complete." From d80af203dc6a982bd38027009ac5150372a21e65 Mon Sep 17 00:00:00 2001 From: Sakib Sadman Shajib Date: Thu, 20 Aug 2026 03:17:29 -0400 Subject: [PATCH 8/9] fix: make the route-split guard fail on the regression it exists for Delta security review. The guard added last commit could not catch the most natural form of its own regression, which is the second time on this branch a test has looked green over a broken protection. It anchored on "@admin", which finds the matcher DEFINITION. Caddy treats a named matcher's definition as position independent and orders only the handle directives, so moving just the `handle @admin` block below the proxy took /auth/v1/admin/* and /auth/v1/invite from refused to proxied while the test still printed OK. It also recognised sites by the literal name caddy-supabase, so a new block on the public port under any other name, or the catch-all importing the internal snippet, put the whole internal route set back on the public listener and passed. The test is now structural. It parses the file into snippets and site blocks, masking Caddy placeholders so a brace inside an address does not cut a header in half, and asks about ports and imports rather than names: nothing bound to the public port may import the internal snippet or proxy without importing the public one. It anchors on `handle @admin`, pins the rewritten header's VALUE rather than its name, and compares the header GoTrue is told to key on against the one the gateway actually rewrites, which are two settings in two files that must agree with nothing else noticing when they stop. Checked against nine deliberate regressions, each applied to the file, run, and reverted. All nine red, the file as written green. The first is the reviewer's: handle @admin moved below the proxy, definition left in place :8080 catch-all imports the internal snippet a differently named site on :8080 imports the internal snippet public site returned to the shared port /rest/v1 exposed on the public snippet X-Forwarded-For rewritten from another request header Sb-Forwarded-For no longer stripped /auth/v1/invite dropped from the admin matcher compose keys rate limits on a header the gateway does not rewrite Sb-Forwarded-For is now stripped at the gateway. GoTrue reads it BEFORE the configured header, gated on a flag that defaults false, so it was one flag away from handing the rate-limit bucket key back to the caller on a browser-facing auth service. Three comments corrected. The overwrite's stated reason was wrong for this pinned Caddy, which replaces rather than appends unless the peer is in trusted_proxies: the line stays, because the cutover will want trusted_proxies set and that is exactly the setting that turns appending back on. The OAuth client listing is not paginated at v2.189.0, so that guard is insurance against a future upstream change rather than a live fix, and its comment now says so. Co-Authored-By: Claude Opus 5 --- .env.example | 8 +- deploy/docker/Caddyfile.supabase | 38 ++- deploy/docker/docker-compose.enterprise.yml | 15 +- scripts/register-owui-oauth-client.py | 14 +- scripts/test_caddy_supabase_routes.py | 290 +++++++++++++++----- 5 files changed, 273 insertions(+), 92 deletions(-) diff --git a/.env.example b/.env.example index fd1281fed..5db57f4f9 100644 --- a/.env.example +++ b/.env.example @@ -654,8 +654,12 @@ ENTERPRISE_RATE_LIMIT_TOKEN_REFRESH=30 ENTERPRISE_PASSWORD_MIN_LENGTH=12 # Header GoTrue keys its rate limits on. Without it none of the four limits # above is ever consulted: GoTrue's limiter returns immediately when the header -# name is empty, so they read as configured while doing nothing. caddy-supabase -# overwrites this header inbound, so a caller cannot pick its own bucket. +# name is empty, so they read as configured while doing nothing. +# +# Leave the value alone unless the gateway changes with it. caddy-supabase +# rewrites X-Forwarded-For specifically, so naming a different header here +# keys GoTrue's limits on whatever the caller sends. The guard test compares +# the two and fails if they drift. ENTERPRISE_RATE_LIMIT_HEADER=X-Forwarded-For # ── Enterprise bring-up escape hatch, and two OAuth toggles ──────────────── diff --git a/deploy/docker/Caddyfile.supabase b/deploy/docker/Caddyfile.supabase index f281213b9..0ded98e22 100644 --- a/deploy/docker/Caddyfile.supabase +++ b/deploy/docker/Caddyfile.supabase @@ -103,19 +103,35 @@ } handle_path /auth/v1/* { reverse_proxy supabase-auth:9999 { - # Replace, never append. GoTrue keys its rate limits on this - # header (GOTRUE_RATE_LIMIT_HEADER), and Caddy's default is to - # append the client address to whatever the client already - # sent, so a caller could hand itself a fresh bucket per - # request just by varying the value it sends. + # GoTrue keys its rate limits on X-Forwarded-For + # (GOTRUE_RATE_LIMIT_HEADER), so what lands here decides whose + # bucket a request spends. # - # ponytail: {remote_host} is the immediate peer, so once a - # Cloudflare Tunnel fronts this listener every user shares one - # bucket. That is a stricter limit rather than a weaker one, so - # it is safe to ship; swap to the tunnel's CF-Connecting-IP - # when the tunnel is actually put in front, which is a cutover - # step and not a bring-up one. + # This Caddy already replaces the header rather than appending, + # because appending only happens for a peer listed in + # trusted_proxies and none is configured. The line is therefore + # not what makes the value trustworthy today; it makes it stay + # trustworthy. The cutover will want trusted_proxies set, so the + # tunnel's client address survives, and that is exactly the + # setting that turns appending back on. Leave this here through + # that change. + # + # Sb-Forwarded-For is stripped because GoTrue reads it BEFORE + # the configured header (performRateLimiting checks + # sbff.GetIPAddress first). It is gated on + # GOTRUE_SB_FORWARDED_FOR_ENABLED, which defaults false, so it + # is inert today and one flag away from handing the bucket key + # back to the caller. + # + # ponytail: {remote_host} is the immediate peer, so behind a + # tunnel this is ONE bucket for the whole deployment, not merely + # a stricter per-user one: 30 token refreshes an hour shared by + # everybody, and one user can lock out the rest. Acceptable + # while nothing is tunnelled here; at cutover set + # trusted_proxies and rewrite from the tunnel's client-IP header + # instead, keeping ENTERPRISE_RATE_LIMIT_HEADER in step. header_up X-Forwarded-For {remote_host} + header_up -Sb-Forwarded-For } } handle { diff --git a/deploy/docker/docker-compose.enterprise.yml b/deploy/docker/docker-compose.enterprise.yml index 59f528140..e0dd87cc4 100644 --- a/deploy/docker/docker-compose.enterprise.yml +++ b/deploy/docker/docker-compose.enterprise.yml @@ -174,9 +174,18 @@ services: # Without this, none of the four limits below is ever consulted: # GoTrue's performRateLimiting falls through to a header-keyed path that # returns immediately when the header name is empty, so the limits read - # as configured while doing nothing at all. caddy-supabase overwrites - # this header inbound (never appends), so its value cannot be chosen by - # the caller. + # as configured while doing nothing at all. + # + # The value has to be trustworthy, and today it is: this Caddy replaces + # a client-sent X-Forwarded-For rather than appending to it, because + # appending happens only for a peer listed in trusted_proxies and none + # is configured. caddy-supabase rewrites it explicitly anyway, so the + # guarantee survives the cutover, which will want trusted_proxies set. + # + # Changing this name alone breaks that: the gateway rewrites + # X-Forwarded-For specifically, so pointing GoTrue at another header + # keys its limits on something the caller sends. The two are checked + # against each other by scripts/test_caddy_supabase_routes.py. GOTRUE_RATE_LIMIT_HEADER: ${ENTERPRISE_RATE_LIMIT_HEADER:-X-Forwarded-For} GOTRUE_RATE_LIMIT_EMAIL_SENT: ${ENTERPRISE_RATE_LIMIT_EMAIL_SENT:-30} # GOTRUE_RATE_LIMIT_VERIFY: max OTP/magic-link verification attempts per hour. diff --git a/scripts/register-owui-oauth-client.py b/scripts/register-owui-oauth-client.py index 4bc68e3ea..9ed198c2c 100755 --- a/scripts/register-owui-oauth-client.py +++ b/scripts/register-owui-oauth-client.py @@ -190,12 +190,16 @@ def main(): payload = build_payload([redirect], client_uri) - # The listing is paginated. Reading only the first page and finding no - # match would register a SECOND client with a second secret, which is - # exactly what the idempotency check exists to prevent. + # Insurance, not a live fix: at v2.189.0 this handler carries an explicit + # "TODO(cemal) :: Add pagination" and returns every row unbounded, so + # per_page is ignored and the response is always complete. The refusal + # below therefore cannot fire on this pin. It stays because the day + # upstream does paginate, reading only the first page would silently + # register a SECOND client with a second secret, which is exactly what + # this check exists to prevent. # ponytail: one large page plus a refusal, rather than a pagination loop. - # This deployment registers one client; if that ever stops being true the - # loop is the upgrade, and until then refusing beats guessing. + # This deployment registers one client; the loop is the upgrade if that + # ever stops being true, and until then refusing beats guessing. page_size = 1000 _, listing = api(base, "/admin/oauth/clients?per_page=" + str(page_size), token) clients = listing.get("clients", listing) if isinstance(listing, dict) else listing diff --git a/scripts/test_caddy_supabase_routes.py b/scripts/test_caddy_supabase_routes.py index 7c897b544..bb7f5ea03 100755 --- a/scripts/test_caddy_supabase_routes.py +++ b/scripts/test_caddy_supabase_routes.py @@ -1,26 +1,37 @@ #!/usr/bin/env python3 """Pin the Caddyfile.supabase route split against silent reopening. -Three properties in this file are load bearing, and all three fail silently: -the config stays valid, Caddy starts, every legitimate request still works, -and the only difference is that something which should be refused is not. +Several properties in that file are load bearing, and every one of them fails +silently: the config stays valid, Caddy starts, every legitimate request still +works, and the only difference is that something which should be refused is +not. The cutover rests on them. - 1. The public site listens on its OWN port. While the public and internal + 1. Nothing on the public port reaches the internal route set. While the two site blocks shared port 80, a request carrying "Host: caddy-supabase" - against the public port received the full internal route set: /rest/v1, - /storage/v1 and the admin API. Host matching is client supplied and case - insensitive, so it enforced nothing. A separate port does. - 2. The @admin refusal precedes the proxy inside the public snippet. handle - blocks are mutually exclusive and evaluated in written order, so swapping - those two blocks moves /auth/v1/admin/users from 404 to a proxied request - with nothing else observable changing, and no other test in this - repository fails. - 3. The public snippet exposes /auth/v1 alone. PostgREST connects to Postgres - as a superuser and a token's role claim selects a database role, so a - publicly reachable /rest/v1 is the whole schema behind whatever grants - happen to exist. Storage is the same argument for objects. - -Same shape as test_caddy_owui_blocklist.py: the file is parsed rather than + against the public port received /rest/v1, /storage/v1 and the admin API, + because a client-supplied Host header was the only thing selecting + between them. + 2. The @admin refusal precedes the proxy INSIDE the public snippet. handle + directives are evaluated in written order, so moving that one block below + the proxy takes /auth/v1/admin/users and /auth/v1/invite from 404 to + proxied, with nothing else observable changing. + 3. The public snippet exposes /auth/v1 alone. PostgREST connects as a + superuser and a token's role claim selects a database role, so a publicly + reachable /rest/v1 is the whole schema behind whatever grants happen to + exist. Storage is the same argument for objects. + 4. The rate-limit bucket key cannot be chosen by the caller: X-Forwarded-For + is rewritten to the peer address, and Sb-Forwarded-For, which GoTrue reads + FIRST, is stripped. + +The checks are structural rather than name based, which matters more than it +sounds. An earlier version of this file looked for the string "@admin" and for +a site block spelled "caddy-supabase". Both passed while the protection was +gone: "@admin" finds the matcher DEFINITION, which Caddy treats as +order-independent, and a new site block on :8080 under any other name imports +whatever it likes. So this parses the file into snippets and site blocks and +asks about ports and imports. + +Same spirit as test_caddy_owui_blocklist.py: the file is parsed rather than duplicated, so editing the Caddyfile is what this measures. No framework, no Docker, no network. Run via `make test-scripts`. """ @@ -29,19 +40,29 @@ import re import sys -CADDYFILE = pathlib.Path(__file__).resolve().parent.parent / "deploy" / "docker" / "Caddyfile.supabase" +HERE = pathlib.Path(__file__).resolve().parent.parent +CADDYFILE = HERE / "deploy" / "docker" / "Caddyfile.supabase" +COMPOSE = HERE / "deploy" / "docker" / "docker-compose.enterprise.yml" PUBLIC_SNIPPET = "supabase_public" INTERNAL_SNIPPET = "supabase_internal" +PUBLIC_PORT = "8080" # Upstream guards /invite with requireAdminCredentials while leaving it outside # the /admin group, so it needs naming separately. Re-read GoTrue's route list -# on every image bump: this list is tracking someone else's invariant. +# on every image bump: this tracks someone else's invariant. At v2.189.0 +# requireAdminCredentials appears exactly twice, on /invite and on the group. REQUIRED_ADMIN_PATHS = ["/auth/v1/admin", "/auth/v1/admin/*", "/auth/v1/invite"] -# Never reachable from the public listener without RLS policies and grants -# that do not exist yet. -FORBIDDEN_PUBLIC_PREFIXES = ["/rest/v1", "/storage/v1"] +# Never reachable from the public listener without the RLS policies and grants +# that would make it safe, and there are none. +INTERNAL_ONLY_PREFIXES = ["/rest/v1", "/storage/v1"] + +# The bucket key GoTrue is told to use, and the value it must carry. +RATE_LIMIT_HEADER = "X-Forwarded-For" +RATE_LIMIT_VALUE = "{remote_host}" +# Read before the configured header by performRateLimiting, so it has to go. +STRIPPED_HEADER = "Sb-Forwarded-For" failures = [] @@ -50,40 +71,104 @@ def fail(msg): failures.append(msg) -def snippet_body(text, name): - """Return the body of a Caddy snippet definition, e.g. (name) { ... }.""" - start = text.index("(" + name + ") {") +def strip_comments(text): + """Drop whole-line comments so a commented-out directive cannot satisfy a + check, and a comment mentioning a prefix cannot trip one.""" + return "\n".join(l for l in text.splitlines() if not l.strip().startswith("#")) + + +# Caddy placeholders are brace delimited too ({$SUPABASE_DOMAIN:...}, +# {remote_host}), and counting them as block braces cuts a site header in half. +# They never contain whitespace, which is what separates them from a real +# block brace, so they are masked out for the depth walk and restored after. +PLACEHOLDER = re.compile(r"\{([^{}\s]*)\}") + + +def mask_placeholders(text): + return PLACEHOLDER.sub(lambda m: "\x01" + m.group(1) + "\x02", text) + + +def unmask_placeholders(text): + return text.replace("\x01", "{").replace("\x02", "}") + + +def parse_blocks(raw): + """Split a Caddyfile into (header, body) pairs at brace depth zero. + + Header is whatever precedes the opening brace: "(snippet_name)" for a + snippet, an address list for a site, empty for the global options block. + """ + text = mask_placeholders(raw) + blocks = [] depth = 0 - for i in range(start, len(text)): - if text[i] == "{": + header_start = 0 + body_start = None + for i, ch in enumerate(text): + if ch == "{": depth += 1 - elif text[i] == "}": + if depth == 1: + header = text[header_start:i].strip() + body_start = i + 1 + elif ch == "}": depth -= 1 if depth == 0: - return text[start:i + 1] - raise AssertionError("unbalanced braces in snippet " + name) + blocks.append((unmask_placeholders(header), unmask_placeholders(text[body_start:i]))) + header_start = i + 1 + elif depth < 0: + raise AssertionError("unbalanced closing brace in the Caddyfile") + if depth != 0: + raise AssertionError("unbalanced opening brace in the Caddyfile") + return blocks -def main(): - text = CADDYFILE.read_text() +def ports_of(address): + """The TCP port a single site address binds. + + ":8080" is a port-only address (any host). Otherwise an explicit :port + suffix wins, and failing that the scheme decides: https is 443, anything + else is 80. Placeholders such as {$SUPABASE_DOMAIN:supabase.localhost} may + contain colons, so the port is read from the tail after the last brace. + """ + addr = address.strip().rstrip(",") + if not addr: + return None + tail = addr[addr.rindex("}") + 1:] if "}" in addr else addr + m = re.search(r":(\d+)$", tail) + if m: + return m.group(1) + return "443" if addr.startswith("https://") else "80" + + +def snippet_body(blocks, name): + for header, body in blocks: + if header == "(" + name + ")": + return body + raise AssertionError("snippet " + name + " is missing from the Caddyfile") - public = snippet_body(text, PUBLIC_SNIPPET) - internal = snippet_body(text, INTERNAL_SNIPPET) - # 1. The admin refusal has to come first, or it never runs. - admin_at = public.find("@admin") +def imports(body): + return set(re.findall(r"^\s*import\s+(\S+)", body, re.M)) + + +def check_public_snippet(public): + # The admin refusal has to come first, or it never runs. Search for the + # handle DIRECTIVE, not the @admin matcher definition: Caddy orders + # directives, and treats a named matcher's definition as position + # independent, so anchoring on the definition passes while the protection + # is gone. + admin_at = public.find("handle @admin") proxy_at = public.find("handle_path /auth/v1/*") if admin_at == -1: - fail("the public snippet has no @admin matcher; the admin API is reachable from outside") + fail("the public snippet has no `handle @admin` block; the admin API is reachable from outside") if proxy_at == -1: fail("the public snippet no longer proxies /auth/v1") if admin_at != -1 and proxy_at != -1 and admin_at > proxy_at: fail( - "the @admin handle now follows handle_path /auth/v1/*, so it never matches: " - "handle blocks are evaluated in written order and the proxy takes the request first" + "`handle @admin` now follows `handle_path /auth/v1/*`, so it never matches: " + "handle directives are evaluated in written order and the proxy takes the " + "request first (/auth/v1/admin/* and /auth/v1/invite become proxied)" ) - # 2. Every admin-credentialed route upstream keeps outside /admin. matcher = re.search(r"@admin\s+path\s+([^\n]+)", public) if not matcher: fail("could not read the @admin path matcher out of the public snippet") @@ -93,45 +178,108 @@ def main(): if want not in declared: fail("the @admin matcher no longer covers " + want) - # 3. Nothing but auth on the public listener. - for prefix in FORBIDDEN_PUBLIC_PREFIXES: + for prefix in INTERNAL_ONLY_PREFIXES: if prefix in public: fail( - "the public snippet mentions " + prefix + ": that backend must not be reachable " - "from a browser without the RLS policies and grants to make it safe" + "the public snippet mentions " + prefix + ": that backend must not be " + "reachable from a browser without the RLS policies and grants to make it safe" ) - for prefix in FORBIDDEN_PUBLIC_PREFIXES: - if prefix not in internal: - fail("the internal snippet no longer routes " + prefix + ", which in-stack callers need") - # 4. The public site keeps its own listener port, and the internal sites - # keep theirs. Host matching alone is not enforcement. - public_site = re.search(r"http://\{\$SUPABASE_DOMAIN[^}]*\}(:\d+)?\s*\{", text) - if not public_site: - fail("could not find the public site block") - elif public_site.group(1) != ":8080": + # Pin the VALUE, not just the header name: rewriting it from another + # request header hands the rate-limit bucket straight back to the caller. + if (RATE_LIMIT_HEADER + " " + RATE_LIMIT_VALUE) not in public: fail( - "the public site no longer binds its own port (found " - + str(public_site.group(1)) - + "): sharing a port with the internal sites makes the route split depend on the " - "client sending an honest Host header" + "the public proxy no longer rewrites " + RATE_LIMIT_HEADER + " to " + + RATE_LIMIT_VALUE + ", so the GoTrue rate-limit bucket may be caller-chosen" ) - if not re.search(r"^:8080\s*\{", text, re.M): - fail("no catch-all on :8080; an unmatched Host there gets Caddy's empty 200") - for line in text.splitlines(): - stripped = line.strip() - if stripped.startswith(("http://caddy-supabase", "https://caddy-supabase")) and ":8080" in stripped: - fail("an internal site is bound to the public port 8080: " + stripped) - - # 5. The rate-limit key must not be caller-controlled. GoTrue keys its - # limits on this header, and Caddy appends to an inbound value unless - # told to replace it. - if "header_up X-Forwarded-For" not in public: + if ("-" + STRIPPED_HEADER) not in public: fail( - "the public proxy no longer overwrites X-Forwarded-For, so a caller can vary it and " - "hand itself a fresh GoTrue rate-limit bucket per request" + "the public proxy no longer strips " + STRIPPED_HEADER + ", which GoTrue reads " + "BEFORE the configured header, so enabling one GoTrue flag would let a caller " + "pick its own rate-limit bucket" ) + +def check_sites(blocks): + """Nothing bound to the public port may reach the internal route set. + + Structural on purpose: a new site block on :8080 under any name at all is + caught, because the question asked is about the port and the imports, not + about the hostname anyone happened to write. + """ + public_sites = [] + for header, body in blocks: + if not header or header.startswith("("): + continue + addresses = [a for a in re.split(r"[,\s]+", header) if a] + on_public = any(ports_of(a) == PUBLIC_PORT for a in addresses) + used = imports(body) + if on_public: + public_sites.append(header) + if INTERNAL_SNIPPET in used: + fail( + "site `" + header + "` is bound to the public port and imports " + + INTERNAL_SNIPPET + ", which puts /rest/v1, /storage/v1 and the admin " + "API back on the public listener" + ) + if PUBLIC_SNIPPET not in used and "reverse_proxy" in body: + fail( + "site `" + header + "` is bound to the public port and proxies without " + "importing " + PUBLIC_SNIPPET + ", so it bypasses the public route set" + ) + elif PUBLIC_SNIPPET in used: + fail("site `" + header + "` imports " + PUBLIC_SNIPPET + " off the public port") + + if not public_sites: + fail( + "no site is bound to port " + PUBLIC_PORT + ": the public route set has to live on " + "its own listener, or the split depends on the client sending an honest Host header" + ) + + domain_site = [h for h in public_sites if "SUPABASE_DOMAIN" in h] + if not domain_site: + fail("the SUPABASE_DOMAIN site is not bound to port " + PUBLIC_PORT) + + catch_all = [h for h in public_sites if h.strip() == ":" + PUBLIC_PORT] + if not catch_all: + fail( + "no catch-all on :" + PUBLIC_PORT + "; an unmatched Host there gets Caddy's " + "empty 200 instead of a refusal" + ) + + +def check_rate_limit_agreement(): + """GoTrue keys its limits on the header named in compose, and the Caddyfile + rewrites a header by name. Two settings that must agree, in two files, with + nothing else noticing when they stop.""" + compose = strip_comments(COMPOSE.read_text()) + m = re.search(r"GOTRUE_RATE_LIMIT_HEADER:\s*\$\{ENTERPRISE_RATE_LIMIT_HEADER:-([^}]+)\}", compose) + if not m: + fail("could not read the GOTRUE_RATE_LIMIT_HEADER default out of the enterprise compose file") + return + configured = m.group(1).strip() + if configured != RATE_LIMIT_HEADER: + fail( + "GoTrue is told to key its rate limits on " + configured + " while the gateway " + "rewrites " + RATE_LIMIT_HEADER + ", so the header GoTrue reads is whatever the " + "caller sent (ENTERPRISE_RATE_LIMIT_HEADER can do this silently at runtime too)" + ) + + +def main(): + text = strip_comments(CADDYFILE.read_text()) + blocks = parse_blocks(text) + + public = snippet_body(blocks, PUBLIC_SNIPPET) + internal = snippet_body(blocks, INTERNAL_SNIPPET) + + check_public_snippet(public) + for prefix in INTERNAL_ONLY_PREFIXES: + if prefix not in internal: + fail("the internal snippet no longer routes " + prefix + ", which in-stack callers need") + check_sites(blocks) + check_rate_limit_agreement() + if failures: print("Caddyfile.supabase route split: FAIL") for f in failures: From 44e6a81f11ea1592719e3bf0f786d91cd616a252 Mon Sep 17 00:00:00 2001 From: Sakib Sadman Shajib Date: Thu, 20 Aug 2026 03:32:01 -0400 Subject: [PATCH 9/9] fix: reject any direct proxy on the public port CodeRabbit, on the guard test. Rejecting a direct reverse_proxy only when the site omitted the public snippet left the obvious hole open: a site on the public port could import supabase_public and then add its own handle_path for /rest/v1, passing an imports-only check while publishing PostgREST. Every proxy definition on that port now has to live in the snippet that is actually reviewed. Added as a tenth mutation, verified red. Also renames a loop variable Ruff flags as ambiguous. Co-Authored-By: Claude Opus 5 --- scripts/test_caddy_supabase_routes.py | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/scripts/test_caddy_supabase_routes.py b/scripts/test_caddy_supabase_routes.py index bb7f5ea03..97fa5ca1a 100755 --- a/scripts/test_caddy_supabase_routes.py +++ b/scripts/test_caddy_supabase_routes.py @@ -74,7 +74,7 @@ def fail(msg): def strip_comments(text): """Drop whole-line comments so a commented-out directive cannot satisfy a check, and a comment mentioning a prefix cannot trip one.""" - return "\n".join(l for l in text.splitlines() if not l.strip().startswith("#")) + return "\n".join(line for line in text.splitlines() if not line.strip().startswith("#")) # Caddy placeholders are brace delimited too ({$SUPABASE_DOMAIN:...}, @@ -222,10 +222,15 @@ def check_sites(blocks): + INTERNAL_SNIPPET + ", which puts /rest/v1, /storage/v1 and the admin " "API back on the public listener" ) - if PUBLIC_SNIPPET not in used and "reverse_proxy" in body: + # Every proxy definition on this port has to live in the public + # snippet, which is the thing under review. A site that imports it + # and then adds its own handle_path would pass an "imports the + # right snippet" check while publishing whatever it likes. + if "reverse_proxy" in body: fail( - "site `" + header + "` is bound to the public port and proxies without " - "importing " + PUBLIC_SNIPPET + ", so it bypasses the public route set" + "site `" + header + "` is bound to the public port and proxies directly; " + "proxy definitions belong in the " + PUBLIC_SNIPPET + " snippet, where the " + "route set is reviewed, not in a site block beside it" ) elif PUBLIC_SNIPPET in used: fail("site `" + header + "` imports " + PUBLIC_SNIPPET + " off the public port")