Update auth configuration to match provisioning (#66)

* Update JWT auth configuration to match console * make integration tests docker compose v2 compatible * use new instance and regional ids * validate the length of the jwt key * add function to easily generate a token * start documentation of env variables and add a mix task to more easily generate authentication tokens * fix spec of format_tokens * commit to try to fix PR tests * add test for generate token task and tweak api * update instance and regional id config * fix tests * change pattern for successful start of vaxine * match both styles of app-up message why are the two so different? * try out-of-order match * add empty default because these are optional * emacs has a "format markdown table" function! * include global cluster id and make dev defaults mirror test defaults
electric-sql · Nov 24, 2022 · 2a36799 · 2a36799
1 parent 3f3f6f2
commit 2a36799
Show file tree

Hide file tree

Showing 23 changed files with 405 additions and 134 deletions.
diff --git a/README.md b/README.md
@@ -71,23 +71,58 @@ make stop_dev_env
 
 The Electric application is configured using environment variables. Everything that doesn't have a default is required to run.
 
-| Variable | Default | Description |
-| --- | --- | --- |
-| `VAXINE_HOST` | | Host of Vaxine instance to connect to |
-| `VAXINE_API_PORT` | `8087` | Port for the regular DB API on Vaxine instance |
-| `VAXINE_REPLICATION_PORT` | `8088` | Port for the replication API on Vaxine instance |
-| `VAXINE_CONNECTION_TIMEOUT` | `5000` | (ms) Timeout waiting while connecting to a Vaxine instance |
-|  |
-| `ELECTRIC_HOST` | | Host of this electric instance for the reverse connection from Postgres. It has to be accessible from postgres instances listed in the `CONNECTORS` |
-| `CONNECTORS` | `""` | Semicolon-separated list of Postgres connection strings for PG instances that will be part of the cluster |
-| |
-| `POSTGRES_REPLICATION_PORT` | `5433` | Port for connections from PG instances as replication followers |
-| `STATUS_PORT` | `5050` | Port to expose health and status API endpoint |
-| `WEBSOCKET_PORT` | `5133` | Port to expose the `/ws` path for the replication over the websocket |
-| |
-| `OFFSET_STORAGE_FILE` | `./offset_storage_data.dat` | Path to the file storing the mapping between connected instances and offsets in Vaxine WAL. Should be persisted between Electric restarts. |
-| `MIGRATIONS_DIR` | | Directory to read the migration SQL files from |
-| `MIGRATIONS_FILE_NAME_SUFFIX` | `/postgres.sql` | Suffix that is appended to the migration name when looking for the migration file |
+| Variable                      | Default                     | Description                                                                                                                                         |
+|-------------------------------|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|
+| `VAXINE_HOST`                 |                             | Host of Vaxine instance to connect to                                                                                                               |
+| `VAXINE_API_PORT`             | `8087`                      | Port for the regular DB API on Vaxine instance                                                                                                      |
+| `VAXINE_REPLICATION_PORT`     | `8088`                      | Port for the replication API on Vaxine instance                                                                                                     |
+| `VAXINE_CONNECTION_TIMEOUT`   | `5000`                      | (ms) Timeout waiting while connecting to a Vaxine instance                                                                                          |
+|                               |                             |                                                                                                                                                     |
+| `ELECTRIC_HOST`               |                             | Host of this electric instance for the reverse connection from Postgres. It has to be accessible from postgres instances listed in the `CONNECTORS` |
+| `CONNECTORS`                  | `""`                        | Semicolon-separated list of Postgres connection strings for PG instances that will be part of the cluster                                           |
+|                               |                             |                                                                                                                                                     |
+| `POSTGRES_REPLICATION_PORT`   | `5433`                      | Port for connections from PG instances as replication followers                                                                                     |
+| `STATUS_PORT`                 | `5050`                      | Port to expose health and status API endpoint                                                                                                       |
+| `WEBSOCKET_PORT`              | `5133`                      | Port to expose the `/ws` path for the replication over the websocket                                                                                |
+|                               |                             |                                                                                                                                                     |
+| `OFFSET_STORAGE_FILE`         | `./offset_storage_data.dat` | Path to the file storing the mapping between connected instances and offsets in Vaxine WAL. Should be persisted between Electric restarts.          |
+|                               |                             |                                                                                                                                                     |
+| `MIGRATIONS_DIR`              |                             | Directory to read the migration SQL files from (see below)                                                                                          |
+| `MIGRATIONS_FILE_NAME_SUFFIX` | `/postgres.sql`             | Suffix that is appended to the migration name when looking for the migration file                                                                   |
+|                               |                             |                                                                                                                                                     |
+| `SATELLITE_AUTH_SIGNING_KEY`  | `""`                        | Authentication token signing/validation secret key. See below.                                                                                      |
+| `SATELLITE_AUTH_SIGNING_ISS`  | `""`                        | Cluster ID which acts as the issuer for the authentication JWT. See below.                                                                          |
+
+**Authentication**
+
+By default, in dev mode, electric uses insecure authentication. This just
+accepts a user id as the authentication token and authorizes the connection as
+that user.
+
+Token based authentication requires a signed JWT token with a `user_id` claim,
+and a valid issuer.
+
+To turn on token-based authentication in dev mode and when running in
+production, set the following environment variables:
+
+- `SATELLITE_AUTH_SIGNING_KEY` - Some random string used as the HMAC signing
+  key. Must be at least 32 bytes long.
+
+- `SATELLITE_AUTH_SIGNING_ISS` - The JWT issuer (the `iss` field in the JWT). 
+
+You can generate a valid token using these configuration values by running `mix electric.gen.token`, e.g:
+
+``` shell
+$ export SATELLITE_AUTH_SIGNING_KEY=00000000000000000000000000000000 
+$ export SATELLITE_AUTH_SIGNING_ISS=my.electric.server
+$ mix electric.gen.token my_user my_other_user
+```
+
+The generated token(s) must be passed in the `token` field of the `SatAuthReq`
+protocol message.
+
+For them to work, you must run the electric server configured with the same
+`SATELLITE_AUTH_SIGNING_KEY` and `SATELLITE_AUTH_SIGNING_ISS` set.
 
 ## Migrations
 

diff --git a/config/config.exs b/config/config.exs
@@ -20,7 +20,8 @@ config :logger, :console,
     :sq_client,
     :vx_consumer,
     :vx_producer,
-    :cluster_id,
+    :instance_id,
+    :regional_id,
     :client_id,
     :user_id
   ]
@@ -40,4 +41,4 @@ config :electric, Electric.Satellite.Auth, provider: {Electric.Satellite.Auth.In
 
 # Import environment specific config. This must remain at the bottom
 # of this file so it overrides the configuration defined above.
-import_config "#{Mix.env()}.exs"
+import_config "#{config_env()}.exs"
diff --git a/config/dev.exs b/config/dev.exs
@@ -73,8 +73,33 @@ config :electric, Electric.Replication.SQConnectors,
   vaxine_connection_timeout: 5000
 
 config :electric,
-  global_cluster_id: System.get_env("GLOBAL_CLUSTER_ID", "electric-development-cluster-0000")
+  global_cluster_id: System.get_env("GLOBAL_CLUSTER_ID", "dev.electric-db"),
+  instance_id: System.get_env("ELECTRIC_INSTANCE_ID", "instance-1.region-1.dev.electric-db"),
+  regional_id: System.get_env("ELECTRIC_REGIONAL_ID", "region-1.dev.electric-db")
 
 config :logger, level: :debug
 
-config :electric, Electric.Satellite.Auth, provider: {Electric.Satellite.Auth.Insecure, []}
+auth_provider =
+  with {:ok, auth_key} <- System.fetch_env("SATELLITE_AUTH_SIGNING_KEY"),
+       {:ok, auth_iss} <- System.fetch_env("SATELLITE_AUTH_SIGNING_ISS") do
+    IO.puts("using JWT auth for issuer #{auth_iss}")
+
+    if byte_size(auth_key) >= 32 do
+      {Electric.Satellite.Auth.JWT, issuer: auth_iss, secret_key: auth_key}
+    else
+      IO.puts(
+        IO.ANSI.format([
+          :bright,
+          :red,
+          "SATELLITE_AUTH_SIGNING_KEY value needs to be 32 bytes or greater. Falling back to insecure auth"
+        ])
+      )
+
+      {Electric.Satellite.Auth.Insecure, []}
+    end
+  else
+    :error ->
+      {Electric.Satellite.Auth.Insecure, []}
+  end
+
+config :electric, Electric.Satellite.Auth, provider: auth_provider
diff --git a/config/runtime.exs b/config/runtime.exs
@@ -86,24 +86,14 @@ if config_env() == :prod do
     dir: System.fetch_env!("MIGRATIONS_DIR"),
     migration_file_name_suffix: System.get_env("MIGRATIONS_FILE_NAME_SUFFIX", "/postgres.sql")
 
-  # set to the database.cluster_slug
-  global_cluster_id = System.fetch_env!("GLOBAL_CLUSTER_ID")
-
   config :electric,
-    global_cluster_id: global_cluster_id
+    global_cluster_id: System.fetch_env!("GLOBAL_CLUSTER_ID"),
+    instance_id: System.fetch_env!("ELECTRIC_INSTANCE_ID"),
+    regional_id: System.fetch_env!("ELECTRIC_REGIONAL_ID")
 
-  # key = :crypto.strong_rand_bytes(32) |> Base.encode64()
-  auth_secret_key = System.fetch_env!("SATELLITE_AUTH_SIGNING_KEY") |> Base.decode64!()
+  auth_key = System.fetch_env!("SATELLITE_AUTH_SIGNING_KEY")
+  auth_iss = System.fetch_env!("SATELLITE_AUTH_SIGNING_ISS")
 
-  # 🐉 DANGER: this "issuer" configuration *MUST* be the same
-  # as the configuration in the console, currently under [:electric, :site_domain]
-  # I'm hard-coding this in all envs ATM  for simplicity
-  # if these config values do not match, the jwt token verification *will fail*
-  # safe option is probably to just remove the `iss` field from the token
   config :electric, Electric.Satellite.Auth,
-    provider:
-      {Electric.Satellite.Auth.JWT,
-       issuer: "electric-sql.com",
-       secret_key: Base.decode64!("AgT/MeUiP3SKzw5gC6BZKXk4t1ulnUvZy2d/O73R0sQ="),
-       global_cluster_id: global_cluster_id}
+    provider: {Electric.Satellite.Auth.JWT, issuer: auth_iss, secret_key: auth_key}
 end
diff --git a/config/test.exs b/config/test.exs
@@ -29,6 +29,13 @@ config :electric, Electric.Replication.SQConnectors,
 
 config :electric, Electric.Migrations, migration_file_name_suffix: "/postgres.sql"
 
-config :electric, global_cluster_id: "electric-development-cluster-0000"
-
-config :electric, Electric.Satellite.Auth, provider: {Electric.Satellite.Auth.Insecure, []}
+config :electric,
+  global_cluster_id: "test.electric-db",
+  instance_id: "instance-1.region-1.test.electric-db",
+  regional_id: "region-1.test.electric-db"
+
+config :electric, Electric.Satellite.Auth,
+  provider:
+    {Electric.Satellite.Auth.JWT,
+     issuer: "dev.electric-db",
+     secret_key: Base.decode64!("AgT/MeUiP3SKzw5gC6BZKXk4t1ulnUvZy2d/O73R0sQ=")}
diff --git a/integration_tests/common.mk b/integration_tests/common.mk
@@ -5,6 +5,7 @@ DOCKER_REGISTRY=europe-docker.pkg.dev/vaxine/vaxine-io
 export ELIXIR_VERSION=1.13.4
 export OTP_VERSION=24.3
 export DEBIAN_VERSION=bullseye-20210902-slim
+export COMPOSE_COMPATIBILITY=true
 
 export UID=$(shell id -u)
 export GID=$(shell id -g)

diff --git a/integration_tests/migrations/shared.luxinc b/integration_tests/migrations/shared.luxinc
@@ -14,7 +14,7 @@
 
 [shell vaxine]
     !make start_vaxine_1
-    ?vx_server started
+    ?(application: vx_server)|(vx_server started)
 
 [shell vaxine_wait]
     [invoke wait_port localhost 8088]

diff --git a/integration_tests/multi_dc/Makefile b/integration_tests/multi_dc/Makefile
@@ -16,6 +16,7 @@ build:
 			-e "s:{PG_PORT}:5432:g" \
 			-e "s:{EL_HOST}:electric_$${num}:g" \
 			-e "s:{EL_PORT}:5433:g" \
+			-e "s:{REGION}:$${num}:g" \
 		< electric.template > electric_$$num.exs; \
 	done
 

diff --git a/integration_tests/multi_dc/electric.template b/integration_tests/multi_dc/electric.template
@@ -35,3 +35,7 @@ config :electric, Electric.Replication.Connectors,
   ]
 
 config :logger, backends: [:console], level: :debug
+
+config :electric,
+  instance_id: "instance-a.region-{REGION}.test.electric-db",
+  regional_id: "region-{REGION}.test.electric-db"
diff --git a/integration_tests/multi_dc/shared.luxinc b/integration_tests/multi_dc/shared.luxinc
@@ -14,11 +14,11 @@
 
 [shell vaxine_1]
     !make start_vaxine_1
-    ?vx_server started
+    ?(application: vx_server)|(vx_server started)
 
 [shell vaxine_2]
     !make start_vaxine_2
-    ?vx_server started
+    ?(application: vx_server)|(vx_server started)
 
 [shell vaxine_wait]
     [invoke wait_port localhost 8087]

diff --git a/integration_tests/single_dc/electric.exs b/integration_tests/single_dc/electric.exs
@@ -71,11 +71,11 @@ config :electric, Electric.Replication.SQConnectors,
 config :logger, backends: [:console], level: :debug
 
 config :electric,
-  global_cluster_id: "fake-global-id-for-tests"
+  instance_id: "instance-a.region-1.test.electric-db",
+  regional_id: "region-1.test.electric-db"
 
 config :electric, Electric.Satellite.Auth,
     provider:
       {Electric.Satellite.Auth.JWT,
        issuer: "dev.electric-sql.com",
-       secret_key: Base.decode64!("AgT/MeUiP3SKzw5gC6BZKXk4t1ulnUvZy2d/O73R0sQ="),
-       global_cluster_id: "fake-global-id-for-tests"}
+       secret_key: Base.decode64!("AgT/MeUiP3SKzw5gC6BZKXk4t1ulnUvZy2d/O73R0sQ=")}
diff --git a/integration_tests/single_dc/electric_b.exs b/integration_tests/single_dc/electric_b.exs
@@ -41,12 +41,12 @@ config :electric, Electric.Replication.SQConnectors,
 
 config :logger, backends: [:console], level: :debug
 
-config :electric, Electric.Satellite,
-  global_cluster_id: "fake-global-id-for-tests"
+config :electric,
+  instance_id: "instance-b.region-1.test.electric-db",
+  regional_id: "region-1.test.electric-db"
 
 config :electric, Electric.Satellite.Auth,
     provider:
     {Electric.Satellite.Auth.JWT,
       issuer: "dev.electric-sql.com",
-      secret_key: Base.decode64!("AgT/MeUiP3SKzw5gC6BZKXk4t1ulnUvZy2d/O73R0sQ="),
-      global_cluster_id: "fake-global-id-for-tests"}
+      secret_key: Base.decode64!("AgT/MeUiP3SKzw5gC6BZKXk4t1ulnUvZy2d/O73R0sQ=")}
diff --git a/integration_tests/single_dc/shared.luxinc b/integration_tests/single_dc/shared.luxinc
@@ -14,7 +14,7 @@
 
 [shell vaxine]
     !make start_vaxine_1
-    ?vx_server started
+    ?(application: vx_server)|(vx_server started)
 
 [shell vaxine_wait]
     [invoke wait_port localhost 8088]

diff --git a/integration_tests/single_dc/sysbench_ws.lux b/integration_tests/single_dc/sysbench_ws.lux
@@ -43,9 +43,10 @@
 [shell ws1]
     [invoke log "Start WS client and start consuming data"]
     [invoke start_elixir_test]
+    # issuer and secret key here must be the same as the issuer in the
+    # config [:electric, Electric.Satellite.Auth, :provider]
     !provider = {Electric.Satellite.Auth.JWT, issuer: "dev.electric-sql.com", \
-                  secret_key: Base.decode64!("AgT/MeUiP3SKzw5gC6BZKXk4t1ulnUvZy2d/O73R0sQ="), \
-                  global_cluster_id: "fake-global-id-for-tests"}
+                  secret_key: Base.decode64!("AgT/MeUiP3SKzw5gC6BZKXk4t1ulnUvZy2d/O73R0sQ=")}
     ?$eprompt
     !Electric.Test.SatelliteWsClient.connect_and_spawn( \
             [ \
@@ -57,7 +58,7 @@
             {:host, "electric_1"}, \
             {:auto_ping, :true} \
             ])
-    ?$eprompt
+    ?+$eprompt
     ?(.*) %Electric.Satellite.SatInStartReplicationReq{__uf__: \[], lsn: "", (.*)
 
 [shell pg_2]
@@ -159,7 +160,7 @@
 #            {:host, "electric_1"}, \
 #            {:auto_ping, :true} \
 #            ])
-#    ?$eprompt
+#    ?+$eprompt
 #    ?(.*) %Electric.Satellite.SatInStartReplicationReq{__uf__: \[], lsn: "3", (.*)
 
 [cleanup]

diff --git a/lib/electric.ex b/lib/electric.ex
@@ -65,10 +65,20 @@ defmodule Electric do
   @doc """
   Every electric cluster belongs to a particular console database instance
 
-  This is that database instance slug
+  This is that database instance id
   """
-  @spec global_cluster_id() :: binary | no_return
-  def global_cluster_id do
-    Application.fetch_env!(:electric, :global_cluster_id)
+  @spec instance_id() :: binary | no_return
+  def instance_id do
+    Application.fetch_env!(:electric, :instance_id)
+  end
+
+  @doc """
+  Identifier that's unique for every electric cluster instance.
+
+  So basically region + instance_id
+  """
+  @spec regional_id() :: binary | no_return
+  def regional_id do
+    Application.fetch_env!(:electric, :regional_id)
   end
 end