Skip to content

Latest commit

 

History

History
187 lines (133 loc) · 10.6 KB

README.md

File metadata and controls

187 lines (133 loc) · 10.6 KB

Erlang CI Status - Alpha

ElectricSQL logo

Electric

This is the replication service for ElectricSQL.

It's an Elixir application that integrates with Postgres over logical replication and Satellite (ElectricSQL's client-side replication component that works with SQLite) via a web socket interface.

Pre-reqs

Docker and Elixir 1.14 compiled with Erlang 24.

Usage

See the Makefile for usage. Setup using:

make deps compile

Run the dependencies using:

make start_dev_env

Run the tests:

make tests

And then develop using:

make shell

This runs active-active replication with Postgres over logical replication and exposes a protocol buffers API over web sockets on localhost:5133.

For example to write some data into one of the Postgres instances:

docker exec -it -e PGPASSWORD=password electric_db_a_1 psql -h 127.0.0.1 -U electric -d electric

There's a second instance, electric-db_b_1, if you want to see data being replicated between them.

Note that you can tear down all the containers with:

make stop_dev_env

Running the release or docker container

The Electric application is configured using environment variables. Everything that doesn't have a default is required to run.

Variable Default Description
VAXINE_HOST Host of Vaxine instance to connect to
VAXINE_API_PORT 8087 Port for the regular DB API on Vaxine instance
VAXINE_REPLICATION_PORT 8088 Port for the replication API on Vaxine instance
VAXINE_CONNECTION_TIMEOUT 5000 (ms) Timeout waiting while connecting to a Vaxine instance
ELECTRIC_HOST Host of this electric instance for the reverse connection from Postgres. It has to be accessible from postgres instances listed in the CONNECTORS
CONNECTORS "" Semicolon-separated list of Postgres connection strings for PG instances that will be part of the cluster
POSTGRES_REPLICATION_PORT 5433 Port for connections from PG instances as replication followers
STATUS_PORT 5050 Port to expose health and status API endpoint
WEBSOCKET_PORT 5133 Port to expose the /ws path for the replication over the websocket
OFFSET_STORAGE_FILE ./offset_storage_data.dat Path to the file storing the mapping between connected instances and offsets in Vaxine WAL. Should be persisted between Electric restarts.
MIGRATIONS_DIR Directory to read the migration SQL files from (see below)
MIGRATIONS_FILE_NAME_SUFFIX /postgres.sql Suffix that is appended to the migration name when looking for the migration file
SATELLITE_AUTH_SIGNING_KEY "" Authentication token signing/validation secret key. See below.
SATELLITE_AUTH_SIGNING_ISS "" Cluster ID which acts as the issuer for the authentication JWT. See below.
GLOBAL_CLUSTER_ID Identifier of the cluster within the Electric cloud. When running locally, you can use any string
ELECTRIC_INSTANCE_ID Unique identifier of this Electric instance when running in the cluster. When running locally, you can use any string
ELECTRIC_REGIONAL_ID Shared identifier for multiple Electric instance that connect to the same Vaxine DC when running in the cluster. When running locally, you can use any string

Authentication

By default, in dev mode, electric uses insecure authentication. This just accepts a user id as the authentication token and authorizes the connection as that user.

Token based authentication requires a signed JWT token with a user_id claim, and a valid issuer.

To turn on token-based authentication in dev mode and when running in production, set the following environment variables:

  • SATELLITE_AUTH_SIGNING_KEY - Some random string used as the HMAC signing key. Must be at least 32 bytes long.

  • SATELLITE_AUTH_SIGNING_ISS - The JWT issuer (the iss field in the JWT).

You can generate a valid token using these configuration values by running mix electric.gen.token, e.g:

$ export SATELLITE_AUTH_SIGNING_KEY=00000000000000000000000000000000
$ export SATELLITE_AUTH_SIGNING_ISS=my.electric.server
$ mix electric.gen.token my_user my_other_user

The generated token(s) must be passed in the token field of the SatAuthReq protocol message.

For them to work, you must run the electric server configured with the same SATELLITE_AUTH_SIGNING_KEY and SATELLITE_AUTH_SIGNING_ISS set.

Migrations

When running locally, you can apply migrations directly using make apply_migration. First make sure you've built your migrations in your application folder, then set the MIGRATIONS_DIR environment variable to the path to the migrations folder:

export MIGRATIONS_DIR='../path/to/migrations'

Now (re)run the electric service (with the env var set):

make shell

You can now apply named migrations using:

make apply_migration name=$MIGRATION_NAME

Where MIGRATION_NAME is the name of a migration folder created using electric migrations new, for example:

make apply_migration name=1666288253_create_items

OSX

Note that if, when running on OSX, you get errors like:

could not connect to the publisher: connection to server at \"host.docker.internal\" (192.168.65.2), port 5433 failed

You may need to adjust your docker networking or run Electric within docker. To run within Docker, you can build the docker image locally:

make docker-build

And then run with the right env vars, e.g.:

docker run -it -p "5433:5433" -p "5133:5133" \
    -e "VAXINE_HOST=host.docker.internal"
    -e "ELECTRIC_HOST=host.docker.internal"
    -e "CONNECTORS=pg1=postgresql://electric:[email protected]:54321/electric;pg2=postgresql://electric:[email protected]:54322/electric" \
    docker.io/library/electric:local-build

Contributing

See the Community Guidelines including the Guide to Contributing and Contributor License Agreement.

Support

We have an open community Discord. If you’re interested in the project, please come and say hello and let us know if you have any questions or need any help or support getting things running.