A sample project demonstrating how to configure and run a Datomic Pro instance using local DynamoDB as storage. The storage, transactor and Datomic Console are running in Docker containers, while the peer runs locally.

Motivation

Now that Datomic is free and with the peer library published to maven, it’s easier than ever to get started with Datomic Pro (what used to be called on-prem). The official Datomic documentation recommends using what is known as the “dev” storage protocol which is easier to get started with, but less similar to an actual production Datomic deployment. The goal of this sample project is to setup a working local development environment that is closer to an actual production Datomic deployment, using DynamoDB as storage provider.

Provisioning local storage

Datomic needs to provision the storage provider so that it can be used by the transactor.

docker-compose run transactor /opt/datomic-pro/bin/datomic \
               ensure-transactor \
               /opt/datomic-pro/config/dev-transactor.properties \
               /opt/datomic-pro/config/dev-transactor.properties

It should output:

{:success /opt/datomic-pro/config/dev-transactor.properties}

A DynamoDB table with the name datomicpro-test will be created in the local DynamoDB service, which you can verify by running:

AWS_ACCESS_KEY_ID=dummy AWS_SECRET_ACCESS_KEY=dummy \
                        aws dynamodb list-tables \
                        --endpoint-url http://localhost:8000

Workflow

Start the storage and transactor services:

docker-compose up storage transactor

Optionally append a -d at the end to start as a background job.

In another terminal start a REPL. An example using Cider:

clj -Sdeps '{:deps {nrepl/nrepl {:mvn/version "1.0.0"} cider/cider-nrepl {:mvn/version "0.36.0"}}}' -M -m nrepl.cmdline --middleware '["cider.nrepl/cider-middleware"]'

Connect to the running REPL and you’re ready to use datomic. Alternatively you can just M-x cider-jack-in if using Emacs/Cider. Here’s a sample REPL session, included in the project as src/datomicpro_test/core.clj:

(require '[datomic.api :as d])

;; create the connection string
;; see docs https://docs.datomic.com/pro/clojure/index.html#datomic.api/connect
(def db-uri "datomic:ddb-local://localhost:8000/datomicpro-test/foo?aws_access_key_id=dummy&aws_secret_key=dummy")

;; create database `foo` as specified in the `db-uri`
(d/create-database db-uri)

;; list databases
;; note how the uri is the same, except the database part is replaced by `*`
(d/get-database-names "datomic:ddb-local://localhost:8000/datomicpro-test/*?aws_access_key_id=dummy&aws_secret_key=dummy")

(def movie-schema [{:db/ident :movie/title
                    :db/valueType :db.type/string
                    :db/cardinality :db.cardinality/one
                    :db/doc "The title of the movie"}

                   {:db/ident :movie/genre
                    :db/valueType :db.type/string
                    :db/cardinality :db.cardinality/one
                    :db/doc "The genre of the movie"}

                   {:db/ident :movie/release-year
                    :db/valueType :db.type/long
                    :db/cardinality :db.cardinality/one
                    :db/doc "The year the movie was released in theaters"}])

;; create a connection
(def conn (d/connect db-uri))

;; transacts the schema. Returns a completed future.
@(d/transact conn movie-schema)

;; sample movie data
(def first-movies [{:movie/title "The Goonies"
                    :movie/genre "action/adventure"
                    :movie/release-year 1985}
                   {:movie/title "Commando"
                    :movie/genre "action/adventure"
                    :movie/release-year 1985}
                   {:movie/title "Repo Man"
                    :movie/genre "punk dystopia"
                    :movie/release-year 1984}])

;; transact movies
@(d/transact conn first-movies)

;; get the current database
(def db (d/db conn))

;; all data from 1985 query
(def all-data-from-1985 '[:find ?title ?year ?genre
                          :where [?e :movie/title ?title]
                          [?e :movie/release-year ?year]
                          [?e :movie/genre ?genre]
                          [?e :movie/release-year 1985]])

(d/q all-data-from-1985 db)
;; => #{["The Goonies" 1985 "action/adventure"] ["Commando" 1985 "action/adventure"]}

;; get id of the `Commando` movie
(def commando-id
  (ffirst (d/q '[:find ?e
                 :where [?e :movie/title "Commando"]]
               db)))

;; update the genre of `Commando`
@(d/transact conn [{:db/id commando-id :movie/genre "future governor"}])

;; get a new db
(def updated-db (d/db conn))

(d/q all-data-from-1985 updated-db)
;; => #{["The Goonies" 1985 "action/adventure"] ["Commando" 1985 "future governor"]}

(d/q all-data-from-1985 db)
;; => #{["The Goonies" 1985 "action/adventure"] ["Commando" 1985 "action/adventure"]}

Datomic Console

The Datomic Console is also included. In can be executed as a separate service and accessed locally through your browser. To run the service:

docker-compose up datomic-console

The output should be:

datomic-console  | Console started on port: 9000
datomic-console  |    dev = datomic:ddb-local://storage:8000/datomicpro-test/
datomic-console  | Open http://localhost:9000/browse in your browser (Chrome recommended)

The datomic console should be available at http://localhost:9000/browse.

How it works / Hacking on it

The system is composed of three services specified, each in its own Docker container (see docker-compose.yml):

1. storage: Local DynamoDB instance. Binds to port 8000.
2. transactor: The Datomic Pro transactor. Binds to port 4334.
3. datomic-console: The Datomic Console. Binds to port 9000.

The Peer is embedded in your own code, running locally. Each service can discover each other using the Docker network.

The transactor is configured to bind to 0.0.0.0 and with alt-host=transactor. The alt-host setting is needed so that the console works correctly. Failing to set the alt-host correctly will cause the console to throw an exception when selecting a database:

datomic-console  | ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219007: Cannot connect to server(s). Tried with all available servers.]

Transactor configuration

The file dev-transactor.properties is used by the transactor. See the volumes section in the transactor service definition:

volumes:
  - ./dev-transactor.properties:/opt/datomic-pro/config/dev-transactor.properties

Connection strings

A connection string for a local DynamoDB provisioned storage looks like this:

datomic:ddb-local://localhost:8000/datomicpro-test/foo?aws_access_key_id=dummy&aws_secret_key=dummy

Dissecting each part:

• datomic:ddb-local: A datomic system using DynamoDB local as storage. An AWS-backed DynamoDB would use datomic:ddb instead.
• localhost:8000: Where to reach DynamoDB local. It has to match your Docker configuration.
• datomicpro-test: The name of the DynamoDB table.
• foo: Name of the Datomic database to use or create. Replace with * when listing all databases.
• ?aws_access_key_id=dummy&aws_secret_key=dummy: Dummy AWS credentials to use, needed when using DynamoDB local.

When in a REPL session, the connection string specifies localhost as endpoint. This is because our code is running locally and the storage service binds itself to localhost:8000.

However the console is running inside a docker container. There, the storage service is reachable as storage:8000 instead. Also note that it’s not necessary to specify AWS credentials in the connection string, since those are passed as environment variables as specified in the service definition in docker-compose.yml.

To update the Datomic version, change transactor/Dockerfile and rebuild the image.