From 691a7de01a2ab0ef36e2d735482294469e40b0b4 Mon Sep 17 00:00:00 2001 From: sdg Date: Fri, 20 Dec 2019 16:48:13 +0100 Subject: [PATCH 1/3] First draft for improving quickstart Co-authored-by: samuelhamard --- README.rst | 42 ++++++- docs/howto/configure.rst | 10 ++ docs/howto/file_naming.rst | 16 +++ docs/quickstart.rst | 235 +++++++++++++++++++++++++++++++++++++ 4 files changed, 300 insertions(+), 3 deletions(-) create mode 100644 docs/howto/configure.rst create mode 100644 docs/howto/file_naming.rst diff --git a/README.rst b/README.rst index 5332ec5..a8541b3 100644 --- a/README.rst +++ b/README.rst @@ -45,16 +45,49 @@ Septentrion supports PostgreSQL 9.6+ and Python 3.6+. .. _South: https://bitbucket.org/andrewgodwin/south/src .. _django-north: https://github.com/peopledoc/django-north -Overview --------- +Very quick start +---------------- - *Step 1*: Write your PostgreSQL migrations, name files and folders according to a convention. + See the folder `example_migrations` for an example: + +.. code-block:: console + + example_migrations/ + ├── 0.1 + ├── 1.0 + │   ├── 1.0-0-version-dml.sql + │   ├── 1.0-author-1-ddl.sql + │   ├── 1.0-author-2-dml.sql + │   ├── 1.0-book-1-ddl.sql + │   └── 1.0-book-2-dml.sql + ├── 1.1 + │   ├── 1.1-0-version-dml.sql + │   ├── 1.1-add-num-pages-1-ddl.sql + │   ├── 1.1-add-num-pages-2-dml.sql + │   └── 1.1-index-ddl.sql + ├── 1.2 + │   ├── 1.2-0-version-dml.sql + │   ├── 1.2-remove-author-dob-ddl.sql + │   └── 1.2-rename-num-pages-ddl.sql + ├── 1.3 + │   ├── 1.3-0-version-dml.sql + │   ├── 1.3-add-readers-ddl.sql + │   ├── 1.3-add-readers-dml.sql + │   ├── 1.3-remove-author-dob-ddl.sql + │   └── 1.3-rename-num-pages-ddl.sql + ├── fixtures + │   ├── fixtures_0.1.sql + │   └── fixtures_1.0.sql + └── schemas + └── schema_0.1.sql + - *Step 2*: .. code-block:: console - $ septentrion migrate + $ septentrion --target-version 1.2 migrate - *Step 3*: That's it. @@ -71,8 +104,11 @@ Where to go from here The complete docs_ is probably the best place to learn about the project. +You can check the quickstart_ guide to start running your first migrations. + If you encounter a bug, or want to get in touch, you're always welcome to open a ticket_. .. _docs: http://septentrion.readthedocs.io/en/latest +.. _quickstart: http://septentrion.readthedocs.io/en/latest/quickstart.html .. _ticket: https://github.com/peopledoc/septentrion/issues/new diff --git a/docs/howto/configure.rst b/docs/howto/configure.rst new file mode 100644 index 0000000..f22480c --- /dev/null +++ b/docs/howto/configure.rst @@ -0,0 +1,10 @@ +Configure database connection settings +-------------------------------------- + +Ensure that your database connection settings are correctly configured. + + - Either your environment variables `PGHOST`, `PGPORT` and `PGUSER` are properly set. + - Or you can set the environment variables `SEPTENTRION_HOST`, `SEPTENTRION_PORT`, `SEPTENTRION_USERNAME`. + - configuration file `septentrion.ini` + - If you don't want to use environment variables, you can use the `--host`, `--port` or `--username` options + when running the migration. \ No newline at end of file diff --git a/docs/howto/file_naming.rst b/docs/howto/file_naming.rst new file mode 100644 index 0000000..6ba502e --- /dev/null +++ b/docs/howto/file_naming.rst @@ -0,0 +1,16 @@ +File naming convention +---------------------- + +We recommend to use the following naming convention for migration files: + +.. code-block:: console + + [VERSION]-[MIGRATION]-[ORDER]-[TYPE].sql + + - version : the release version + + - migration : a free name for the migration + + - order : a number used to order the files inside the migration + + - type : ddl or dml, ddl if the file contains schema migration, dml if the file contains data migration. \ No newline at end of file diff --git a/docs/quickstart.rst b/docs/quickstart.rst index 33dc72c..3cde437 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -1,6 +1,241 @@ Quickstart ========== +Welcome to the *Septentrion* quickstart documentation. + +In this tutorial, you will learn how to write migrations in the expected format, how to visualize them and how to run +them against a PostgreSQL database. + +Prepare your database +--------------------- + +If you already have a PostgreSQL database around, make sure to note the connection +parameters. Otherwise, we'll create one together with Docker_: + +.. _Docker: https://docs.docker.com/ + +.. code-block:: console + + $ docker run --detach --rm -p 5432:5432 --name septentrion_db postgres + +.. note:: + + If you need to stop the docker at some point, use ``docker stop septentrion_db``. + +.. note:: + + If you stop the Docker or reboot your machine, the Postgres database will be trashed and you will need to re-create + one and add some data again. + + +You'll also need ``psycopg2``, which is notoriously complex to install. Septentrion +will install the ``psycopg2`` python package, but will expect the system to already +have the prerequisites (on ``Ubuntu``):: + + $ sudo apt install libpq-dev python-dev postgresql-client + +.. note:: + On a different OS, if you experiment difficulties, we'll be grateful if you can tell + us via an issue so that we improve this documentation. + +Install and configure Septentrion +--------------------------------- + +Within a virtualenv_, install Septentrion with: + +.. _virtualenv: https://packaging.python.org/tutorials/installing-packages/#creating-virtual-environments + +.. code-block:: console + + (venv) $ pip install septentrion + +Next we will configure the connection to the PostgreSQL database. We can do this either with command line flags, +environment variables or a configuration file. +In this tutorial, we will use a configuration file, ``septentrion.ini``. + +.. code-block:: ini + + [septentrion] + migrations_root=migrations + host=localhost + username=postgres + # port=5432 + +.. note:: + + With the Docker setup described above, you should be good to go. + If you need additional configuration parameters to connect to your database, have a look at + :ref:`advanced configuration options `. + + +Write migrations +---------------- + +Migrations are SQL files that are executed in order. Migrations are grouped together in successive versions. +All migrations in the same version are placed in a folder named after the version number. + +Let's create our migration folder, and a first version folder. + +.. code-block:: console + + $ mkdir migrations + $ mkdir migrations/1.0 + +Now we can add a migration file in ``migrations/1.0/`` named ``00-author.ddl.sql``: + +.. code-block:: sql + + BEGIN; + CREATE TABLE "author" ( + "id" serial NOT NULL PRIMARY KEY, + "name" varchar(100) NOT NULL, + "birth_date" varchar(10) NOT NULL + ); + COMMIT; + +.. note:: + + Migrations are executed in alphabetical order, so an easy way to control the order is to prefix filenames with two + digits. + +.. note:: + + ``ddl`` stands for *Data Definition Language* and corresponds to all the operations that change the structure of the + database (``CREATE``, ``DROP``, ...). + +Congratulations, you have now written your first *Septentrion* migration. Let's see how to run it! + + +Run migrations +-------------- + +First, we want to visualize what is going to happen, without making any change to our data yet. + +.. code-block:: console + + $ septentrion --target-version 1.0 show-migrations + + Current version is None + Target version is 1.0 + Version 1.0 + [ ] 00-author.ddl.sql + + +# TODO add a schema_0.1.sql else septentrion is crashing + +Great, we can now run it for real: + +.. code-block:: console + + $ septentrion --target-version 1.0 migrate + + Loading schema + [X] Applied 0.1 + Applying migrations + Version 0.1 + Version 1.0 + + + +.. note:: + + You should run *septentrion* in your root directory, where your ``migrations`` folder is. + +.. note:: + + The ``--target-version`` flag is a required option (it might change in the future). + + +If something is not working as it should be, you probably want to check the :ref:`troubleshooting guide ` +or the :ref:`advanced options `. + +At this point, the ``author`` table has been created in the database. We can check that and simulate our application +by creating a few rows in the table. + + +.. code-block:: console + + $ psql --host localhost --user postgres postgres + + postgres=# \d + + List of relations + Schema | Name | Type | Owner + --------+-------------------------------+----------+---------- + public | author | table | postgres + public | author_id_seq | sequence | postgres + public | septentrion_migrations | table | postgres + public | septentrion_migrations_id_seq | sequence | postgres + public | sql_version | table | postgres + (5 rows) + + postgres=# INSERT INTO author (name, birth_date) + VALUES + ('Victor Hugo', '1802-02-26'), + ('George Gordon Byron', '1788-01-22'), + ('JRR Tolkien', '1892-01-03'); + INSERT 0 3 + + postgres=# SELECT * FROM author; + id | name | birth_date + ----+---------------------+--------------- + 0 | Victor Hugo | 1802-02-26 + 1 | George Gordon Byron | 1788-01-22 + 2 | JRR Tolkien | 1892-01-03 + (3 rows) + + + + +More complex migrations +----------------------- + +For version ``2.0`` of our application, we want to change ``birth_date`` from ``varchar`` to the ``date`` type. + +We create a new folder for the version. + +.. code-block:: console + + $ mkdir migrations/2.0 + +We can add the migration file in ``migrations/2.0/`` named ``00-change_birth_date_type.ddl.sql``: + +.. code-block:: sql + + BEGIN; + ALTER TABLE author + ALTER COLUMN birth_date + TYPE DATE USING to_date(birth_date, 'YYYY-MM-DD'); + COMMIT; + + +We launch the migration. + +.. code-block:: console + + $ septentrion --target-version 2.0 migrate + + Applying migrations + Version 0.1 + Version 1.0 + [X] Already applied + Version 2.0 + [X] Applied 00-change_birth_date_type.ddl.sql + +Now we can check that our migration successfully changed the column type in the ``author`` table. + +.. code-block:: console + + $ psql --host localhost --user postgres postgres + postgres=# SELECT column_name, data_type FROM information_schema.columns WHERE table_name = 'author'; + column_name | data_type + ---------------+------------------- + id | integer + name | character varying + date_of_birth | date + (3 rows) + + Going further ------------- From 79c5522dd9b97157eb2c50b1d7d783c523fe9811 Mon Sep 17 00:00:00 2001 From: sdg Date: Mon, 3 Feb 2020 18:00:14 +0100 Subject: [PATCH 2/3] Update very quick start --- README.rst | 43 +++++++++---------------------------------- 1 file changed, 9 insertions(+), 34 deletions(-) diff --git a/README.rst b/README.rst index a8541b3..5ea82d9 100644 --- a/README.rst +++ b/README.rst @@ -48,46 +48,21 @@ Septentrion supports PostgreSQL 9.6+ and Python 3.6+. Very quick start ---------------- -- *Step 1*: Write your PostgreSQL migrations, name files and folders according to - a convention. - See the folder `example_migrations` for an example: +- *Step 1*: Create a folder for the version, and add some migration files. .. code-block:: console - example_migrations/ - ├── 0.1 - ├── 1.0 - │   ├── 1.0-0-version-dml.sql - │   ├── 1.0-author-1-ddl.sql - │   ├── 1.0-author-2-dml.sql - │   ├── 1.0-book-1-ddl.sql - │   └── 1.0-book-2-dml.sql - ├── 1.1 - │   ├── 1.1-0-version-dml.sql - │   ├── 1.1-add-num-pages-1-ddl.sql - │   ├── 1.1-add-num-pages-2-dml.sql - │   └── 1.1-index-ddl.sql - ├── 1.2 - │   ├── 1.2-0-version-dml.sql - │   ├── 1.2-remove-author-dob-ddl.sql - │   └── 1.2-rename-num-pages-ddl.sql - ├── 1.3 - │   ├── 1.3-0-version-dml.sql - │   ├── 1.3-add-readers-ddl.sql - │   ├── 1.3-add-readers-dml.sql - │   ├── 1.3-remove-author-dob-ddl.sql - │   └── 1.3-rename-num-pages-ddl.sql - ├── fixtures - │   ├── fixtures_0.1.sql - │   └── fixtures_1.0.sql - └── schemas - └── schema_0.1.sql - -- *Step 2*: + migrations/ + └── 1.0 +   ├── 1.0-0-version-dml.sql +    ├── 1.0-author-1-ddl.sql +    └── 1.0-author-2-dml.sql + +- *Step 2*: Run septentrion .. code-block:: console - $ septentrion --target-version 1.2 migrate + $ septentrion --target-version 1.0 migrate - *Step 3*: That's it. From e008a6a8c9c235774490a6664881f419c2c4b3b9 Mon Sep 17 00:00:00 2001 From: sdg Date: Mon, 3 Feb 2020 18:15:13 +0100 Subject: [PATCH 3/3] Add placeholder files in howto --- docs/howto/advanced-options.rst | 1 + docs/howto/configure.rst | 2 ++ docs/howto/file_naming.rst | 3 +++ docs/howto/troubleshoot.rst | 1 + docs/quickstart.rst | 21 +++++++++------------ 5 files changed, 16 insertions(+), 12 deletions(-) create mode 100644 docs/howto/advanced-options.rst create mode 100644 docs/howto/troubleshoot.rst diff --git a/docs/howto/advanced-options.rst b/docs/howto/advanced-options.rst new file mode 100644 index 0000000..bb9f609 --- /dev/null +++ b/docs/howto/advanced-options.rst @@ -0,0 +1 @@ +# TODO This file will list advanced options for septentrion usage (eg using flags or env variables instead of a .ini file) \ No newline at end of file diff --git a/docs/howto/configure.rst b/docs/howto/configure.rst index f22480c..5199ef3 100644 --- a/docs/howto/configure.rst +++ b/docs/howto/configure.rst @@ -1,6 +1,8 @@ Configure database connection settings -------------------------------------- +# TODO expand documentation on database connection + Ensure that your database connection settings are correctly configured. - Either your environment variables `PGHOST`, `PGPORT` and `PGUSER` are properly set. diff --git a/docs/howto/file_naming.rst b/docs/howto/file_naming.rst index 6ba502e..992d9c2 100644 --- a/docs/howto/file_naming.rst +++ b/docs/howto/file_naming.rst @@ -1,6 +1,9 @@ File naming convention ---------------------- +# TODO explain the file naming convention +# The following extract was inspired by the DBA documentation. + We recommend to use the following naming convention for migration files: .. code-block:: console diff --git a/docs/howto/troubleshoot.rst b/docs/howto/troubleshoot.rst new file mode 100644 index 0000000..fd132e4 --- /dev/null +++ b/docs/howto/troubleshoot.rst @@ -0,0 +1 @@ +# TODO This file will contain help with the most common issues. \ No newline at end of file diff --git a/docs/quickstart.rst b/docs/quickstart.rst index 3cde437..28b8445 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -65,7 +65,7 @@ In this tutorial, we will use a configuration file, ``septentrion.ini``. With the Docker setup described above, you should be good to go. If you need additional configuration parameters to connect to your database, have a look at - :ref:`advanced configuration options `. + :ref:`advanced configuration options `. Write migrations @@ -121,7 +121,6 @@ First, we want to visualize what is going to happen, without making any change t [ ] 00-author.ddl.sql -# TODO add a schema_0.1.sql else septentrion is crashing Great, we can now run it for real: @@ -129,10 +128,7 @@ Great, we can now run it for real: $ septentrion --target-version 1.0 migrate - Loading schema - [X] Applied 0.1 Applying migrations - Version 0.1 Version 1.0 @@ -146,8 +142,8 @@ Great, we can now run it for real: The ``--target-version`` flag is a required option (it might change in the future). -If something is not working as it should be, you probably want to check the :ref:`troubleshooting guide ` -or the :ref:`advanced options `. +If something is not working as it should be, you probably want to check the :ref:`troubleshooting guide ` +or the :ref:`advanced options `. At this point, the ``author`` table has been created in the database. We can check that and simulate our application by creating a few rows in the table. @@ -186,9 +182,8 @@ by creating a few rows in the table. - -More complex migrations ------------------------ +A more complex migration +------------------------ For version ``2.0`` of our application, we want to change ``birth_date`` from ``varchar`` to the ``date`` type. @@ -216,7 +211,6 @@ We launch the migration. $ septentrion --target-version 2.0 migrate Applying migrations - Version 0.1 Version 1.0 [X] Already applied Version 2.0 @@ -236,10 +230,13 @@ Now we can check that our migration successfully changed the column type in the (3 rows) + +Congratulations, you can now run migrations with *Septentrion*! + Going further ------------- -To continue with practical steps, head to the :ref:`How-to... ` section. +To continue with practical steps, head to the :ref:`How-to... ` section. If you want to better understand some design decisions, head to the :ref:`Discussions ` sections.