Add Docker Compose Support by jalessio · Pull Request #2406 · openstreetmap/openstreetmap-website

jalessio · 2019-10-20T02:19:35Z

This PR is primarily intended as a way of concretely demonstrating potential improvements / responding to reviewers in PR #2272 by @fazlerabbi37.

Notable changes

Reorganized the directory structure based on experience using this structure successfully on several other projects.
~~Use a postgis Docker image instead of postgres since we definitely need PostGIS.~~
Combine RUN commands to reduce the number of layers in Docker images.
~~Don't compile Ruby or install gems in the db image. We just need a single header file in order to build libpgosm.so.~~
Turns out the docker_postgres.sh script which installs the OSM-specific Postgres extensions wasn't actually being run because of a missing slash. Fixed that _(and renamed the file to include openstreetmap in the name to make its purpose clear).
Switched from ruby:2.5-slim to ruby:2.5 since we're installing so much stuff anyway (ok, fine - it's mostly because it has curl installed by default which saves some bootstrapping trouble 😛). I'm happy to be talked out of this one.

TODO

Review and update DOCKER.md. ~~I haven't touched this other than moving it to the top-level directory.~~
I wonder if just using an Ubuntu image for web would be easier since we'd get sane Ruby and NodeJS defaults? Additionally, we could just copy/paste the current instructions from INSTALL.md and what is used for Vagrant instead of inventing a separate set of installation instructions.
Get feedback on the addition of a Makefile at the top level for running docker-compose commands (doesn't seem like that great of an idea).
Deal with Postgres persistence. Instructions for this included at https://hub.docker.com/_/postgres
"create dedicated user & group to run the server, don't just use root." per @mmd-osm in Add docker support #2272 (comment)
Get tests running
Get tests PASSING locally
Clean up commit history

simonpoole · 2019-10-20T07:39:27Z

* Use a `postgis` Docker image instead of `postgres` since we definitely need PostGIS.

What's the rationale for that? Given that the rails-port definitely doesn't require PostGIS.

mmd-osm · 2019-10-20T08:16:24Z

With #2383 in place, most of docker/postgres/Dockerfile and docker/postgres/openstreetmap-postgres-init.sh will no longer be needed.

jalessio · 2019-10-20T18:00:47Z

* Use a `postgis` Docker image instead of `postgres` since we definitely need PostGIS.
What's the rationale for that? Given that the rails-port definitely doesn't require PostGIS.

@simonpoole During the course of working on this yesterday I somehow managed to convince myself that PostGIS was in fact needed... but you're right, it's not!

I reverted back to Postgres in 3e5b8ce.

jalessio · 2019-10-20T18:38:49Z

With #2383 in place, most of docker/postgres/Dockerfile and docker/postgres/openstreetmap-postgres-init.sh will no longer be needed.

@mmd-osm PR #2383 just merged by @tomhughes so I removed the additional PostgreSQL functions from the db Docker image. After rebase from master the changes are in commit b0d9d03.

zerebubuth · 2019-10-21T15:25:24Z

Hi! Looks great so far. I was able to get a couple of containers up using your instructions. Please let me know if there's anything I can do to help 😄

jalessio · 2019-10-21T16:15:45Z

Hi! Looks great so far. I was able to get a couple of containers up using your instructions. Please let me know if there's anything I can do to help 😄

@zerebubuth Any thoughts on how to handle persistence of Postgres data? There is some background information in the "Where to Store Data" section here: https://hub.docker.com/_/postgres

I personally prefer the "Let Docker manage the storage" approach but I have found that people find that confusing since their data vanishes when you run docker-compose down. For me, that's a feature but it seems to be surprising to people and can lead to unexpected data destruction which nobody likes.

Because of that I'm inclined to do with the "Create a data directory on the host system (outside the container)" option. I find it weird to have a PGDATA directory on my host system that is only used in a container, but I suspect I'm in the minority.

Thoughts?

btw, right now it's using the "Let Docker manage the storage" approach

Firefishy · 2019-10-21T16:37:10Z

Use a docker volume (managed via docker-compose.yml) for postgres persistance. In "production" one would not use postgres in the same way. A volume is not automatically removed on a docker-compose down

Firefishy · 2019-10-21T16:45:19Z

Quick example of docker-compose.yml volume with postgres: https://gist.github.com/Firefishy/c6ba34c1eab11d5c3426137011d4869a

mmd-osm · 2019-10-21T20:11:05Z

Dockerfile

+    apt-get install --no-install-recommends -y \
+      build-essential \
+      curl \
+      imagemagick \


I wonder why imagemagick is still in the list. GPX preview image generation was moved to gd2-ffij earlier this year, and I'm not aware of other parts of the website depending on imagemagick.

The imagemagick requirements are still listed in the "Minimum Requirements" section of INSTALL.md so if they are dropped here they should probably be dropped there as well.

https://github.com/openstreetmap/openstreetmap-website/blob/master/INSTALL.md#minimum-requirements

Right, it seems the INSTALL.md file also needs a bit of cleanup.

@mmd-osm is there anything in the test suite which would verify whether this dependency is needed? If not, any suggestions on a suitable test I could add?

I thought we got rid of imagemagick, but it seems that 6c20244 introduced mini_magick, which depends on imagemagick. This is only used to resize avatar images and might be a bit heavyweight for the task at hand. Maybe @tomhughes knows more.

Well I don't believe there is any other supported approach. It only uses it via the command line though so there is no need for the development package.

mmd-osm · 2019-10-21T20:11:28Z

Dockerfile

+      imagemagick \
+      libarchive-dev \
+      libffi-dev \
+      libmagickwand-dev \


This one should also be reviewed, not sure it is still needed.

jalessio · 2019-10-22T19:36:13Z

Use a docker volume (managed via docker-compose.yml) for postgres persistance. In "production" one would not use postgres in the same way. A volume is not automatically removed on a docker-compose down

I switched over to use a Docker volume for Postgres in 8d84170

added Dockerfile for the ruby on rails app inside a separate docker directory. this will build a image from the official ruby (specifically ruby:2.5-slim).

…rror postgresql-client throws a dpkg error when it can't create a softlink in for man page and aborts installation. found a solve at stack overflow and modified Dockerfile to add a comment to mention the problem with slove source.

…ed jobs from bundle install modified Dockerfile to separate apt-get cache cleaning command, added a comment explaining why and how npm is added and removed --jobs flag from bundle install command

added docker_postgres.sh for PostgreSQL extension and functions that installs PostgreSQL Btree-gist Extension, compiles libpgosm.so and installs maptile_for_point, tile_for_point and xid_to_int4 functions

added Dockerfile.postgres for the postgres database app. this will build a image from the official postgres (specifically postgres:9.4) image.

…ntomjs via npm modified Dockerfile to changes postgresql-client package from postgresql-client-10 to postgresql-client, and added phantomjs installation via npm

modified Dockerfile.postgres file to changes docker_postgres.sh location from db to docker

modified Dockerfile.postgres file to add app location setup and gem install via bundle

Until the Docker build context proves to be big enough to actually be a performance problem.

Fallout from changing build context for this image

This is no longer needed as of PR openstreetmap#2383

Without this there are permissions issues on the host filesystem.

Make it easy to get up and running quickly

jalessio · 2019-10-22T19:59:54Z

I'm closing this PR out and replacing it with PR #2409, which has a significantly cleaner commit history. For outstanding issues in the comments of this PR I propose we move them to Issues and address them in follow-on PRs if/when #2409 merges.

tomhughes · 2019-10-22T20:08:38Z

It doesn't matter but if you're not aware you don't have to open a new PR just because you've rewritten the branch history - you can just force push and the existing PR will get the new clean history.

jalessio changed the title ~~[WIP] Add Docker Compose Support~~ Add Docker Compose Support Oct 20, 2019

jalessio mentioned this pull request Oct 20, 2019

Add docker support #2272

Closed

8 tasks

jalessio force-pushed the docker-compose branch from 07dc7b5 to b0d9d03 Compare October 20, 2019 18:36

mmd-osm reviewed Oct 21, 2019

View reviewed changes

fazlerabbi37 added 17 commits October 22, 2019 19:45

Add Dockerfile for rails app

5fdf823

added Dockerfile for the ruby on rails app inside a separate docker directory. this will build a image from the official ruby (specifically ruby:2.5-slim).

Modify Dockerfile to separate apt cache clean, npm comment, and remov…

39269fd

…ed jobs from bundle install modified Dockerfile to separate apt-get cache cleaning command, added a comment explaining why and how npm is added and removed --jobs flag from bundle install command

Add docker_postgres.sh for PostgreSQL extension and functions

1e5dfea

added docker_postgres.sh for PostgreSQL extension and functions that installs PostgreSQL Btree-gist Extension, compiles libpgosm.so and installs maptile_for_point, tile_for_point and xid_to_int4 functions

Add Dockerfile.postgres for postgres app

c03fcd0

added Dockerfile.postgres for the postgres database app. this will build a image from the official postgres (specifically postgres:9.4) image.

Add docker-compose.yml to build the complete app

c66841a

modify Dockerfile.postgres to fix script path

b19e46b

modify Dockerfile.postgres to upgrade postgres image to 11

e8d364b

modify Dockerfile to change postgresql-client package and install pha…

5d685e7

…ntomjs via npm modified Dockerfile to changes postgresql-client package from postgresql-client-10 to postgresql-client, and added phantomjs installation via npm

modify docker-compose.yml to fix context path

91b69b1

add DOCKER.md file with docker setup instructions

a1ab596

add .dockerignore file to list ignorable file for docker

59b399e

modify DOCKER.md to add database population

cf63d94

modify DOCKER.md to add app config

afdc9e9

moved docker_postgres.sh to docker directory

ed004c7

modify Dockerfile.postgres file to change pgsql script location

d382d4c

modified Dockerfile.postgres file to changes docker_postgres.sh location from db to docker

modify Dockerfile.postgres file to add app location and gem install

da0b13c

modified Dockerfile.postgres file to add app location setup and gem install via bundle

jalessio added 25 commits October 22, 2019 19:45

Use currently 'supported' version of Postgres

50dce9a

Move 'ADD' later in build process to preserve layer caches

8c4a16f

Dockerfile cleanup to minimize image layers

2e887d0

Drop unneeded image names from docker-compose.yml

a62ead6

Update build context for 'db'

6b77b8d

More re-org of main Dockerfile for better layer caching

c68ce09

Remove dockerignore file for less build surprises

56019ec

Until the Docker build context proves to be big enough to actually be a performance problem.

Fix db Dockerfile path

2179344

Fallout from changing build context for this image

Port forward Postgres to a less common port to avoid conflicts

2104509

Rename Postgres init script

2bd5abb

Build shared object in Dockerfile, not at run time

707da60

Use PG11. Don't install Ruby - just get headers for PG extension.

b8a2a9a

Add db name to database URL for docker-compose

4105547

Temporary(?) Makefile for docker-compose shortcuts

609afda

This project doesn't require PostGIS

c9e8721

Remove line from Makefile which was not working as expected

7fcfbe9

Remove need for shared library PostgreSQL functions

a3a989f

This is no longer needed as of PR openstreetmap#2383

First pass modifications on Docker README

e28ab32

Use Docker volume to persist Postgres data on host filesystem

8be4a40

Exlude PGDATA directory from Docker build context

37142f1

Without this there are permissions issues on the host filesystem.

Use 'openstreetmap' user to connect to db

b397c05

Add a helper command for populating the database

c0ec8bc

Make it easy to get up and running quickly

DOCKER readme tweaks

7748dd4

Move db config to yml file

ba08eec

Add helper command for running tests w/Docker

cb2881c

jalessio force-pushed the docker-compose branch from c528202 to cb2881c Compare October 22, 2019 19:45

jalessio mentioned this pull request Oct 22, 2019

Add Docker Compose Support for Development Environment #2409

Merged

jalessio closed this Oct 22, 2019

Conversation

jalessio commented Oct 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notable changes

TODO

Uh oh!

simonpoole commented Oct 20, 2019

Uh oh!

mmd-osm commented Oct 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jalessio commented Oct 20, 2019

Uh oh!

jalessio commented Oct 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zerebubuth commented Oct 21, 2019

Uh oh!

jalessio commented Oct 21, 2019

Uh oh!

Firefishy commented Oct 21, 2019

Uh oh!

Firefishy commented Oct 21, 2019

Uh oh!

mmd-osm Oct 21, 2019

Choose a reason for hiding this comment

Uh oh!

jalessio Oct 21, 2019

Choose a reason for hiding this comment

Uh oh!

mmd-osm Oct 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jalessio Oct 22, 2019

Choose a reason for hiding this comment

Uh oh!

mmd-osm Oct 22, 2019

Choose a reason for hiding this comment

Uh oh!

tomhughes Oct 22, 2019

Choose a reason for hiding this comment

Uh oh!

mmd-osm Oct 21, 2019

Choose a reason for hiding this comment

Uh oh!

jalessio commented Oct 22, 2019

Uh oh!

jalessio commented Oct 22, 2019

Uh oh!

tomhughes commented Oct 22, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jalessio commented Oct 20, 2019 •

edited

Loading

mmd-osm commented Oct 20, 2019 •

edited

Loading

jalessio commented Oct 20, 2019 •

edited

Loading

mmd-osm Oct 22, 2019 •

edited

Loading