Provide installation script

[matkoniecz]

Version for Ubuntu 14.04: #657 (comment)

Current setup instructions require unnecessary figuring out what and how it should be done.

But it is possible to assume that user have certain Linux (for example in VM) and provide installation script. It would be possible to provide entire installation instruction also in form:

install Lubuntu 14.04, it may be a virtual machine
$ sudo apt-get install git
$ git clone openstreetmap-carto
$ sudo sh ./openstreetmap-carto/lubuntu14.04_installation_script.sh

script would include

• set up of PostGIS database
• installing listed dependencies and fonts
• running get-shapefiles.sh
• fetching some small area suitable for testing and loading this data into database

Note that currently some people even make pull requests without testing them (see for example #654 ).

I may make and test the installation script (it would be mostly copy of http://switch2osm.org/loading-osm-data/ ).

[matthijsmelissen]

@kocio-pl I think something like this should work. Let me know if anything is missing. I would like to have an install script, but I think we first need to have a tested procedural description.

• Install Tilemill
• Load OSM data. You can skip the Section 'Assorted software'.
  If you want to contribute to the stylesheet, first create a fork by clicking the Fork button on Github, and then in the 'Stylesheet' section, instead execute

git clone https://github.com/username/openstreetmap-carto.git

where 'username' is your Github username.

In the 'Loading with osm2pgsql' section, use instead the command

osm2pgsql -d gis ~/osm/liechtenstein-latest.osm.pbf --style openstreetmap-carto.style

• Install some dependencies:

sudo apt-get install mapnik-utils ogr2ogr python-yaml

• Run

./get-shapefiles.sh
ogr2ogr ne_10m_populated_places_fixed.shp ne_10m_populated_places.shp

• To install non-Western fonts, run

    sudo apt-get install ttf-dejavu fonts-droid ttf-unifont fonts-sipa-arundina fonts-sil-padauk fonts-khmeros \
    ttf-indic-fonts-core ttf-tamil-fonts ttf-kannada-fonts

[pnorman]

Install some dependencies:
sudo apt-get mapnik-utils ogr2ogr
pip install pyyaml

It's better to use the python-yaml package.

[matthijsmelissen]

Updated my original comment.

[gravitystorm]

Having an osm2pgsql database loaded with osm data is a common task and certainly not exclusive to this style. It can also be fairly platform-specific.

I'm wary of creating Yet Another Guide(s) to this repo which will, inevitably, bitrot. Could we not just link to an existing guide elsewhere?

[matthijsmelissen]

I think the description in my comment above is the minimum description we need to give, even when we link to existing guides whenever possible.

[matthijsmelissen]

@pnorman Would it be possible to drop the 'Stylesheet' section from http://switch2osm.org/loading-osm-data/? The section doesn't really fit within 'Loading OSM data', and is insufficient as it doesn't talk about fonts, shapefiles etc.

It would requiring removing the '--style ~/osm/openstreetmap-carto/openstreetmap-carto.style' part from the osm2pgsql line though.

[kocio-pl]

For now I just try to achieve the success, but I will try to make a clean room installation some other time to check if all instructions are correct. However I remember "pip" package should be replaced with "python-pip" and "ogr2ogr" with "gdal-bin".

I guess "Yet Another Guide(s) to this repo" is highly recommended, because (a) Ubuntu 14.04 is here to stay for 5 years, so bitrot won't get it too soon =} , (b) process of installing is long and consists of many different and unusual steps, so one can easy get distracted - I'm hardly a newbie in Linux/FLOSS world, but I still couldn't get the environment ready in one go today.

[kocio-pl]

First success is here: I managed to fulfill the database! Now the TileMill is awaiting...

One more thing: there should be some "cd" to the right directories in the instructions. I realized, where I should go to issue some command, by trial and error, but not everybody can effectively handle such guessing.

[kocio-pl]

So I try to create PostGIS project in TileMill and I have even found the correct "crashcourse" part, I guess (https://www.mapbox.com/tilemill/docs/guides/postgis-work/), but I don't know what is the db password, since (using tips from http://switch2osm.org/loading-osm-data/ ) I just created the user and no password was set by me:

sudo -u postgres createuser -s $USER

but when I omit the password, it can't connect to the database.

I don't know what the ID is (the same as the database name - "gis"?). I also don't know what tables should I import and how should I calculate the extent, given that I just get only one country (Poland).

[pnorman]

but when I omit the password, it can't connect to the database

Omit the user, password, hostname, and port, giving only the dbname. This will use local unix sockets to connect.

For extents use -20037508,-20037508,20037508,20037508. This is perhaps slightly sub-optimal for only poland, but will still work.

Also use 900913 for the SRS, not the custom string they gave.

Overall reading through https://www.mapbox.com/tilemill/docs/guides/postgis-work/ I see numerous problems with it.

My guide to setting up TileMill to work with OpenStreetMap Carto is https://hackpad.com/TileMill-I6rxRVszKMv but I haven't had time to finish it off.

[kocio-pl]

OK, so my next right step was adding layer like this:

1. Select PostGIS tab [I probably tried SQLite by mistake before, now the extents field is not here at all, and the extent field doesn't need to be touched]
2. ID field: carto [just sounds nice]
3. Connection field: dbname=gis [as you said!]
4. Table or subquery field: (select * from planet_osm_line where highway like 'path') as paths [example query from the hint under the field]
   ...and Save.

But I needed to add styling, so I just copied the #countries settings in style.mss as:

#carto {
::outline {
line-color: #85c5d3;
line-width: 2;
line-join: round;
}
polygon-fill: #fff;
}

and after saving it worked!

So, now I have working database filled with current data and I can use it inside the TileMill at the basic level - double success! But what I really want is testing real carto style. Let's try it then...

[kocio-pl]

Yep:

ln -s ~/osm/openstreetmap-carto ~/Documents/MapBox/project/

did the trick! Now I have my forked style as a separate TileMill project and it starts loading...

However today I won't wait to see how long it takes and if my computer has enough of resources to work with entire country data. It's just a nice point to stop for now. Combined instructions and my guessing were OK, now let's make them simpler and no-guessing.

[pnorman]

You should be able to just run through the commands at https://hackpad.com/TileMill-I6rxRVszKMv, which will get you all the way to running TileMill, which depends on your setup, and opening the project from within TileMill, which is fairly simple.

I have no interest in writing a guide which runs through the steps without explaining, as then you have no idea why you've done what you've done, and if the slightest thing changes the person has no knowledge of what's going on. I have shell scripts which run all the commands to set up an osm2pgsql database from scratch and install TileMill, but they're extremely fragile.

[kocio-pl]

Hm, I'm a bit confused now: this ticket is all about installation script - do you think it is completely unnecessary or you just have some minor/technical issues?

What parts are so fragile (for example I don't think package installation is such a part)?

Maybe the script should have some conditionalities with hints ("Which source do you chose to download data from: 1..., 2..., 3..., skip downloading [press Enter for default: 1]")?

[pnorman]

Given that switch2osm will be getting a guide on setting up the stylesheet when the writing is finished, I think it's unnecessary.

A script will bitrot unless written very carefully. Writing robust scripts isn't trivial. This is based on experience, and answering support questions, as well as having my scripts break.

[pnorman]

@pnorman Would it be possible to drop the 'Stylesheet' section from http://switch2osm.org/loading-osm-data/? The section doesn't really fit within 'Loading OSM data', and is insufficient as it doesn't talk about fonts, shapefiles etc.

No. We need a style file from a stylesheet as one shouldn't expect the default.style to work with a particular stylesheet. I've chosen to go with the openstreetmap-carto.style file instead of one of the others.

[matkoniecz]

IMPORTANT: Current version resides in #657 (comment)

@math1985 - new version: edits in the beginning, working ogr2ogr installation, missing cd, symbolic link.

Needs testing, preferably by somebody new to developing OSM styles.

---

Instructions for installation on Ubuntu 14.04

• Install Tilemill

sudo add-apt-repository ppa:developmentseed/mapbox
sudo apt-get update
sudo apt-get install tilemill libmapnik nodejs

• Load OSM data as descibed on switch2osm. You may skip the Section 'Assorted software', 'Getting ready to load' and 'Tuning'. One should not attempt 'Loading the full planet' as start - 'Loading an extract' is clearly superior for the beginner.
• Open Tilemill and close it (it generates its files in ~/Documents)
• Add openstreetmap-carto to Tilemill projects using symbolic link

ln -s ~/osm/openstreetmap-carto ~/Documents/MapBox/project/

If you want to contribute to the stylesheet, first create a fork by clicking the Fork button on Github, and then in the 'Stylesheet' section in switch2osm guide, instead execute

git clone https://github.com/username/openstreetmap-carto.git

where 'username' is your Github username.

In the 'Loading with osm2pgsql' section, use instead the command

osm2pgsql -d gis ~/osm/liechtenstein-latest.osm.pbf --style ~/osm/openstreetmap-carto/openstreetmap-carto.style

• Install some dependencies:

sudo apt-get install mapnik-utils gdal-bin python-yaml

• Run

cd ~/osm/openstreetmap-carto
./get-shapefiles.sh
ogr2ogr ne_10m_populated_places_fixed.shp ne_10m_populated_places.shp

• To install non-Western fonts, run

    sudo apt-get install ttf-dejavu fonts-droid ttf-unifont fonts-sipa-arundina fonts-sil-padauk fonts-khmeros \
    ttf-indic-fonts-core ttf-tamil-fonts ttf-kannada-fonts

• It is now possible to open Tilemill and see rendered output. Changes in style definition files will result in a changed style of generated map.

[matkoniecz]

I tested and improved previous comment (mostly wording - and two instructions "Open Tilemill and close it" + final one).

[matthijsmelissen]

As far as I'm concerned, we add this to https://github.com/math1985/openstreetmap-carto/blob/master/INSTALL.md .

[pnorman]

I have had nothing but problems with ppa:developmentseed/mapbox causing conflicts with other versions.

[matkoniecz]

@pnorman

I have had nothing but problems with ppa:developmentseed/mapbox causing conflicts with other versions.

So what is the proper way of installing Tilemill? There are recommending this method on their website.

Or is there a suitable replacement for Tilemill?

[pnorman]

I'd prefer recommending Kosmtik over Tilemill.

• Kosmtik is better suited to large complex stylesheets. Tilemill tries to be a text editor, but can't deal with more than 3-4 .mss files
• Tilemill is essentially abandoned. Kosmtik is more of a community driven effort
• Kosmtik is smaller

The main differences with Kosmtik install and setup are

• You need to install it from source (@yohanboniface, any progress there?)
• You point it directly at the project.yaml file
• While running, it only deals with one project

For a new user creating new projects which will only involve 1 MSS file, I could see recommending Tilemill, as the built-in text editor makes it easier to start, but it just doesn't work on large projects.

[yohanboniface]

You need to install it from source (@yohanboniface, any progress there?)

Nope, sadly. Creating a Debian package is not really an issue in se, but I need to sit down a bit to find the best way to deal with Mapnik.

[matkoniecz]

Kosmtik sounds nice (I am not fan of Tilemill but I was unaware that alternative exists), lets hope that standard installation from source is straightforward.

BTW, is there any performance difference compared to Tilemill?

[kocio-pl]

I try to install from source on Ubuntu Utopic, but I get this error (I got node package from regular repositories) - is it the same as @yohanboniface was talking?:

$ npm install
npm WARN package.json [email protected] No README data
-
> [email protected] install /home/kocio/devel/kosmtik/node_modules/mapnik
> node-pre-gyp install --fallback-to-build

[pnorman]

BTW, is there any performance difference compared to Tilemill?

For map development, not really. You're still bottlenecked by Mapnik, which on machines with slow disks and low RAM means you're bottlenecked by the database. There's also the metatile size tradeoff between throughput and responsiveness. I can't speak to time to export tilesets or images.

It's probably marginally quicker in some ways, as it does less.

[pnorman]

Is this the right way to think about it, or am I doing something wrong? @pnorman @kocio-pl

Have a look at openstreetmap/openstreetmap-website#1290

[kocio-pl]

Default Ubuntu image is currently 14.04, so I would use rather FROM ubuntu:16.04. But we can also use minimal image, which is supposed to solve some Docker-specific issues, so probably FROM phusion/baseimage.

[kocio-pl]

There are also known best practices for using apt-get in Docker, related to its extensive caching. Basically - always use apt-get update in the same line as apt-get install.

If we want to always have fresh git content from osm-carto (every time the container is started, not only when building new image), we should update it through ENTRYPOINT.

And when we have all the needed containers, we have to start them in a coordinated way - it can be made with a simple script, but there also exists Vagrant, which is popular VM "orchestrating" software. See this example.

[matthijsmelissen]

@Kocio DId you get the geo-data image to work? I repeatedly get '2016-10-14 20:42:37 UTC FATAL: could not access private key file "/etc/ssl/private/ssl-cert-snakeoil.key": Permission denied'.

[kocio-pl]

No, I didn't try it because of lack of the license - I don't like to waste the time with such uncertain project. I can test it however in a few days, if it would be useful to you in any way.

[kocio-pl]

I have some problems building, but you could also try with a fork which has quite fresh build available:

$ docker run xingfuryda/openstreetmap-tiles-docker

[pnorman]

Here's the dockerfile I have

FROM debian:testing

# Install style dependencies and Kosmtik dependencies
RUN apt-get update && apt-get install --no-install-recommends -y \
    nodejs nodejs-legacy npm \
    zip unzip curl git ca-certificates \
    fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted ttf-unifont \
    && rm -rf /var/lib/apt/lists/*

RUN npm install kosmtik

RUN mkdir -p /openstreetmap-carto
WORKDIR /openstreetmap-carto

# Get the shapefiles
ADD get-shapefiles.sh ./
RUN ./get-shapefiles.sh

Next steps would be to make a docker-compose file which

• Maps . to /openstreetmap-carto
• Exposes port 6789
• Has a container for PostGIS
• Runs Kosmtik on start

The data loading also requires another dockerfile for osm2pgsql which will load the data into the PostGIS container.

I'm doing an OS upgrade so might not get back to this for a few days.

[kocio-pl]

I was not aware of Docker Compose, but it looks like a better way than Bash scripts or Vagrant rules - thanks!

[pnorman]

OS upgrade finished, and I have something that partially works

master...pnorman:docker starts two containers, one for the DB, the other for Kosmtik, and docker-compose up brings them up together. I'm still working on the import side.

to-do before opening a PR for discussion

• use a data volume for shapefiles
• get importing going
• add documentation

[kocio-pl]

Why don't you use Joxit/docker-kosmtik, which you mentioned yourself, or kosmtik/kosmtik for example?

[pnorman]

Why don't you use Joxit/docker-kosmtik, which you mentioned yourself, or kosmtik/kosmtik for example?

I'd have needed to do just as much work modifying it, or end up with another container which does the work of downloading the shapefiles.

[matthijsmelissen]

So currently the import process fails (amongst others) because it has no createdb executable. Should we add postgresql-client to the import Dockerfile? Still trying to understand how this works.

[pnorman]

So currently the import process fails (amongst others) because it has no createdb executable. Should we add postgresql-client-9.3 to the import Dockerfile? Still trying to understand how this works.

Yes. Well, not that version, postgresql-client is what should be used, but I've done this change locally and haven't pushed the changes.

[matthijsmelissen]

Thanks. Do you have any other local changes? Half implementations are fine.

[matthijsmelissen]

The issue I'm not having is:

import_1 | createdb: could not connect to database template1: FATAL: the database system is starting up

In other words, at the moment createdb is executed, the psql server is not yet fully running. Is there a way to handle this?

[pnorman]

In other words, at the moment createdb is executed, the psql server is not yet fully running. Is there a way to handle this?

Not sure.

[matthijsmelissen]

I added a sleep 10 now. There must be a better way though...

[pnorman]

https://docs.docker.com/compose/startup-order/

[matthijsmelissen]

Good point.

[matthijsmelissen]

Next problem: the import Docker file contains WORKDIR /openstreetmap-carto and RUN npm install kosmtik, but when doing a CMD, /openstreetmap-carto/node_modules does not exist. What am I missing?

[kocio-pl]

I would try to look at this, since I consider having easy toolset for developers to be quite important thing - getting to know some details takes me years, so it's good to have some developers to take care for some easier tasks, so we have more people ready to work with the code in the longer perspective.

[matthijsmelissen]

See also https://www.reddit.com/r/openstreetmap/comments/5i6tif/how_to_attract_new_mappers_for_osm/db6kupn/

[Ircama]

Sometimes this could be of help...

[matthijsmelissen]

Would https://shinesolutions.com/2016/12/30/generating-high-res-maps-with-mapnik-and-docker/ help us with the docker work?

[pnorman]

they're using a flag file to indicate if the DB is initialized and hard-coding the location of the data that gets imported. It doesn't help with what we still need to work out.

[nebulon42]

I have updated the branch (master...nebulon42:docker) and added some documentation. It basically works. The only thing I have not looked at is if we can prevent get-shapefiles.py from indexing every time and if there need to be other improvements with this script.

Another thing is that the shapefile data currently is owned by the user with UID 1000. First it was root which was even worse. We have to think of something for systems where 1000 is not the default user.

We also have to think about if we should build the containers and upload them to the registry to avoid that everybody has to re-build them.