Find that Postcode

This project creates an elasticsearch index based on the UK postcode file, and runs a webserver on top of it for making queries. It's like a more lightweight and less sophisticated version of MapIt.

Setup

1. Setup elasticsearch

Follow the instructions to download and install elasticsearch.

Run the server. At the moment the scripts only work on a default server of localhost:9200 but a future version will have configurable host and port.

2. Install python dependencies

You'll need to install the python elasticsearch and flask libraries, either directly through pip or by running a virtual environment and running:

pip install -r requirements.txt

The code is written in python 3 and hasn't been tested in python 2.

3. Point flask to the app

Flask needs to know which app it's running. The easiest way to do this is to create a file called .env in the project directory, and add the following contents:

FLASK_APP=findthatpostcode
FLASK_ENV=development

# S3 credentials for boundaries
S3_REGION=XXXXXXXX
S3_ENDPOINT=XXXXXXXX
S3_ACCESS_ID=XXXXXXXX
S3_SECRET_KEY=XXXXXXXX
S3_BUCKET=XXXXXXXX

4. Create elasticsearch indexes

Run flask init-db to create the needed index and mappings before data import.

5. Import postcodes

Run the following to import the data and save postcodes to the elasticsearch index:

flask import nspl --url https://example.com/url-to-nspl

Replace https://example.com/url-to-nspl with the URL to the latest NSPL file. This file can be found through a search on the ONS Geoportal. On the page for the file copy the link shown in the "Download" button on the right hand side.

This will then run the import process. It takes a while to run as there are over 2.5 million postcodes. The data will be around 1.3 GB in size on the disk.

6. Import area codes

Run the following to import the code history database and register of geographic codes.

flask import rgc
flask import chd
flask import msoanames # imports the names for MSOAs from House of Commons Library

The URL of the files used can be customised with the --url parameter. Unfortunately the ONS geoportal doesn't provide a persistent URL to the latest data.

6. Import boundaries (optional)

Boundaries are uploaded as individual area files to S3 storage.

Boundary files can be found on the ONS Geoportal. Generally the "Generalised Clipped" versions should be used to minimise the file size. Open each boundary file link and find the "API" link on the right hand side, and copy the GeoJSON link, or download the file.

These files are the latest available at April 2017:

• Countries: https://opendata.arcgis.com/datasets/b789ba2f70fe45eb92402cee87092730_0.geojson
• Westminster Parliamentary Constituencies: https://opendata.arcgis.com/datasets/094f326b0b1247e3bcf1eb7236c24679_0.geojson
• Counties and unitary authorities: https://opendata.arcgis.com/datasets/0de4288db3774cb78e45b8b74e9eab31_0.geojson
• Local Authority Districts: https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/Local_Authority_Districts_May_2022_UK_BGC_V3/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson
• Regions: https://opendata.arcgis.com/datasets/284d82f437554938b0d0fbb3c6522007_0.geojson
• CCGs: https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/Clinical_Commissioning_Groups_April_2021_EN_BGC/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson
• European electoral regions: https://opendata.arcgis.com/datasets/20595dbf22534e20944c9cee42c665b3_0.geojson
• Local Enterprise Partnerships: https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/LEP_MAY_2021_EN_BGC_V2/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson
• NHS Commissioning Regions: https://opendata.arcgis.com/datasets/edcbf58c70004d0f8d44501d07c38fe9_0.geojson
• National Parks: https://opendata.arcgis.com/datasets/f41bd8ff39ce4a2393c2f454006ea60a_0.geojson
• Police Force areas: https://opendata.arcgis.com/datasets/282af275c1a24c2ea64ff9e05bdd7d7d_0.geojson
• Travel to Work Areas: https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/Travel_to_Work_Areas_December_2011_UK_BGC_v2/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson
• Major Towns and Cities: https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/TCITY_2015_EW_BGG_V2/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson
• Combined Authorities: https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/Combined_Authorities_December_2021_EN_BGC/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson

These files are large:

• Parishes (11,000): https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/Parishes_May_2022_EW_BGC/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson --code-field=par22cd
• Wards (8,900): https://opendata.arcgis.com/datasets/d2dce556b4604be49382d363a7cade72_0.geojson - Didn't work
• LSOAs (35,000): https://opendata.arcgis.com/datasets/e993add3f1944437bc91ec7c76100c63_0.geojson - Didn't work
• MSOAs (7,200): https://opendata.arcgis.com/datasets/29fdaa2efced40378ce8173b411aeb0e_2.geojson - Didn't work
• Built-up Areas (5,800): https://opendata.arcgis.com/datasets/f6684981be23404e83321077306fa837_0.geojson
• Built-up Area Sub-divisions (1,800): https://opendata.arcgis.com/datasets/1f021bb824ee4820b353b4b58fab6df5_0.geojson

Import the boundary files by running:

flask import boundaries "https://opendata.arcgis.com/datasets/094f326b0b1247e3bcf1eb7236c24679_0.geojson"

You can add more than one URL to each import script.

7. Import placenames (optional)

A further related dataset is placenames. The ONS has a list of these which can be imported using the import placenames command. An entry for each placename is added to the geo_placenames elasticsearch index.

flask import placenames

The --url parameter can be used to customise the URL used.

7. Import statistics (optional)

Statistics can be added to areas, using ONS data. The available statistics are added to LSOAs, but could also be added to other areas.

flask import imd2019
flask import imd2015

The --url parameter can be used to customise the URL used to get the data.

Run tests

python -m pytest tests

Using the data

Run the server

The project comes with a simple server (using the flask framework) allowing you to look at postcodes. The server returns either html pages (using .html) or json data by default.

Run the server by:

flask run

By default the server is available at http://localhost:5000/.

Server endpoints

The server has a number of possible uses:

• /postcodes/SW1A+1AA.html gives information about a particular postcode.
• /areas/E09000033.html gives information about an area, including example postcodes.
• /areas/search.html?q=Winchester finds any areas containing a search query.
• /areatypes/laua.html gives information about a type of area, including lists of example codes.
• /areatypes.html lists all the possible area types.
• /points/53.490911,-2.095804.html gives details of the postcode closest to the latitude, longitude point. If it's more than 10km from the nearest postcode it's assumed to be outside the UK.

Elasticsearch REST api

The data is also now available in an elasticsearch index to be used in other local applications using the elasticsearch REST api.

Find details on a postcode

curl "http://localhost:9200/geo_postcode/_doc/SW1A+1AA?pretty"

{
  "_index": "postcode",
  "_type": "postcode",
  "_id": "SW1A 1AA",
  "_version": 1,
  "found": true,
  "_source": {
    "bua11": "E34004707",
    "oac11": "2C3",
    "park": "E99999999",
    "osnrth1m": 179645,
    "buasd11": "E35000546",
    "lsoa11": "E01004736",
    "lsoa21": "E01004736",
    "pcon": "E14000639",
    "pct": "E16000057",
    "nuts": "E05000644",
    "pcds": "SW1A 1AA",
    "ccg": "E38000031",
    "osgrdind": 1,
    "eer": "E15000007",
    "hlthau": "E18000007",
    "imd": 16419,
    "ward": "E05000644",
    "wz11": "E33031119",
    "ctry": "E92000001",
    "oseast1m": 529090,
    "pcd2": "SW1A 1AA",
    "laua": "E09000033",
    "rgn": "E12000007",
    "location": {
      "lon": -0.141588,
      "lat": 51.501009
    },
    "lat": 51.501009,
    "usertype": 1,
    "cty": "E99999999",
    "ttwa": "E30000234",
    "lep1": "E37000023",
    "pcd": "SW1A1AA",
    "teclec": "E24000014",
    "dointr": "1980-01-01T00:00:00",
    "oa11": "E00023938",
    "oa21": "E00023938",
    "long": -0.141588,
    "pfa": "E23000001",
    "ru11ind": "A1",
    "hro": "E19000003",
    "msoa11": "E02000977",
    "msoa21": "E02000977",
    "lep2": null,
    "doterm": null
  }
}

Find details on an area

curl "http://localhost:9200/geo_area/_doc/E00046056?pretty"

{
  "_index": "postcode",
  "_type": "code",
  "_id": "E00046056",
  "_version": 2,
  "found": true,
  "_source": {
    "code": "E00046056",
    "name": "",
    "name_welsh": null,
    "statutory_instrument_id": "1111/1001",
    "statutory_instrument_title": "GSS re-coding strategy",
    "date_start": "2009-01-01T00:00:00",
    "date_end": null,
    "parent": "E01009081",
    "entity": "E00",
    "owner": "ONS",
    "active": true,
    "areaehect": 3.75,
    "areachect": 3.75,
    "areaihect": 0,
    "arealhect": 3.75,
    "sort_order": "E00046056",
    "predecessor": [
        "00CNFN0006"
    ],
    "successor": [],
    "equivalents": {
        "ons": "00CNFN0006"
    }
  }
}

Todo / future features

• Find areas containing a point

Dokku setup

# create app
dokku apps:create find-that-postcode

# add permanent data storage
dokku storage:mount find-that-postcode /var/lib/dokku/data/storage/find-that-postcode:/data

# enable domain
dokku domains:enable find-that-postcode
dokku domains:add find-that-postcode findthatpostcode.uk

# elasticsearch
sudo dokku plugin:install https://github.com/dokku/dokku-elasticsearch.git elasticsearch
dokku elasticsearch:create find-that-postcode-es
dokku elasticsearch:link find-that-postcode-es find-that-postcode

# SSL
sudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git
dokku config:set --no-restart find-that-postcode [email protected]
dokku letsencrypt find-that-postcode
dokku letsencrypt:cron-job --add

2. Add as a git remote and push

On local machine:

git remote add dokku dokku@SERVER_HOST:find-that-postcode
git push dokku main

3. Setup and run import

On Dokku server run:

# setup and run import
dokku config:set find-that-postcode FLASK_APP=findthatpostcode
dokku config:set find-that-postcode S3_REGION=XXXXXXXX
dokku config:set find-that-postcode S3_ENDPOINT=XXXXXXXX
dokku config:set find-that-postcode S3_ACCESS_ID=XXXXXXXX
dokku config:set find-that-postcode S3_SECRET_KEY=XXXXXXXX
dokku config:set find-that-postcode S3_BUCKET=XXXXXXXX
dokku run find-that-postcode flask init-db
dokku run find-that-postcode flask import nspl --year=2011
dokku run find-that-postcode flask import nspl --year=2021
dokku run find-that-postcode flask import rgc
dokku run find-that-postcode flask import chd
dokku run find-that-postcode flask import msoanames
dokku run find-that-postcode flask import imd2019
dokku run find-that-postcode flask import imd2015
dokku run find-that-postcode flask import placenames

# import boundaries
dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/7be6a3c1be3b4385951224d2f522470a_0.geojson
dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/094f326b0b1247e3bcf1eb7236c24679_0.geojson
dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/0de4288db3774cb78e45b8b74e9eab31_0.geojson
dokku run find-that-postcode flask import boundaries https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/Local_Authority_Districts_May_2022_UK_BGC_V3/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson
dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/284d82f437554938b0d0fbb3c6522007_0.geojson
dokku run find-that-postcode flask import boundaries https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/Clinical_Commissioning_Groups_April_2021_EN_BGC/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson
dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/20595dbf22534e20944c9cee42c665b3_0.geojson
dokku run find-that-postcode flask import boundaries https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/LEP_MAY_2021_EN_BGC_V2/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson
dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/edcbf58c70004d0f8d44501d07c38fe9_0.geojson
dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/f41bd8ff39ce4a2393c2f454006ea60a_0.geojson
dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/282af275c1a24c2ea64ff9e05bdd7d7d_0.geojson
dokku run find-that-postcode flask import boundaries https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/Travel_to_Work_Areas_December_2011_UK_BGC_v2/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson
dokku run find-that-postcode flask import boundaries https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/TCITY_2015_EW_BGG_V2/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson
dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/c6bd4568af5947519cf266b80a94de2e_0.geojson

# large boundary files
dokku run find-that-postcode flask import boundaries --code-field=par18cd https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/Parishes_May_2022_EW_BGC/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson
dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/d2dce556b4604be49382d363a7cade72_0.geojson
dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/e993add3f1944437bc91ec7c76100c63_0.geojson
dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/29fdaa2efced40378ce8173b411aeb0e_2.geojson
dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/f6684981be23404e83321077306fa837_0.geojson
dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/1f021bb824ee4820b353b4b58fab6df5_0.geojson