Raw Data API

Documentation: hotosm.github.io/raw-data-api

Source Code: github.com/hotosm/raw-data-api

API Documentation : https://api-prod.raw-data.hotosm.org/v1/redoc

Raw Data API is a set of high-performant APIs for transforming and exporting OpenStreetMap (OSM) data in different GIS file formats.

Features

• Fast: Built on top of FastAPI

• OAUTH 2.0 Authentication with OpenStreetMap(OSM)

• Multiple GIS formats support via GDAL's ogr2ogr - see table below for currently supported formats. Out of which , GeoJSON Follows Own Raw Data API conversion script

  Formats	Status
  Esri Shapefile	✔️
  KML	✔️
  Mbtiles	✔️
  FlatGeoBuf	✔️
  CSV	✔️
  GeoPackage	✔️
  PGDUMP	✔️
  GeoJSON	✔️
  Pmtiles	✔️
  Geoparquet	✔️

Product Roadmap

Status	Feature
✅	Yaml support for rawdatapi request
✅	Support for country exports
✅	Vector tiles support
✅	Tasking Manager exports support
✅	Export completeness information integration
✅	Independent exports link to be shared accross various stakeholders using hdx
⚙️	Metadata of exports such as no of features / total length etc
⚙️	p-code support for exports
⚙️	yaml request connection with export tool
⚙️	Migration to generic cron jobs from hdx jobs
	Improvements in data exports using h3 index research
	Raw Data Backend Upgrade

Installation

Raw Data API consists of two elements:

• A backend database, tools, and scripts: used to import OSM data into a specific database structure and keep it updated.
• An API that is used to serve data from the backend database.

To setup the backend see Backend Installation

Raw Data API can be installed through docker or locally on your computer.

• To install with docker see docker installation.
• To install locally, see local installation.

Configurations

Setup the necessary configurations for Raw Data API from configurations. Setup config.txt in project root or include your env variables in .env

Basic Usage

• Confirm that Authetication works

  1. Hit the /auth/login/ endpoint
  2. Hit the url returned on the response
  3. You will get an access_token
  4. You can use the access_token in all endpoints that requires authentication.
  5. To check token pass token in /auth/me/. It should return your OpenStreetMap (OSM) profile

• Try extracting some data:

  You can use the /raw-data/current-snapshot/ endpoint with the following input to check both authentication, database connection and download the export:

curl -d '{"geometry":{"type":"Polygon","coordinates":[[[83.96919250488281,28.194446860487773],[83.99751663208006,28.194446860487773],   [83.99751663208006,28.214869548073377],[83.96919250488281,28.214869548073377],[83.96919250488281,28.194446860487773]]]}}' -H 'Content-Type: application/json'   http://127.0.0.1:8000/v2/raw-data/current-snapshot/

Introduction of sozip

We use sozip optimization while we zip the files , SOZip makes it possible to access large compressed files directly from a .zip file without prior decompression. It is not a new file format, but a profile of the existing ZIP format, done in a fully backward compatible way. ZIP readers that are non-SOZip aware can read a SOZip-enabled file normally and ignore the extended features that support efficient seek capability. Learn more about it in attached link

Tests

• Raw Data API uses pytest for tests, navigate to the root directory and install package in editable mode:

pip install -e .

• At this point you should have PostgreSQL + PostGIS extension enabled on your computer, now run Pytest:

py.test -v -s

• Running individual tests

py.test -k test function name

Contribution & Development

Learn about current priorities and work going through Roadmap & see here CONTRIBUTING

Roadmap

see Roadmap

License

see LICENSE

Version Control

For version control, We use commitizen, follow docs

Authors

Created by HOTOSM and Friends