Start with the openaddresses code

Author: orangejulius

CONTRIBUTING.md

@@ -0,0 +1,53 @@
+Pelias can't succeed without contributions from community members like you! Contributions come in many different shapes and sizes. In this file we provide guidance around two of the most common types of contributions: opening issues and opening pull requests.
+
+# Community Values 
+
+We ask that you are respectful when contributing to Pelias or engaging with our community. As a community, we appreciate the fact that contributors might be approaching the project from a different perspective and background. We hope that beginners as well as advanced users will be able to use and contribute back to Pelias. We want to encourage contributions and feedback from all over the world, which means that English might not be a contributor's native language, and sometimes we may encounter cultural differences. Contructive disagreements can be essential to moving a project forward, but disrespectful language or behavior will not be tolerated. 
+
+Above all, be patient, be respectful, and be kind!
+
+# Submitting Issues
+
+All issues for Pelias are housed in the [pelias/pelias](https://github.com/pelias/pelias) repo. Before opening an issue, be sure to search the repository to see if someone else has asked your question before. If not, go ahead and [open a new issue](https://github.com/pelias/pelias/issues/new).
+
+## Submitting technical bugs
+
+When submitting bug reports, please be sure to give us as much context as possible so that we can reproduce the error you encountered. Be sure to include:
+- System conditions (OS, browser, etc)
+- Steps to reproduce
+- Expected outcome
+- Actual outcome
+- Screenshots, if applicable
+- Code that exposes the bug, if you have it (such as a failing test or a barebones script)
+
+## Submitting issues around search result quality
+
+It's important to get feedback about the quality of local search results. When it comes to things like address structure, capitalization, and spelling errors, your local knowledge will make it easier for us to understand the problem. When submitting issues be sure to include:
+- Where in the world you were searching
+- Your search query
+- Your expected result
+- Your actual result
+
+
+# Pull Requests Welcome!
+
+## Project standards overview
+
+Pelias has several miscellaneous standards:
+
+- we use [JSHint](http://jshint.com/docs/) for linting
+- we use [TravisCI](https://travis-ci.org/) for continuous integration
+- we use [Winston](https://www.npmjs.com/package/winston) for logging
+- we *love* tests, especially when written with [tape](https://github.com/substack/tape)
+- we use [semver](http://semver.org/) for package versioning
+- we *loosely* use [JSDoc](http://usejsdoc.org/index.html) for documenting code, as described [here](in_code_documentation_guidelines.md)
+
+`jshint` and any unit tests in a project will be automatically invoked when you commit to an existing project; make
+sure they exit successfully!
+
+## Active contributors
+
+We'll gladly invite active contributors to become members of the [Pelias organization](https://github.com/pelias). New
+members will gain direct write permissions, *and with great power comes great responsibility*. To ensure that any new
+repositories that you create conform to Pelias standards, we developed [pelias-init](https://github.com/pelias/init), a
+simple project generator that will initialize all of the boilerplate needed to get started on something new.

Dockerfile

@@ -0,0 +1,19 @@
+# base image
+FROM pelias/baseimage
+
+# downloader apt dependencies
+# note: this is done in one command in order to keep down the size of intermediate containers
+RUN apt-get update && apt-get install -y unzip && rm -rf /var/lib/apt/lists/*
+
+# change working dir
+ENV WORKDIR /code/pelias/openaddresses
+WORKDIR ${WORKDIR}
+
+# copy code into image
+ADD . ${WORKDIR}
+
+# install npm dependencies
+RUN npm install
+
+# run tests
+RUN npm test

LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2014 Mapzen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,99 @@
+>This repository is part of the [Pelias](https://github.com/pelias/pelias) project. Pelias is an
+>open-source, open-data geocoder built by [Mapzen](https://www.mapzen.com/) that also powers
+>[Mapzen Search](https://mapzen.com/projects/search). Our official user documentation is
+>[here](https://mapzen.com/documentation/search/).
+
+# Pelias OpenAddresses importer
+
+[![Greenkeeper badge](https://badges.greenkeeper.io/pelias/openaddresses.svg)](https://greenkeeper.io/)
+
+[![Build Status](https://travis-ci.org/pelias/openaddresses.svg?branch=master)](https://travis-ci.org/pelias/openaddresses)
+
+## Overview
+
+The OpenAddresses importer is used to process data from [OpenAddresses](http://openaddresses.io/)
+for import into the Pelias geocoder.
+
+## Requirements
+
+Node.js 4 or higher is required.
+
+## Installation
+```bash
+git clone https://github.com/pelias/openaddresses
+cd openaddresses
+npm install
+```
+
+## Data Download
+Use the `imports.openaddresses.files` configuration option to limit the download to just the OpenAddresses files of interest.
+Refer to the [OpenAddresses data listing]( http://results.openaddresses.io/?runs=all#runs) for file names.
+
+> see the 'Configuration' section below for a more detailed example of how to use `imports.openaddresses.files`
+
+```bash
+npm run download
+```
+
+## Usage
+```bash
+# show full command line options
+node import.js --help
+
+# run an import
+npm start
+```
+
+## Admin Lookup
+OpenAddresses records do not contain information about which city, state (or
+other region like province), or country that they belong to. Pelias has the
+ability to compute these values from [Who's on First](http://whosonfirst.mapzen.com/) data.
+For more info on how admin lookup works, see the documentation for
+[pelias/wof-admin-lookup](https://github.com/pelias/wof-admin-lookup). By default,
+adminLookup is enabled.  To disable, set `imports.adminLookup.enabled` to `false` in Pelias config.
+
+**Note:** Admin lookup requires loading around 5GB of data into memory.
+
+## Configuration
+This importer can be configured in [pelias-config](https://github.com/pelias/config), in the `imports.openaddresses`
+hash. A sample configuration file might look like:
+
+```javascript
+{
+  "esclient": {
+    "hosts": [
+      {
+        "env": "development",
+        "protocol": "http",
+        "host": "localhost",
+        "port": 9200
+      }
+    ]
+  },
+  "logger": {
+    "level": "debug"
+  },
+  "imports": {
+    "whosonfirst": {
+      "datapath": "/mnt/data/whosonfirst/",
+      "importPostalcodes": false,
+      "importVenues": false
+    },
+    "openaddresses": {
+      "datapath": "/mnt/data/openaddresses/",
+      "files": [ "us/ny/city_of_new_york.csv" ]
+    }
+  }
+}
+```
+
+The following properties are recognized:
+
+This importer is configured using the [`pelias-config`](https://github.com/pelias/config) module.
+The following configuration options are supported by this importer.
+
+| key | required | default | description |
+| --- | --- | --- | --- |
+| `datapath` | yes | | The absolute path of the directory containing OpenAddresses files. Must be specified if no directory is given as a command-line argument. |
+| `files` | no | | An array of the names of the files to download/import. If specified, *only* these files will be downloaded and imported, rather than *all* `.csv` files in the given directory. **If the array is empty, all files will be downloaded and imported.** Refer to the [OpenAddresses data listing]( http://results.openaddresses.io/?runs=all#runs) for file names.|
+| `deduplicate` | no | `false` | Boolean flag to enable deduplication (deprecated. See [pelias/address-deduplicator](https://github.com/pelias/address-deduplicator) for more info). |

bin/parallel

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+# grab the number of workers count
+count=$1
+
+# remove the first argument from the arguments array ($@)
+shift
+
+# only do anything if count is a valid integer >= 1
+if [[ $count -ge 1 ]]; then
+	echo "starting $count parallel builds"
+
+	# spawn $count parallel builds, passing correct params and all arguments
+	for i in `seq 0 $(($count-1))`; do
+		cmd="npm start -- --parallel-count $count --parallel-id $i $@ "
+		$cmd &
+	done
+
+	# don't let this script finish until all parallel builds have finished
+	wait
+fi

import.js

@@ -0,0 +1,53 @@
+/**
+ * @file Entry-point script for the OpenAddresses import pipeline.
+ */
+
+'use strict';
+
+var peliasConfig = require( 'pelias-config' ).generate(require('./schema'));
+
+var logger = require( 'pelias-logger' ).get( 'openaddresses' );
+
+var parameters = require( './lib/parameters' );
+var importPipeline = require( './lib/importPipeline' );
+
+const adminLookupStream = require('pelias-wof-admin-lookup');
+var deduplicatorStream = require('./lib/streams/deduplicatorStream');
+
+var addressDeduplicator = require('pelias-address-deduplicator');
+
+
+// Pretty-print the total time the import took.
+function startTiming() {
+  var startTime = new Date().getTime();
+  process.on( 'exit', function (){
+    var totalTimeTaken = (new Date().getTime() - startTime).toString();
+    var seconds = totalTimeTaken.slice(0, totalTimeTaken.length - 3);
+    var milliseconds = totalTimeTaken.slice(totalTimeTaken.length - 3);
+    logger.info( 'Total time taken: %s.%ss', seconds, milliseconds );
+  });
+}
+
+var args = parameters.interpretUserArgs( process.argv.slice( 2 ) );
+
+const adminLayers = ['neighbourhood', 'borough', 'locality', 'localadmin',
+  'county', 'macrocounty', 'region', 'macroregion', 'dependency', 'country',
+  'empire', 'continent'];
+
+if( 'exitCode' in args ){
+  ((args.exitCode > 0) ? console.error : console.info)( args.errMessage );
+  process.exit( args.exitCode );
+} else {
+  startTiming();
+
+  if (peliasConfig.imports.openaddresses.hasOwnProperty('adminLookup')) {
+    logger.info('imports.openaddresses.adminLookup has been deprecated, ' +
+                'enable adminLookup using imports.adminLookup.enabled = true');
+  }
+
+  var files = parameters.getFileList(peliasConfig, args);
+
+  var deduplicator = deduplicatorStream.create(peliasConfig, addressDeduplicator);
+
+  importPipeline.create( files, args.dirPath, deduplicator, adminLookupStream.create(adminLayers) );
+}

lib/cleanup.js

@@ -0,0 +1,25 @@
+var _ = require('lodash');
+
+function removeLeadingZerosFromStreet(token) {
+  return token.replace(/^(?:0*)([1-9]\d*(st|nd|rd|th))/,'$1');
+}
+
+function capitalizeProperly(streetname){
+  if (streetname.toUpperCase() === streetname || streetname.toLowerCase() === streetname){
+    streetname = _.capitalize(streetname);
+  }
+  return streetname;
+}
+
+function cleanupStreetName(input) {
+  return input.split(/\s/)
+  .map(removeLeadingZerosFromStreet)
+  .filter(function(part){
+    return part.length > 0;
+  }).map(capitalizeProperly)
+  .join(' ');
+}
+
+module.exports = {
+  streetName: cleanupStreetName
+};

lib/importPipeline.js

@@ -0,0 +1,37 @@
+var logger = require( 'pelias-logger' ).get( 'openaddresses' );
+var recordStream = require('./streams/recordStream');
+var model = require( 'pelias-model' );
+var peliasDbclient = require( 'pelias-dbclient' );
+var isUSorCAHouseNumberZero = require( './streams/isUSorCAHouseNumberZero' );
+
+/**
+ * Import all OpenAddresses CSV files in a directory into Pelias elasticsearch.
+ *
+ * @param {array of string} files An array of the absolute file-paths to import.
+ * @param {object} opts Options to configure the import. Supports the following
+ *    keys:
+ *
+ *      deduplicate: Pass address object through `address-deduplicator-stream`
+ *        to perform deduplication. See the documentation:
+ *        https://github.com/pelias/address-deduplicator-stream
+ *
+ *      adminValues: Add admin values to each address object (since
+ *        OpenAddresses doesn't contain any) using `admin-lookup`. See the
+ *        documentation: https://github.com/pelias/admin-lookup
+ */
+function createFullImportPipeline( files, dirPath, deduplicatorStream, adminLookupStream, finalStream ){ // jshint ignore:line
+  logger.info( 'Importing %s files.', files.length );
+
+  finalStream = finalStream || peliasDbclient();
+
+  recordStream.create(files, dirPath)
+    .pipe(deduplicatorStream)
+    .pipe(adminLookupStream)
+    .pipe(isUSorCAHouseNumberZero.create())
+    .pipe(model.createDocumentMapperStream())
+    .pipe(finalStream);
+}
+
+module.exports = {
+  create: createFullImportPipeline
+};

lib/isValidCsvRecord.js

@@ -0,0 +1,46 @@
+var _ = require('lodash');
+
+/*
+ * Return true if a record has all of LON, LAT, NUMBER and STREET defined
+ */
+function isValidCsvRecord( record ){
+  return hasAllProperties(record) &&
+          !houseNumberIsExclusionaryWord(record) &&
+          !streetContainsExclusionaryWord(record) &&
+          !latLonAreOnNullIsland(record);
+}
+
+/*
+ * Return false if record.NUMBER is literal word 'NULL', 'UNDEFINED',
+ * or 'UNAVAILABLE' (case-insensitive)
+*/
+function houseNumberIsExclusionaryWord(record) {
+  return ['NULL', 'UNDEFINED', 'UNAVAILABLE'].indexOf(_.toUpper(record.NUMBER)) !== -1;
+}
+
+/*
+ * Return false if record.STREET contains literal word 'NULL', 'UNDEFINED',
+ * or 'UNAVAILABLE' (case-insensitive)
+*/
+function streetContainsExclusionaryWord(record) {
+  return /\b(NULL|UNDEFINED|UNAVAILABLE)\b/i.test(record.STREET);
+}
+
+function hasAllProperties(record) {
+  return [ 'LON', 'LAT', 'NUMBER', 'STREET' ].every(function(prop) {
+    return record[ prop ] && record[ prop ].length > 0;
+  });
+}
+
+// returns true when LON and LAT are both parseable as 0
+// > parseFloat('0');
+// 0
+// > parseFloat('0.000000');
+// 0
+// > parseFloat('0.000001');
+// 0.000001
+function latLonAreOnNullIsland(record) {
+  return ['LON', 'LAT'].every(prop => parseFloat(record[prop]) === 0);
+}
+
+module.exports = isValidCsvRecord;