Wigle Wardriving Research

This page now serves as a historic set of notes about the Wigle.net database and SHOULD NOT be used with this project. A description of the working Wigle.net batch download method used with the ProbeRequestCollectorsKit can be found on this wiki page.

Scraping the Wigle.net Database

Below are miscellaneous notes and various findings from attempts to batch download data from Wigle.net. The website has an undocumented and unsupported API that allows registered users to query their database using an HTTP/REST style api and receive JSON results. Results seem to be limited per day/per search (haven't quite figured this out yet).

A somewhat out of date (although potentially fully functional) API Wigle.net reference exists here: http://www5.musatcha.com/musatcha/computers/wigleapi.htm

JIGLE (A Java Wigle.net client) is available for download from this page: https://wigle.net/tools

There has been significant discussion of using JIGLE to batch download data here: https://wigle.net/phpbb/viewtopic.php?t=1376

The above method is fairly "janky", however I have successfully used TomThumbUK's method to use the .auto cache file to download results as CSV although I have not found that method to be completely reproducible (although it may be worth another look) and features a 10,000 result limit as mentioned in the forum post above..

A quick GitHub search reveals various recent (and ancient) attempts to do exactly what we want to do, so perhaps a scrutinizing look into these is in order. When reading code, note that the https://wigle.net/api/v1/jsonSearch URL that Wigle.net's search page is a different URL than the API reference/JIGLE seem to use (which is http://WiGLE.net/gpsopen/gps/GPSDB/confirmquery) and most likely has different limits (I've found 5,000 results to be the max).

Here is a small batch download (14MB) I scraped using a custom nodejs scraper tool I whipped out that is similar to wigle.js or wigle-api. This data was downloaded an combined from https://wigle.net/api/v1/jsonSearch.

CURL Download

Pulling code from the wigle2kml.sh script it is trivial to query the Wigle.net API which returns a max of 10,001 results per request (and often takes over a minute to return the respond).

1. Login and save the session cookie

# perhaps we should set noexpire=on
curl -s -c cookie.txt -d "credential_0=USERNAME&credential_1=PASSWORD&noexpire=off" https://wigle.net/gps/gps/GPSDB/login/

2. Query the API

# see API reference for all perameters
curl -b cookie.txt -o results.txt "https://wigle.net/gpsopen/gps/GPSDB/confirmquery?longrange1=LONGRANGE1&longrange2=LONGRANGE2&latrange1=LATRANGE1&latrange2=LATRANGE2&simple=true"

Results are returned in a tilde-delimited list (wtf) that can easily be converted to a CSV on the command line with sed

cat results.txt | sed "s/~/,/g" > results.txt