Skip to content
This repository has been archived by the owner on Jul 29, 2023. It is now read-only.

Script to generate a PostgreSQL schema from GTFS #3

Merged
merged 2 commits into from
Sep 14, 2019
Merged

Script to generate a PostgreSQL schema from GTFS #3

merged 2 commits into from
Sep 14, 2019

Conversation

hfs
Copy link

@hfs hfs commented Sep 7, 2019

This implements the idea already mentioned in the README.md: Write a script to generate a schema based on the contents of an actual GTFS file.

The bash script acts on a directory with the extracted contents of a GTFS file. It checks which files are present and prepares one table for each of them. Then it checks which columns are present and in which order and generates a column defintiion for each of them. Finally, the contents of the CSV files are imported into the database using PostgreSQL's \COPY mode.

Closes #1

The script acts on a directory with the extracted contents of a GTFS
file. It checks which files are present and prepares one table for each
of them. Then it checks which columns are present and in which order and
generates a column defintiion for each of them. Finally, the contents of
the CSV files are imported into the database using PostgreSQL's \COPY
mode.
@tyleragreen
Copy link
Owner

This seems fantastic @hfs! Is there a bash version requirement?

I am running the following command on 3.2.57(1)-release of bash for macOS High Sierra.

./gtfs_schema.sh ~/Downloads/mbta_gtfs/ > mbta_schema.sql

where ~/Downloads/mbta_gtfs is the unzipped data from MBTA GTFS from OpenMobilityData.

I receive the following errors:

./gtfs_schema.sh: line 35: declare: -A: invalid option
declare: usage: declare [-afFirtx] [-p] [name[=value] ...]
./gtfs_schema.sh: line 180: agency.txt/agency_id: syntax error: invalid arithmetic operator (error token is ".txt/agency_id")
./gtfs_schema.sh: line 299: declare: -A: invalid option
declare: usage: declare [-afFirtx] [-p] [name[=value] ...]
./gtfs_schema.sh: line 305: agency.txt/agency_idnagency_namenagency_ulnagency_timezonenagency_langnagency_phone: syntax error: invalid arithmetic operator (error token is ".txt/agency_idnagency_namenagency_ulnagency_timezonenagency_langnagency_phone")
./gtfs_schema.sh: line 314: agency.txt/agency_name: syntax error: invalid arithmetic operator (error token is ".txt/agency_name")
./gtfs_schema.sh: line 329: agency.txt/agency_idnagency_namenagency_ulnagency_timezonenagency_langnagency_phone: syntax error: invalid arithmetic operator (error token is ".txt/agency_idnagency_namenagency_ulnagency_timezonenagency_langnagency_phone")

@hfs
Copy link
Author

hfs commented Sep 11, 2019

Yes, the script requires bash version >= 4. declare -A creates an associative array, a feature which was introduced with bash 4.0. I went with bash because I thought it’s ubiquitous… didn’t know about macOS.

Do you have the option of installing bash 4 as additional package via e.g. Homebrew or Macports?

I could add a version check with an explanation at the top of the script if you want?

Alternatively I could try to rewrite it using only bash 3, but then the script would become quite ugly.

@tyleragreen
Copy link
Owner

Got it! I was able to confirm functionality using bash 5.0 in the image from Dockerhub. Very cool! 🙌

I don't think we need a version check in the script. Want to just throw the version requirement into the README and this will be good to go?

@hfs
Copy link
Author

hfs commented Sep 14, 2019

Sure, I added a note to the README

Copy link
Owner

@tyleragreen tyleragreen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @hfs 🙌

@tyleragreen tyleragreen merged commit 10ca434 into tyleragreen:master Sep 14, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pre-processing script to dynamically create a schema from a GTFS feed
2 participants