project-brian

Data pipeline for ONS website

This service converts CSDB or CSV files to JSON for use by time series datasets.

Context

Zebedee converts uploaded CSDB or CSV files in time series datasets into separate individual time-series at collection approval stage. It does this by POSTing the csdb or csv file to Brian and receiving a JSON file in return which it then processes in DataPublication.java (Link correct as at zebedee v2.13.0)

How to debug locally

First obtain a valid CSDB or CSV file. For example…

• emp.csdb - ('structured text') on a time series like Average weekly earnings
• upload-pop.csv - ('csv') on Population estimates time series dataset

To run project brian, use:

make debug

Now, post the downloaded file to the service using:

curl -X POST -F "[email protected]" http://localhost:8083/Services/ConvertCSDB -o csdb.json

or for CSV…

curl -X POST -F "[email protected]" http://localhost:8083/Services/ConvertCSV -o csv.json

The output will be the JSON that would be sent back to zebedee.