Pandas interface for observation timeseries

[lkangas]

Having the data output as a flat pandas dataframe greatly improves usability and versatility of the open data output.

Example of turning a result from a weather observations multipointcoverage result into a DF:

rows = []

for date in obs.data.keys():
    for loc in obs.data[date].keys():
        row = obs.data[date][loc].copy()
        for key in row.keys():
            row[key] = row[key]['value']
        row['date'] = date
        row['location'] = loc
        rows.append(row)

df = pd.DataFrame(rows)```

[adriennn]

the above doesn't work as such, the times columns is a list, so you'll need to do check if the entry bein processed has the 'values' key.

[mikaelhg]

Here's what I've come up with:

args = ['timeseries=True', f"starttime={start_time}", f"endtime={end_time}"]
obs = download_stored_query(query, args=args)

cols = set([v for p in obs.data for v in obs.data[p]])
cols.remove('times')

dfs = []
for name in obs.data:
    data = {k: obs.data[name][k]['values'] for k in cols}
    idx = pd.DatetimeIndex(name='hour', data=obs.data[name]['times'])
    idx0 = pd.CategoricalIndex(name='place', data=[name]*idx.size)
    df = pd.DataFrame(data=data, index=[idx0, idx], columns=cols, dtype='float64')
    dfs.append(df)

df = pd.concat(dfs)

df.attrs.update({'location_metadata': obs.location_metadata})

df.to_parquet('data/airquality.parquet')

If you wanted to compress this, unfortunately Pandas and numpy don't offer compressed fixed point formats, but you can use int16 and multiply the values by 10 to achieve that.

Alternative indexing:

    mi = pd.MultiIndex.from_product([[name], obs.data[name]['times']], names=['place', 'hour'])
    df = pd.DataFrame(data=data, index=mi, columns=cols, dtype='float64')