Scope out SDG indicator database ingestion workload

[kzollove]

The SDG indicator database will be a very valuable resource, specifically for the IDSR / CODATA use case.

SDG has a well-documented API for accessing the SDG indicator database.
https://unstats.un.org/SDGAPI/swagger/

Scope out a plan for ingesting the SDG indicator database into Gaia data source and variable source records.

First figure out the logical approach from gaia perspective:

• What subset of countries to bring in?
• What subset of indicators to bring in?
• How to break down data sources (years, countries, regions, other?)

Then handle the technical side of things (API support in gaia)

• GaiaDB data source records support API ingest

Then handle the scripting for the API calls

• How to structure an API call to use in the data_source record?
• How to structure an API call to get variable source information? This would be used to help generate the infromation for the variable_source record, but the call itself would (likely) not be stored

Depends on:

• Data Request: DIVA-GIS (country- and global-level geometry data) #31

[rtmill]

At a glance, this seems to be country-level at the most granular? If that is the case, do we have use cases for content that broad?

[jaygee-on-github]

@rtmill, actually there are small area estimates for many of the SDGs. These estimates can go to the regional and "traditional authority" (aka community) levels. If you are ok with this kind of granularity, maybe I can proceed with the" logical approach from gaia perspective" as outlined by @kzollove. What do you think?

[jaygee-on-github]

@kzollove, @tibbben, @rtmill...

Here is a framework I propose for hosting the SDG indicators:

Each indicator would have its own catalog entry. The catalog entry would refer to a dataset in which the rows are regions and the columns successively disaggregate an indicator. Depending on the indicator, a disaggregation might not include males or females as appropriate.

Note that in this framework a region flattens a hierarchy. Depending on granularity, a country may contain areas which may contain an areas. In this event there would be a row for each country / area / area.

In the first sprint we would NOT tackle the SDMX SDG API. Instead we would prepare one or more datasets -- again, one for each indicator.

So far I haven't tried to specify the table, column and cell level metadata we might support.