Skip to content

The easiest way to write advanced queries against the Get Information About Schools (GIAS) database

Notifications You must be signed in to change notification settings

DFE-Digital/gias-query-tool

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

61 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

The [Unofficial] GIAS Query Toolkit

Ever wanted quickly query the Get Information About Schools dataset but couldn't because it's provided in an unwieldy CSV file that's not properly-encoded and has too many columns?

Great, you're in the right place!

This set of tools downloads and imports the data into a locally-running PostgreSQL database and lets you take advantage of PostGIS to supercharge your queries.

Getting up and running

Prerequisites

  • GNU Make, used to run the automatic download and import
  • GNU Iconv, fix file encoding
  • GNU Wget for downloading the GIAS CSV file
  • PostgreSQL with an local superuser account
  • PostGIS for geographic query goodness

Running the command

To download, cleanse, import and build the data objects only a single command should be required.

make

When debugging, use make refresh to run the import steps without repeatedly downloading the export file.

make refresh

Manual importing

The entire import, apart from file cleansing, is written in standard SQL. Executing the statements needs to be done in the correct order, the Makefile is the best place to get a feel for how it works.

Sending this data to BigQuery

Dependencies

pip install bigquery-schema-generator
brew install gcloud-cli
gcloud auth login
gcloud config set project PROJECT_ID

The script assumes you have a gias dataset in your BigQuery project and a Google Cloud Storage bucket called rugged-abacus-uploads.

Uploading

First build the tables in your local database

make

Then run the import

make load_to_bq

Tables and views

The importer creates the following database objects:

Name Type Description
schools table All schools, both open and closed
deprivation_pupil_premium table DPP information broken down by school
open_schools materialized view Only open schools
regions table England's regions and associated gegoraphic information
local_authorities table England's local authorities and associated gegoraphic information
establishment type School types (eg. Foundation school, Free school)
establishment_group type School categories (eg. Independent Schools, Universities, Colleges
gender type School gender policies (eg. Boys, Girls, Mixed)
ofsted_rating type All Ofsted ratings, including deprecated ones
phase type School phases (eg. Secondary, Primary, 16 plus)
rural_urban_classification type Classification of a school's setting, source links in definition

FAQs

Why use enumerated types when you could've just used a varchar?

Efficiency aside, the main reason is to allow ordering by rank rather than alphabetic position.

I want to query a column that's not included in the import, how do I add it?

  1. Add your desired column to the create statement in ddl/tables/create_schools.sql. Ensure it is the correct datatype.
  2. Then add your new column name to the list of target columns in dml/import_schools.sql and add a the corresponding column in the source data (i.e. the source CSV) to the select part of the statement. For datatypes other than varchar, cast appropriately with ::new_datatype There are plenty of examples of casting in the file already
  3. See if it worked, run make refresh to drop everything and re-import

Where can I find and how do I import other geographic data?

There are plenty of great sources for educational geographic data:

Once you've found a useful dataset, select the Shapefile download option if available. Now we can use shp2pgsql to import it. It comes with a GUI which makes the process very simple.

shp2pgsql spatial data loader

Alternatively, and more-flexibly (if the dataset you want isn't available as a Shapefile), you can use GDAL's ogr2ogr.

Nomenclature

Word Definition
EduBase The old name for Get information about schools (GIAS)
Ofsted The Office for Standards in Education, Children's Services and Skills (Ofsted) is a non-ministerial department of the UK government, reporting to Parliament.A
URN A six-digit number used by the UK government to identify educational establishments in the United Kingdom.

Example queries

"What's the breakdown of school genders by Ofsted rating?" πŸ˜•

select
  os.ofsted_rating as "Ofsted rating",
  os.gender,
  count(*)
from
  open_schools os
group by
  os.ofsted_rating,
  os.gender
order by
  os.ofsted_rating,
  os.gender
\crosstabview

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚    Ofsted rating     β”‚ Boys β”‚ Girls β”‚ Mixed β”‚ (null) β”‚ Not applicable β”‚
β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•ͺ══════β•ͺ═══════β•ͺ═══════β•ͺ════════β•ͺ════════════════║
β”‚ Outstanding          β”‚   58 β”‚    95 β”‚  3343 β”‚      1 β”‚                β”‚
β”‚ Good                 β”‚  139 β”‚   107 β”‚ 13844 β”‚      1 β”‚                β”‚
β”‚ Requires improvement β”‚   37 β”‚    19 β”‚  2017 β”‚        β”‚                β”‚
β”‚ Inadequate           β”‚   20 β”‚    13 β”‚    70 β”‚        β”‚                β”‚
β”‚ Serious Weaknesses   β”‚    2 β”‚     1 β”‚    97 β”‚        β”‚                β”‚
β”‚ Special Measures     β”‚    7 β”‚     1 β”‚   165 β”‚        β”‚                β”‚
β”‚ (null)               β”‚  166 β”‚   225 β”‚  4886 β”‚    256 β”‚           1345 β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

"Find all the schools within 3km of Stonehenge" πŸ€”

select
  os.urn,
  os.name
from
  open_schools os
where st_dwithin(
  os.coordinates,                         -- Database column that holds the school's location
  st_setsrid(
    st_makepoint(-1.826194, 51.178868),   -- Stonehenge's coords
    4326                                  -- World Geodetic System
  ),
  3000                                    -- Search radius in metres
);


β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  urn   β”‚                     name                      β”‚
β•žβ•β•β•β•β•β•β•β•β•ͺ═══════════════════════════════════════════════║
β”‚ 145545 β”‚ Larkhill Primary School                       β”‚
β”‚ 143006 β”‚ St Michael's Church of England Primary School β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Obligatory sense check 🧐

larkhill_primary

Looks good!

"I'd like a percentile summary of the twenty local authorities with the lowest average deprivation pupil premium, excluding authorities with fewer than fifteen qualifying schools" 🀭

select
    os.local_authority,
    percentile_disc(0.4) within group (order by dpp.allocation) as "P40", -- discrete percentile at 0.4 (40%)
    percentile_disc(0.5) within group (order by dpp.allocation) as "P50",
    percentile_disc(0.6) within group (order by dpp.allocation) as "P60",
    percentile_disc(0.7) within group (order by dpp.allocation) as "P70",
    percentile_disc(0.8) within group (order by dpp.allocation) as "P80",
    percentile_disc(0.9) within group (order by dpp.allocation) as "P90"
from
    deprivation_pupil_premium dpp
inner join
    open_schools os on dpp.urn = os.urn
group by
    os.local_authority
having
    count(*) > 15                                                         -- only select local authorities with more than fifteen schools
order by
    avg(dpp.allocation::decimal) asc                                      -- order by DPP allocation ascending, we want the lowest
limit
    20
;

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚     local_authority      β”‚    P40     β”‚    P50     β”‚    P60     β”‚    P70     β”‚     P80     β”‚     P90     β”‚
β•žβ•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•β•ͺ════════════β•ͺ════════════β•ͺ════════════β•ͺ════════════β•ͺ═════════════β•ͺ═════════════║
β”‚ Rutland                  β”‚ Β£13,200.00 β”‚ Β£15,840.00 β”‚ Β£18,480.00 β”‚ Β£23,760.00 β”‚  Β£43,560.00 β”‚  Β£70,125.00 β”‚
β”‚ North Yorkshire          β”‚ Β£10,560.00 β”‚ Β£15,895.00 β”‚ Β£25,080.00 β”‚ Β£40,920.00 β”‚  Β£62,645.00 β”‚  Β£98,175.00 β”‚
β”‚ Cumbria                  β”‚ Β£14,520.00 β”‚ Β£20,570.00 β”‚ Β£31,680.00 β”‚ Β£48,620.00 β”‚  Β£64,680.00 β”‚ Β£118,800.00 β”‚
β”‚ West Berkshire           β”‚ Β£19,800.00 β”‚ Β£25,080.00 β”‚ Β£38,280.00 β”‚ Β£52,800.00 β”‚  Β£74,800.00 β”‚ Β£100,320.00 β”‚
β”‚ Windsor and Maidenhead   β”‚ Β£26,400.00 β”‚ Β£31,680.00 β”‚ Β£43,560.00 β”‚ Β£60,720.00 β”‚  Β£74,800.00 β”‚ Β£100,045.00 β”‚
β”‚ Wokingham                β”‚ Β£22,440.00 β”‚ Β£34,320.00 β”‚ Β£43,560.00 β”‚ Β£58,080.00 β”‚  Β£73,865.00 β”‚  Β£91,080.00 β”‚
β”‚ Herefordshire, County of β”‚ Β£21,120.00 β”‚ Β£26,400.00 β”‚ Β£34,320.00 β”‚ Β£50,490.00 β”‚  Β£62,040.00 β”‚ Β£118,800.00 β”‚
β”‚ Wiltshire                β”‚ Β£21,120.00 β”‚ Β£31,680.00 β”‚ Β£44,880.00 β”‚ Β£59,400.00 β”‚  Β£81,840.00 β”‚ Β£114,840.00 β”‚
β”‚ Buckinghamshire          β”‚ Β£22,440.00 β”‚ Β£29,040.00 β”‚ Β£37,895.00 β”‚ Β£47,685.00 β”‚  Β£64,680.00 β”‚ Β£135,465.00 β”‚
β”‚ Shropshire               β”‚ Β£18,480.00 β”‚ Β£23,760.00 β”‚ Β£40,920.00 β”‚ Β£62,645.00 β”‚  Β£83,215.00 β”‚ Β£125,290.00 β”‚
β”‚ Central Bedfordshire     β”‚ Β£22,440.00 β”‚ Β£35,640.00 β”‚ Β£50,160.00 β”‚ Β£63,360.00 β”‚  Β£89,760.00 β”‚ Β£128,095.00 β”‚
β”‚ Oxfordshire              β”‚ Β£22,440.00 β”‚ Β£30,360.00 β”‚ Β£42,240.00 β”‚ Β£61,380.00 β”‚  Β£88,440.00 β”‚ Β£134,173.00 β”‚
β”‚ Cheshire East            β”‚ Β£19,800.00 β”‚ Β£29,040.00 β”‚ Β£47,520.00 β”‚ Β£68,640.00 β”‚ Β£100,320.00 β”‚ Β£144,925.00 β”‚
β”‚ South Gloucestershire    β”‚ Β£26,400.00 β”‚ Β£34,320.00 β”‚ Β£43,560.00 β”‚ Β£62,040.00 β”‚  Β£81,840.00 β”‚ Β£136,043.00 β”‚
β”‚ Devon                    β”‚ Β£21,120.00 β”‚ Β£26,400.00 β”‚ Β£44,880.00 β”‚ Β£60,720.00 β”‚  Β£92,400.00 β”‚ Β£141,240.00 β”‚
β”‚ Dorset                   β”‚ Β£22,440.00 β”‚ Β£31,790.00 β”‚ Β£47,520.00 β”‚ Β£68,640.00 β”‚  Β£92,400.00 β”‚ Β£130,680.00 β”‚
β”‚ Gloucestershire          β”‚ Β£23,760.00 β”‚ Β£33,000.00 β”‚ Β£49,060.00 β”‚ Β£66,000.00 β”‚  Β£96,305.00 β”‚ Β£133,705.00 β”‚
β”‚ Leicestershire           β”‚ Β£30,360.00 β”‚ Β£44,880.00 β”‚ Β£58,080.00 β”‚ Β£71,280.00 β”‚  Β£91,080.00 β”‚ Β£134,640.00 β”‚
β”‚ Somerset                 β”‚ Β£27,720.00 β”‚ Β£36,960.00 β”‚ Β£51,425.00 β”‚ Β£72,600.00 β”‚  Β£90,695.00 β”‚ Β£153,120.00 β”‚
β”‚ Surrey                   β”‚ Β£33,000.00 β”‚ Β£42,240.00 β”‚ Β£56,540.00 β”‚ Β£75,240.00 β”‚  Β£99,110.00 β”‚ Β£141,240.00 β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Try doing that in Excel πŸ˜…

"List all the currently-open schools in London excluding those in Kensington and Chelsea, Southwark, and Tower Hamlets" 🀨

with local_authorities_to_exclude as (
  select
    st_union(la.edge) as edges               -- union multiple edges into a single geometry
  from
    local_authorities la
  where
    name in (
      'Kensington and Chelsea',
      'Southwark',
      'Tower Hamlets'
    )
)
select
  distinct on (urn)
  os.urn,
  os.name,
  os.coordinates
from
  open_schools os
inner join                                   -- join on region containing coordinates
  regions r
    on st_contains(
      r.edge,
      os.coordinates::geometry
    )
inner join                                   -- exclude the named LAs from above
  local_authorities la
    on not st_contains(
      (select edges from local_authorities_to_exclude),
      os.coordinates::geometry
    )
where
  r.name = 'London'
;

There are too many results to list, but here's a screenshot displaying the results in QGIS. Note that QGIS fully supports PostGIS, all queries that include a geospatial column can be displayed and manipulated by the software and used to create reports or perform advanced queries.

Schools in London minus Kensington and Chelsea, Tower Hamlets and Southwark

About

The easiest way to write advanced queries against the Get Information About Schools (GIAS) database

Topics

Resources

Code of conduct

Security policy

Stars

Watchers

Forks

Languages