Skip to content

ms32035/inspector

Repository files navigation

Inspector

Inspector was created because ETL pipelines never get enough testing.

The main features are:

  • Checks - run SQL queries within your database or across two databases
    • supports for any SqlAlchemy compatible database
    • comparing results against:
      • another SQL query
      • constant values (dates, numbers, strings)
      • Python and SQL expressions
  • Check history - see how your checks behaved in the past
  • Multiple environments - run checks against different instances of the same system
  • Tagging - tag checks to easily group them and execute multiple checks at once
  • Profiling - rapid profiling of database tables using ydata-profiling

You can use Inspector for:

  • monitoring correctness of data in production
  • automated testing of ETL processes
  • confirming that testing has actually been done ;)
  • profiling your data

Some key technical details:

  • Built with Django
  • Uses Celery for asynchronous tasks
  • All functionality is exposed via REST API (Django REST Framework)
  • Fully dockerized
  • SqlAlchemy for database connections
  • ydata-profiling for profiling
  • React frontend - available separately, non-open source, contact the author for more details

Documentation

https://data-inspector.readthedocs.io/

Using docker images

Local development, only Postgres and Redis in Docker

Install local dependencies for all services

pip install -r requirements/local.txt

Run a local django server

export DJANGO_READ_DOT_ENV_FILE=True
export DJANGO_ENV_FILE=.envs/.local/.inspector
docker-compose -f docker-compose-develop.yml up -d postgres redis
./manage.py runserver

You may also use

Docker development

docker-compose -f docker-compose-develop.yml up -d

Access Inspector at http://localhost:8000

Docker-compose - production image

Simply

docker-compose up -d

and Inspector will be available at http://localhost:5000

Obviously, you might want to customize parameters, so check out the files in .envs/example and prepare your own docker-compose.yml with HTTPS reverse proxy in front

User interface

  • Check list

docs/_static/check_list.png

  • Check definition

docs/_static/check_definition.png

  • Check execution history

docs/_static/check_run_history.png

  • Table list

docs/_static/table_list.png

  • Profiling history

docs/_static/profiling_history.png

  • Profiling report (Pandas Profiling)

docs/_static/profiling_report.png

Contributing

You are more than welcome to submit a PR