Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

convert manual steps into a Makefile chain #18

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

dvdotsenko
Copy link

  • separated destination DB from container that connects and loads data
  • separated Ruby runner container from DB image. Now all images are separete and pure as they come from Docker Hub
  • reduced whole download, convert, upload process into 2 make commands

Goals:

  1. Intentionally separate destination DB instance from all the Ruby-based conversion (and runtime) code. Intentionally separated psql runner from destination DB instance (now psql runs in a separate container and quits). This is to allow destination DB to be any instance of Postgres. Just feed credentials to this script and send data to destination Postgres instance.
  2. Use pure vendor images for everything: Ruby, Postgres. Makes things run faster (no image build step) and allows easy swap of versions of Ruby and Postgres runtimes.
  3. I tend to use this a lot now to bootstrap a demo server. Needed full automation and as few commands as possible. Also needed reentrant run (where if AdventureWorks.zip is already downloaded, script automatically just goes to next step. This allowed air-gapped use on internet-limited machines.

- separated destination DB from container that connects and loads data
- separated Ruby runner container from DB image. Now all images are separete and pure as they come from Docker Hub
- reduced whole process to 2 `make` commands
Names are too short to be intuitive. They don't join or transform anything - just copy of table from another schema. They pollute schema definitions in data lineage tools
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant