Add versioned migrations

[GAlexIHU]

Summary

• We have to use atlas to generate the diffs based on the ent schema we declare
• Atlas doesn't support multiple histories in a single database/schema (plain english: you can't change the lock table) so for migration running we have to use stg else (golang-migrate in this case)
• Ent's Schema.Migrate is replaced by tool/migrate custom utility for running migrations as part of Application Startup. This behavior is configurable via PostgresConfig.AutoMigrate field/type.

Current Workflow

1. Do your changes in openmeter/ent, then atlas migrate --env local diff <diffName> creates the new migration files
2. You can apply your migrations with

migrate -database "postgres://postgres:postgres@localhost:5432/postgres?sslmode=disable&x-migrations-table=schema_om" -path ./tools/migrate/migrations up

Or you can set postgres.autoMigrate=migration to run migrations at startup

Other useful commands

# Lint migrations (atlas linting is quite good)
atlas migrate --env local lint
atlas migrate --env local-om lint # For Cloud only referencing the OM migrations

# Validate migrations directory
atlas migrate --env local validate

Added CI & Consistency Cheks

• We validate that all code is generated (meaning the ent schema is in sync with the generated client)
• We validate that the migrations directory is in sync with the schema
• We run linting on the migrations directory to adhere guard against destructive changes
• We validate migrations directory consistency via atlas checksum
• We test that the migrations do run (up-down-up on an empty DB)

Future Improvements on Current Workflow

• Using x-migrations-table in the connection string is a burden running manually. tools/migrate sets it consistently when invoking from code. A possible direction going forward could be to build on that custom tooling for both the local dev scripts as well as CI & Deploy automations (with all the drawbacks of having custom tooling for this)

Notes for Reviwers

• There's a sister PR to this one, you can check it here
• postgres.autoMigrate has a default value of ent which is same as the current default. However config.example.yaml promotes using the migration value

[turip]

I think you did a great job with this. 🙇

I would love if we could make the migration generation part of the make generate flow, just for ease of use. I suppose that's not done due to the naming requirement, right?

I would love if @sagikazarmark would also take a look, before merging.

[chrisgacsal]

Awesome job! Would love to have others also taking a look on it, but I am happy with the current solution.