Fix `make test` for running integration tests locally #591

dbeatty10 · 2022-05-16T23:02:31Z

Resolves #344 resolves #564

This is a:

All pull requests from community contributors should target the main branch (default).

How to try it out

Clone this repo
Change branches to fix/344-564-make-test
Follow the instructions within integration_tests/README.md

Description & motivation

It should be possible to run integration tests locally by following a set of written instructions. There are instructions right now, but they don't work as written.

Implementation details

Re-write of the Makefile, which includes dropping the test-all target -- each adapter will need to be tested individually
Pulled Python venv creation out of *.sh scripts and into written instructions (similar to how dbt-core has it)
Dropped all Docker containers other than a Postgres database container
Renamed environment variables within integration_tests/.env/* to align with variable names in CircleCI
Checked that the written instructions for work for me :) -- now need to have others confirm/deny

Checklist

I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)
- BigQuery
- Postgres
- Redshift
- Snowflake
~~I have updated the README.md (if applicable)~~
~~I have added tests & descriptions to my models (and macros if applicable)~~
~~I have added an entry to CHANGELOG.md~~

…pinned patch version)

dbeatty10 · 2022-05-16T23:35:37Z

integration_tests/README.md

+#### Set up profiles
+Do either one of the following:
+1. Use `DBT_PROFILES_DIR`
+    ```shell
+    cp integration_tests/ci/sample.profiles.yml integration_tests/profiles.yml
+    export DBT_PROFILES_DIR=$(cd integration_tests && pwd)
+    ```
+2. Use `~/.dbt/profiles.yml`
+    - Copy contents from `integration_tests/ci/sample.profiles.yml` into `~/.dbt/profiles.yml`.


When given the choice, I prefer including a profiles.yml file within each dbt project and then setting DBT_PROFILES_DIR. See example within run_test.sh

Of course this relies on profiles.yml to be completely devoid of actual secrets (which seems eminently doable since we can use environment variables instead).

Any cautions for me before I rename integration_tests/ci/sample.profiles.yml to integration_tests/profiles.yml?

dbeatty10 · 2022-05-16T23:35:58Z

Makefile

+    echo "Install dbt-$(target)..."; \
+    pip install --upgrade pip setuptools; \
+	pip install --pre "dbt-$(target)" -r dev-requirements.txt;


Do we want this to always be --pre? This will include pre-release versions of the dbt adapters (like release candidates).

Or would we want this to typically not include --pre by default, but allow it to be configurable?

dbeatty10 · 2022-05-16T23:47:50Z

integration_tests/README.md

+### Configure credentials
+Edit the env file for your TARGET in `integration_tests/.env/[TARGET].env`.


Do we want to consolidate all the integration_tests/.env/[TARGET].env files into a single test.env file instead?

dbeatty10 · 2022-05-17T00:02:15Z

integration_tests/.env/bigquery.env

I don't think there's anything preventing folks from accidentally committing secrets that they write to locations like integration_tests/.env/bigquery.env

dbeatty10 · 2022-05-17T00:08:01Z

integration_tests/README.md

+Load the environment variables:
+```shell
+set -a; source integration_tests/.env/[TARGET].env; set +a
+```

-To run the integration tests on your local machine, like they will get run in the CI (using CircleCI), you can do the following:
+or more specific:
+```shell
+set -a; source integration_tests/.env/postgres.env; set +a


Is there any more preferable way to store and load environment variables containing secrets?

For non-secret project-specific environment variables, I like using direnv

robertdefilippi · 2022-05-22T15:55:20Z

Just trying out the examples in the README.md and still running into the issues.

Looks like this test is failing after running through the examples in the PR. I'm thinking it is because the database source name is not getting correctly set.

make test target=postgres models=sql.test_star seeds=sql.data_star

/Users/me/.pyenv/shims/dbt
Core:
  - installed: 1.1.1-rc1
  - latest:    1.1.0     - Ahead of latest version!

Plugins:
  - postgres: 1.1.1rc1 - Ahead of latest version!
15:47:18  Running with dbt=1.1.1-rc1
dbt version: 1.1.1-rc1
python version: 3.9.11
python path: /Users/me/.pyenv/versions/3.9.11/envs/dbt-utils/bin/python3.9
os info: macOS-12.3.1-arm64-arm-64bit
Using profiles.yml file at /Users/me/development/sql/dbt-utils/integration_tests/profiles.yml
Using dbt_project.yml file at /Users/me/development/sql/dbt-utils/integration_tests/dbt_project.yml

Configuration:
  profiles.yml file [OK found and valid]
  dbt_project.yml file [OK found and valid]

Required dependencies:
 - git [OK found]

Connection:
  host: localhost
  port: 5432
  user: root
  database: circle_test
  schema: dbt_utils_integration_tests_postgres
  search_path: None
  keepalives_idle: 0
  sslmode: None
  Connection test: [OK connection ok]

All checks passed!
15:47:19  Running with dbt=1.1.1-rc1
15:47:19  Installing ../
15:47:19    Installed from <local @ ../>
15:47:20  Running with dbt=1.1.1-rc1
15:47:20  Partial parse save file not found. Starting full parse.
15:47:21  
Warning: the `surrogate_key` macro now takes a single list argument instead of multiple string arguments. Support for multiple string arguments will be deprecated in a future release of dbt-utils. The dbt_utils_integration_tests.test_surrogate_key model triggered this warning. 
15:47:21  
Warning: the `surrogate_key` macro now takes a single list argument instead of multiple string arguments. Support for multiple string arguments will be deprecated in a future release of dbt-utils. The dbt_utils_integration_tests.test_surrogate_key model triggered this warning. 
15:47:21  
            Warning: the `unpivot` macro no longer accepts a `table` parameter.             This parameter will be deprecated in a future release of dbt-utils. Use the `relation` parameter instead.             The dbt_utils_integration_tests.test_unpivot_original_api model triggered this warning.             
15:47:21  
    Warning: `dbt_utils.unique_where` is no longer supported.
    Starting in dbt v0.20.0, the built-in `unique` test supports a `where` config.
    
15:47:21  
    Warning: `dbt_utils.not_null_where` is no longer supported.
    Starting in dbt v0.20.0, the built-in `not_null` test supports a `where` config.
    
15:47:22  Found 52 models, 91 tests, 0 snapshots, 0 analyses, 389 macros, 0 operations, 80 seed files, 0 sources, 0 exposures, 0 metrics
15:47:22  
15:47:22  Concurrency: 5 threads (target='postgres')
15:47:22  
15:47:22  1 of 1 START seed file dbt_utils_integration_tests_postgres.data_star .......... [RUN]
15:47:22  1 of 1 OK loaded seed file dbt_utils_integration_tests_postgres.data_star ...... [CREATE 3 in 0.10s]
15:47:22  
15:47:22  Finished running 1 seed in 0.50s.
15:47:22  
15:47:22  Completed successfully
15:47:22  
15:47:22  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
15:47:23  Running with dbt=1.1.1-rc1
15:47:24  Found 52 models, 91 tests, 0 snapshots, 0 analyses, 389 macros, 0 operations, 80 seed files, 0 sources, 0 exposures, 0 metrics
15:47:24  
15:47:24  Concurrency: 5 threads (target='postgres')
15:47:24  
15:47:24  1 of 1 START view model dbt_utils_integration_tests_postgres.test_star ......... [RUN]
15:47:24  1 of 1 OK created view model dbt_utils_integration_tests_postgres.test_star .... [CREATE VIEW in 0.19s]
15:47:24  
15:47:24  Finished running 1 view model in 0.53s.
15:47:24  
15:47:24  Completed successfully
15:47:24  
15:47:24  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
15:47:25  Running with dbt=1.1.1-rc1
15:47:25  Found 52 models, 91 tests, 0 snapshots, 0 analyses, 389 macros, 0 operations, 80 seed files, 0 sources, 0 exposures, 0 metrics
15:47:25  
15:47:26  Concurrency: 5 threads (target='postgres')
15:47:26  
15:47:26  1 of 1 START test dbt_utils_equality_test_star_ref_data_star_expected_ ......... [RUN]
15:47:26  1 of 1 ERROR dbt_utils_equality_test_star_ref_data_star_expected_ .............. [ERROR in 0.14s]
15:47:26  CANCEL query test.dbt_utils_integration_tests.dbt_utils_equality_test_star_ref_data_star_expected_.b45cbb2e87  [CANCEL]
15:47:26  CANCEL query list_circle_test_dbt_utils_integration_tests_postgres_events ...... [CANCEL]
15:47:26  
15:47:26  Database Error in test dbt_utils_equality_test_star_ref_data_star_expected_ (models/sql/schema.yml)
15:47:26    relation "dbt_utils_integration_tests_postgres.data_star_expected" does not exist
15:47:26    LINE 36:     select * from "circle_test"."dbt_utils_integration_tests...
15:47:26                               ^
15:47:26    compiled SQL at target/run/dbt_utils_integration_tests/models/sql/schema.yml/dbt_utils_equality_test_star_ref_data_star_expected_.sql
15:47:26  Encountered an error:
FailFast Error in test dbt_utils_equality_test_star_ref_data_star_expected_ (models/sql/schema.yml)
  Failing early due to test failure or runtime error
make: *** [test] Error 1

dbeatty10 · 2022-06-13T19:00:15Z

Yes, thanks for pointing that out, @robertdefilippi !

Doing it in this order worked for me, so I'm updating the instructions accordingly:

Run all tests:
- make test target=postgres
Run tests for a single model:
- make test target=postgres models=sql.test_star seeds=sql.data_star

robertdefilippi · 2022-06-13T19:05:30Z

Ah makes sense! Thank you 🙂

…ions

dbeatty10 added 7 commits May 15, 2022 09:04

Remove venv from CircleCI

37fcf76

Utilize latest minor version of Python 3.9 for CircleCI (rather than …

ec669af

…pinned patch version)

Align local environment variables with CircleCI

163894f

Ignore changes related to running integration tests

b6ed778

Move the make file to the project root

810c897

Refactor make commands to run integration tests

af5c96d

Update instructions for running tests

a965f72

dbeatty10 commented May 16, 2022

View reviewed changes

dbeatty10 commented May 17, 2022

View reviewed changes

dbeatty10 marked this pull request as ready for review May 17, 2022 00:11

dbeatty10 requested a review from jtcohen6 May 17, 2022 00:11

This was referenced May 17, 2022

Cannot build docker for postgres integration test #344

Closed

make test example in /integration_test not running correctly #564

Closed

Implementation guidelines

ad37406

dbeatty10 requested a review from callum-mcdata May 26, 2022 13:28

Switch order of testing all models vs. a single model in the instruct…

fd8eb91

…ions

dbeatty10 merged commit 67ef54b into main Jun 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `make test` for running integration tests locally #591

Fix `make test` for running integration tests locally #591

dbeatty10 commented May 16, 2022 •

edited

Loading

dbeatty10 May 16, 2022

dbeatty10 May 16, 2022

dbeatty10 May 16, 2022

dbeatty10 May 17, 2022

dbeatty10 May 17, 2022

robertdefilippi commented May 22, 2022

dbeatty10 commented Jun 13, 2022

robertdefilippi commented Jun 13, 2022

		### Configure credentials
		Edit the env file for your TARGET in `integration_tests/.env/[TARGET].env`.

Fix make test for running integration tests locally #591

Fix make test for running integration tests locally #591

Conversation

dbeatty10 commented May 16, 2022 • edited Loading

How to try it out

Description & motivation

Implementation details

Checklist

dbeatty10 May 16, 2022

Choose a reason for hiding this comment

dbeatty10 May 16, 2022

Choose a reason for hiding this comment

dbeatty10 May 16, 2022

Choose a reason for hiding this comment

dbeatty10 May 17, 2022

Choose a reason for hiding this comment

dbeatty10 May 17, 2022

Choose a reason for hiding this comment

robertdefilippi commented May 22, 2022

dbeatty10 commented Jun 13, 2022

robertdefilippi commented Jun 13, 2022

Fix `make test` for running integration tests locally #591

Fix `make test` for running integration tests locally #591

dbeatty10 commented May 16, 2022 •

edited

Loading