-
Notifications
You must be signed in to change notification settings - Fork 294
Add sample dbt_packages to validate incremental dbt deps
#1669
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
181 changes: 181 additions & 0 deletions
181
dev/dags/dbt/simple/dbt_packages/dbt_date/.circleci/config.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,181 @@ | ||
| version: 2.1 | ||
|
|
||
| jobs: | ||
| integration-tests-core: | ||
| docker: | ||
| - image: cimg/python:3.9.9 | ||
| - image: cimg/postgres:14.0 | ||
|
|
||
| resource_class: small | ||
|
|
||
| environment: | ||
| DBT_PROFILES_DIR: ./integration_tests/ci | ||
| DBT_PROJECT_DIR: ./integration_tests | ||
| BIGQUERY_SERVICE_KEY_PATH: "/home/circleci/bigquery-service-key.json" | ||
| DBT_VERSION: 1.8.* | ||
|
|
||
| steps: | ||
| - checkout | ||
| - run: &pip-install-core | ||
| name: Install core Python packages & dbt-core | ||
| command: | | ||
| python3 -m venv venv | ||
| . venv/bin/activate | ||
| pip install -U pip setuptools wheel | ||
| pip install "dbt-core==$DBT_VERSION" | ||
|
|
||
| - run: | ||
| name: Install dbt adapter packages | ||
| command: | | ||
| python3 -m venv venv | ||
| . venv/bin/activate | ||
| pip install "dbt-postgres==$DBT_VERSION" "dbt-bigquery==$DBT_VERSION" "dbt-snowflake==$DBT_VERSION" | ||
| pip install "dbt-duckdb==$DBT_VERSION" | ||
|
|
||
| - run: &dbt-deps | ||
| name: Install dbt dependencies | ||
| command: | | ||
| . venv/bin/activate | ||
| dbt deps --project-dir $DBT_PROJECT_DIR | ||
|
|
||
| - run: | ||
| name: "Run Tests - Postgres" | ||
| environment: | ||
| POSTGRES_HOST: localhost | ||
| POSTGRES_TEST_USER: postgres | ||
| POSTGRES_TEST_PASSWORD: "" | ||
| POSTGRES_TEST_PORT: 5432 | ||
| POSTGRES_TEST_DATABASE: circle_test | ||
| POSTGRES_TEST_SCHEMA: dbt_date_integration_tests | ||
| command: | | ||
| . venv/bin/activate | ||
| dbt build -t postgres --project-dir $DBT_PROJECT_DIR | ||
|
|
||
| - run: | ||
| name: "Set up GCP credentials" | ||
| command: | | ||
| echo "Writing to $BIGQUERY_SERVICE_KEY_PATH" | ||
| echo $BIGQUERY_SERVICE_KEY > $BIGQUERY_SERVICE_KEY_PATH | ||
| FILESIZE=$(stat -c%s "$BIGQUERY_SERVICE_KEY_PATH") | ||
| echo "Size of $BIGQUERY_SERVICE_KEY_PATH = $FILESIZE bytes." | ||
| echo "BIGQUERY_TEST_DATABASE = $BIGQUERY_TEST_DATABASE" | ||
|
|
||
| - run: | ||
| name: "Run Tests - BigQuery" | ||
| command: | | ||
| . venv/bin/activate | ||
| dbt build -t bigquery --project-dir $DBT_PROJECT_DIR | ||
|
|
||
| - run: | ||
| name: "Run Tests - Snowflake" | ||
| command: | | ||
| . venv/bin/activate | ||
| dbt build -t snowflake --project-dir $DBT_PROJECT_DIR | ||
|
|
||
| - run: | ||
| name: "Run Tests - DuckDB" | ||
| command: | | ||
| . venv/bin/activate | ||
| dbt build -t duckdb --project-dir $DBT_PROJECT_DIR | ||
|
|
||
| - store_artifacts: | ||
| path: ./logs | ||
|
|
||
| integration-tests-spark-thrift: | ||
| docker: | ||
| - image: cimg/python:3.9.9 | ||
| - image: godatadriven/spark:3.1.1 | ||
| environment: | ||
| WAIT_FOR: localhost:5432 | ||
| command: > | ||
| --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 | ||
| --name Thrift JDBC/ODBC Server | ||
| - image: postgres:9.6.17-alpine | ||
| environment: | ||
| POSTGRES_USER: dbt | ||
| POSTGRES_PASSWORD: dbt | ||
| POSTGRES_DB: metastore | ||
|
|
||
| resource_class: small | ||
|
|
||
| environment: | ||
| DBT_PROFILES_DIR: ./integration_tests/ci | ||
| DBT_PROJECT_DIR: ./integration_tests | ||
| DBT_VERSION: 1.8.* | ||
|
|
||
| steps: | ||
| - checkout | ||
| - run: | ||
| name: Install Ubuntu packages | ||
| command: | | ||
| sudo apt-get update | ||
| sudo apt-get install libsasl2-dev libsasl2-2 | ||
| - run: *pip-install-core | ||
| - run: | ||
| name: Install dbt adapter packages | ||
| command: | | ||
| python3 -m venv venv | ||
| . venv/bin/activate | ||
| pip install "dbt-spark==$DBT_VERSION" "dbt-spark[PyHive]==$DBT_VERSION" | ||
| - run: *dbt-deps | ||
| - run: | ||
| name: Wait for Spark-Thrift | ||
| command: dockerize -wait tcp://localhost:10000 -timeout 15m -wait-retry-interval 5s | ||
| - run: | ||
| name: "Run Tests - Spark" | ||
| command: | | ||
| . venv/bin/activate | ||
| dbt build -t spark --project-dir $DBT_PROJECT_DIR | ||
|
|
||
| - store_artifacts: | ||
| path: ./logs | ||
|
|
||
| integration-tests-trino: | ||
| docker: | ||
| - image: cimg/python:3.11 | ||
| - image: trinodb/trino:431 | ||
|
|
||
| resource_class: small | ||
|
|
||
| environment: | ||
| DBT_PROFILES_DIR: ./integration_tests/ci | ||
| DBT_PROJECT_DIR: ./integration_tests | ||
| DBT_VERSION: 1.8.* | ||
|
|
||
| steps: | ||
| - checkout | ||
| - run: *pip-install-core | ||
| - run: | ||
| name: Install dbt adapter packages | ||
| command: | | ||
| python3 -m venv venv | ||
| . venv/bin/activate | ||
| pip install "dbt-trino==$DBT_VERSION" | ||
| - run: *dbt-deps | ||
| - setup_remote_docker | ||
| - run: | ||
| name: Run Trino server | ||
| command: | | ||
| docker run --name trino -p 8080:8080 -d -v `pwd`/integration_tests/docker/trino/catalog:/etc/trino/catalog trinodb/trino:431 | ||
| timeout 5m bash -c -- 'while ! docker logs trino 2>&1 | tail -n 1 | grep "SERVER STARTED"; do sleep 2; done' | ||
| - run: | ||
| name: "Run Tests - Trino" | ||
| command: | | ||
| . venv/bin/activate | ||
| dbt build -t trino --project-dir $DBT_PROJECT_DIR | ||
|
|
||
| workflows: | ||
| version: 2 | ||
| test-all: | ||
| jobs: | ||
| - hold: | ||
| type: approval | ||
| - integration-tests-core: | ||
| requires: | ||
| - hold | ||
| - integration-tests-spark-thrift: | ||
| requires: | ||
| - hold | ||
| - integration-tests-trino: | ||
| requires: | ||
| - hold | ||
39 changes: 39 additions & 0 deletions
39
dev/dags/dbt/simple/dbt_packages/dbt_date/.github/ISSUE_TEMPLATE/bug_report.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| --- | ||
| name: Bug report | ||
| about: I think I've found a bug | ||
| title: "[BUG]" | ||
| labels: "" | ||
| assignees: "" | ||
| --- | ||
|
|
||
| ### Is this a new bug in dbt-date? | ||
|
|
||
| - [ ] I believe this is a new bug in dbt-date | ||
| - [ ] I have searched the existing issues, and I could not find an existing issue for this bug | ||
|
|
||
| ### Current Behavior | ||
|
|
||
| ### Expected Behavior | ||
|
|
||
| ### Steps To Reproduce | ||
|
|
||
| ### Relevant log output | ||
|
|
||
| ### Environment | ||
|
|
||
| ```markdown | ||
| - OS: | ||
| - Python: | ||
| - dbt: | ||
| - dbt-expectations: | ||
| ``` | ||
|
|
||
| ### Which database adapter are you using with dbt? | ||
|
|
||
| Note: dbt-date currently does not support database adapters other than the ones listed below. | ||
|
|
||
| - Postgres | ||
| - Snowflake | ||
| - BigQuery | ||
|
|
||
| ### Additional Context |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
|
|
||
| target/ | ||
| dbt_packages/ | ||
| logs/ | ||
| .python-version | ||
| integration_tests/.spark-warehouse | ||
| integration_tests/.hive-metastore |
27 changes: 27 additions & 0 deletions
27
dev/dags/dbt/simple/dbt_packages/dbt_date/.pre-commit-config.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| --- | ||
| repos: | ||
| - repo: https://github.com/pre-commit/pre-commit-hooks | ||
| rev: v5.0.0 | ||
| hooks: | ||
| - id: trailing-whitespace | ||
| - id: check-json | ||
| - id: check-ast | ||
| - id: check-merge-conflict | ||
| - id: check-toml | ||
| - id: check-yaml | ||
| args: [--unsafe] | ||
| - id: debug-statements | ||
| - id: detect-private-key | ||
| - id: end-of-file-fixer | ||
| exclude: macros/calendar_date/week_of_year.sql | ||
| - repo: https://github.com/pre-commit/mirrors-prettier | ||
| rev: "v3.1.0" | ||
| hooks: | ||
| - id: prettier | ||
| types_or: [json, markdown, yaml] | ||
| - repo: https://github.com/tconbeer/sqlfmt | ||
| rev: v0.26.0 | ||
| hooks: | ||
| - id: sqlfmt | ||
| language_version: python | ||
| additional_dependencies: [".[jinjafmt]"] |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.