Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support projections #342

Merged
merged 14 commits into from
Aug 22, 2024
Merged

Support projections #342

merged 14 commits into from
Aug 22, 2024

Conversation

BentsiLeviav
Copy link
Contributor

Summary

close #319

This PR introduces support for projections in dbt-clickhouse. Projections are added to the table and distributed_table materializations as a model setting. For distributed tables, the projection is applied to the _local tables, not to the distributed proxy table.

Tests

I added a test for each materialization. During test development, I encountered two challenges:

  1. To verify that the projection was used by the query, we need to query system.query_log. However, this table takes time to create, which necessitates adding a sleep statement.
  2. Currently, there's no straightforward way to pass a query_id parameter to ClickHouse, making it difficult to retrieve the exact query I want to check. As a workaround, I queried system.query_log using a LIKE statement.

@BentsiLeviav BentsiLeviav requested a review from genzgd August 18, 2024 18:24
@BentsiLeviav BentsiLeviav changed the title Support projectionsxx Support projections Aug 18, 2024
@BentsiLeviav
Copy link
Contributor Author

Writing it here for community awareness - the feature was added both for table and distributed_table materialization, but it doesn't fully function with distributed_table. I marked the distributed test with @pytest.mark.xfail.

@BentsiLeviav BentsiLeviav merged commit b1f6651 into ClickHouse:main Aug 22, 2024
21 checks passed
@arun11299
Copy link

arun11299 commented Feb 6, 2025

@BentsiLeviav Would this be a problem when using Clickhouse Cloud ? Also can you please add an example configuration too ?

@BentsiLeviav
Copy link
Contributor Author

@arun11299 it is supposed to work on ClickHouse Cloud with no problems.
For an example please refer to this changelog update
https://github.com/ClickHouse/dbt-clickhouse/blob/main/CHANGELOG.md#release-182-2024-08-22

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Adding PROJECTION to models
4 participants