Fix for inconsistent Bigint/Long datatype that makes dbt think there was a schema change #358

francescomucio · 2022-05-16T14:34:48Z

resolves #642

Description

This PR fixes the inconsistent datatype returned for the Bigint columns (bigint vs long) by Spark sql.

Checklist

I have signed the CLA
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have updated the CHANGELOG.md and added information about my change to the "dbt-spark next" section.

cla-bot · 2022-05-16T14:34:50Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Francesco Mucio.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

jtcohen6 · 2022-05-16T15:04:36Z

@cla-bot check

cla-bot · 2022-05-16T15:04:39Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Francesco Mucio.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2022-05-16T15:04:41Z

The cla-bot has been summoned, and re-checked this pull request!

jtcohen6 · 2022-05-16T15:05:20Z

@francescomucio Thanks for the PR!

Could you rebase/squash your commits and force push? It looks like the first one doesn't have a git identity set, which is why the CLA bot isn't happy

cla-bot · 2022-05-16T15:47:58Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Francesco Mucio.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

* More consistent results from get_columns_in_relation * Not dispatched, full name * Add changelog entry

McKnight-42 · 2022-06-06T19:17:37Z

@francescomucio sorry for the delay this is looking very good, do you think you will have time to update the branch to clear up the conflicts? would love to keep your local and remote up to date with one another.

francescomucio · 2022-06-08T14:09:26Z

@McKnight-42 sorry for the late reply, I will try to work on this today. BTW I reached out to the Databricks support and they told me that the problem is also in opensource Apache Spark. Should I prepare a PR also for that adapter?

jtcohen6 · 2022-06-15T08:41:39Z

CHANGELOG.md

-### Under the hood
- Add `DBT_INVOCATION_ENV` environment variable to ODBC user agent string ([#366](https://github.com/dbt-labs/dbt-spark/pull/366))
-


Not sure how this changelog entry disappeared — mind adding it back, and adding one of your own? :)

ups, sure, I will do it

francescomucio · 2022-06-29T20:30:22Z

@McKnight-42 @jtcohen6 sorry for being late, I finally found the time to fix this PR. I guess this will go in 1.2.0, not in 1.1.1

McKnight-42 · 2022-07-25T17:06:02Z

@francescomucio This is looking fantastic seems once you did some pushing up the pre-commit test started failing for end-of-line spacing. if you could rerun the pre-commit command locally again and add those eof changes should fix it.

francescomucio · 2022-07-26T07:56:26Z

@McKnight-42 I think I fixed it

McKnight-42 · 2022-07-26T18:58:01Z

@francescomucio thank you so much, i'm sorry about this we just released some new versions for dbt-core and the adapters this morning, if you could pull down the latest changes should fix the version mismatch issue and get the ci running again.

) * Not dropping table for incremental full refresh with delta * Updated changelog * Simplified conditional logic according to suggestion * Updated changelog * Only drop table if not delta table Co-authored-by: Jeremy Cohen <[email protected]> * Update changelog, trigger CircleCI tests Co-authored-by: Jeremy Cohen <[email protected]> Co-authored-by: Jeremy Cohen <[email protected]>

* Run tests for data type macros. Fine tune numeric_type * Hard code seed loading types for float + int * Repoint, fixup, changelog entry

cla-bot · 2022-09-23T16:37:23Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Francesco Mucio.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2022-09-23T16:39:06Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Francesco Mucio.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2022-09-23T16:40:15Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Francesco Mucio.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2022-09-23T16:42:56Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Francesco Mucio.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2022-09-23T16:44:20Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Francesco Mucio.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2022-09-23T16:44:40Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Francesco Mucio.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2022-09-23T16:45:32Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Francesco Mucio.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

McKnight-42 · 2022-10-04T18:36:43Z

@jtcohen6 would this be by setting the on_schema_change to "sync_all_columns" so that it does the type checking?

https://docs.getdbt.com/docs/building-a-dbt-project/building-models/configuring-incremental-models#what-if-the-columns-of-my-incremental-model-change

francescomucio · 2022-11-17T09:55:13Z

@McKnight-42 @jtcohen6 any update on this?

We are trying to upgrade to dbt version 1.3 and I still need to apply this fix.

ueshin

What's the status of this change?

ueshin · 2022-12-07T21:43:08Z

dbt/adapters/spark/column.py

@@ -46,7 +50,7 @@ def numeric_type(cls, dtype: str, precision: Any, scale: Any) -> str:
            return "{}({},{})".format("decimal", precision, scale)

    def __repr__(self) -> str:
-        return "<SparkColumn {} ({})>".format(self.name, self.data_type)
+        return "<SparkColumn {} ({})>".format(self.name, self.translate_type(self.data_type))


self.data_type is already translated?

github-actions · 2023-09-14T01:43:30Z

This PR has been marked as Stale because it has been open with no activity as of late. If you would like the PR to remain open, please comment on the PR or else it will be closed in 7 days.

github-actions · 2024-08-06T01:50:37Z

This PR has been marked as Stale because it has been open with no activity as of late. If you would like the PR to remain open, please comment on the PR or else it will be closed in 7 days.

fixing biging/long unconsistent datatype

26c9d42

francescomucio force-pushed the main branch from ab535ac to 23ac892 Compare May 16, 2022 15:47

More consistent results from get_columns_in_relation (dbt-labs#355)

98ea5e3

* More consistent results from get_columns_in_relation * Not dispatched, full name * Add changelog entry

francescomucio force-pushed the main branch from 23ac892 to 98ea5e3 Compare May 16, 2022 19:03

cla-bot bot added the cla:yes label May 16, 2022

McKnight-42 requested review from VersusFacit and McKnight-42 May 17, 2022 18:45

Merge branch 'main' into main

6bb2f82

jtcohen6 reviewed Jun 15, 2022

View reviewed changes

jtcohen6 added the ready_for_review Externally contributed PR has functional approval, ready for code review from Core engineering label Jun 15, 2022

francescomucio added 2 commits June 29, 2022 22:20

merged upstream

4c944e3

added entry in CHANGELOG.md

ccfb64f

fixed end-of-file

a8e4302

jtcohen6 and others added 5 commits July 26, 2022 21:58

Revert dbt-labs#367 (dbt-labs#378)

d39b0a3

Data type macros (dbt-labs#380)

cfda7ed

* Run tests for data type macros. Fine tune numeric_type * Hard code seed loading types for float + int * Repoint, fixup, changelog entry

Fix changelog for spark upgrade feature (dbt-labs#385)

3675494

Use lowercase file ext for CONTRIBUTING.md (dbt-labs#384)

af7fae3

cla-bot bot removed the cla:yes label Sep 23, 2022

francescomucio force-pushed the main branch from a138fd7 to 7462b03 Compare September 23, 2022 16:39

francescomucio force-pushed the main branch from b58adf2 to 7a652fe Compare September 23, 2022 16:44

francescomucio force-pushed the main branch from dd6c05c to d90d54e Compare September 23, 2022 16:45

francescomucio force-pushed the main branch from d90d54e to 477e537 Compare September 23, 2022 16:48

cla-bot bot added the cla:yes label Sep 23, 2022

francescomucio added 3 commits September 23, 2022 18:52

Fixed changelog

179311f

fixed typo

5e6a7cd

changie-ed

61611ed

ueshin reviewed Dec 7, 2022

View reviewed changes

ueshin mentioned this pull request Dec 16, 2022

Use show table extended with table name list for get_catalog. databricks/dbt-databricks#237

Merged

Fleid added the pr_tracked label Mar 13, 2023

nathaniel-may requested a review from a team as a code owner March 16, 2023 21:20

github-actions bot added the Stale label Sep 14, 2023

Merge branch 'main' into main

f04993f

github-actions bot removed the Stale label Sep 15, 2023

Merge branch 'main' into main

389fc73

mikealfare removed the pr_tracked label Feb 7, 2024

github-actions bot added the Stale label Aug 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix for inconsistent Bigint/Long datatype that makes dbt think there was a schema change #358

Fix for inconsistent Bigint/Long datatype that makes dbt think there was a schema change #358

francescomucio commented May 16, 2022 •

edited by McKnight-42

Loading

cla-bot bot commented May 16, 2022

jtcohen6 commented May 16, 2022

cla-bot bot commented May 16, 2022

cla-bot bot commented May 16, 2022

jtcohen6 commented May 16, 2022 •

edited

Loading

cla-bot bot commented May 16, 2022

McKnight-42 commented Jun 6, 2022

francescomucio commented Jun 8, 2022

jtcohen6 Jun 15, 2022

francescomucio Jun 15, 2022

francescomucio commented Jun 29, 2022

McKnight-42 commented Jul 25, 2022

francescomucio commented Jul 26, 2022

McKnight-42 commented Jul 26, 2022

cla-bot bot commented Sep 23, 2022

cla-bot bot commented Sep 23, 2022

cla-bot bot commented Sep 23, 2022

cla-bot bot commented Sep 23, 2022

cla-bot bot commented Sep 23, 2022

cla-bot bot commented Sep 23, 2022

cla-bot bot commented Sep 23, 2022

McKnight-42 commented Oct 4, 2022

francescomucio commented Nov 17, 2022

ueshin left a comment

ueshin Dec 7, 2022

github-actions bot commented Sep 14, 2023

github-actions bot commented Aug 6, 2024

		### Under the hood
		- Add `DBT_INVOCATION_ENV` environment variable to ODBC user agent string ([#366](https://github.com/dbt-labs/dbt-spark/pull/366))

Fix for inconsistent Bigint/Long datatype that makes dbt think there was a schema change #358

Are you sure you want to change the base?

Fix for inconsistent Bigint/Long datatype that makes dbt think there was a schema change #358

Conversation

francescomucio commented May 16, 2022 • edited by McKnight-42 Loading

Description

Checklist

cla-bot bot commented May 16, 2022

jtcohen6 commented May 16, 2022

cla-bot bot commented May 16, 2022

cla-bot bot commented May 16, 2022

jtcohen6 commented May 16, 2022 • edited Loading

cla-bot bot commented May 16, 2022

McKnight-42 commented Jun 6, 2022

francescomucio commented Jun 8, 2022

jtcohen6 Jun 15, 2022

Choose a reason for hiding this comment

francescomucio Jun 15, 2022

Choose a reason for hiding this comment

francescomucio commented Jun 29, 2022

McKnight-42 commented Jul 25, 2022

francescomucio commented Jul 26, 2022

McKnight-42 commented Jul 26, 2022

cla-bot bot commented Sep 23, 2022

cla-bot bot commented Sep 23, 2022

cla-bot bot commented Sep 23, 2022

cla-bot bot commented Sep 23, 2022

cla-bot bot commented Sep 23, 2022

cla-bot bot commented Sep 23, 2022

cla-bot bot commented Sep 23, 2022

McKnight-42 commented Oct 4, 2022

francescomucio commented Nov 17, 2022

ueshin left a comment

Choose a reason for hiding this comment

ueshin Dec 7, 2022

Choose a reason for hiding this comment

github-actions bot commented Sep 14, 2023

github-actions bot commented Aug 6, 2024

francescomucio commented May 16, 2022 •

edited by McKnight-42

Loading

jtcohen6 commented May 16, 2022 •

edited

Loading