Track Upstream Bug in databricks-sql-python (Issue #525) Affecting dbt-databricks on LTS Clusters

[azachar]

Description:
When using dbt-databricks on Databricks LTS clusters (e.g., versions 12.2, 14.3, and 15.4), we encountered an error originating from the underlying databricks-sql-python library. The issue stems from a bug where the retry logic attempts to access attributes from urllib3 2.x (e.g., Retry.backoff_jitter), which are not present in the older urllib3 versions shipped with these LTS runtimes.

This problem has been reported upstream in databricks-sql-python Issue #525.

Steps to Reproduce:

1. Deploy dbt-databricks on a Databricks LTS cluster (versions 12.2, 14.3, or 15.4).
2. Execute operations that trigger the retry logic in databricks-sql-python (such as connection establishment or query execution).
3. The following error is raised:

   AttributeError: type object 'Retry' has no attribute 'backoff_jitter'
   

Workaround (Not Recommended):
A suggested workaround is to manually upgrade urllib3 to version 2.3.0. However, this can introduce dependency conflicts and may compromise the stability of the Databricks runtime.

Desired Outcome:
It would be ideal if the retry logic in databricks-sql-python were updated to check the installed version of urllib3 and use a backward-compatible approach when necessary. Tracking and resolving this upstream issue will help ensure dbt-databricks continues to work reliably on all supported Databricks runtimes.

Additional Context:

• This issue is critical for users running dbt-databricks on LTS clusters.

We appreciate your attention and look forward to a resolution that maintains compatibility without forcing manual dependency upgrades.