chore(deps): upgrade databricks connectors

[drummerwolli]

SUMMARY

upgrade databricks connectors

sqlalchemy support for databricks was first moved to databricks-sql-connector and then outsourced again into a new package name, hence the rename:

https://github.com/crflynn/sqlalchemy-databricks?tab=readme-ov-file#%EF%B8%8F-this-project-is-archived-a-new-sqlalchemy-dialect-is-available-directly-from-databricks

https://github.com/databricks/databricks-sql-python?tab=readme-ov-file#sqlalchemy

DEPENDENCY ISSUE

databricks-sqlalchemy is available for SQLAlchemy 1 and 2. I pinpointed this version now to 1.0.5 which is the version which is for SQLAlchemy 1.

This version 1.0.5 needs though pyarrow<17. pyarrow in Superset was upgrade in #31476 to require at least v18. So we either upgrade SQLAlchemy to v2 (tricky and big, is someone working on this?) or downgrade pyarrow to 16 (which is still an upgrade to v14 which is included in 5.0.0, and the feature required by the PR creator vor v18 is already included in v16). The latter is proposed here: #34693

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

n/a

TESTING INSTRUCTIONS

1. install databricks drivers
2. set up a databricks connection and check that it works

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[drummerwolli]

fyi, i hope the tests fail somewhere to identify the version mismatch.

[drummerwolli]

just to iterate here once more: this needs first #34693 to get merged, since the new Databricks version has an upper version limit for pyarrow

[drummerwolli]

@sadpandajoe @betodealmeida any chance to still get this one into 6.0.0?

[sadpandajoe]

@drummerwolli so this is really a fix to get databricks connector working because

1. of the pyarrow change
2. support being moved

Am I understanding this right?

[drummerwolli]

@sadpandajoe yes, kind of. the databricks connector is currently working, but it is very outdated. And in order to work on proper Oauth support for Databricks, I would first like to get the databricks connector to the latest versions, in order to rely on the most up-to-date codebase.

The pyarrow change was necessary to do the upgrade proposed in this PR.

[sadpandajoe]

@drummerwolli since it isn't a fix and existing databricks connector is working, it won't be pulled into 6.0 as we've already cut and we only take in fixes that will stabilize the release. It'll go out with the following major or minor release then.

[drummerwolli]

@sadpandajoe understood. but the 6.0 release is already cut from master, right? can we then merge this to master at least? anything missing to merge to master?

[iercan]

Recently I had a problem with databricks connector. I've solved my problem by upgrading to official library. We've discussed about it here with @sfirke.

It would be enough to add just databricks-sqlalchemy~=1.0. Official databricks document also suggest to install it as.
I also want to mention that this is a breaking change because I needed to edit connection string.