Destination databricks: switch to oss jdbc driver

[edgao]

closes https://github.com/airbytehq/airbyte-internal-issues/issues/9120

also it seems like we don't need the databricks sdk at all?

The new driver has a slightly different interface (you can't directly supply a URL, it forces you to supply individual fields/properties). I tried to port over our existing stuff, but removed the transportMode=http and EnableArrow=0 things to see if they're still needed.

Databricks documentation doesn't even describe how to do oauth, it only says how to do PAT (https://docs.gcp.databricks.com/en/integrations/jdbc/oss.html#authenticate-the-driver). I copied our old stuff to the new interfaces naively, but it doesn't work

• DatabricksSQLException: Communication link failure. Failed to connect to server. :https://dbc-6aebf761-f8d6.cloud.databricks.com:443accessToken must be defined
• DatabricksSQLException: Communication link failure. Failed to connect to server. :https://dbc-6aebf761-f8d6.cloud.databricks.com:443Cannot invoke "com.databricks.sdk.core.oauth.OpenIDConnectEndpoints.getTokenEndpoint()" because "jsonResponse" is null).

notable changes in the oss driver:

• timestamps with timezone now have a timezone directly from the driver
• timestamps without timezone have .000 precision
• Inline byte limit exceeded. Statements executed with disposition=INLINE can have a result size of at most 26214400 bytes. Please execute the statement with disposition=EXTERNAL_LINKS if you want to download the full result

which means:

• the destinationhandler can parse directly to an Instant, instead of needing to go through LocalDateTime
• tons of changes in the expected records

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Sep 4, 2024 4:56pm

[edgao]

• Destination databricks: switch to oss jdbc driver #44033 👈
• Destinations CDK: add test for mixed-case stream name #44505 : 1 other dependent PR (#44506 )
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @edgao and the rest of your teammates on Graphite

[edgao]

@gisripa do you remember what the /${config.database} thing is supposed to do? It sounds like we're passing in the database (i.e. unity catalog name?)

but the oss driver's docs sound like this is actually supposed to be the schema, which sounds super weird. I.e. their url format is jdbc:databricks://<server-hostname>:<port>/<schema>;

(I'm seeing some weird test failures with ConcurrentModificationException and am very confused about how that could be happening, no idea if this is related)

[cgardens]

We are closing this PR due to inactivity. We regularly close PRs that were created more than 6 months ago. If you'd like to keep working on it, please feel free reopen it. We'd be happy to keep working with you.