[FEATURE]: Allow table mapping to be retrieved from DB SQL

[rwforest]

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

Currently table mapping is stored in csv format, which is great for a small number of tables, it will not be scalable for thousands of tables in a single workspace. Csv files are also difficult to manage for versioning.

Proposed Solution

Allow users to store the mapping into a queryable storage, like DB SQL. It will enhance the ability for collaboration

Additional Context

No response

[nfx]

The whole idea of CSV file in the workspace is that people can edit it: in the browser, excel, google sheets, etc. Databricks doesn’t have a UI for editing data row-by-row

[rwforest]

@nfx DB SQL is just an example. SQLite is another example, but it will introduce additional dependencies. If you tracked issues with Excel, you know it's not ideal. They can edit in Excel, or whatever, but still be able to push back and merge, keeping a single version of the truth. Otherwise people will end up sending csv / xls / google sheets to each other.

[nfx]

We might want to add a small WebUI through a localhost webserver, but it's not yet a priority

[rwforest]

@nfx makes sense, but I'd suggest giving some guidelines on how to manage versioning and single source of the truth. You can't expect every team in every organization got a clean mapping of their before and after. With an additional layer of catalog, a lot of people are taking advantage of this process to redesign their data catalog. Information is all over the place, like in sharepoint site, someone's inbox, etc. Downstream will suffer badly if we don't have a single version of the truth.

[nfx]

The CSV file per workspace is the source of truth