Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve dashboard as code #108

Merged
merged 80 commits into from
May 31, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
5a11735
Add counter
JCZuurmond May 27, 2024
ec9b9de
Rename Dashboards to Dashboard
JCZuurmond May 27, 2024
a14a81d
Remove unused import
JCZuurmond May 27, 2024
2563c3e
Make dashboard singular
JCZuurmond May 27, 2024
cef1967
Format
JCZuurmond May 27, 2024
e17b6ea
Add integration test and endpoint to deploy dashboard from code
JCZuurmond May 27, 2024
9c3a88e
Handle WidgetSpec v0 table
JCZuurmond May 27, 2024
1871246
Add TODO
JCZuurmond May 27, 2024
858edbd
Rewrite test to count queries
JCZuurmond May 27, 2024
2f926cc
Create datasets from queries
JCZuurmond May 27, 2024
5da22cb
Handle counter v0
JCZuurmond May 27, 2024
f424402
Add test for creating counter specs
JCZuurmond May 27, 2024
f00f5c3
Add a counter widget for each query
JCZuurmond May 27, 2024
0002be3
Deploy lakeview dashboard
JCZuurmond May 27, 2024
af41d9c
Remove unused import
JCZuurmond May 27, 2024
b46999c
Split dashboard create and deploy
JCZuurmond May 27, 2024
cc7ffb0
Add count field
JCZuurmond May 27, 2024
1654cb3
Refactor to clean test dashboards
JCZuurmond May 27, 2024
1cd743c
Test the lakeview dicts
JCZuurmond May 27, 2024
5872649
Create random id
JCZuurmond May 27, 2024
f23b25e
Refactor dashboard to plural
JCZuurmond May 27, 2024
64bf3d8
Refactor create to create_dashboard
JCZuurmond May 27, 2024
fc1b0e9
Refactor deploy to deploy_dashboard
JCZuurmond May 27, 2024
8fe59d2
Remove name collision
JCZuurmond May 27, 2024
89eb08d
Change counter height to three
JCZuurmond May 27, 2024
a685425
Add working dashboard
JCZuurmond May 28, 2024
8e9fede
Add disaggregated field
JCZuurmond May 28, 2024
b9df7cb
Add counter field encoding
JCZuurmond May 28, 2024
871e39b
Name query
JCZuurmond May 28, 2024
7bd6ab7
Format
JCZuurmond May 28, 2024
e73f5db
Add comment
JCZuurmond May 28, 2024
d687ce8
Remove create random id
JCZuurmond May 28, 2024
4219f59
Replace fields in testing
JCZuurmond May 28, 2024
98f5368
Format
JCZuurmond May 28, 2024
b4d5414
Raise value error when missing diplay name and dashboard id
JCZuurmond May 28, 2024
25d2b94
Raise value error when stating display name and dashboard id
JCZuurmond May 28, 2024
d283c82
Call create when deploy with display name
JCZuurmond May 28, 2024
246e792
Call update when deploy with dashboard id
JCZuurmond May 28, 2024
5626f82
Improve tests
JCZuurmond May 28, 2024
21bbe8d
Move unit tests to unit test folder
JCZuurmond May 28, 2024
5cbef30
Move queries into dashboards
JCZuurmond May 28, 2024
8bfd949
Rename dashboard_client to dashboards
JCZuurmond May 28, 2024
704ea05
Refactor to separate out better names
JCZuurmond May 28, 2024
6e0eb24
Make create dashboard a static method
JCZuurmond May 28, 2024
66c0b04
Move deploy integration test up
JCZuurmond May 28, 2024
6461810
Rewrite integration test to save dashboard to folder
JCZuurmond May 28, 2024
7228664
Add type hint
JCZuurmond May 28, 2024
d8b478d
Let save to folder expect Dashboard
JCZuurmond May 28, 2024
adb4237
Move integration test to unit test
JCZuurmond May 28, 2024
97d80d2
Fix test names
JCZuurmond May 28, 2024
57c6970
Update lvdash yaml name
JCZuurmond May 28, 2024
8d8cb5f
Reused with better names
JCZuurmond May 28, 2024
f4517c3
Replace name for Widget
JCZuurmond May 28, 2024
d73edb9
Replace all names
JCZuurmond May 28, 2024
240938f
Format tests
JCZuurmond May 28, 2024
42ff42a
Remove unused function
JCZuurmond May 28, 2024
b05dc7a
Fix typo
JCZuurmond May 28, 2024
163f351
Move test up
JCZuurmond May 28, 2024
e4b3551
Test for yml to be created
JCZuurmond May 28, 2024
5ce4cce
Short test
JCZuurmond May 28, 2024
b615416
Test dataset names
JCZuurmond May 28, 2024
de98f16
Test page names
JCZuurmond May 28, 2024
06f6a10
Test query names
JCZuurmond May 28, 2024
794ba6d
Move ugly dashboard into fixtures
JCZuurmond May 28, 2024
e6f4a18
Add test for replacing counter names
JCZuurmond May 28, 2024
9e883aa
Remove unused imports
JCZuurmond May 28, 2024
30d3959
Refactor lakeview_dashboard to dashboard
JCZuurmond May 28, 2024
b2bc6df
Reuse dataclasses methods
JCZuurmond May 28, 2024
d44ea96
Format
JCZuurmond May 28, 2024
1eecdc0
Remove redundant if
JCZuurmond May 28, 2024
8ece444
Remove redundant if
JCZuurmond May 28, 2024
322b0a8
Refactor format query
JCZuurmond May 28, 2024
03667bb
Save files with names
JCZuurmond May 28, 2024
52f8ad6
Make format query private
JCZuurmond May 31, 2024
820bbde
Return early
JCZuurmond May 31, 2024
df37440
Make with better names private
JCZuurmond May 31, 2024
550134a
Move dataset replace names into replace names method
JCZuurmond May 31, 2024
a45788e
Move dataset and page replace names into replace names method
JCZuurmond May 31, 2024
4c52d15
Test replace names through public save to folder
JCZuurmond May 31, 2024
14a6352
Update dashboard folder name
JCZuurmond May 31, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 63 additions & 45 deletions src/databricks/labs/lsql/dashboards.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import dataclasses
import json
from pathlib import Path
from typing import ClassVar, Protocol, runtime_checkable
from typing import TypeVar

import sqlglot
import yaml
Expand All @@ -24,41 +25,56 @@
Widget,
)


@runtime_checkable
class _DataclassInstance(Protocol):
__dataclass_fields__: ClassVar[dict]
T = TypeVar("T")


class Dashboards:
def __init__(self, ws: WorkspaceClient):
self._ws = ws

def get_dashboard(self, dashboard_path: str):
def get_dashboard(self, dashboard_path: str) -> Dashboard:
with self._ws.workspace.download(dashboard_path, format=ExportFormat.SOURCE) as f:
raw = f.read().decode("utf-8")
as_dict = json.loads(raw)
return Dashboard.from_dict(as_dict)

def save_to_folder(self, dashboard_path: str, local_path: Path):
def save_to_folder(self, dashboard: Dashboard, local_path: Path) -> Dashboard:
local_path.mkdir(parents=True, exist_ok=True)
dashboard = self.get_dashboard(dashboard_path)
better_names = {}
dashboard = self._with_better_names(dashboard)
for dataset in dashboard.datasets:
name = dataset.display_name
better_names[dataset.name] = name
query_path = local_path / f"{name}.sql"
sql_query = dataset.query
self._format_sql_file(sql_query, query_path)
lvdash_yml = local_path / "lvdash.yml"
with lvdash_yml.open("w") as f:
first_page = dashboard.pages[0]
self._replace_names(first_page, better_names)
page = first_page.as_dict()
yaml.safe_dump(page, f)
assert True
query = self._format_query(dataset.query)
with (local_path / f"{dataset.name}.sql").open("w") as f:
f.write(query)
for page in dashboard.pages:
JCZuurmond marked this conversation as resolved.
Show resolved Hide resolved
with (local_path / f"{page.name}.yml").open("w") as f:
yaml.safe_dump(page.as_dict(), f)
return dashboard

@staticmethod
def _format_query(query: str) -> str:
try:
parsed_query = sqlglot.parse(query)
except sqlglot.ParseError:
return query
statements = []
for statement in parsed_query:
if statement is None:
continue
# see https://sqlglot.com/sqlglot/generator.html#Generator
statements.append(
statement.sql(
dialect="databricks",
normalize=True, # normalize identifiers to lowercase
pretty=True, # format the produced SQL string
normalize_functions="upper", # normalize function names to uppercase
max_text_width=80, # wrap text at 120 characters
)
)
formatted_query = ";\n".join(statements)
return formatted_query

def create_dashboard(self, dashboard_folder: Path) -> Dashboard:
@staticmethod
def create_dashboard(dashboard_folder: Path) -> Dashboard:
"""Create a dashboard from code, i.e. configuration and queries."""
datasets, layouts = [], []
for query_path in dashboard_folder.glob("*.sql"):
Expand Down Expand Up @@ -99,42 +115,44 @@ def deploy_dashboard(
)
return dashboard

def _format_sql_file(self, sql_query, query_path):
with query_path.open("w") as f:
try:
for statement in sqlglot.parse(sql_query):
# see https://sqlglot.com/sqlglot/generator.html#Generator
pretty = statement.sql(
dialect="databricks",
normalize=True, # normalize identifiers to lowercase
pretty=True, # format the produced SQL string
normalize_functions="upper", # normalize function names to uppercase
max_text_width=80, # wrap text at 120 characters
)
f.write(f"{pretty};\n")
except sqlglot.ParseError:
f.write(sql_query)
def _with_better_names(self, dashboard: Dashboard) -> Dashboard:
"""Replace names with human-readable names."""
better_names = {}
for dataset in dashboard.datasets:
JCZuurmond marked this conversation as resolved.
Show resolved Hide resolved
if dataset.display_name is not None:
better_names[dataset.name] = dataset.display_name
for page in dashboard.pages:
if page.display_name is not None:
better_names[page.name] = page.display_name
return self._replace_names(dashboard, better_names)

def _replace_names(self, node: _DataclassInstance, better_names: dict[str, str]):
# walk evely dataclass instance recursively and replace names
if isinstance(node, _DataclassInstance):
for field in node.__dataclass_fields__.values():
def _replace_names(self, node: T, better_names: dict[str, str]) -> T:
# walk every dataclass instance recursively and replace names
if dataclasses.is_dataclass(node):
for field in dataclasses.fields(node):
value = getattr(node, field.name)
if isinstance(value, list):
setattr(node, field.name, [self._replace_names(item, better_names) for item in value])
elif isinstance(value, _DataclassInstance):
elif dataclasses.is_dataclass(value):
setattr(node, field.name, self._replace_names(value, better_names))
if isinstance(node, Query):
if isinstance(node, Dataset):
node.name = better_names.get(node.name, node.name)
elif isinstance(node, Page):
node.name = better_names.get(node.name, node.name)
elif isinstance(node, Query):
node.dataset_name = better_names.get(node.dataset_name, node.dataset_name)
elif isinstance(node, NamedQuery) and node.query:
# 'dashboards/01eeb077e38c17e6ba3511036985960c/datasets/01eeb081882017f6a116991d124d3068_...'
if node.name.startswith("dashboards/"):
parts = [node.query.dataset_name]
for field in node.query.fields:
parts.append(field.name)
for query_field in node.query.fields:
parts.append(query_field.name)
new_name = "_".join(parts)
better_names[node.name] = new_name
node.name = better_names.get(node.name, node.name)
elif isinstance(node, ControlFieldEncoding):
node.query_name = better_names.get(node.query_name, node.query_name)
elif isinstance(node, Widget):
if node.spec is not None:
node.name = node.spec.as_dict().get("widgetType", node.name)
return node
74 changes: 14 additions & 60 deletions tests/integration/test_dashboards.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
import json
from dataclasses import fields, is_dataclass
from pathlib import Path

import pytest

from databricks.labs.lsql.dashboards import Dashboards
from databricks.labs.lsql.lakeview.model import CounterSpec, Dashboard
from databricks.labs.lsql.lakeview.model import Dashboard


@pytest.fixture
Expand All @@ -20,68 +19,23 @@ def dashboard_id(ws, make_random):
ws.lakeview.trash(dashboard.dashboard_id)


def test_load_dashboard(ws):
dashboard = Dashboards(ws)
src = "/Workspace/Users/[email protected]/Trivial Dashboard.lvdash.json"
dst = Path(__file__).parent / "sample"
dashboard.save_to_folder(src, dst)


def test_dashboard_creates_one_dataset_per_query(ws):
queries = Path(__file__).parent / "queries"
dashboard = Dashboards(ws).create_dashboard(queries)
assert len(dashboard.datasets) == len([query for query in queries.glob("*.sql")])


def test_dashboard_creates_one_counter_widget_per_query(ws):
queries = Path(__file__).parent / "queries"
dashboard = Dashboards(ws).create_dashboard(queries)

counter_widgets = []
for page in dashboard.pages:
for layout in page.layout:
if isinstance(layout.widget.spec, CounterSpec):
counter_widgets.append(layout.widget)

assert len(counter_widgets) == len([query for query in queries.glob("*.sql")])


def replace_recursively(dataklass, replace_fields):
for field in fields(dataklass):
value = getattr(dataklass, field.name)
if is_dataclass(value):
new_value = replace_recursively(value, replace_fields)
elif isinstance(value, list):
new_value = [replace_recursively(v, replace_fields) for v in value]
elif isinstance(value, tuple):
new_value = (replace_recursively(v, replace_fields) for v in value)
else:
new_value = replace_fields.get(field.name, value)
setattr(dataklass, field.name, new_value)
return dataklass


def test_dashboard_deploys_dashboard(ws, dashboard_id):
queries = Path(__file__).parent / "queries"
dashboard_client = Dashboards(ws)
lakeview_dashboard = dashboard_client.create_dashboard(queries)

dashboard = dashboard_client.deploy_dashboard(lakeview_dashboard, dashboard_id=dashboard_id)
deployed_lakeview_dashboard = dashboard_client.get_dashboard(dashboard.path)

replace_name = {"name": "test", "dataset_name": "test"} # Dynamically created names
lakeview_dashboard_wo_name = replace_recursively(lakeview_dashboard, replace_name)
deployed_lakeview_dashboard_wo_name = replace_recursively(deployed_lakeview_dashboard, replace_name)

assert lakeview_dashboard_wo_name.as_dict() == deployed_lakeview_dashboard_wo_name.as_dict()


def test_dashboards_deploys_exported_dashboard_definition(ws, dashboard_id):
dashboard_file = Path(__file__).parent / "dashboards" / "dashboard.json"
with dashboard_file.open("r") as f:
lakeview_dashboard = Dashboard.from_dict(json.load(f))

dashboard_client = Dashboards(ws)
dashboard = dashboard_client.deploy_dashboard(lakeview_dashboard, dashboard_id=dashboard_id)
dashboards = Dashboards(ws)
dashboard = dashboards.deploy_dashboard(lakeview_dashboard, dashboard_id=dashboard_id)

assert ws.lakeview.get(dashboard.dashboard_id)


def test_dashboard_deploys_dashboard_the_same_as_created_dashboard(ws, dashboard_id):
queries = Path(__file__).parent / "dashboards" / "one_counter"
dashboards = Dashboards(ws)
dashboard = dashboards.create_dashboard(queries)

sdk_dashboard = dashboards.deploy_dashboard(dashboard, dashboard_id=dashboard_id)
new_dashboard = dashboards.get_dashboard(sdk_dashboard.path)

assert dashboards._with_better_names(dashboard).as_dict() == dashboards._with_better_names(new_dashboard).as_dict()
1 change: 1 addition & 0 deletions tests/unit/queries/counter.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
SELECT 6217 AS count
Loading
Loading