Skip to content

Common conf routes #400

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Feb 20, 2025
Merged

Common conf routes #400

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
58198e4
Create a common route for all confs (joins, group bys, models, and st…
techniq Feb 19, 2025
12cc490
fix(ConfProperties): Hande `model.source`
techniq Feb 19, 2025
8f8357e
fix(ConfProperties): Handle remaining Model properties
techniq Feb 19, 2025
210d704
Remove model details from (join) observability
techniq Feb 19, 2025
6e57db0
Add "Open" button to drilldown dialog
techniq Feb 19, 2025
c959003
Hide metaDataLabel when shown at top of conf/entity page to reduce ve…
techniq Feb 20, 2025
39290a0
Update tween duration to match CSS transition duration
techniq Feb 20, 2025
9f54d00
Allow overview lineage DAG to use remaining viewport height
techniq Feb 20, 2025
e1ac1e3
Fix gap above observability sticky filters. Refine overflow context
techniq Feb 20, 2025
8c71813
Refine fallback date range handling
techniq Feb 20, 2025
9e7d38b
Remove unimplemented action buttons (only leaving sort for now)
techniq Feb 20, 2025
eda4b1e
Add TODO to trigger CI
techniq Feb 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 33 additions & 2 deletions frontend/src/lib/api/api.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import type {
} from '$lib/types/codegen';
import { ConfType, DriftMetric, Status } from '$lib/types/codegen';
import type { ConfListResponse } from '$lib/types/codegen/ConfListResponse';
import { joinToLineage } from './utils';
import { confToLineage } from './utils';

export type ApiOptions = {
base?: string;
Expand Down Expand Up @@ -160,7 +160,38 @@ export class Api {

// TODO: Remove this once we have the API endpoint
return this.getJoin(name!).then((join) => {
return joinToLineage(join);
return confToLineage(join);
});
}

async getGroupByLineage({
name
// type,
// branch,
// direction
}: ILineageRequestArgs): Promise<ILineageResponse> {
// TODO: Remove this once we have the API endpoint
return this.getGroupBy(name!).then((groupBy) => {
return confToLineage(groupBy);
});
}

async getModelLineage({
name
// type,
// branch,
// direction
}: ILineageRequestArgs): Promise<ILineageResponse> {
// TODO: Remove this once we have the API endpoint
return this.getModel(name!).then((model) => {
return confToLineage(model);
});
}

async getStagingQueryLineage({ name }: ILineageRequestArgs): Promise<ILineageResponse> {
// TODO: Remove this once we have the API endpoint
return this.getStagingQuery(name!).then((stagingQuery) => {
return confToLineage(stagingQuery);
});
Comment on lines +167 to 195
Copy link
Contributor

@coderabbitai coderabbitai bot Feb 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Reduce duplication in lineage methods.

Similar pattern repeated across getGroupByLineage, getModelLineage, and getStagingQueryLineage.

+private async getConfLineage<T>(name: string, getConfFn: (name: string) => Promise<T>): Promise<ILineageResponse> {
+  return getConfFn(name).then(confToLineage);
+}

 async getGroupByLineage({ name }: ILineageRequestArgs): Promise<ILineageResponse> {
-  return this.getGroupBy(name!).then((groupBy) => {
-    return confToLineage(groupBy);
-  });
+  return this.getConfLineage(name!, this.getGroupBy.bind(this));
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
async getGroupByLineage({
name
// type,
// branch,
// direction
}: ILineageRequestArgs): Promise<ILineageResponse> {
// TODO: Remove this once we have the API endpoint
return this.getGroupBy(name!).then((groupBy) => {
return confToLineage(groupBy);
});
}
async getModelLineage({
name
// type,
// branch,
// direction
}: ILineageRequestArgs): Promise<ILineageResponse> {
// TODO: Remove this once we have the API endpoint
return this.getModel(name!).then((model) => {
return confToLineage(model);
});
}
async getStagingQueryLineage({ name }: ILineageRequestArgs): Promise<ILineageResponse> {
// TODO: Remove this once we have the API endpoint
return this.getStagingQuery(name!).then((stagingQuery) => {
return confToLineage(stagingQuery);
});
private async getConfLineage<T>(name: string, getConfFn: (name: string) => Promise<T>): Promise<ILineageResponse> {
return getConfFn(name).then(confToLineage);
}
async getGroupByLineage({ name }: ILineageRequestArgs): Promise<ILineageResponse> {
// TODO: Remove this once we have the API endpoint
return this.getConfLineage(name!, this.getGroupBy.bind(this));
}
async getModelLineage({
name
// type,
// branch,
// direction
}: ILineageRequestArgs): Promise<ILineageResponse> {
// TODO: Remove this once we have the API endpoint
return this.getModel(name!).then((model) => {
return confToLineage(model);
});
}
async getStagingQueryLineage({ name }: ILineageRequestArgs): Promise<ILineageResponse> {
// TODO: Remove this once we have the API endpoint
return this.getStagingQuery(name!).then((stagingQuery) => {
return confToLineage(stagingQuery);
});
}

}

Expand Down
81 changes: 60 additions & 21 deletions frontend/src/lib/api/utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,36 +2,83 @@ import { InternMap } from 'd3';
import {
LogicalType,
type IJoin,
type IJoinPart,
type ILineageResponse,
type ILogicalNode,
type INodeKey,
type ISource,
type NodeGraph
} from '../types/codegen';
import { getLogicalNodeType, type CombinedLogicalNode } from '../types/LogicalNode';

/** Convert Join to LineageResponse by walking joinParts */
export function joinToLineage(join: IJoin, excludeLeft = false): ILineageResponse {
export function confToLineage(conf: CombinedLogicalNode, excludeLeft = false): ILineageResponse {
// Use `InternMap` insteaad of `Map` to support object keys (instances will be different once serialized/fetched from API) - https://d3js.org/d3-array/intern
// @ts-expect-error: Bad typing
const connections: NodeGraph['connections'] = new InternMap([], JSON.stringify);
// @ts-expect-error: Bad typing
const infoMap: NodeGraph['infoMap'] = new InternMap([], JSON.stringify);

const joinNodeKey: INodeKey = {
name: join.metaData?.name,
logicalType: LogicalType.JOIN
const logicalType = getLogicalNodeType(conf);

const confNodeKey: INodeKey = {
name:
'metaData' in conf && conf.metaData
? conf.metaData.name
: 'table' in conf
? conf.table
: 'Unknown',
logicalType
};
infoMap.set(joinNodeKey, {
conf: join as ILogicalNode
infoMap.set(confNodeKey, {
conf: conf as ILogicalNode
});
const joinParents: INodeKey[] = [];
connections.set(joinNodeKey, { parents: joinParents });
const confParents: INodeKey[] = [];
connections.set(confNodeKey, { parents: confParents });

/*
* Join
*/
if ('left' in conf && conf.left && !excludeLeft) {
processSource(conf.left, infoMap, connections, confParents);
}

if ('joinParts' in conf && conf.joinParts) {
processJoinParts(conf.joinParts, infoMap, connections, confParents);
}

/*
* GroupBy
*/
if ('sources' in conf && conf.sources) {
for (const source of conf.sources ?? []) {
processSource(source, infoMap, connections, confParents);
}
}

if (join.left && !excludeLeft) {
processSource(join.left, infoMap, connections, joinParents);
/*
* Model
*/
if ('source' in conf && conf.source) {
processSource(conf.source, infoMap, connections, confParents);
}

for (const jp of join.joinParts ?? []) {
return {
nodeGraph: {
connections,
infoMap
},
mainNode: confNodeKey
};
}

function processJoinParts(
joinParts: IJoinPart[],
infoMap: NonNullable<NodeGraph['infoMap']>,
connections: NonNullable<NodeGraph['connections']>,
parents: INodeKey[]
) {
for (const jp of joinParts ?? []) {
if (jp.groupBy) {
const groupByNodeKey: INodeKey = {
name: jp.groupBy.metaData?.name,
Expand All @@ -40,7 +87,7 @@ export function joinToLineage(join: IJoin, excludeLeft = false): ILineageRespons
infoMap.set(groupByNodeKey, {
conf: jp.groupBy as ILogicalNode
});
joinParents.push(groupByNodeKey);
parents.push(groupByNodeKey);

const groupByParents: INodeKey[] = [];
connections.set(groupByNodeKey, { parents: groupByParents });
Expand All @@ -50,14 +97,6 @@ export function joinToLineage(join: IJoin, excludeLeft = false): ILineageRespons
}
}
}

return {
nodeGraph: {
connections,
infoMap
},
mainNode: joinNodeKey
};
}

function processSource(
Expand Down Expand Up @@ -99,7 +138,7 @@ function processSource(
parents.push(joinNodeKey);

// Transfer connections and infoMap from joinSource join to root join graph
const joinSourceLineage = joinToLineage(source.joinSource.join as IJoin);
const joinSourceLineage = confToLineage(source.joinSource.join as IJoin);

for (const [key, nodeConnections] of joinSourceLineage.nodeGraph?.connections ?? []) {
connections.set(key === joinSourceLineage.mainNode ? joinNodeKey : key, nodeConnections);
Expand Down
47 changes: 6 additions & 41 deletions frontend/src/lib/components/ActionButtons.svelte
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,10 @@
import { cn } from '$lib/utils';

import IconArrowsUpDown from '~icons/heroicons/arrows-up-down-16-solid';
import IconPlus from '~icons/heroicons/plus-16-solid';
import IconSquare3Stack3d from '~icons/heroicons/square-3-stack-3d-16-solid';
import IconXMark from '~icons/heroicons/x-mark-16-solid';

import { getSortParamKey, type SortContext, getSortParamsConfig } from '$lib/util/sort';

let {
showCluster = false,
class: className,
showSort = false,
context = 'drift'
Expand All @@ -23,8 +19,6 @@
context?: SortContext;
} = $props();

let activeCluster = showCluster ? 'GroupBys' : null;

const sortKey = $derived(getSortParamKey(context));
const params = $derived(
queryParameters(getSortParamsConfig(context), { pushHistory: false, showDefaults: false })
Expand All @@ -35,40 +29,11 @@
}
</script>

<div class={cn('flex flex-wrap gap-3', className)}>
<!-- Active Items Section -->
{#if activeCluster}
<div class="flex flex-wrap gap-[1px]">
<Button variant="secondaryAlt" size="sm" icon="leading" disabled>
<IconSquare3Stack3d />
Cluster by
</Button>
<Button variant="secondaryAlt" size="sm" disabled>
{activeCluster}
</Button>
<Button variant="secondaryAlt" size="sm" class="p-2" disabled>
<IconXMark />
</Button>
</div>
{/if}

<!-- Action Buttons Section -->
<div class="flex gap-3">
{#if showSort}
<Button variant="secondary" size="sm" icon="leading" on:click={toggleSort}>
<IconArrowsUpDown />
Sort {params[sortKey] === 'asc' ? 'A-Z' : 'Z-A'}
</Button>
{/if}
<Button variant="secondary" size="sm" icon="leading" disabled>
<IconPlus />
Filter
<div class={cn('inline-flex flex-wrap gap-3', className)}>
{#if showSort}
<Button variant="secondary" size="sm" icon="leading" on:click={toggleSort}>
<IconArrowsUpDown />
Sort {params[sortKey] === 'asc' ? 'A-Z' : 'Z-A'}
</Button>
{#if showCluster}
<Button variant="secondary" size="sm" icon="leading" disabled>
<IconSquare3Stack3d />
Cluster
</Button>
{/if}
</div>
{/if}
</div>
31 changes: 12 additions & 19 deletions frontend/src/lib/components/ChartControls.svelte
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,8 @@
import DriftMetricToggle from '$lib/components/DriftMetricToggle.svelte';
import DateRangeSelector from '$lib/components/DateRangeSelector.svelte';
import ActionButtons from '$lib/components/ActionButtons.svelte';
import * as Alert from '$lib/components/ui/alert/index.js';
import { formatDate } from '$lib/util/format';
import type { SortContext } from '$lib/util/sort';
import { fromAbsolute, getLocalTimeZone } from '@internationalized/date';

let {
isZoomed = false,
Expand All @@ -29,30 +28,24 @@
</script>

<div class="space-y-4">
{#if isUsingFallbackDates}
<div class="w-fit">
<Alert.Root variant="warning">
<Alert.Description>
No data for that date range. Showing data between {formatDate(dateRange.startTimestamp)} and
{formatDate(dateRange.endTimestamp)}</Alert.Description
>
</Alert.Root>
</div>
{/if}

<div class="flex items-center space-x-6">
{#if isZoomed}
<ResetZoomButton onClick={onResetZoom} />
{/if}
<DateRangeSelector />
<DateRangeSelector
fallbackDateRange={isUsingFallbackDates
? {
start: fromAbsolute(dateRange.startTimestamp, getLocalTimeZone()),
end: fromAbsolute(dateRange.endTimestamp, getLocalTimeZone())
}
: undefined}
/>
{#if context === 'drift'}
<DriftMetricToggle />
{/if}
</div>

{#if showActionButtons}
<div>
{#if showActionButtons}
<ActionButtons {showCluster} {showSort} {context} />
</div>
{/if}
{/if}
</div>
</div>
Loading