Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(web-console): new monitoring widgets #352

Merged
merged 157 commits into from
Jan 21, 2025
Merged
Changes from 1 commit
Commits
Show all changes
157 commits
Select commit Hold shift + click to select a range
2124412
Core logic for metrics tabs
insmac Oct 30, 2024
0261900
Basic implementation of charts
insmac Oct 31, 2024
6b99213
updates
insmac Oct 31, 2024
d53de21
Refetch on duration change
insmac Oct 31, 2024
230f973
fix restoring metrics buffer
insmac Oct 31, 2024
b1747fb
fix: buffer payload for metrics
insmac Nov 1, 2024
9d0d9bb
Table does not exist error
insmac Nov 2, 2024
9e27ea2
fix: metrics layout
insmac Nov 4, 2024
72f9a3c
Support more timespans, fixes
insmac Nov 4, 2024
fd3f89b
Fix resize and unnecessary re-render
insmac Nov 4, 2024
f1db733
Fix: bring back buffer change re-render
insmac Nov 4, 2024
60dea6c
charts: align fix
insmac Nov 4, 2024
7984efd
Show timezone in toolbar
insmac Nov 4, 2024
dcb83dc
fix: editor buffer restore
insmac Nov 4, 2024
7db5bb1
fix: restore metrics tabs
insmac Nov 4, 2024
9a60782
dashboard tabs PoC
insmac Nov 6, 2024
414e526
Fix: switching between metric tabs
insmac Nov 6, 2024
1e4fff6
fix: metrics loading indicator
insmac Nov 6, 2024
a3d5189
Telemetry unavailable zero state
insmac Nov 7, 2024
627ad90
Buffer naming support
insmac Nov 7, 2024
c475e01
Save buffer data, updates
insmac Nov 7, 2024
4951eb8
fix: new metric duration
insmac Nov 7, 2024
ae49a2b
Metric widget actions
insmac Nov 7, 2024
e89e859
no metrics zero state
insmac Nov 8, 2024
d389d78
fix: filter out non-wal tables in dialog
insmac Nov 8, 2024
2a2b3c6
Change the add metric flow
insmac Nov 13, 2024
bea00ee
WIP: table search select
insmac Nov 14, 2024
ea147f8
small styling update
insmac Nov 14, 2024
50f7dac
temp: disable loader
insmac Nov 14, 2024
4b09527
Update metrics unavailable, libs update
insmac Nov 14, 2024
9080eab
Fetch metrics only when needed
insmac Nov 14, 2024
184929d
fix: wait for telemetry config to be loaded
insmac Nov 14, 2024
9e27ad7
submodule update
insmac Nov 14, 2024
59c9b5e
Add auto metric refresh
insmac Nov 15, 2024
0c1de0b
cleanup metric update
insmac Nov 15, 2024
f964926
Add refresh on focus
insmac Nov 15, 2024
c1ac6a5
Fix: focus listeners
insmac Nov 15, 2024
21e9eb4
Custom styling for metric tabs
insmac Nov 15, 2024
ee974d5
remove `react-select-search`
insmac Nov 18, 2024
cac35be
New table selector
insmac Nov 18, 2024
c642147
Table selector: keyboard nav
insmac Nov 18, 2024
5f3aa13
fix: backspace
insmac Nov 18, 2024
05455d7
Update mouse over keyIndex
insmac Nov 19, 2024
97b1354
simply the tab selector input handling
insmac Nov 19, 2024
7f63f1c
fix: placeholder
insmac Nov 19, 2024
0c475ae
support disabled tables
insmac Nov 19, 2024
95bc485
Simplify action bar
insmac Nov 19, 2024
dac4f2f
list limiter
insmac Nov 20, 2024
46b0d13
Merge branch 'main' into web-console/chart-tabs
insmac Nov 22, 2024
df3bf11
Color palette for metrics
insmac Nov 22, 2024
1026dcc
Organize QuestDB client into files
insmac Nov 25, 2024
0e99fab
Update palette
insmac Nov 25, 2024
cfab887
Merge branch 'main' into web-console/chart-tabs
insmac Nov 26, 2024
4ca8664
Date formatting
insmac Nov 26, 2024
6bedfcc
fixes, date adjustments
insmac Nov 26, 2024
7329494
Clear auto refresh on exit
insmac Nov 27, 2024
9117074
Change wording for latency
insmac Nov 27, 2024
b0aa844
fix: crash when all datapoints are identical
insmac Nov 27, 2024
ee6537f
Implement zoom to data
insmac Nov 29, 2024
a1de63a
fix: setup listeners on new tab creation
insmac Nov 29, 2024
73d2b47
submodule update
insmac Nov 29, 2024
c7e623d
fix: conditional for `lastNotNull`
insmac Nov 29, 2024
4ee9314
sync cursor crosshairs
insmac Nov 29, 2024
d50ee55
fix axes values
insmac Nov 29, 2024
1db382e
convert nulls in avg write amplification
insmac Nov 29, 2024
1ea0d14
round values
insmac Dec 2, 2024
9b16745
fixes: sorting, clean up side effect hooks
insmac Dec 2, 2024
bc507c8
code cleanup
insmac Dec 2, 2024
7ff2152
Default timespan to 1h
insmac Dec 2, 2024
afdc72c
unify y axis values
insmac Dec 3, 2024
ce8db48
Abstract out widgets, add Commit Rate
insmac Dec 3, 2024
ea281ba
fix table selector issues
insmac Dec 4, 2024
3d3d44d
New latency query, update time filter
insmac Dec 4, 2024
4c71d5f
User configurable refresh rates
insmac Dec 4, 2024
bd1a8f8
Refresh all widgets CTA
insmac Dec 4, 2024
d7bae00
disable focus listener on refresh off
insmac Dec 4, 2024
a6d4ff6
disable refetch on focus when set to all off
insmac Dec 4, 2024
ac11a54
cleanup, change sampling config
insmac Dec 4, 2024
c6d0e22
Refactor listeners, simplify, cleanup
insmac Dec 6, 2024
e155064
blur listener deregister
insmac Dec 6, 2024
fe3da6f
Cleanup, add `rollingAppendRowLimit` logic for data refresh
insmac Dec 6, 2024
e8abb4b
View modes in dashboards, sample by in commit rate
insmac Dec 6, 2024
e9164bc
fix: auto refresh bug
insmac Dec 6, 2024
4871103
improve no data conditionals
insmac Dec 9, 2024
eae0eb1
rolling append logic - disabled for now
insmac Dec 10, 2024
1253285
Configurable SAMPLE BY
insmac Dec 10, 2024
e0eaf13
Add metric loaders
insmac Dec 10, 2024
e5b0cb9
Refactor time filter to use FROM TO, refactor rolling append
insmac Dec 12, 2024
ae93d16
Refactor time filters to use eventBus and rely on FROM/TO
insmac Dec 12, 2024
215c853
fix from/to logic
insmac Dec 13, 2024
ad2e638
auto unit fixes
insmac Dec 13, 2024
99c6e5a
Fix: autosize
insmac Dec 13, 2024
3ed7981
fix: first click on `zoom to data` doesn't work
insmac Dec 16, 2024
559faae
cleanup unused file
insmac Dec 16, 2024
7071d0c
Fix: spuriously appearing `zoom to data`
insmac Dec 16, 2024
844d222
Fix: table selector autosizer
insmac Dec 18, 2024
b78e4ad
Remove sample by selector, use automated sampling at all times
insmac Dec 18, 2024
4afefdf
Implement datepicker with ranges
insmac Dec 23, 2024
8e24991
date validator
insmac Dec 23, 2024
7f16e10
Format date in `onChange`
insmac Dec 30, 2024
5ae129a
Datepicker: Validation of custom dates
insmac Dec 30, 2024
ae2b15f
fix: full refetch on widget table change
insmac Dec 30, 2024
b00dd92
submodule update
insmac Dec 30, 2024
10fca7d
Calendar updates
insmac Dec 31, 2024
e918a5d
Fix: Do not re-render X axis when refresh is set to OFF
insmac Dec 31, 2024
04ac986
Fix: x axis formatter
insmac Dec 31, 2024
76b45dc
force refresh on duration change and refresh CTA
insmac Jan 2, 2025
6b9a71d
submodule update
insmac Jan 3, 2025
4b6f2f8
Fixes around dates
insmac Jan 8, 2025
171b95d
Enhance validation
insmac Jan 8, 2025
aaaa582
fix: memleak advisory
insmac Jan 8, 2025
7fd1566
Widget UI updates, validation updates
insmac Jan 8, 2025
ed5fba8
Apply changes on Enter
insmac Jan 9, 2025
358ebe1
Set fetch mode in event payload
insmac Jan 9, 2025
cedcc35
refactor: abstract out types and utils
insmac Jan 9, 2025
851c995
Refactor: Abstract out `DataRange`
insmac Jan 9, 2025
ba41c1d
delayed loader
insmac Jan 10, 2025
9ffd680
fix: refresh after table drops
insmac Jan 10, 2025
de9bb00
validate dates out of range
insmac Jan 10, 2025
148bfbb
Set limit on the metric level
insmac Jan 13, 2025
e7a7195
submodule update
insmac Jan 13, 2025
c6eb370
Merge branch 'main' into web-console/chart-tabs
insmac Jan 13, 2025
aeb081d
Merge branch 'main' into web-console/chart-tabs
insmac Jan 13, 2025
b677d5b
post-merge fixes
insmac Jan 13, 2025
6b1430b
rix: rolling append
insmac Jan 14, 2025
c755955
New sampling rate logic
insmac Jan 14, 2025
b806a45
Add sample rate info
insmac Jan 14, 2025
8cbec45
Cap date range at 7 days
insmac Jan 15, 2025
46f9508
Change when dataframe is fully fetched
insmac Jan 15, 2025
c187356
small cleanup
insmac Jan 15, 2025
3f587b4
fix: limit
insmac Jan 15, 2025
8aac450
remove LIMIT, use last known timestamp
insmac Jan 15, 2025
099a0a4
cleanup
insmac Jan 15, 2025
3310f1e
fix: don't force refresh on tab focus
insmac Jan 15, 2025
eef590f
iterating
nwoolmer Jan 15, 2025
e2574b8
iterating
nwoolmer Jan 15, 2025
916536c
configurable scale distribution
insmac Jan 15, 2025
af959cf
start log scale at 1
insmac Jan 16, 2025
9040a0d
fix: start log y scale at 1
insmac Jan 16, 2025
09845ef
candidate descriptions, will need tuning and substitution of values
nwoolmer Jan 16, 2025
6935043
Interpolate widget descriptions
insmac Jan 16, 2025
32b9835
remove unused values from write throughput SQL
insmac Jan 16, 2025
cd4628f
submodule update
insmac Jan 16, 2025
0c8f9d0
Add unit tests for Metrics utils
insmac Jan 17, 2025
27680a1
set tx to 0 if no writes
insmac Jan 17, 2025
3badcc3
Add fetch error indicator
insmac Jan 17, 2025
ef365ff
adjust tooltip in view mode toggle
insmac Jan 20, 2025
55cda9a
post-merge fix
insmac Jan 20, 2025
3317726
view mode action changes
insmac Jan 20, 2025
f926025
SQL cleanup
insmac Jan 21, 2025
6e8b0b6
inline SQL queries
insmac Jan 21, 2025
13bce54
fix
nwoolmer Jan 21, 2025
b265005
cleanup debris
insmac Jan 21, 2025
9dd608b
submodule update
insmac Jan 21, 2025
742ed43
Bump `@questdb/sql-grammar` to 1.2.2
insmac Jan 21, 2025
e547b3e
submodule update
insmac Jan 21, 2025
c1b9861
Merge branch 'main' into web-console/chart-tabs
insmac Jan 21, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Basic implementation of charts
insmac committed Oct 31, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 026190041c74f2aac781e81470f01ba1164847a1
134 changes: 134 additions & 0 deletions packages/web-console/src/scenes/Editor/Metrics/graph.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
import React, { useContext, useEffect, useRef } from "react"
import styled from "styled-components"
import { MetricDuration } from "./utils"
import { useGraphOptions } from "./useGraphOptions"
import uPlot from "uplot"
import UplotReact from "uplot-react"
import { useSelector } from "react-redux"
import { selectors } from "../../../store"
import { ThemeContext } from "styled-components"
import { Box } from "@questdb/react-components"

const NO_DATA_TEXT = "No data available for this period"
const TELEMETRY_DISABLED_TEXT = "Enable Telemetry to see metrics"

const Root = styled(Box).attrs({
align: "center",
flexDirection: "column",
gap: 0,
})`
position: relative;
background-color: ${({ theme }) => theme.color.backgroundLighter};
padding: 1rem;
`

const Header = styled.span`
font-size: 1.4rem;
font-weight: 600;
margin-bottom: 1rem;
`

const Label = styled.div`
position: absolute;
bottom: 1rem;
display: flex;
gap: 0.5rem;
font-family: ${({ theme }) => theme.font};
`

const LabelValue = styled.span`
color: ${({ theme }) => theme.color.cyan};
`

type Props = {
label: string
data: uPlot.AlignedData
duration: MetricDuration
yValue: (rawValue: number) => string
}

export const Graph = ({ label, data, duration, yValue }: Props) => {
const timeRef = useRef(null)
const valueRef = useRef(null)
const uPlotRef = useRef<uPlot>()
const telemetryConfig = useSelector(selectors.telemetry.getConfig)
const theme = useContext(ThemeContext)

let featureUnavailableText: string
if (!telemetryConfig?.enabled) {
featureUnavailableText = TELEMETRY_DISABLED_TEXT
} else {
featureUnavailableText = NO_DATA_TEXT
}

const resizeObserver = new ResizeObserver((entries) => {
uPlotRef.current?.setSize({
width: entries[0].contentRect.width,
height: 180,
})
})

const graphOptions = useGraphOptions({
data,
duration,
timeRef,
valueRef,
xValue: (rawValue, index, ticks) =>
index === 0 || index === ticks.length - 1
? new Date(rawValue).toLocaleTimeString(navigator.language, {
hour: "2-digit",
minute: "2-digit",
hourCycle: "h23",
})
: null,
yValue,
})

const graphRootRef = useRef<HTMLDivElement>(null)

useEffect(() => {
if (graphRootRef.current) {
resizeObserver.observe(graphRootRef.current)
}

return () => {
resizeObserver.disconnect()
}
}, [graphRootRef.current])

return (
<Root ref={graphRootRef}>
<Header>{label}</Header>
<UplotReact
options={{
...graphOptions,
height: 200,
width: graphRootRef.current?.clientWidth
? graphRootRef.current.clientWidth - 22
: 0,
}}
data={data}
onCreate={(uplot) => {
uPlotRef.current = uplot
if (uplot.data[0].length === 0 || !telemetryConfig?.enabled) {
const noData = document.createElement("div")
noData.innerText = featureUnavailableText
noData.style.position = "absolute"
noData.style.left = "50%"
noData.style.top = "50%"
noData.style.transform = "translate(-50%, -50%)"
noData.style.color = theme.color.gray2
noData.style.fontSize = "1.2rem"
noData.style.width = "100%"
noData.style.textAlign = "center"
uplot.over.appendChild(noData)
}
}}
/>
<Label>
<span ref={timeRef} />
<LabelValue ref={valueRef} />
</Label>
</Root>
)
}
204 changes: 201 additions & 3 deletions packages/web-console/src/scenes/Editor/Metrics/index.tsx
Original file line number Diff line number Diff line change
@@ -1,13 +1,211 @@
import React from "react"
import React, { useContext, useEffect, useState } from "react"
import styled from "styled-components"
import { Box, Select } from "@questdb/react-components"
import { Text } from "../../../components"
import { useEditor } from "../../../providers"
import { useSelector } from "react-redux"
import { selectors } from "../../../store"
import { MetricDuration } from "./utils"
import type { Latency, RowsApplied } from "./utils"
import * as QuestDB from "../../../utils/questdb"
import { QuestContext } from "../../../providers"
import { rowsApplied as rowsAppliedSQL, latency as latencySQL } from "./queries"
import { Graph } from "./graph"

const Root = styled.div`
width: 100%;
height: 100%;
background: #2c2e3d;
padding: 2rem 2.5rem;
padding-bottom: calc(4.5rem + 2.5rem + 2.5rem);
`

const Toolbar = styled(Box).attrs({
align: "center",
justifyContent: "space-between",
})`
width: 100%;
height: 4.5rem;
padding: 0 2.5rem;
border-bottom: 1px solid ${({ theme }) => theme.color.backgroundDarker};
box-shadow: 0 2px 10px 0 rgba(23, 23, 23, 0.35);
`

const Header = styled(Text)`
font-size: 1.8rem;
font-weight: 600;
color: ${({ theme }) => theme.color.foreground};
`

const Charts = styled.div`
display: grid;
grid-template-columns: 1fr 1fr;
align-content: flex-start;
gap: 2.5rem;
padding: 2.5rem;
overflow-y: auto;
height: 100%;
`

export const Metrics = () => {
return <Root>metrics charts here</Root>
const { quest } = useContext(QuestContext)
const { activeBuffer, updateBuffer } = useEditor()
const tables = useSelector(selectors.query.getTables)
const [metricDuration, setMetricDuration] = useState<MetricDuration>(
(activeBuffer?.metricsViewState?.metricDuration as MetricDuration) ??
MetricDuration.SEVEN_DAYS,
)
const [rowsAppliedLoading, setRowsAppliedLoading] = useState(false)
const [latencyLoading, setLatencyLoading] = useState(false)
const [rowsApplied, setRowsApplied] = useState<RowsApplied[]>([])
const [lastNotNullRowsApplied, setLastNotNullRowsApplied] =
useState<RowsApplied>()
const [latency, setLatency] = useState<Latency[]>([])
const [lastNotNullLatency, setLastNotNullLatency] = useState<Latency>()

const formatDurationLabel = (duration: MetricDuration) => `Last ${duration}`

const table = tables.find(
(table) => table.id === activeBuffer?.metricsViewState?.tableId,
)

const fetchRowsApplied = async (tableId: number) => {
try {
const response = await quest.query<RowsApplied>(
rowsAppliedSQL(tableId, metricDuration),
)
if (response && response.type === QuestDB.Type.DQL) {
setRowsApplied(response.data)
const lastNotNullRowsApplied = response.data
.slice()
.reverse()
.find(
(l) =>
l.avgWalAmplification !== null && l.numOfRowsWritten !== null,
)
setLastNotNullRowsApplied(lastNotNullRowsApplied)
}
} catch (e) {
console.error(e)
} finally {
setRowsAppliedLoading(false)
}
}

const fetchLatency = async (tableId: number) => {
try {
const response = await quest.query<Latency>(
latencySQL(tableId, metricDuration),
)
if (response && response.type === QuestDB.Type.DQL) {
setLatency(response.data)
const lastNotNullLatency = response.data
.slice()
.reverse()
.find((l) => l.numOfWalApplies !== null && l.avg_latency !== null)
setLastNotNullLatency(lastNotNullLatency)
}
} catch (e) {
console.error(e)
} finally {
setLatencyLoading(false)
}
}

useEffect(() => {
if (activeBuffer.metricsViewState) {
fetchLatency(activeBuffer.metricsViewState.tableId)
fetchRowsApplied(activeBuffer.metricsViewState.tableId)
setMetricDuration(
(activeBuffer.metricsViewState.metricDuration as MetricDuration) ??
MetricDuration.SEVEN_DAYS,
)
}
}, [activeBuffer])

useEffect(() => {
if (!activeBuffer.id || !activeBuffer.metricsViewState || !metricDuration)
return

updateBuffer(activeBuffer.id, {
...activeBuffer,
metricsViewState: {
...activeBuffer.metricsViewState,
metricDuration,
},
})
}, [metricDuration])

if (!table) return null

return (
<Root>
<Toolbar>
<Header>WAL metrics for {table.table_name}</Header>
<Box align="center" gap="1rem">
<Select
name="duration"
value={metricDuration}
options={[
{
label: formatDurationLabel(MetricDuration.TWENTY_FOUR_HOURS),
value: MetricDuration.TWENTY_FOUR_HOURS,
},
{
label: formatDurationLabel(MetricDuration.THREE_DAYS),
value: MetricDuration.THREE_DAYS,
},
{
label: formatDurationLabel(MetricDuration.SEVEN_DAYS),
value: MetricDuration.SEVEN_DAYS,
},
]}
onChange={(e) =>
setMetricDuration(e.target.value as MetricDuration)
}
/>
</Box>
</Toolbar>
<Charts>
{!latencyLoading && (
<div>
<Graph
label="Read latency in ms"
duration={metricDuration}
data={[
latency.map((l) => new Date(l.time).getTime()),
latency.map((l) => parseFloat(l.avg_latency)),
]}
yValue={(rawValue: number) => (+rawValue).toFixed(2) + "ms"}
/>
</div>
)}
{!rowsAppliedLoading && (
<div>
<Graph
label="Write throughput"
duration={metricDuration}
data={[
rowsApplied.map((l) => new Date(l.time).getTime()),
rowsApplied.map((l) => parseFloat(l.numOfRowsWritten)),
]}
yValue={(rawValue: number) => (+rawValue).toFixed(0)}
/>
</div>
)}
{!rowsAppliedLoading && (
<div>
<Graph
label="Write amplification"
duration={metricDuration}
data={[
rowsApplied.map((l) => new Date(l.time).getTime()),
rowsApplied.map((l) => parseFloat(l.avgWalAmplification)),
]}
yValue={(rawValue: number) => (+rawValue).toFixed(0) + "x"}
/>
</div>
)}
</Charts>
</Root>
)
}
4 changes: 2 additions & 2 deletions packages/web-console/src/scenes/Editor/Metrics/queries.ts
Original file line number Diff line number Diff line change
@@ -7,7 +7,7 @@ import {
} from "./utils"

export const rowsApplied = (
id: string,
id: number,
metricDuration: MetricDuration,
sampleBy?: SampleBy,
) => `
@@ -27,7 +27,7 @@ and created < date_trunc('minute', now())
sample by ${sampleBy ?? mappedSampleBy[metricDuration]}`

export const latency = (
id: string,
id: number,
metricDuration: MetricDuration,
sampleBy?: SampleBy,
) => `
Loading