-
Notifications
You must be signed in to change notification settings - Fork 17
feat: DAG data-flow timeline — per-operator distribution protocol + playhead UI #393
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
rapids-bot
merged 15 commits into
rapidsai:main
from
felipeblazing:claude/dag-data-flow-timeline-d6f6e4
Jul 20, 2026
Merged
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
fedc7cd
feat(analyzer,server): generic data-flow distribution timeline protocol
felipeblazing 81d2ffe
feat(ui): DAG data-flow timeline playhead and per-node distribution bars
felipeblazing 9497334
Merge remote-tracking branch 'origin/main' into claude/dag-data-flow-…
felipeblazing fb08a6c
feat(ui): in-segment value labels and dual-measure totals on DAG flow…
felipeblazing c8e8ffa
fix(ui): pipe separator for node flow totals
felipeblazing 783e578
feat(ui): data-flow label measure, labeled tier bar, and tier selection
felipeblazing c52f37f
refactor(query-engine): address PR #393 review feedback (Rust)
felipeblazing 79a2e86
refactor(ui): address PR #393 review feedback
felipeblazing 1af4855
style: rustfmt; a11y headers on the data-flow matrix
felipeblazing ce70c65
fix(ui): collision-free colors for synthetic data-flow states
felipeblazing b250582
feat(ui): analyzer-declared default measure for distribution timelines
felipeblazing c50058f
feat(ui): honor default_measure; per-tier totals in the DAG legend
felipeblazing 2d0438b
refactor(analyzer,ui): address maintainer review on the data-flow pro…
felipeblazing 85969a3
refactor(ui): adapt to the categorical-timeline protocol rename
felipeblazing c7571a8
style(docs): collapse double blank line left by section removal
felipeblazing File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,171 @@ | ||
| // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| //! Binned timelines of weighted values keyed by categories. | ||
| //! | ||
| //! A categorical timeline describes, per opaque series (e.g. an operator in a | ||
| //! query engine), how some weighted quantity (a "measure", e.g. an entity | ||
| //! count) breaks down over the states of a finite state machine and an | ||
| //! application-defined dimension (e.g. which resource holds the entity's | ||
| //! data), for each time bin of a window. Bin values are absolute, | ||
| //! time-weighted quantities — not normalized shares. | ||
| //! | ||
| //! This module is application-agnostic: series, measures, states, and | ||
| //! dimension keys are opaque to the aggregation. Downstream analyzers decide | ||
| //! what they mean and are expected to keep dimension keys a small enumerable | ||
| //! set. | ||
|
|
||
| use std::hash::Hash; | ||
|
|
||
| use rustc_hash::FxHashMap as HashMap; | ||
|
|
||
| use quent_time::{SpanNanoSec, bin::BinnedSpan}; | ||
|
|
||
| use crate::{ | ||
| AnalyzerResult, | ||
| timeline::binned::{BinnedTimelineAggregator, KeyedAggregator}, | ||
| }; | ||
|
|
||
| /// Identity of one aggregation cell of a categorical timeline. | ||
| #[derive(Clone, Debug, PartialEq, Eq, Hash)] | ||
| pub struct CategoricalKey<S, M, St, D> { | ||
| /// Opaque series the sample belongs to (e.g. an operator id downstream). | ||
| pub series: S, | ||
| /// The measure this weight contributes to (e.g. an entity count). | ||
| pub measure: M, | ||
| /// The FSM state name during the span. | ||
| pub state: St, | ||
| /// Application-defined dimension key (opaque to the aggregation). | ||
| pub dimension: D, | ||
| } | ||
|
|
||
| /// A binned timeline of weighted (state, dimension) values for multiple | ||
| /// series and measures. | ||
| #[derive(Clone, Debug)] | ||
| pub struct CategoricalTimeline<S, M, St, D> { | ||
| pub config: BinnedSpan, | ||
| pub data: HashMap<CategoricalKey<S, M, St, D>, Vec<f64>>, | ||
| } | ||
|
|
||
| /// Builds a [`CategoricalTimeline`] from weighted samples. | ||
| /// | ||
| /// Aggregation is span-weighted: each sample contributes | ||
| /// `weight * overlap_fraction` to every bin its span intersects, so bin values | ||
| /// are time-weighted averages over the bin, not instantaneous snapshots. | ||
| pub struct CategoricalTimelineBuilder<S, M, St, D> { | ||
| aggregator: KeyedAggregator<CategoricalKey<S, M, St, D>>, | ||
| } | ||
|
|
||
| impl<S, M, St, D> CategoricalTimelineBuilder<S, M, St, D> | ||
| where | ||
| S: Eq + Hash, | ||
| M: Eq + Hash, | ||
| St: Eq + Hash, | ||
| D: Eq + Hash, | ||
| { | ||
| pub fn new(config: BinnedSpan) -> Self { | ||
| Self { | ||
| aggregator: KeyedAggregator::new(config), | ||
| } | ||
| } | ||
|
|
||
| /// Return the configuration of the binned timeline. | ||
| pub fn config(&self) -> BinnedSpan { | ||
| self.aggregator.config() | ||
| } | ||
|
|
||
| /// Attempt to push one weighted sample spanning `span` into the timeline. | ||
| pub fn try_push( | ||
| &mut self, | ||
| key: CategoricalKey<S, M, St, D>, | ||
| span: SpanNanoSec, | ||
| weight: f64, | ||
| ) -> AnalyzerResult<()> { | ||
| self.aggregator.try_push(span, (key, weight)) | ||
| } | ||
|
|
||
| pub fn build(self) -> CategoricalTimeline<S, M, St, D> { | ||
| CategoricalTimeline { | ||
| config: self.aggregator.config(), | ||
| data: self.aggregator.finish(), | ||
| } | ||
| } | ||
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod tests { | ||
| use std::num::NonZero; | ||
|
|
||
| use super::*; | ||
|
|
||
| fn test_config() -> BinnedSpan { | ||
| BinnedSpan::try_new( | ||
| SpanNanoSec::try_new(0, 1000).unwrap(), | ||
| NonZero::try_from(10).unwrap(), | ||
| ) | ||
| .unwrap() | ||
| } | ||
|
|
||
| fn key<'a>( | ||
| series: u32, | ||
| measure: &'a str, | ||
| state: &'a str, | ||
| dimension: &'a str, | ||
| ) -> CategoricalKey<u32, &'a str, &'a str, &'a str> { | ||
| CategoricalKey { | ||
| series, | ||
| measure, | ||
| state, | ||
| dimension, | ||
| } | ||
| } | ||
|
|
||
| /// Samples differing in any key component land in distinct cells; the | ||
| /// binning math itself is covered by the aggregator's own tests. | ||
| #[test] | ||
| fn distinct_series_measures_states_dimensions() -> AnalyzerResult<()> { | ||
| let mut builder = CategoricalTimelineBuilder::new(test_config()); | ||
| let span = SpanNanoSec::try_new(0, 1000).unwrap(); | ||
|
|
||
| builder.try_push(key(1, "count", "a", "x"), span, 1.0)?; | ||
| builder.try_push(key(2, "count", "a", "x"), span, 1.0)?; | ||
| builder.try_push(key(1, "bytes", "a", "x"), span, 100.0)?; | ||
| builder.try_push(key(1, "count", "b", "x"), span, 1.0)?; | ||
| builder.try_push(key(1, "count", "a", "y"), span, 1.0)?; | ||
|
|
||
| let timeline = builder.build(); | ||
| assert_eq!(timeline.data.len(), 5); | ||
| assert_eq!( | ||
| timeline.data.get(&key(1, "bytes", "a", "x")).unwrap()[..], | ||
| [100.0; 10] | ||
| ); | ||
| assert_eq!( | ||
| timeline.data.get(&key(2, "count", "a", "x")).unwrap()[..], | ||
| [1.0; 10] | ||
| ); | ||
| Ok(()) | ||
| } | ||
|
|
||
| /// Non-string key components only need `Eq + Hash` — no stringification. | ||
| #[test] | ||
| fn non_string_key_components() -> AnalyzerResult<()> { | ||
| #[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)] | ||
| enum Measure { | ||
| Count, | ||
| } | ||
|
|
||
| let mut builder: CategoricalTimelineBuilder<u32, Measure, u8, u16> = | ||
| CategoricalTimelineBuilder::new(test_config()); | ||
| let cell = CategoricalKey { | ||
| series: 7u32, | ||
| measure: Measure::Count, | ||
| state: 3u8, | ||
| dimension: 9u16, | ||
| }; | ||
| builder.try_push(cell.clone(), SpanNanoSec::try_new(0, 1000).unwrap(), 2.0)?; | ||
|
|
||
| let timeline = builder.build(); | ||
| assert_eq!(timeline.data.get(&cell).unwrap()[..], [2.0; 10]); | ||
| Ok(()) | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| //! Requests and responses for categorical timelines: binned timelines of | ||
| //! weighted values broken down by (FSM state, application-defined dimension), | ||
| //! for one or more application-declared measures. Bin values are absolute | ||
| //! time-weighted quantities, not normalized shares. | ||
| //! | ||
| //! All semantics are declared by the downstream analyzer: which FSM the states | ||
| //! belong to, what the dimension keys mean, and which measures exist. The UI | ||
| //! renders the declared names verbatim. | ||
|
|
||
| use std::collections::HashMap; | ||
|
|
||
| use serde::{Deserialize, Serialize}; | ||
| use ts_rs::TS; | ||
|
|
||
| use crate::{quantity::CapacityKind, timeline::request::TimelineConfig}; | ||
|
|
||
| /// Request for a categorical timeline. | ||
| #[derive(TS, Debug, Clone, Serialize, Deserialize)] | ||
| pub struct CategoricalTimelineRequest<GlobalParams> { | ||
| /// Names of the measures to compute. Empty means all declared measures. | ||
| pub measures: Vec<String>, | ||
| /// The configuration of the window and number of bins. | ||
| pub config: TimelineConfig, | ||
| /// Global application-specific parameters, e.g. filters. | ||
| pub app_params: GlobalParams, | ||
| } | ||
|
|
||
| /// A measure declared by the downstream analyzer for a categorical timeline, | ||
| /// e.g. an entity count or a number of bytes. | ||
| #[derive(TS, Debug, Clone, Serialize)] | ||
| pub struct MeasureDecl { | ||
| /// Unique name; key into [`CategoricalSeries::values`]. | ||
| pub name: String, | ||
| /// Human-friendly display name. | ||
| pub display_name: String, | ||
| /// Key into the application's quantity specs map for unit formatting. | ||
| pub quantity: String, | ||
| /// Display semantics of a bin value (Occupancy = time-weighted level). | ||
| pub kind: CapacityKind, | ||
| } | ||
|
|
||
| /// One key of the application-defined dimension of a categorical timeline. | ||
| #[derive(TS, Debug, Clone, Serialize)] | ||
| pub struct DimensionKeyDecl { | ||
| /// The key used in [`CategoricalSeries::values`]. | ||
| pub key: String, | ||
| /// Human-friendly display name. | ||
| pub display_name: String, | ||
| } | ||
|
|
||
| /// Presentation metadata for a categorical timeline, declared by the | ||
| /// downstream analyzer. | ||
| /// | ||
| /// Dimension keys are expected to be a small enumerable set; unbounded key | ||
| /// cardinality is a downstream misuse. | ||
| #[derive(TS, Debug, Clone, Serialize)] | ||
| pub struct CategoricalDecl { | ||
| /// The FSM type whose states are broken down. References an entry in the | ||
| /// application's FSM type declarations (e.g. `QueryBundle` fsm_types) for | ||
| /// the state graph, names, and ordering. | ||
| pub entity_type_name: String, | ||
| /// Display name of the dimension, e.g. an application may use a dimension | ||
| /// describing where an entity's data resides. | ||
| pub dimension_name: String, | ||
| /// The dimension keys in stable stacking/legend order. | ||
| pub dimension_keys: Vec<DimensionKeyDecl>, | ||
| /// The measures present in this response. | ||
| pub measures: Vec<MeasureDecl>, | ||
| /// The measure the UI should select by default; must name an entry in | ||
| /// `measures`. `None` means the first declared measure. | ||
| pub default_measure: Option<String>, | ||
| } | ||
|
|
||
| /// Binned values of one categorical timeline series: | ||
| /// measure name -> state name -> dimension key -> one value per time bin. | ||
| /// | ||
| /// Absent inner entries mean all-zero bins. | ||
| #[derive(TS, Debug, Clone, Default, Serialize)] | ||
| pub struct CategoricalSeries { | ||
| pub values: HashMap<String, HashMap<String, HashMap<String, Vec<f64>>>>, | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The tests here seem a bit superfluous as they test functionality of the inner aggregator more than any functionality expressed in this source. If these tests do not exist on the aggregator yet we should move them there.