feat(studio): add experiment trace comparison view [ASE-186]

[walston]

Closes ASE-186

Screen.Recording.2026-07-22.at.08.50.28.mov

Recording above predates the run-based redesign — will refresh before merge.

Summary

Evaluation trace detail could only show one run of a test case at a time. This adds a side-by-side comparison of the same test case across different runs.

A "Compare against evaluation run" selector in the page header lists every run (session) of the current test case across the experiment group's evaluations. Because the sessions endpoint is scoped to a single evaluation, the list is built client-side by fanning out one query per evaluation (GET /apis/intake/v2/workspaces/{ws}/evaluations/{name}/sessions?filter[test_case_id]=<id>) and flattening — a group-scoped endpoint is the eventual replacement (fast-follow). Picking a run adds ?compareWith=<traceId> to the URL and splits into a "Test case comparison — Test case <id>" page with two columns.

Each column shows:

• the run label — <evaluation-name> · Trial <last 5 of session id>
• expand-all / collapse-all controls
• a per-session metrics card: Cost, Latency, Tkns In, Tkns Out, Cached Tkns, and every evaluator score (not experiment rollups)
• the trace in list mode

If the selected run has no matching session, the column shows "Test case not available".

Selector UX: single-trial evaluations collapse onto one selectable line; multi-trial evaluations render the evaluation as a subdued heading with each trial indented beneath (↳ Trial XXXXX). Every option carries an aria-label of the full <evaluation> · Trial XXXXX so screen readers get self-describing options. Loading shows a disabled control with "Loading other runs"; no siblings shows "No runs to compare to".

Rendering two trace views also required two supporting fixes:

• TraceSpanAccordions is controlled — single view binds span selection to ?spanId= (deep links unchanged); compare columns render independently in list mode. Expand/collapse surface to the column header via a SlotHeaderRenderProp.
• Refresh invalidation is scoped to the specific trace's query key (was invalidating every span-list query in the workspace).

Test plan

• Open an evaluation trace (/workspaces/<ws>/experiment/<group>/<evaluation>/traces/<traceId>) → "Compare against evaluation run" selector appears in the page header (top right)
• Open the selector → runs of this test case across the group's evaluations, grouped by evaluation; single-trial evals collapsed to one line, multi-trial evals nested with ↳; disabled + "Loading other runs" while loading, "No runs to compare to" when there are none
• Pick a run → URL gains ?compareWith=<traceId>, page shows "Test case comparison — Test case <id>" and splits into two columns, each with the run label, per-session metrics (Cost, Latency, Tkns In/Out, Cached, evaluator scores), and the trace in list mode
• Pick a different run → the compare column swaps (no backtracking); Clear returns to the single trace view
• Pick a run whose session isn't found → right column shows "Test case not available"
• Single-trace view: click spans → ?spanId= updates in the URL; reload the deep link → same span selected

Reviewer note (dev data): most experiment groups have a single trial per evaluation (all collapsed). For a dropdown showing both collapsed and nested entries, use workspace bugnemo-mc, group opt-97d7f6d1-02b5-4f34-82db-a09b3feefc0f, test case ttyb-028-...bug-432.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-nwalston-ase-186-trace-comparison-view.docs.buildwithfern.com/nemo-platform

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	27113/34832	77.8%	62.1%
Integration Tests	15892/33443	47.5%	19.9%

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Changes

Experiment trace comparison

Layer / File(s)	Summary
Controlled span selection web/packages/studio/src/components/IntakeDetail/TraceDetailView.tsx, web/packages/studio/src/components/IntakeDetail/TraceSpanAccordions.tsx, web/packages/studio/src/routes/IntakeTraceDetailRoute/index.tsx	Span selection is parent-controlled, synchronized through spanId, and breadcrumb updates can be disabled for embedded traces.
Comparison session lookup web/packages/studio/src/routes/ExperimentTraceDetailRoute/useCompareSession.ts, web/packages/studio/src/routes/ExperimentTraceDetailRoute/useCompareSession.test.ts	Sessions are queried by test case and represented as loading, missing-ID, not-found, or found states.
Comparison route and navigation web/packages/studio/src/routes/ExperimentTraceDetailRoute/index.tsx, web/packages/studio/src/routes/ExperimentTraceDetailRoute/CompareExperimentSelect.tsx, web/packages/studio/src/routes/constants.ts, web/packages/studio/src/routes/utils.ts	The route filters comparable sibling experiments, adds compareWith navigation, and renders a controlled comparison selector.
Side-by-side comparison view web/packages/studio/src/routes/ExperimentTraceDetailRoute/ExperimentTraceCompare.tsx, web/packages/studio/src/routes/ExperimentTraceDetailRoute/index.test.tsx	Primary and matched traces render in separate columns with loading, unavailable, and missing-test-case states covered by route tests.

Sequence Diagram(s)

sequenceDiagram
  participant Browser
  participant ExperimentTraceDetailRoute
  participant useCompareSession
  participant ExperimentTraceCompare
  participant IntakeTraceDetailView
  Browser->>ExperimentTraceDetailRoute: Select compareWith experiment
  ExperimentTraceDetailRoute->>useCompareSession: Query matching test_case_id
  useCompareSession->>ExperimentTraceCompare: Return comparison state
  ExperimentTraceCompare->>IntakeTraceDetailView: Render primary and matched traces
  IntakeTraceDetailView->>Browser: Update linked span selection

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: adding an experiment trace comparison view.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch nwalston/ase-186-trace-comparison-view

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 11

🧹 Nitpick comments (3)

docs/superpowers/plans/2026-07-07-experiment-column-reordering.md (2)

1-33: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Align this page with the Markdown documentation guidelines.

The page mixes architecture explanation, implementation how-to steps, and reference material. It also lacks an explicit prerequisites section at the top and a Next Steps section at the end. Either classify this as an internal implementation-plan exception or restructure it into one Diataxis quadrant with the required sections.

Also applies to: 544-554

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/superpowers/plans/2026-07-07-experiment-column-reordering.md` around
lines 1 - 33, Restructure the document into a single Diataxis category,
preferably an internal implementation plan, instead of mixing architecture,
procedural instructions, and reference material. Add an explicit Prerequisites
section near the top and a Next Steps section at the end, and reorganize or
remove content that does not fit the chosen category while preserving the
implementation plan’s actionable task structure.

Source: Coding guidelines

---

500-540: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Add focused automated coverage for the opt-in behavior.

Existing DataView tests and manual checks do not lock down the new contract. Add tests for the disabled default, pinned-column exclusion, reordered columnOrder, body-cell movement, sorting via the label, and column visibility.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/superpowers/plans/2026-07-07-experiment-column-reordering.md` around
lines 500 - 540, Add focused automated tests for the DataView column-reordering
contract, covering the disabled default, exclusion of pinned columns, reordered
columnOrder, corresponding body-cell movement, sorting through the column label,
and column visibility. Place the tests with the existing DataView test suite and
use the relevant TableColumnHeader, TableContent, and ExperimentGroupDataView
behaviors to verify both opt-in and unchanged default behavior.

web/packages/studio/src/routes/ExperimentTraceDetailRoute/index.test.tsx (1)

125-173: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider adding a test for the select → navigate flow.

Current compare-view tests assert rendered states but not that choosing an experiment from CompareExperimentSelect (or clicking "Clear") actually updates the URL/route via handleCompareChange/handleClearCompare.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@web/packages/studio/src/routes/ExperimentTraceDetailRoute/index.test.tsx`
around lines 125 - 173, Add a compare-view interaction test that selects another
experiment through CompareExperimentSelect and verifies navigation updates the
compareWith query parameter via handleCompareChange; also cover the Clear action
and assert it removes the compare parameter or returns to the primary route via
handleClearCompare.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/superpowers/plans/2026-07-07-experiment-column-reordering.md`:
- Around line 180-185: Update the validation commands in the documented steps,
including the “Verify types pass” checks and the referenced validation sections,
to avoid masking failures through pipelines. Run typecheck/lint commands
directly where possible, or prepend `set -o pipefail` and ensure filtering
commands cannot replace the original command’s exit status.
- Around line 522-531: Update “Step 3: Run existing DataView tests to confirm
nothing regressed” to invoke the repository’s declared JavaScript test command
unconditionally, removing the uv run pytest fallback chain; retain the Vitest
command only if it is the documented web test runner, and ensure the command
targets packages/common/src/components/DataView.
- Around line 27-31: The column reordering logic must preserve pinned column IDs
in the table’s full column order. Update the DnD setup and reorder handler
around TableContent, SortableContext, and arrayMove so only non-pinned columns
are draggable while table.setColumnOrder() always receives the complete order;
merge reordered non-pinned IDs back with the original pinned IDs rather than
replacing the order with a filtered list.
- Around line 298-320: Update DragAlongCell to accept and forward the existing
TableDataCell props and children by extending ComponentProps<typeof
TableDataCell>; merge the sortable DnD styles with the incoming style rather
than replacing it, and render the provided children instead of reconstructing
content with flexRender. Apply the same forwarding pattern to the related
implementation noted around the second occurrence.
- Around line 82-109: Update the header drag wiring and DndContext setup to
configure KeyboardSensor with the required coordinate getter, pass
setActivatorNodeRef through DragProps, and apply it to the drag handle button.
Also set the handle’s type to "button" to prevent form submission, updating
related props and rendering paths such as TableColumnHeaderProps as needed.
- Around line 296-320: Wrap the body-cell rendering that contains DragAlongCell
in a SortableContext using items={columnOrder} and
horizontalListSortingStrategy, or move an equivalent context to a shared parent
that covers both headers and body cells. Ensure every DragAlongCell receives the
same column order and sorting strategy as the dragged headers so row cells
remain aligned during column reordering.

In `@scripts/seed_local_experiments.py`:
- Around line 34-36: Add a shared timeout constant and pass it to both
urllib.request.urlopen calls in the relevant request helpers, including get, so
endpoint requests cannot hang indefinitely.
- Around line 107-113: Update the session-selection logic in the experiment
seeding flow to fetch both experiment session lists, retain only sessions with
non-empty test_case_id values, and intersect them by shared test_case_id rather
than relying on page order or session_id fallback. Before seeding, fail
explicitly when fewer than three shared test cases remain, and use the matched
test_case_id values to select the corresponding sessions for both experiments.
- Around line 113-114: Update the agent.output construction in the
session-processing logic to derive its value from the already normalized `inp`
variable, using `inp[:500]` instead of calling `session.get("input", "")[:500]`,
so null inputs are handled safely and the existing truncation is reused.

In
`@web/packages/studio/src/routes/ExperimentTraceDetailRoute/ExperimentTraceCompare.tsx`:
- Around line 108-131: The compare column shows the no-test-case state while the
primary trace is still loading. Thread the primary trace loading flag from the
parent into the comparison logic, and update renderRightColumn and
useCompareSession usage to return the loading UI until the primary trace
resolves; only treat a falsy testCaseId as no-test-case-id after loading
completes. Use the existing renderRightColumn and useCompareSession symbols.

In
`@web/packages/studio/src/routes/ExperimentTraceDetailRoute/useCompareSession.test.ts`:
- Line 10: Replace the relative useCompareSession import in the test with the
project’s configured absolute alias path, preserving the existing symbol and
import behavior.

---

Nitpick comments:
In `@docs/superpowers/plans/2026-07-07-experiment-column-reordering.md`:
- Around line 1-33: Restructure the document into a single Diataxis category,
preferably an internal implementation plan, instead of mixing architecture,
procedural instructions, and reference material. Add an explicit Prerequisites
section near the top and a Next Steps section at the end, and reorganize or
remove content that does not fit the chosen category while preserving the
implementation plan’s actionable task structure.
- Around line 500-540: Add focused automated tests for the DataView
column-reordering contract, covering the disabled default, exclusion of pinned
columns, reordered columnOrder, corresponding body-cell movement, sorting
through the column label, and column visibility. Place the tests with the
existing DataView test suite and use the relevant TableColumnHeader,
TableContent, and ExperimentGroupDataView behaviors to verify both opt-in and
unchanged default behavior.

In `@web/packages/studio/src/routes/ExperimentTraceDetailRoute/index.test.tsx`:
- Around line 125-173: Add a compare-view interaction test that selects another
experiment through CompareExperimentSelect and verifies navigation updates the
compareWith query parameter via handleCompareChange; also cover the Clear action
and assert it removes the compare parameter or returns to the primary route via
handleClearCompare.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 34625f0e-0e3c-46f1-b783-ef1a39731a7e

📥 Commits

Reviewing files that changed from the base of the PR and between 9903a5f and 205203a.

📒 Files selected for processing (13)

• docs/superpowers/plans/2026-07-07-experiment-column-reordering.md
• scripts/seed_local_experiments.py
• web/packages/studio/src/components/IntakeDetail/TraceDetailView.tsx
• web/packages/studio/src/components/IntakeDetail/TraceSpanAccordions.tsx
• web/packages/studio/src/routes/ExperimentTraceDetailRoute/CompareExperimentSelect.tsx
• web/packages/studio/src/routes/ExperimentTraceDetailRoute/ExperimentTraceCompare.tsx
• web/packages/studio/src/routes/ExperimentTraceDetailRoute/index.test.tsx
• web/packages/studio/src/routes/ExperimentTraceDetailRoute/index.tsx
• web/packages/studio/src/routes/ExperimentTraceDetailRoute/useCompareSession.test.ts
• web/packages/studio/src/routes/ExperimentTraceDetailRoute/useCompareSession.ts
• web/packages/studio/src/routes/IntakeTraceDetailRoute/index.tsx
• web/packages/studio/src/routes/constants.ts
• web/packages/studio/src/routes/utils.ts

[coderabbitai]

🧹 Nitpick comments (1)

docs/superpowers/plans/2026-07-07-experiment-column-reordering.md (1)

1-9: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Keep this page in one Diátaxis quadrant.

This page mixes a HOW-TO, architecture explanation, reference table, and manual checklist. Add prerequisites before the goal, classify the page (likely HOW-TO), split reference/architecture material into linked pages if needed, and add a Next Steps section with cross-links.

As per coding guidelines, documentation pages must use one Diátaxis quadrant, list prerequisites first, apply progressive disclosure, and include Next Steps.

Also applies to: 544-554

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/superpowers/plans/2026-07-07-experiment-column-reordering.md` around
lines 1 - 9, The implementation plan mixes Diátaxis content types instead of
staying in one quadrant. Rework this page as a HOW-TO: add prerequisites before
the Goal, keep the implementation steps and manual checklist as the primary
content, move architecture and reference details to linked documentation where
appropriate, and add a Next Steps section with relevant cross-links.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@docs/superpowers/plans/2026-07-07-experiment-column-reordering.md`:
- Around line 1-9: The implementation plan mixes Diátaxis content types instead
of staying in one quadrant. Rework this page as a HOW-TO: add prerequisites
before the Goal, keep the implementation steps and manual checklist as the
primary content, move architecture and reference details to linked documentation
where appropriate, and add a Next Steps section with relevant cross-links.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ffe3de71-8e4a-4c0d-8e2f-f7fa772f57df

📥 Commits

Reviewing files that changed from the base of the PR and between 205203a and 8d296fc.

📒 Files selected for processing (15)

• docs/superpowers/plans/2026-07-07-experiment-column-reordering.md
• scripts/seed_local_experiments.py
• web/packages/studio/src/components/IntakeDetail/IntakeComponents/spanKeyValues.test.tsx
• web/packages/studio/src/components/IntakeDetail/TraceDetailView.tsx
• web/packages/studio/src/components/IntakeDetail/TraceSpanAccordions.tsx
• web/packages/studio/src/routes/ExperimentTraceDetailRoute/CompareExperimentSelect.tsx
• web/packages/studio/src/routes/ExperimentTraceDetailRoute/ExperimentTraceCompare.tsx
• web/packages/studio/src/routes/ExperimentTraceDetailRoute/index.test.tsx
• web/packages/studio/src/routes/ExperimentTraceDetailRoute/index.tsx
• web/packages/studio/src/routes/ExperimentTraceDetailRoute/useCompareSession.test.ts
• web/packages/studio/src/routes/ExperimentTraceDetailRoute/useCompareSession.ts
• web/packages/studio/src/routes/IntakeTraceDetailRoute/index.tsx
• web/packages/studio/src/routes/constants.ts
• web/packages/studio/src/routes/utils.ts
• web/packages/studio/src/util/intakeTelemetry.ts

✅ Files skipped from review due to trivial changes (3)

• web/packages/studio/src/routes/utils.ts
• web/packages/studio/src/components/IntakeDetail/IntakeComponents/spanKeyValues.test.tsx
• web/packages/studio/src/routes/constants.ts

🚧 Files skipped from review as they are similar to previous changes (9)

• web/packages/studio/src/routes/ExperimentTraceDetailRoute/CompareExperimentSelect.tsx
• web/packages/studio/src/routes/IntakeTraceDetailRoute/index.tsx
• web/packages/studio/src/routes/ExperimentTraceDetailRoute/useCompareSession.ts
• web/packages/studio/src/routes/ExperimentTraceDetailRoute/useCompareSession.test.ts
• web/packages/studio/src/routes/ExperimentTraceDetailRoute/index.test.tsx
• web/packages/studio/src/routes/ExperimentTraceDetailRoute/ExperimentTraceCompare.tsx
• web/packages/studio/src/components/IntakeDetail/TraceDetailView.tsx
• web/packages/studio/src/components/IntakeDetail/TraceSpanAccordions.tsx
• web/packages/studio/src/routes/ExperimentTraceDetailRoute/index.tsx

[walston]

quick sync with @rrhyne has some UI and UX issues:

1. The headers are wrong: use proper headings.
2. Column layout doesn't have room for "Tree" view. switch to List view and hide/disable the toggle
3. Breadcrumbs should render "[Experiment Name] > Compare > [Experiment Name]+"
4. Compare To... dropdown should be in the page header (See figma).
5. Compare To... dropdown should be populated by experiment names (it might be already, doublecheck).
6. Columns need a "close" button
7. Support 3-column layout
   • when closing 3rd column, switch to 2-column.
   • when closing last remaining column switch to an empty state view with "Compare To..." drop down

[shanaiabuggy]

🚀