fix(intake): performance improvements for trace list

[BrianNewsom]

Use a two phase query to hydrate the trace list view instead of the current single query approach, which requires reading all input/output data across spans.

Summary by CodeRabbit

• Bug Fixes

  • Improved trace listing pagination and multi-step loading to keep results consistent across modes.
  • Added reconciliation when hydration omits items, including warning logs and corrected total counts.
  • Refined preview payload handling to ensure proper truncation without impacting detailed views.
  • Empty preview pages now avoid unnecessary follow-up loading.

• Tests

  • Expanded assertions for preview and detailed query behavior, including aggregation expressions and payload formatting.
  • Added cases for empty pages and reconciliation when hydrated results miss requested refs.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2918b72a-a4a1-4d41-8ee5-d8a1fb846169

📥 Commits

Reviewing files that changed from the base of the PR and between 4b541a5 and 27ea1b3.

📒 Files selected for processing (2)

• services/intake/src/nmp/intake/repository/clickhouse/trace.py
• services/intake/tests/test_traces_clickhouse_repository.py

---

📝 Walkthrough

Walkthrough

Non-summary trace listing now retrieves page roots first, hydrates selected traces through page-scoped ClickHouse queries, reconciles missing hydrated rows, logs dropped references, and adjusts totals. Tests cover preview, detailed, empty-page, ordering, and query-shape behavior.

Changes

Trace page hydration

Layer / File(s)	Summary
Staged page retrieval and reconciliation services/intake/src/nmp/intake/repository/clickhouse/trace.py, services/intake/tests/test_traces_clickhouse_repository.py	Non-summary listing creates page references, skips hydration for empty pages, restores page order after hydration, logs missing references, and adjusts pagination totals.
Mode-specific root projections services/intake/src/nmp/intake/repository/clickhouse/trace.py, services/intake/tests/test_traces_clickhouse_repository.py	Root SELECT columns and mode-specific parameters are centralized; preview truncation and detailed span reconstruction are validated.
Page-scoped span aggregation services/intake/src/nmp/intake/repository/clickhouse/trace.py	Hydration composes page-root and aggregate subqueries, reconstructs span fields, counts spans, and selects unique model/provider attributes.

Sequence Diagram(s)

sequenceDiagram
  participant ClickHouseTraceRepository
  participant ClickHouseClient
  participant PageRootQuery
  ClickHouseTraceRepository->>ClickHouseClient: Execute page-root query
  ClickHouseClient->>PageRootQuery: Select page trace roots
  PageRootQuery-->>ClickHouseClient: Return page references
  ClickHouseClient-->>ClickHouseTraceRepository: Return page rows
  ClickHouseTraceRepository->>ClickHouseClient: Execute page-scoped hydration query
  ClickHouseClient-->>ClickHouseTraceRepository: Return hydrated rows
  ClickHouseTraceRepository->>ClickHouseTraceRepository: Reconcile order and dropped refs

Loading

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#725: Updates related preview payload handling in intake trace listing.
• NVIDIA-NeMo/nemo-platform#941: Changes the same trace repository and ClickHouse query layer.

Suggested labels: refactor

Suggested reviewers: shanaiabuggy, a2bondar, aahunt-nv

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title matches the main change: speeding up trace list retrieval in intake.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch brnewsom/fix-ch-traces-query

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@services/intake/src/nmp/intake/repository/clickhouse/trace.py`:
- Around line 135-165: Reconcile hydration results with the page references in
the trace listing flow: after the `traces.list.hydrate` query, detect page refs
that were not returned because concurrent filtering or deletion excluded them,
and explicitly track or log those dropped refs rather than silently shrinking
the page. Keep `total_results` and pagination behavior consistent with the
chosen reconciliation policy.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2fab7cf1-b5b9-4294-a510-d69eb6cb9940

📥 Commits

Reviewing files that changed from the base of the PR and between 9c36aac and 4b541a5.

📒 Files selected for processing (2)

• services/intake/src/nmp/intake/repository/clickhouse/trace.py
• services/intake/tests/test_traces_clickhouse_repository.py

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	28406/36247	78.4%	62.8%
Integration Tests	16988/34965	48.6%	21.1%