perf(analytics): materialize git observations as sorted table

[aleksdotbar]

git_metric_observations ran its full pipeline (FINAL dedup, joins, ten measure branches) on every metric query, though silver inputs only change when a connector sync runs dbt. Identical results recomputed per query, multiplied by request concurrency, against a fixed ClickHouse memory budget — the source of the metric-results OOM failures.

Materialize it as a MergeTree table instead:

• Pipeline runs once per dbt build — the only time results can change. Freshness unchanged.
• Metric queries become flat scans; the ordering key (source_key, measure_key, entity_id, metric_date) mirrors the runtime filter shape, so single-measure queries read index-pruned ranges.
• Runtime untouched: same relation name and columns; the schema probe sees a table.
• Rebuild rides the existing sync-triggered dbt selection (tag:gitlab+); dbt handles the view-to-table transition on the next run.

Result parity verified (row-level checksum); model dbt tests pass.

Refs #1733.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Performance Improvements
  • Git metric observations are now stored in an optimized, sorted table.
  • Metric queries can benefit from faster, index-pruned reads, particularly for single-measure lookups.
  • Observation processing is performed once during each data build rather than on every metric query.

[coderabbitai]

📝 Walkthrough

Walkthrough

The git metric observations dbt model changes from a view to an ordered MergeTree table. Its documentation now describes dbt-build execution, index-pruned reads, retroactive changes, and revised deduplication memory behavior.

Changes

Git metric observation storage

Layer / File(s)	Summary
Ordered MergeTree model src/ingestion/gold/git_metric_observations.sql	The model uses MergeTree table materialization with ordering by source, measure, entity, and metric date; comments document the updated build, read, and deduplication behavior.

Estimated code review effort: 2 (Simple) | ~5 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: converting git observations into a sorted materialized table for performance.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch perf/git-observations-materialize

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (2)

src/ingestion/gold/git_metric_observations.sql (2)

2-4: 🩺 Stability & Availability | 🔵 Trivial

Consider adding a PARTITION BY expression for data lifecycle management.

No partition_by is configured, so all data resides in a single partition. While metric_date in order_by enables granule-level index pruning for date-range queries, a partition expression (e.g., toYYYYMM(metric_date)) would also enable partition-level pruning and efficient partition drops for data retention. This is especially relevant now that the table is materialized and accumulates full-history data per build.

This can be deferred if data volume is manageable or retention is handled externally.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/ingestion/gold/git_metric_observations.sql` around lines 2 - 4, Consider
adding a partition expression to the materialized table configuration, using the
existing metric_date column (for example, monthly partitioning via
toYYYYMM(metric_date)) alongside order_by. If retention or data volume does not
require partition management, document or preserve the decision to defer this
change.

---

2-4: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Move tenant_id ahead of source_key in order_by — tenant_id is the tenant-isolation field, while source_key is fixed to 'git' and adds no selectivity. Leading with tenant_id gives tenant-scoped reads better pruning.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/ingestion/gold/git_metric_observations.sql` around lines 2 - 4, The
order_by configuration in the git metric observations table currently leads with
the non-selective source_key. Update it to place tenant_id before source_key,
preserving the remaining columns and their order to improve tenant-scoped read
pruning.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/ingestion/gold/git_metric_observations.sql`:
- Around line 2-4: Consider adding a partition expression to the materialized
table configuration, using the existing metric_date column (for example, monthly
partitioning via toYYYYMM(metric_date)) alongside order_by. If retention or data
volume does not require partition management, document or preserve the decision
to defer this change.
- Around line 2-4: The order_by configuration in the git metric observations
table currently leads with the non-selective source_key. Update it to place
tenant_id before source_key, preserving the remaining columns and their order to
improve tenant-scoped read pruning.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 46cedcec-255f-48a6-807c-32f869561ad8

📥 Commits

Reviewing files that changed from the base of the PR and between a905112 and c751b92.

📒 Files selected for processing (1)

• src/ingestion/gold/git_metric_observations.sql