feat: migrate git metrics to unified metric system (backend)

[aleksdotbar]

Migrate git metric family to unified metrics, served through POST /v1/metric-results. Adds the git runtime vocab, a dbt gold observation model, and registry seeds.

Closes #1694. Part of #1561. Builds on #1681.

Additive, safe ahead of the frontend: legacy git gold views (git_bullet_rows, ic_kpis, ic_chart_loc) and catalog seeds untouched. Legacy path retires after frontend cutover (separate task).

Changes

• ObservationSource enum → ObservationRelation newtype. Relation name is data in metric_sources.source_ref, validated against the <family>_metric_observations shape on load, runtime-gated by the schema probe. New source = dbt model + registry rows, no code. No DB migration.
• median computation — quantileExact(0.5) over per-event values (new event_measure dbt macro). Single + batched paths. Peer view swaps only the per-entity aggregate; cohort scoping, disclosure floor, honest-null unchanged. Never zero-filled.
• histogram view — ten server-owned fixed-width bins over an entity's exact [min, max], last bin closed, equal values collapse, event-less entities empty. Deterministic, not the adaptive histogram(). Median metrics only. SQL owns membership, builder owns edges.
• Forward-only migration widens the computation_type enum, re-adds CHECK same name (idempotent DROP CONSTRAINT IF EXISTS).
• insight.git_metric_observations gold view over the class contracts (not legacy fct_/mtr_) + shared file-classification macro. Ten git.* definitions.

Metrics (source git)

metric	computation	notes
git.commits	sum	distinct non-merge commit hashes
git.code_lines	sum	lines added to code-category files
git.lines_added	sum	all lines added, split by file category
git.prs_created	sum
git.prs_merged	sum	merge-dated throughput
git.merge_rate	ratio	share of PRs created in the period that merged (created-cohort)
git.commits_per_active_day	ratio
git.commit_size	median	typical commit diff size
git.pr_size	median	typical PR diff size
git.pr_cycle_time_h	median	typical open→merge hours

Dimensions: source (github/gitlab/bitbucket_cloud), category (code/test/config/docs). File class computed in the gold view via a shared macro → taxonomy changes apply retroactively.

PR author attribution

PR rows often lack an author email. Two tiers, most-precise first:

1. Direct author_email (e.g. GitLab, from the users stream).
2. Per-PR commit crosswalk — dominant author email among the PR's linked commits (tie → unresolved). commit_hash join, account-agnostic, carries the bulk.

Unresolved excluded, never misattributed. Account-handle-keyed resolution is a follow-up.

Deltas vs legacy

• merge_rate created-cohort: numerator and denominator both created-dated → bounded to 100% (legacy mixed a merge-dated numerator with a create-dated denominator, could exceed 100%).
• commit_size median, not the legacy mean — outlier-robust, symmetric with pr_size.
• cycle time / PR size medians (legacy label said "average"; SQL already computed a median).
• Merge commits excluded from commit measures and PR identity election.
• docs split out of what legacy counted as code; legacy spec label corrected to test (matches its patterns).

Verified

• Adversarial review, multiple passes; confirmed findings fixed (migration idempotency, merge-rate cohort, entity-id parity, distinct-commit vote, merge-commit exclusion).
• cargo test + clippy clean; migration CHECK-name pinned by test.
• dbt build green incl. integrity tests (entity-id shape, dimension vocabs, non-negativity).
• End-to-end vs a local warehouse: all metrics × all views, error cases, legacy-vs-unified parity.

Follow-ups

• Frontend git-group cutover.
• Retire legacy git gold views + catalog seeds.
• Account-handle-keyed PR identity resolution.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added median-based metric calculations, including commit size, pull request size, and cycle time metrics.
  • Added histogram views for median metrics with 10 deterministic bins per entity.
  • Added unified Git metrics covering commits, pull requests, file changes, categories, and data sources.
  • Added clearer Git labels for code, tests, configuration, documentation, and supported providers.

• Bug Fixes

  • Improved validation for metric sources, unsupported histogram requests, and oversized result requests.
  • Median metrics with no observations now return an empty value instead of zero.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR adds median computation and histogram result support, replaces enum-based observation sources with validated relation names, and introduces dbt-based git observation models, metric registrations, integrity tests, and silver placeholder schema updates.

Changes

Median & Histogram Backend

Layer / File(s)	Summary
Metric contracts and observation relations src/backend/services/analytics/src/domain/metric_definitions/{definition.rs,repository.rs,validator.rs,mod.rs}	Adds median computation contracts and replaces ObservationSource with validated ObservationRelation handling.
Builtin metrics and computation migration src/backend/services/analytics/src/domain/metric_definitions/{builtin.rs,seeds.rs}, src/backend/services/analytics/src/migration/*	Registers git and median metrics, persists relation names, and widens the database computation enum.
Median SQL and histogram execution src/backend/services/analytics/src/domain/metric_results/{compiler.rs,batch.rs,builder.rs}, src/backend/services/analytics/src/api/metric_results.rs	Compiles median and deterministic histogram queries, plans histogram execution, and builds result payloads.
Histogram DTOs and validation src/backend/services/analytics/src/domain/metric_results/{dto.rs,view.rs,validation.rs}	Adds histogram request/response types, restricts histogram views to median metrics, and applies fixed-bin row limits.
Metric design specification docs/domain/metrics/specs/DESIGN.md	Documents relation validation, median semantics, histogram binning, measure grain, and authoring rules.

Git Metrics Ingestion Pipeline

Layer / File(s)	Summary
Git observation macros and gold model src/ingestion/dbt/macros/*, src/ingestion/gold/git_metric_observations.sql	Adds git taxonomy and event-grain macros and produces unified commit, file-change, and pull-request observations.
Git observation contract and integrity tests src/ingestion/gold/schema.yml, src/ingestion/dbt/tests/gold/*	Defines the git observation schema and validates dimensions, entity identifiers, and non-negative values.
Silver placeholders and ingestion documentation src/ingestion/scripts/create-bronze-placeholders.sh, src/ingestion/silver/git/README.md	Updates placeholder schemas and documents the unified metrics ingestion path.

Estimated code review effort: 4 (Complex) | ~75 minutes

Possibly related issues

• INGEST 1: Unified metric system (Metrics catalog) #1561: Covers the broader unified metric system and git metric migration implemented by this PR.
• feat: migrate wiki metrics to unified metric system (backend) #1700: Covers overlapping unified-metrics gold observation models and builtin metric registration patterns.

Possibly related PRs

• constructorfabric/insight#1656: Touches the same metric definition, compiler, builder, and validation paths extended here.
• constructorfabric/insight#1701: Overlaps with the metric-results batching and planned execution paths used for histogram dispatch.

Suggested reviewers: ktursunov, cyberantonz, mitasovr

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 30.30% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately summarizes the main change: migrating git metrics to the unified metric system.
Linked Issues check	✅ Passed	The PR adds the git gold observation model, registry definitions, dimensions/cohorts mapping, and unit/dbt tests required by #1694.
Out of Scope Changes check	✅ Passed	The changes stay within the backend git-metrics migration and supporting docs/tests, with no clear unrelated scope added.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/unified-metrics-git

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 5

🧹 Nitpick comments (2)

src/ingestion/silver/git/git_actor_emails.sql (1)

1-19: 🚀 Performance & Scalability | 🔵 Trivial

Full-history view recomputed on every read of the identity bridge.

This view joins full commit/PR history plus identity_inputs FINAL on every read, and is itself joined by git_metric_observations for every query that resolves PR authors. The retroactivity rationale is sound, but as commit history grows this could become a meaningful cost on the metrics read path. Worth keeping an eye on materialization strategy (e.g. periodic snapshot table + incremental refresh) if query latency becomes an issue.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/ingestion/silver/git/git_actor_emails.sql` around lines 1 - 19, The
`git_actor_emails` view is being recomputed from full history on every read,
which can become expensive as commit volume grows. Review the materialization
approach in `git_actor_emails` and its downstream use in
`git_metric_observations`, and consider a snapshot or incremental-refresh
pattern instead of a pure view if read latency becomes a problem.

src/ingestion/gold/git_metric_observations.sql (1)

101-141: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Nested aggregate-in-window is correct but non-obvious; consider using the alias.

max(uniqExact(commits.commit_hash)) OVER (...) (lines 122-125) re-invokes the exact aggregate expression already computed as email_count (line 121) instead of referencing that alias inside the window function. This likely works in ClickHouse (window functions apply over the grouped result and the identical aggregate expression is resolved consistently), but it's an unusual pattern worth double-checking under dbt build since window+GROUP BY interactions have had ClickHouse version-specific quirks. Using the email_count alias directly would be clearer and avoid any ambiguity about whether the aggregate is recomputed.

♻️ Suggested clarity fix

-            max(uniqExact(commits.commit_hash)) OVER (
+            max(email_count) OVER (
                 PARTITION BY links.tenant_id, links.source_id,
                              links.project_key, links.repo_slug, links.pr_id
             ) AS max_count

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/ingestion/gold/git_metric_observations.sql` around lines 101 - 141, The
`pr_commit_emails` CTE is reusing the full `uniqExact(commits.commit_hash)`
expression inside the window instead of the already-computed `email_count`
alias, which is harder to read and may be brittle in ClickHouse. Update the
`max_count` calculation in `pr_commit_emails` to reference the grouped
`email_count` result directly (or otherwise factor the aggregate once), keeping
the same `PARTITION BY` over `links.tenant_id`, `links.source_id`,
`links.project_key`, `links.repo_slug`, and `links.pr_id`.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/domain/metrics/specs/DESIGN.md`:
- Around line 412-415: Case 3 still refers to the removed ObservationSource enum
and its from_ref/table_ref mapping, so update the instructions to match the
current ObservationRelation/source_ref model. In the DESIGN.md Case 3 section,
remove the step that tells developers to add an enum variant and table-name
mapping in definition.rs, and renumber the remaining steps so the guidance
reflects that adding a source is now just dbt plus registry seed changes. Also
align the referenced db_strings_round_trip test description with
ObservationRelation round-trips instead of ObservationSource pairs.

In `@src/ingestion/gold/git_metric_observations.sql`:
- Around line 8-14: The header contract in git_metric_observations.sql currently
says no vendor-specific label mappings may appear, but commits_source and
pull_requests_source both hardcode the same data_source-to-product label mapping
in two places. Fix this by either updating the header to match the implemented
behavior or, preferably, extracting the mapping into a shared macro like
git_file_category_label so the logic is centralized and the contract remains
accurate.
- Around line 47-77: The commits_source CTE currently uses LIMIT 1 BY without a
deterministic ORDER BY, so the surviving row for forked commits can vary across
runs and change the project_key/repo_slug used later in the file_changes_source
join. Update commits_source to sort deterministically on the identifying commit
fields before the LIMIT 1 BY clause, using the same symbols already present in
the CTE (such as tenant_id, data_source, commit_hash, project_key, and
repo_slug) so the chosen row is stable.

In `@src/ingestion/scripts/create-bronze-placeholders.sh`:
- Around line 666-689: The identity.identity_inputs placeholder is created in
create-bronze-placeholders.sh but is not covered by
drop_silver_placeholders_at_start, so it can linger and block the real model.
Update the cleanup path to also remove this placeholder, using the same
identity.identity_inputs symbol and the INSIGHT_PLACEHOLDER_v1 table marker,
either by extending the drop logic to include identity-tagged placeholders or by
adding an explicit drop for this table before the real build runs.

In `@src/ingestion/silver/git/git_actor_emails.sql`:
- Around line 131-161: The outer filter in git_actor_emails.sql is accidentally
coupled to the SELECT alias assumeNotNull(email), so the null/empty check may
resolve against the aliased value instead of the raw nullable column. Update the
query around the outer SELECT/WHERE to isolate the filter from the alias by
using a separate subquery or by renaming the projected email expression, and
make sure the WHERE condition in the final projection applies to the underlying
nullable email value from the inner query.

---

Nitpick comments:
In `@src/ingestion/gold/git_metric_observations.sql`:
- Around line 101-141: The `pr_commit_emails` CTE is reusing the full
`uniqExact(commits.commit_hash)` expression inside the window instead of the
already-computed `email_count` alias, which is harder to read and may be brittle
in ClickHouse. Update the `max_count` calculation in `pr_commit_emails` to
reference the grouped `email_count` result directly (or otherwise factor the
aggregate once), keeping the same `PARTITION BY` over `links.tenant_id`,
`links.source_id`, `links.project_key`, `links.repo_slug`, and `links.pr_id`.

In `@src/ingestion/silver/git/git_actor_emails.sql`:
- Around line 1-19: The `git_actor_emails` view is being recomputed from full
history on every read, which can become expensive as commit volume grows. Review
the materialization approach in `git_actor_emails` and its downstream use in
`git_metric_observations`, and consider a snapshot or incremental-refresh
pattern instead of a pure view if read latency becomes a problem.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 1120bfc2-43d9-4fc1-8028-28e7d08f0dad

📥 Commits

Reviewing files that changed from the base of the PR and between 0cf4fef and c75208d.

📒 Files selected for processing (28)

• docs/domain/metrics/specs/DESIGN.md
• src/backend/services/analytics/src/api/metric_results.rs
• src/backend/services/analytics/src/domain/metric_definitions/builtin.rs
• src/backend/services/analytics/src/domain/metric_definitions/definition.rs
• src/backend/services/analytics/src/domain/metric_definitions/mod.rs
• src/backend/services/analytics/src/domain/metric_definitions/repository.rs
• src/backend/services/analytics/src/domain/metric_definitions/seeds.rs
• src/backend/services/analytics/src/domain/metric_definitions/validator.rs
• src/backend/services/analytics/src/domain/metric_results/builder.rs
• src/backend/services/analytics/src/domain/metric_results/compiler.rs
• src/backend/services/analytics/src/domain/metric_results/dto.rs
• src/backend/services/analytics/src/domain/metric_results/mod.rs
• src/backend/services/analytics/src/domain/metric_results/validation.rs
• src/backend/services/analytics/src/domain/metric_results/view.rs
• src/backend/services/analytics/src/migration/m20260709_000001_metric_median_computation.rs
• src/backend/services/analytics/src/migration/mod.rs
• src/ingestion/dbt/macros/git_file_category.sql
• src/ingestion/dbt/macros/metric_observation_measures.sql
• src/ingestion/dbt/tests/git/assert_git_actor_emails_unique.sql
• src/ingestion/dbt/tests/gold/assert_git_observations_dimension_values.sql
• src/ingestion/dbt/tests/gold/assert_git_observations_entity_id_shape.sql
• src/ingestion/dbt/tests/gold/assert_git_observations_nonnegative.sql
• src/ingestion/gold/git_metric_observations.sql
• src/ingestion/gold/schema.yml
• src/ingestion/scripts/create-bronze-placeholders.sh
• src/ingestion/silver/git/README.md
• src/ingestion/silver/git/git_actor_emails.sql
• src/ingestion/silver/git/schema.yml

[coderabbitai]

🧹 Nitpick comments (2)

src/ingestion/dbt/macros/git_file_category.sql (1)

29-39: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Minor: package-lock\.json lacks end-anchor in config regex.

In the git_file_category config branch (line 14), package-lock\.json is the only pattern without a $ end-anchor, so a hypothetical package-lock.json.bak would classify as config. The sibling yarn.lock is covered by \.lock$ so it's fine, but for consistency consider anchoring: package-lock\\.json$.

This is a nitpick with negligible real-world impact — flagging only for regex consistency.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/ingestion/dbt/macros/git_file_category.sql` around lines 29 - 39, Anchor
the package-lock pattern in the git_file_category config branch by changing
package-lock\.json to package-lock\.json$, keeping the existing classification
behavior while preventing suffix matches such as package-lock.json.bak.

src/backend/services/analytics/src/domain/metric_results/batch.rs (1)

108-115: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add regression coverage for unbatched histogram planning.

Add a test asserting that multiple histogram views each produce a distinct PlannedQuery::Single with UnbatchedView::Histogram. This protects the distinct execution contract established in validation.rs:41-54 and consumed by builder.rs:158-215.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/backend/services/analytics/src/domain/metric_results/batch.rs` around
lines 108 - 115, Add regression coverage for the unbatched histogram branch in
the metric planning test suite: construct a request containing multiple
histogram views, invoke the relevant planning function, and assert each produces
a distinct PlannedQuery::Single with the correct metric_index, view_index, and
UnbatchedView::Histogram.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/backend/services/analytics/src/domain/metric_results/batch.rs`:
- Around line 108-115: Add regression coverage for the unbatched histogram
branch in the metric planning test suite: construct a request containing
multiple histogram views, invoke the relevant planning function, and assert each
produces a distinct PlannedQuery::Single with the correct metric_index,
view_index, and UnbatchedView::Histogram.

In `@src/ingestion/dbt/macros/git_file_category.sql`:
- Around line 29-39: Anchor the package-lock pattern in the git_file_category
config branch by changing package-lock\.json to package-lock\.json$, keeping the
existing classification behavior while preventing suffix matches such as
package-lock.json.bak.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 366d1a9d-9e22-4c0d-8b11-867ebfd10c6d

📥 Commits

Reviewing files that changed from the base of the PR and between ed3489d and 83d52bd.

📒 Files selected for processing (11)

• docs/domain/metrics/specs/DESIGN.md
• src/backend/services/analytics/src/api/metric_results.rs
• src/backend/services/analytics/src/domain/metric_definitions/definition.rs
• src/backend/services/analytics/src/domain/metric_results/batch.rs
• src/backend/services/analytics/src/domain/metric_results/builder.rs
• src/backend/services/analytics/src/domain/metric_results/compiler.rs
• src/backend/services/analytics/src/domain/metric_results/mod.rs
• src/backend/services/analytics/src/domain/metric_results/validation.rs
• src/ingestion/dbt/macros/git_file_category.sql
• src/ingestion/gold/git_metric_observations.sql
• src/ingestion/scripts/create-bronze-placeholders.sh

✅ Files skipped from review due to trivial changes (1)

• src/backend/services/analytics/src/domain/metric_results/mod.rs

🚧 Files skipped from review as they are similar to previous changes (8)

• src/backend/services/analytics/src/api/metric_results.rs
• src/ingestion/scripts/create-bronze-placeholders.sh
• docs/domain/metrics/specs/DESIGN.md
• src/backend/services/analytics/src/domain/metric_definitions/definition.rs
• src/ingestion/gold/git_metric_observations.sql
• src/backend/services/analytics/src/domain/metric_results/validation.rs
• src/backend/services/analytics/src/domain/metric_results/builder.rs
• src/backend/services/analytics/src/domain/metric_results/compiler.rs