seed: build silver+gold via real deploy scripts; fix gold data conformance

[ktursunov]

What

Rebuilds the sample-data seed so table + gold-layer setup uses the same mechanism as a real deployment instead of a Python reimplementation that skipped the dbt-owned gold layer.

silver.py now runs the exact scripts the k8s clickhouse-migrate Hook Job runs:

1. create-bronze-placeholders.sh — creates the silver tables
2. seed generators — insert synthetic rows
3. apply-ch-migrations.sh — migrations (gold views) + staging repair + dbt run --select tag:gold
4. refresh the task refreshable MVs

Seeding sits between steps 1 and 3 so the one materialized='table' gold model (git_metric_observations) is built over real seeded rows rather than empty placeholders. Re-running the placeholder script inside step 3 is a no-op (IF NOT EXISTS), and the dbt drop_silver_placeholders_at_start hook can't fire on a tag:gold-only run, so seeded rows survive.

Why

The old seed reimplemented DDL/migrations in Python and never built the 3 dbt-owned gold observation models — so insight.ai_metric_observations, git_metric_observations, and metric_entity_cohorts_current didn't exist on a seeded stack, and any metric routed through the unified observation runtime was blank.

Changes

Mechanism

• Dockerfile: pin python:3.12-slim (matches the toolbox dbt image), add bash; add dbt-clickhouse to requirements.txt.
• docker-compose.yml: mount the full ./src/ingestion at /ingestion (dbt needs the whole project), set CLICKHOUSE_DATABASE, redirect dbt target//logs/ to /tmp.
• clickhouse-user-defaults.xml: enable allow_experimental_refreshable_materialized_view — the migrations SET it per-statement but the deploy runner (run_ch) is stateless; production CH allows it server-side.

Data conformance (so the gold relations actually populate)

• git.py: PR state → MERGED/OPEN; add tenant_id + data_source; commit data_source.
• ai.py: assistant surface → chat; add conversation_count + tool/surface display labels.
• people.py: seed workspace_id (was collapsing metric_entity_cohorts_current to zero rows).
• task.py: seed class_task_statuses dimension + carry status_id on status events (fixes jira_closed_tasks = 0).
• base.py: deterministic_int for Int64 id columns (pr_id).

Verification

Ran end-to-end against a local compose stack (CH 24.8):

• dbt gold build Completed successfully, 5 gold tests PASS.
• metric_entity_cohorts_current: 25 rows / 5 cohorts (was 0).
• git_metric_observations: pr_merged 270, pr_cycle_hours 9728 (were 0/NULL); source dimension github/GitHub.
• ai_metric_observations: dev_conversations 4594, chat_assistant_conversations 2110 (were 0); labels populated.
• jira_closed_tasks: 212 rows / 236 tasks closed (was 0).
• tenant_id consistent across all three relations; observation↔cohort join resolves.

Notes / out of scope

• class_git_pull_requests_commits link table left unseeded — it feeds a PR-author-email fallback that never fires (PR author_email is always seeded).
• Cohorts keyed by department name (correct for the analytics layer's org-unit representation).

Summary by CodeRabbit

• New Features

  • Added a urls command to display local service addresses, including optional frontend, APIs, databases, and supporting services.
  • Expanded seeded AI usage data with clearer tool and surface labels.
  • Added complete task status data for more accurate task histories and reporting.

• Bug Fixes

  • Improved sample data accuracy for people, Git activity, pull requests, and tenant associations.
  • Improved deployment and seeding reliability, including materialized view creation and refresh behavior.
  • Updated development container setup for more consistent script execution.

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR aligns local seed execution with ingestion deployment scripts, expands seeded silver-table data, adds task status dimensions, updates container and ClickHouse defaults, and introduces a dev-compose.sh urls command for reporting service endpoints.

Changes

Seed pipeline

Layer / File(s)	Summary
Runtime and deployment wiring deploy/compose/clickhouse-user-defaults.xml, deploy/seed/Dockerfile, deploy/seed/requirements.txt, docker-compose.yml	ClickHouse refreshable materialized views are enabled by default; the seed image uses Python 3.12 with bash; dbt and ingestion mount settings are added.
Seeded silver data shapes deploy/seed/generators/*	AI usage labels, Git source and PR fields, tenant-linked people rows, deterministic integer IDs, and task status dimensions are added or updated.
Script-driven seed orchestration deploy/seed/silver.py	Seeding now runs bronze placeholder creation, row generation, ClickHouse migrations, gold builds, and dependent task materialized-view refreshes in deployment order.

Compose service URL reporting

Layer / File(s)	Summary
Service URL command dev-compose.sh	Adds the urls subcommand, environment-file handling, frontend detection, and conditional service endpoint output.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SeedRunner
  participant CreateBronzePlaceholders
  participant SilverGenerators
  participant ApplyCHMigrations
  participant TaskMaterializedViews
  SeedRunner->>CreateBronzePlaceholders: run create-bronze-placeholders.sh
  CreateBronzePlaceholders-->>SeedRunner: create placeholders
  SeedRunner->>SilverGenerators: seed silver rows
  SilverGenerators-->>SeedRunner: populate silver tables
  SeedRunner->>ApplyCHMigrations: run apply-ch-migrations.sh
  ApplyCHMigrations-->>SeedRunner: build gold models and views
  SeedRunner->>TaskMaterializedViews: refresh dependent views

Loading

Possibly related PRs

• constructorfabric/insight#1468: Updates the bronze placeholder script used by the revised seed flow.
• constructorfabric/insight#1480: Adds the ClickHouse migration script executed by the revised orchestration.
• constructorfabric/insight#1732: Extends task status-category handling used by the new task status seeding.

Suggested reviewers: cyberantonz, mitasovr

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 52.17% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: rebuilding seeding to use real deploy scripts and fixing gold-layer data conformance.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

deploy/seed/generators/ai.py (1)

72-99: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Leave conversation_count NULL for Cursor rows deploy/seed/generators/ai.py:72-99
Cursor maps this column to NULL in Silver, and dev_conversations sums it directly downstream. Set conversation_count only for tools that report conversations; Cursor should write None here.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/seed/generators/ai.py` around lines 72 - 99, Update the row
construction in the _DEV_TOOLS loop so the conversation_count value is None when
tool is Cursor, while retaining the existing sessions-based value for tools that
report conversations. Keep all other generated metrics unchanged.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/seed/silver.py`:
- Around line 51-66: Correct the host-run fallback in _ingestion_scripts_dir()
by resolving from the repository root rather than the deploy directory, so it
returns <repo>/src/ingestion/scripts. Keep the /ingestion/scripts
mounted-directory preference unchanged.

In `@dev-compose.sh`:
- Around line 403-412: Update the env_file assignment in the surrounding command
function to propagate a nonzero result from resolve_env_file immediately,
preserving its original error instead of continuing with an empty path. Remove
the frontend_up variable and the FRONTEND_MODE == "none" conditional, and call
report_service_urls using its default behavior.

---

Outside diff comments:
In `@deploy/seed/generators/ai.py`:
- Around line 72-99: Update the row construction in the _DEV_TOOLS loop so the
conversation_count value is None when tool is Cursor, while retaining the
existing sessions-based value for tools that report conversations. Keep all
other generated metrics unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 2f4f4e05-88eb-47f4-ab63-afb98f440b4e

📥 Commits

Reviewing files that changed from the base of the PR and between bcd0875 and 4028e40.

📒 Files selected for processing (11)

• deploy/compose/clickhouse-user-defaults.xml
• deploy/seed/Dockerfile
• deploy/seed/generators/ai.py
• deploy/seed/generators/base.py
• deploy/seed/generators/git.py
• deploy/seed/generators/people.py
• deploy/seed/generators/task.py
• deploy/seed/requirements.txt
• deploy/seed/silver.py
• dev-compose.sh
• docker-compose.yml