Skip to content

fix(ai-cost): heal seat_status onto the AI dev-usage contract on upgrade - #2629

Merged
Gregory91G merged 1 commit into
constructorfabric:mainfrom
Gregory91G:fix/ai-dev-usage-seat-status-upgrade
Aug 18, 2026
Merged

fix(ai-cost): heal seat_status onto the AI dev-usage contract on upgrade#2629
Gregory91G merged 1 commit into
constructorfabric:mainfrom
Gregory91G:fix/ai-dev-usage-seat-status-upgrade

Conversation

@Gregory91G

@Gregory91G Gregory91G commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Closes #2617.

Two defects from #2607: one broke helm upgrade, the other breaks a connector sync.

The column never reached an existing installation

seat_status entered the class contract in three places — the staging projections, the DDL snapshot, the gold model that reads it — and in no migration. So only a fresh cluster ever had it: the snapshot is CREATE TABLE IF NOT EXISTS, and dbt appends the column (on_schema_change=append_new_columns) only when the silver model runs, which is connector-sync time. The migrate hook builds gold in between, ai_metric_evidence reads the column, and the hook fails the release.

A migration adds it to silver.class_ai_dev_usage, AFTER _version — the position it holds in the staging projections and in the snapshot, and the one the positional incremental insert requires. MODIFY follows the ADD so an instance where an out-of-band ALTER placed it elsewhere converges too, which is the state at least one environment is in.

A class heals one contributor at a time, and the union does not tolerate that

The silver class unions its staging members with a positional SELECT * UNION ALL, and each member gains the column on its own connector's sync. Between the first such sync and the last, the union has members of different widths:

DB::Exception: UNION different number of columns in queries. (TYPE_MISMATCH)

So a migration on silver alone would have traded the deploy failure for a sync failure on any installation with more than one AI vendor. The staging heal already in the hook now carries the same two ALTERs, converging all four contributors in one deploy. Staging is healed there rather than by a migration because those tables are absent from the DDL snapshot and exist only after their connector's first sync, so the statements need the ch_table_exists guard.

Scope

An earlier revision also had the hook build the silver classes before gold, so that a release adding a silver column could not fail on its own gold model. Dropped on review: the migration and the heal are sufficient (re-verified below with the selector unchanged), and the hook change bought little — the column enters the contract through staging, which heals only on its own sync, so the ALTER has to be written by hand either way. It cost a three-term selector, 48 incremental models on the deploy path, and a wider blast radius for a mechanism drop_silver_placeholders_at_start already records a plan to retire.

No DDL snapshot regeneration: silver.sql already carries seat_status from #2607, and the AI staging tables are deliberately absent from the snapshot. Gold needs nothing — every gold model is materialized: table, rebuilt whole on each run.

Test plan

Against a throwaway clickhouse/clickhouse-server:25.7 and the deployed toolbox image (insight-toolbox:2026.08.18.02.16-0297cb1) with this tree mounted, running the real apply-ch-migrations.sh end to end — the same script the Hook Job runs, with its selector untouched:

  • Fresh cluster — empty ClickHouse: exit 0. Nothing to heal; the snapshot creates the table with the column
  • Pre-change cluster, code from main — a warm install rolled back to the pre-fix(ai-cost): stop the Claude Team path losing data it already has #2607 shape (silver and both staging tables at 27 columns, rows in silver): exit 1, PASS=18 ERROR=1 SKIP=2, code: 47 … Unknown expression or function identifier seat_status … ai_metric_evidence. The reported failure, reproduced
  • Pre-change cluster, this branch — same state: exit 0; silver and both staging tables at 28 columns, position 28, Nullable(String); the rows that were in silver survived; ai_metric_evidence rebuilt
  • Staging desync — silver and one staging member healed, the other not, code from main: the silver build fails with UNION different number of columns (TYPE_MISMATCH)
  • Column already added out of band — silver healed, staging not (the state one environment is in): exit 0, the heal converges both staging tables, and a subsequent run of the silver model — the connector sync that used to fail — passes
  • Idempotence — re-run over the converged state: exit 0
  • bash -n, git diff --check

Not verified: a real helm upgrade, which needs a release through gitops.

@Gregory91G
Gregory91G requested a review from a team as a code owner August 18, 2026 08:44
@Gregory91G Gregory91G added bug Something isn't working component:ingestion dbt / approved connector scope labels Aug 18, 2026
@Gregory91G Gregory91G self-assigned this Aug 18, 2026
@Gregory91G Gregory91G added bug Something isn't working component:ingestion dbt / approved connector scope labels Aug 18, 2026
@coderabbitai

coderabbitai Bot commented Aug 18, 2026

Copy link
Copy Markdown

Important

Review available on request

  • 🔍 Trigger review

Reviews should be triggered manually for repositories with fewer than 10 stars. Select Trigger review above or comment @coderabbitai review to review the latest changes. For a full review, comment @coderabbitai full review.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: bb6ae8ff-6935-40ba-9b89-43d249e470bb


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@aleksdotbar

Copy link
Copy Markdown
Contributor

I don't think we need to build the silver models
Building gold during deploy is already not a good solution on its own
Building full silver will make deploy even heavier
I think healing migrations should be enough for now

@Gregory91G

constructorfabric#2607 added the column to the class contract but to no migration, so only a
fresh cluster got it: the DDL snapshot is CREATE TABLE IF NOT EXISTS, and dbt
appends it (on_schema_change=append_new_columns) only when the silver model
runs — which happens at connector-sync time, after the deploy. The migrate
hook builds gold in between, and ai_metric_evidence reads seat_status, so
upgrading an existing installation failed the hook and rolled the release back.

The staging heal is not optional. A class unions its staging members
positionally, and each member gains the column on its own connector's sync, so
an install with two AI vendors gets `UNION different number of columns` on the
first sync after the upgrade — a sync failure traded for the deploy failure.
Healing all four contributors in one deploy closes that window.

AFTER _version, matching the position the column holds in the staging
projections and in the DDL snapshot; MODIFY converges an instance where an
out-of-band ALTER placed it elsewhere.

Signed-off-by: Gregory Gogin <grigoriy.gogin@constructor.tech>
@Gregory91G
Gregory91G force-pushed the fix/ai-dev-usage-seat-status-upgrade branch from 69c105f to 79afca4 Compare August 18, 2026 10:39
@Gregory91G
Gregory91G enabled auto-merge August 18, 2026 12:00
@Gregory91G
Gregory91G force-pushed the fix/ai-dev-usage-seat-status-upgrade branch from 79afca4 to 0819878 Compare August 18, 2026 15:15
@Gregory91G Gregory91G changed the title fix(ingestion): heal seat_status on upgrade and build silver before gold fix(ai-cost): heal seat_status onto the AI dev-usage contract on upgrade Aug 18, 2026
@Gregory91G

Copy link
Copy Markdown
Contributor Author

I don't think we need to build the silver models Building gold during deploy is already not a good solution on its own Building full silver will make deploy even heavier I think healing migrations should be enough for now

@Gregory91G

fixed

@Gregory91G
Gregory91G added this pull request to the merge queue Aug 18, 2026
Merged via the queue into constructorfabric:main with commit 55c816a Aug 18, 2026
54 checks passed
@Gregory91G
Gregory91G deleted the fix/ai-dev-usage-seat-status-upgrade branch August 18, 2026 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working component:ingestion dbt / approved connector scope

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upgrading an existing installation fails at the ClickHouse migrate hook, and the AI usage metrics stop refreshing

2 participants