refactor(crm): static schemas + raw_data for salesforce/hubspot

.claude/skills/connector/workflows/create.md

@@ -676,13 +676,11 @@ cd src/ingestion/scripts/bootstrap-db
 ./generate-connectors-config.sh '<category>/<name>'
 ```
 
-**NEVER regenerate the whole file** (`./generate-connectors-config.sh` with no
-argument): it overwrites the four `env:` credential references
-(`HUBSPOT_ACCESS_TOKEN`, `SALESFORCE_CLIENT_ID`, `SALESFORCE_CLIENT_SECRET`,
-`SALESFORCE_INSTANCE_URL`) with fake `value:` entries.
+Regenerate only your own fragment: a whole-file regeneration rewrites every
+connector's entry and buries your change in unrelated churn.
 
-Fake credentials in your own fragment are fine — bootstrap only calls
-`discover`, which reads the static spec, not the live API.
+Fake credentials are fine — bootstrap only calls `discover`, which builds the
+catalog from the connector's static schemas, not from the live API.
 
 ### 2. Keep shared silver class column types identical
 

.github/workflows/connectors-ddl.yml

@@ -22,11 +22,9 @@ name: connectors-ddl
 # are each green apart — one adding a gold table, one regenerating the snapshot —
 # can still leave main drifted, and only a run on the merged tree sees that.
 #
-# PRs are gated on the branch living in this repository: the job runs PR code and
-# needs the HubSpot / Salesforce credentials whose CDK `discover` calls a live
-# API, and secrets are not available to fork PRs. The trigger is `pull_request`,
-# never `pull_request_target` — that would run fork code with access to those
-# secrets. Pushes always carry secrets, so they need no such guard.
+# Every connector's `discover` runs on fake config values, so the gate needs no
+# secrets and fork PRs validate like any other. The trigger is `pull_request`,
+# never `pull_request_target`.
 #
 # The gate job validates; it never commits. On PR drift the separate regen-pr
 # job delivers the regenerated snapshot as a reviewable stacked PR against the
@@ -51,10 +49,6 @@ permissions:
 jobs:
   connectors-ddl:
     name: connectors-ddl snapshot + field parity
-    # Fork PR: no secrets, so `discover` for hubspot/salesforce cannot run. Skip
-    # up front instead of dying 20 minutes into the bootstrap. A push to main is
-    # always in-repo, so it has no head repository to compare.
-    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository
     runs-on: ubuntu-latest
     timeout-minutes: 60
     outputs:
@@ -92,28 +86,6 @@ jobs:
           yq --version
           jq --version
 
-      - name: Require connector credentials
-        env:
-          HUBSPOT_ACCESS_TOKEN: ${{ secrets.HUBSPOT_ACCESS_TOKEN }}
-          SALESFORCE_CLIENT_ID: ${{ secrets.SALESFORCE_CLIENT_ID }}
-          SALESFORCE_CLIENT_SECRET: ${{ secrets.SALESFORCE_CLIENT_SECRET }}
-          SALESFORCE_INSTANCE_URL: ${{ secrets.SALESFORCE_INSTANCE_URL }}
-        run: |
-          set -euo pipefail
-          # HubSpot and Salesforce build their catalogue from a live API, so their
-          # `discover` cannot run on fake values. Without them seed-connectors.sh
-          # skips both connectors, their bronze never lands, and dbt dies — 20
-          # minutes later, with a confusing error. Fail in 5 seconds instead,
-          # naming exactly what is missing.
-          missing=()
-          for var in HUBSPOT_ACCESS_TOKEN SALESFORCE_CLIENT_ID SALESFORCE_CLIENT_SECRET SALESFORCE_INSTANCE_URL; do
-            [[ -n "${!var:-}" ]] || missing+=("$var")
-          done
-          if (( ${#missing[@]} > 0 )); then
-            echo "::error title=Missing repository secrets::${missing[*]} — hubspot/salesforce discover calls a live API and cannot run on fake values. Add them as repository secrets, or narrow connectors-config.yaml."
-            exit 1
-          fi
-
       - name: Start an empty ClickHouse
         run: |
           set -euo pipefail
@@ -128,11 +100,6 @@ jobs:
 
       - name: Bootstrap from connectors + dbt + migrations
         id: bootstrap
-        env:
-          HUBSPOT_ACCESS_TOKEN: ${{ secrets.HUBSPOT_ACCESS_TOKEN }}
-          SALESFORCE_CLIENT_ID: ${{ secrets.SALESFORCE_CLIENT_ID }}
-          SALESFORCE_CLIENT_SECRET: ${{ secrets.SALESFORCE_CLIENT_SECRET }}
-          SALESFORCE_INSTANCE_URL: ${{ secrets.SALESFORCE_INSTANCE_URL }}
         run: |
           set -euo pipefail
           "${BOOTSTRAP_DIR}/bootstrap-db.sh" "${BOOTSTRAP_DIR}/connectors-config.yaml"
@@ -195,11 +162,10 @@ jobs:
         run: docker logs --tail 200 "${CLICKHOUSE_CONTAINER}" 2>&1 || true
 
   # On snapshot drift in a PR, deliver the regenerated snapshot as a REVIEWABLE
-  # stacked pull request against the PR's own branch, instead of telling the
-  # author to regenerate locally — which requires the HubSpot/Salesforce
-  # credentials most contributors do not have. The author reviews the DDL diff
-  # and merges it; that merge is a human push, so the required checks re-run
-  # normally (a bot push with GITHUB_TOKEN would not trigger them).
+  # stacked pull request against the PR's own branch, saving the author a local
+  # bootstrap run. The author reviews the DDL diff and merges it; that merge is
+  # a human push, so the required checks re-run normally (a bot push with
+  # GITHUB_TOKEN would not trigger them).
   #
   # This job never executes code from the PR: it checks out the head branch only
   # as a git base and replaces the snapshot directory with the artifact the gate
@@ -211,8 +177,11 @@ jobs:
   regen-pr:
     name: Open a regen PR on snapshot drift
     needs: connectors-ddl
+    # Same-repo only: this job pushes a branch and opens a PR, neither of which
+    # it can do against a fork's head. A fork PR still gets the gate result and
+    # the regenerated snapshot as a downloadable artifact.
     if: >-
-      always() && github.event_name == 'pull_request' && needs.connectors-ddl.result == 'failure' && needs.connectors-ddl.outputs.drift == 'true'
+      always() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && needs.connectors-ddl.result == 'failure' && needs.connectors-ddl.outputs.drift == 'true'
     runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:

docs/domain/connector/specs/ADR/0004-static-stream-schemas-with-raw-data.md

@@ -0,0 +1,168 @@
+---
+status: proposed
+date: 2026-08-05
+---
+
+# ADR-0004: Static Stream Schemas with Full-Record `raw_data`
+
+**ID**: `cpt-insightspec-adr-connector-static-schema-raw-data`
+
+<!-- toc -->
+
+- [Context and Problem Statement](#context-and-problem-statement)
+- [Decision Drivers](#decision-drivers)
+- [Considered Options](#considered-options)
+- [Decision Outcome](#decision-outcome)
+  - [Consequences](#consequences)
+  - [Confirmation](#confirmation)
+- [Pros and Cons of the Options](#pros-and-cons-of-the-options)
+  - [Option 1: Static columns plus full-record `raw_data`](#option-1-static-columns-plus-full-record-raw_data)
+  - [Option 2: Discovery-derived columns](#option-2-discovery-derived-columns)
+  - [Option 3: Static columns, residual-only overflow blob](#option-3-static-columns-residual-only-overflow-blob)
+  - [Option 4: `raw_data` only, no typed columns](#option-4-raw_data-only-no-typed-columns)
+- [More Information](#more-information)
+- [Traceability](#traceability)
+
+<!-- /toc -->
+
+## Context and Problem Statement
+
+A connector whose advertised schema is computed from a live discovery call
+against the source instance makes the Bronze table shape a function of that
+instance. Two consequences follow. Bronze DDL cannot be produced without
+credentials, so tables cannot be created before the first sync. And two
+instances of the same source yield different column sets for the same stream,
+so downstream models cannot rely on a column existing.
+
+Restricting columns to a curated list fixes the shape but discards every field
+outside the list — including fields the source adds later. Fields that were
+never captured cannot be recovered retroactively, which forecloses metrics
+that would otherwise be derivable from data already collected.
+
+What schema contract should a connector advertise so that Bronze is
+instance-independent and no source field is lost?
+
+## Decision Drivers
+
+* Offline DDL — bronze tables must be creatable from the repository alone, with no credentials and no discovery call
+* Instance independence — the same stream must produce the same columns against any instance of the source
+* No field loss — a field the source returns must reach Bronze even when no column is declared for it
+* Retroactive analysis — a metric conceived after ingestion must be answerable from history already stored
+* Query ergonomics — the fields models actually consume should be plain typed columns, not JSON extraction
+
+## Considered Options
+
+* Option 1: Static columns plus full-record `raw_data`
+* Option 2: Discovery-derived columns
+* Option 3: Static columns, residual-only overflow blob
+* Option 4: `raw_data` only, no typed columns
+
+## Decision Outcome
+
+Chosen option: "Static columns plus full-record `raw_data`".
+
+A connector declares each stream's columns statically, in the repository. The
+declaration is the source of truth for both the advertised catalog and the
+generated Bronze DDL. Every record additionally carries `raw_data`: the whole
+source record, serialized as a compact JSON string.
+
+Rules:
+
+1. The advertised schema is loaded from a static declaration. Discovery calls
+   may build fetch lists (which fields to request from the API), never the
+   schema.
+2. `raw_data` holds the record as received, minus source metadata envelopes
+   that carry no data. It is present on every stream.
+3. Fields the stream does not declare are emitted only inside `raw_data`, never
+   as top-level keys. Emitting them would let the destination create columns
+   for them and restore instance-dependent drift.
+4. String values inside `raw_data` are capped per value. The serialized blob is
+   never truncated, so it always parses.
+5. Adding a column is a repository change: extend the static declaration.
+
+### Consequences
+
+* Good, because Bronze DDL derives from the repository and needs no credentials
+* Good, because the column set is identical across instances of a source
+* Good, because a field with no declared column still reaches Bronze and stays
+  available to models written later
+* Good, because the fields models consume stay typed columns, so existing
+  queries need no JSON extraction
+* Bad, because declared values are stored twice, once as a column and once
+  inside `raw_data`
+* Bad, because a source that adds a field no longer surfaces it as a column
+  automatically; promoting it is a deliberate repository change
+* Neutral, because `raw_data` is a JSON string rather than a native JSON
+  column — the destination materializes both identically today
+
+### Confirmation
+
+* Building a stream catalog performs no network call, and repeated builds
+  against different instances produce byte-identical schemas
+* Every stream's advertised properties equal the static declaration plus the
+  envelope fields
+* A record carrying a field with no declared column emits no top-level key for
+  it, and the field is present in the parsed `raw_data`
+
+## Pros and Cons of the Options
+
+### Option 1: Static columns plus full-record `raw_data`
+
+Columns declared in the repository; whole record additionally preserved as JSON.
+
+* Good, because it satisfies offline DDL and no-field-loss simultaneously
+* Good, because typed columns keep the common query path ergonomic
+* Bad, because declared values are duplicated inside the blob
+
+### Option 2: Discovery-derived columns
+
+The advertised schema is computed per instance from a live discovery call.
+
+* Good, because a newly added source field becomes a column with no code change
+* Bad, because Bronze DDL cannot be produced without credentials
+* Bad, because column sets differ across instances of the same source
+* Bad, because a fetch failure during discovery fails the whole sync
+
+### Option 3: Static columns, residual-only overflow blob
+
+Only undeclared fields go to the blob; declared values are not duplicated.
+
+* Good, because it avoids duplicate storage
+* Bad, because the record cannot be reconstructed from one column; consumers
+  must join columns and blob and know which is which
+* Bad, because promoting a field to a column changes where historical values
+  live, so a query must read both shapes
+
+### Option 4: `raw_data` only, no typed columns
+
+Envelope plus one JSON column; all fields extracted downstream.
+
+* Good, because Bronze DDL becomes identical for every stream
+* Good, because it stores each value once
+* Bad, because every existing downstream model must be rewritten to extract
+  from JSON
+* Bad, because wide blobs flowing through sort buffers are a known source of
+  memory exhaustion in downstream aggregation
+
+## More Information
+
+Connectors whose column set is already curated and instance-independent satisfy
+this ADR by adding `raw_data`; their existing declaration becomes the static
+schema. Connectors that compute schemas from discovery must move the
+declaration into the repository.
+
+`raw_data` is the only overflow carrier. A connector emits no second blob for a
+subset of the record — a column holding just the instance-defined fields is
+contained in `raw_data`, and two representations of the same values drift:
+they are written by different rules and can disagree on truncation, on null
+handling, and on which fields they consider in scope. A consumer that wants
+only the instance-defined subset filters `raw_data` at read time.
+
+## Traceability
+
+This decision directly addresses the following requirements or design elements:
+
+* `cpt-insightspec-fr-cn-custom-fields` — a field with no declared column is
+  preserved in `raw_data`
+* `cpt-insightspec-adr-connector-responsibility-scope` — the connector emits the
+  full payload alongside extracted fields, as that ADR requires

docs/domain/ingestion-data-flow/specs/DESIGN.md

@@ -276,7 +276,7 @@ Raw API ingestion. Connector writes minimally-transformed JSON-decoded rows into
 
 ##### Why this component exists
 
-Per-connector cleanup, type coercion, projection, and dedup. Staging models hide bronze idiosyncrasies (Airbyte JSON envelopes, raw timestamp strings, custom_fields blobs) behind a normalized per-connector schema that silver can union without further per-source logic.
+Per-connector cleanup, type coercion, projection, and dedup. Staging models hide bronze idiosyncrasies (Airbyte JSON envelopes, raw timestamp strings, raw_data blobs) behind a normalized per-connector schema that silver can union without further per-source logic.
 
 ##### Responsibility scope
 

src/ingestion/connectors/crm/hubspot/README.md

@@ -1,6 +1,6 @@
 # HubSpot Connector
 
-CDK-based Python connector for HubSpot CRM. Pulls live data via CRM v3 Search API with v4 associations and archived data via list + batch_read; only an allowlisted subset of `hubspotDefined` standard properties (the curated `ALLOWED_PROPERTIES_BY_OBJECT`) surfaces as typed Bronze columns. Tenant-defined (`hubspotDefined=false`) properties are folded into a single `custom_fields` JSON column so Bronze stays stable across portals and bounded in width regardless of customization depth.
+CDK-based Python connector for HubSpot CRM. Pulls live data via CRM v3 Search API with v4 associations and archived data via list + batch_read; the allowlist in `ALLOWED_PROPERTIES_BY_OBJECT` surfaces as dedicated Bronze columns and defines the advertised schema statically, so the table shape is identical across portals. Every other property still ships in the `raw_data` JSON column, keeping Bronze width bounded without losing values.
 
 Streams sync sequentially. HubSpot's search endpoint is rate-limited to 4 rps portal-wide so a single thread saturates the cap; concurrency would only redistribute the same 4 rps across more 429 retries.
 
@@ -114,7 +114,9 @@ HubSpot's CRM Search endpoint caps at `after = 10,000`. The connector sorts ever
 - `5xx`, chunked-encoding, connection resets — retried with exponential backoff.
 
 ### Property scope
-Bronze advertises **only the curated `ALLOWED_PROPERTIES_BY_OBJECT` allowlist** of `hubspotDefined` standard properties as typed `properties_*` columns; standard properties outside the allowlist are skipped. Tenant-defined (`hubspotDefined=False`) properties land in the `custom_fields` JSON column with null/empty values dropped and per-value byte cap applied (see envelope). This keeps Bronze width bounded regardless of portal customization depth — typical width is 5–15 typed columns per object plus `custom_fields`, instead of the 50–250+ you'd get from projecting every standard property. To project a new standard column, add it to `ALLOWED_PROPERTIES_BY_OBJECT[object_type]` in `constants.py`.
+Bronze advertises **exactly the `ALLOWED_PROPERTIES_BY_OBJECT` allowlist** as `properties_*` columns. The advertised schema is static — derived from `constants.py`, not from a portal describe — so every portal produces the same table shape and an allowlisted property the portal doesn't define is simply NULL. To project a new column, add it to `ALLOWED_PROPERTIES_BY_OBJECT[object_type]` in `constants.py`.
+
+Syncs still request **every** property the portal defines. Nothing is discarded: the full record — nested `properties` object plus association ids — is serialized into the `raw_data` JSON column, so a property with no dedicated column stays recoverable downstream. The per-value byte cap (see envelope) applies before serializing, so the blob itself stays parseable. This keeps the typed width bounded regardless of portal customization depth.
 
 ### Deleted / archived records
 Each archived stream runs as **client-side incremental on `archivedAt`** — page the full archived set, drop records at-or-below the prior cursor state, batch_read full properties for the survivors. After the first sync, only newly-archived rows write to Bronze. Silver UNIONs the live and archived sources and ranks rows by `greatest(updatedAt, archivedAt)` so an archive event outranks the prior live update. The `archived: true` flag is still surfaced on Silver rows via the `metadata` JSON column.

src/ingestion/connectors/crm/hubspot/dbt/hubspot__crm_accounts.sql

@@ -44,8 +44,6 @@ WITH src AS (
             'annualrevenue',     coalesce(toString(properties_annualrevenue), ''),
             'archived',          toString(coalesce(archived, false))
         ))                                              AS metadata,
-        -- Envelope parity with salesforce__crm_* (no HubSpot custom-fields blob).
-        '{}'                                            AS custom_fields,
         createdAt                                       AS created_at,
         updatedAt                                       AS updated_at,
         data_source,