Skip to content

ci(connectors-ddl): deliver drift as a stacked regen PR; retire the reminder #2133

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mitasovr merged 3 commits into from
Aug 3, 2026
Merged

ci(connectors-ddl): deliver drift as a stacked regen PR; retire the reminder #2133

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
308fe32
ci(connectors-ddl): deliver drift as a stacked regen PR; retire the r…
Aug 3, 2026
e00ab12
ci(connectors-ddl): SHA-pin the actions on the write path
Aug 3, 2026
fef1539
Merge branch 'main' into feat/connectors-ddl-regen-pr
mitasovr Aug 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 0 additions & 72 deletions .github/workflows/connectors-ddl-reminder.yml
View file
Open in desktop

This file was deleted.

112 changes: 109 additions & 3 deletions .github/workflows/connectors-ddl.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,10 @@ name: connectors-ddl
# never `pull_request_target` — that would run fork code with access to those
# secrets. Pushes always carry secrets, so they need no such guard.
#
# The job validates; it never commits. A stale snapshot is the author's to
# regenerate (see the failure message).
# The gate job validates; it never commits. On PR drift the separate regen-pr
# job delivers the regenerated snapshot as a reviewable stacked PR against the
# PR's own branch — see its header for the trust boundary. Drift on a main push
# stays a red run: there is no PR branch to stack onto.

on:
pull_request:
Expand All @@ -55,6 +57,10 @@ jobs:
if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
timeout-minutes: 60
outputs:
# Written by the drift step before its exit 1 — step outputs survive the
# step's own failure, which is how regen-pr below learns it has work.
drift: ${{ steps.drift.outputs.drift }}
env:
BOOTSTRAP_DIR: src/ingestion/scripts/bootstrap-db
CLICKHOUSE_CONTAINER: connectors-ddl-clickhouse
Expand Down Expand Up @@ -132,6 +138,7 @@ jobs:
"${BOOTSTRAP_DIR}/bootstrap-db.sh" "${BOOTSTRAP_DIR}/connectors-config.yaml"

- name: Re-dump the snapshot and fail on drift
id: drift
run: |
set -euo pipefail
"${BOOTSTRAP_DIR}/dump-ddl.sh"
Expand All @@ -140,7 +147,8 @@ jobs:
git add --intent-to-add --all -- src/ingestion/scripts/connectors-ddl
git diff -- src/ingestion/scripts/connectors-ddl > "$RUNNER_TEMP/connectors-ddl.diff"
if [[ -s "$RUNNER_TEMP/connectors-ddl.diff" ]]; then
echo "::error title=connectors-ddl snapshot is stale::Regenerate it: ${BOOTSTRAP_DIR}/bootstrap-db.sh ${BOOTSTRAP_DIR}/connectors-config.yaml && ${BOOTSTRAP_DIR}/dump-ddl.sh — then commit the result. Full diff in the connectors-ddl-drift artifact."
echo "drift=true" >> "$GITHUB_OUTPUT"
echo "::error title=connectors-ddl snapshot is stale::The regen-pr job opens a pull request against this branch with the regenerated snapshot — review its DDL diff and merge it. Full diff also in the connectors-ddl-drift artifact."
git diff --stat -- src/ingestion/scripts/connectors-ddl
head -n 200 "$RUNNER_TEMP/connectors-ddl.diff"
exit 1
Expand All @@ -155,6 +163,15 @@ jobs:
path: ${{ runner.temp }}/connectors-ddl.diff
if-no-files-found: ignore

- name: Upload the regenerated snapshot
# Input for the regen-pr job below: the freshly dumped *.sql files, not
# the diff — the diff may not apply if the author pushes meanwhile.
if: failure() && steps.drift.outputs.drift == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: connectors-ddl-regenerated
path: src/ingestion/scripts/connectors-ddl/

- name: Field parity
# Runs even when the snapshot drifted: the two gates are independent, and a
# PR that breaks both should learn both in one run instead of one per push.
Expand All @@ -176,3 +193,92 @@ jobs:
- name: ClickHouse logs
if: failure()
run: docker logs --tail 200 "${CLICKHOUSE_CONTAINER}" 2>&1 || true

# On snapshot drift in a PR, deliver the regenerated snapshot as a REVIEWABLE
# stacked pull request against the PR's own branch, instead of telling the
# author to regenerate locally — which requires the HubSpot/Salesforce
# credentials most contributors do not have. The author reviews the DDL diff
# and merges it; that merge is a human push, so the required checks re-run
# normally (a bot push with GITHUB_TOKEN would not trigger them).
#
# This job never executes code from the PR: it checks out the head branch only
# as a git base and replaces the snapshot directory with the artifact the gate
# job produced. That separation is why THIS job may hold write tokens while
# the gate job runs PR code with contents: read.
#
# Convergence terminates: the re-dump on an unchanged tree is byte-identical,
# so once the regen commit is merged the next run finds no drift.
regen-pr:
name: Open a regen PR on snapshot drift
needs: connectors-ddl
if: >-
always() && github.event_name == 'pull_request' && needs.connectors-ddl.result == 'failure' && needs.connectors-ddl.outputs.drift == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: write
pull-requests: write
env:
HEAD_REF: ${{ github.event.pull_request.head.ref }}
REGEN_BRANCH: regen-ddl/${{ github.event.pull_request.head.ref }}
PR_NUMBER: ${{ github.event.pull_request.number }}
DDL_DIR: src/ingestion/scripts/connectors-ddl
GH_TOKEN: ${{ github.token }}
steps:
- uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5
with:
ref: ${{ github.event.pull_request.head.ref }}
# This checkout keeps credentials on purpose: the push below uses them.

- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: connectors-ddl-regenerated
path: ${{ runner.temp }}/regenerated

- name: Commit the snapshot and push the regen branch
id: commit
run: |
set -euo pipefail
# Full replace, not overlay: a relation the code no longer produces
# must disappear from the snapshot, and cp alone cannot delete.
rm -f "${DDL_DIR}"/*.sql
cp "${RUNNER_TEMP}/regenerated"/*.sql "${DDL_DIR}/"
git add --all -- "${DDL_DIR}"
if git diff --cached --quiet; then
# The author pushed a regeneration while the gate was running.
echo "changed=false" >> "$GITHUB_OUTPUT"
exit 0
fi
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git commit --signoff -m "chore(connectors-ddl): regenerate the snapshot for ${HEAD_REF}" \
-m "Produced by the connectors-ddl gate from a full bootstrap of this branch: real connector discover, destination-clickhouse, dbt, migrations, then dump-ddl.sh."
# Force: this branch is bot-owned and rebuilt from scratch on every
# drifting run, so its only history is the latest regeneration.
git push --force origin "HEAD:refs/heads/${REGEN_BRANCH}"
echo "changed=true" >> "$GITHUB_OUTPUT"

- name: Open or reuse the stacked PR + sticky comment
if: steps.commit.outputs.changed == 'true'
run: |
set -euo pipefail
url="$(gh pr list --head "${REGEN_BRANCH}" --base "${HEAD_REF}" --state open --json url -q '.[0].url // empty')"
if [[ -z "${url}" ]]; then
url="$(gh pr create --base "${HEAD_REF}" --head "${REGEN_BRANCH}" \
--title "chore(connectors-ddl): regenerate the snapshot for ${HEAD_REF}" \
--body "The connectors-ddl gate on #${PR_NUMBER} found the committed snapshot stale against what this branch's connectors + dbt + migrations actually produce. This PR carries the regenerated snapshot — review the DDL diff and merge it into \`${HEAD_REF}\`; the gate then re-runs and goes green. Closes automatically if #${PR_NUMBER} is merged without it (the base branch disappears).")"
fi
echo "regen PR: ${url}"
Comment on lines +261 to +271

@coderabbitai coderabbitai Bot Aug 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

Does GitHub retarget or close a pull request when its base branch is deleted after a merge?

💡 Result:

No, GitHub does not automatically retarget (update the base branch of) an open pull request if its base branch is deleted [1][2]. Instead, if a pull request's base branch is deleted, the pull request is typically closed automatically [2]. It is important to distinguish this behavior from GitHub's automatic retargeting of "dependent" pull requests [1]: 1. Automatic Retargeting of Dependent Pull Requests: When a pull request is merged and its head branch is deleted, GitHub automatically checks for other open pull requests in the same repository that used that now-deleted branch as their base [3][4]. If it finds any, GitHub automatically updates their base branch to the merged pull request's new base branch [3][4]. This is a specific feature designed to maintain dependency chains [1][4]. 2. Deletion of a Base Branch: If you manually delete a branch that is currently serving as the base for an open pull request (not necessarily a head branch that was just merged), GitHub does not attempt to "guess" a new base and retarget the pull request [2]. Because the pull request no longer has a valid target branch, it is automatically closed [2]. If you find yourself in a situation where a pull request has been closed due to its base branch being deleted, you can restore the base branch, update the pull request's base to a different, active branch, and then re-open the pull request [2].

Citations:

🌐 Web query:

GitHub docs dependent pull requests automatically base updated when head branch merged deleted

💡 Result:

Yes, when a pull request is merged and its head branch is deleted, GitHub automatically updates any open pull requests in the same repository that were using that deleted branch as their base [1]. In such cases, GitHub re-targets those dependent pull requests to the merged pull request's base branch [1]. This behavior occurs automatically when the deletion is performed via the GitHub website (for example, using the "Delete branch" button after merging) or if the repository is configured to automatically delete head branches upon merge [1][2]. However, there are important limitations and nuances to this process: 1. CLI and API behavior: If the branch is deleted via the Git command line or certain API methods (like manually calling a delete reference endpoint) rather than through GitHub's standard merge/delete UI, GitHub may close the dependent pull requests instead of re-targeting their base branch [3][2]. 2. Stacked Pull Requests: GitHub now has native support for stacked pull requests [4][5]. In a stack, when a middle pull request is merged, the pull requests above it are automatically updated and re-targeted to the stack's base branch [4]. 3. Manual intervention: If a dependent pull request is not automatically updated, you can manually change its base branch through the GitHub interface by selecting "Edit" next to the pull request title [6]. In summary, GitHub's automated base branch update is designed to keep dependent workflows intact when branches are removed during standard repository cleanup, provided the deletion is handled through GitHub's native tools [1][2].

Citations:

🌐 Web query:

GitHub dependent pull requests base branch automatically updated documentation

💡 Result:

GitHub does not have a native feature that automatically updates standard pull request branches with changes from the base branch on a continuous, background basis [1][2]. While GitHub provides tools to manage synchronization, the action remains manual or triggered by specific workflow configurations [1][3]. Key mechanisms for keeping pull requests up to date include: 1. Manual Updates via UI: On a pull request page, if the branch is out of sync and there are no merge conflicts, users can click the Update branch button [1][2]. This performs either a traditional merge (merging the base branch into the head branch) or a rebase, depending on the user's choice [1][4]. 2. Repository Settings: Administrators can enable the Always suggest updating pull request branches setting [3][5]. When enabled, this ensures the Update branch button is always available on the pull request page whenever the branch is behind, rather than only when required by branch protection rules [3][5]. 3. Specialized Handling for Dependabot: Dependabot attempts to automatically rebase its pull requests when they become out of date, though this is a best-effort behavior and not guaranteed to be instantaneous [6][7]. 4. Stacked (Dependent) Pull Requests: GitHub now supports stacked pull requests [8]. When you merge a pull request at the bottom of a stack, the remaining branches are automatically rebased so that they target the updated base branch [8]. 5. Automated Workflows: For general pull requests, teams often implement custom automation—such as GitHub Actions—to automatically update or "retarget" pull request branches when the base branch changes [9]. This is commonly done using the GitHub REST API (specifically pulls.updateBranch) to perform an automatic update [10]. In summary, GitHub provides the interface to request an update but relies on manual user action, explicit repository settings, or custom automation to perform the synchronization for standard pull requests [1][3][9].

Citations:

Remove the “Closes automatically” statement from the regenerated PR body.

When ${HEAD_REF} is deleted after the original PR merges, GitHub does not close the stacked regen PR because its base branch now points at a deleted branch. Update the text to describe that the regenerated PR may remain open and needs to be updated against the new base, or verify the repository’s stacked PR/base-branch behavior before relying on GitHub actions.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/connectors-ddl.yml around lines 261 - 271, Update the PR
body constructed in the “Open or reuse the stacked PR + sticky comment” step to
remove the claim that the regen PR closes automatically when #${PR_NUMBER}
merges. Replace it with accurate guidance that the PR may remain open after
${HEAD_REF} is deleted and must be updated against the appropriate new base,
without changing the surrounding PR creation behavior.


marker='<!-- connectors-ddl-regen-pr -->'
body="$(printf '%s\n### :robot: connectors-ddl snapshot drift\n\n%s\n\n%s' \
"${marker}" \
"The committed snapshot does not match what this branch actually produces. The regenerated snapshot is waiting in ${url} — review the DDL diff there and merge it into this branch; the gate re-runs on your merge." \
"_Refreshed on every drifting gate run (the regen branch is force-pushed), so it reflects this branch as of the last completed run._")"
existing="$(gh api "repos/${GITHUB_REPOSITORY}/issues/${PR_NUMBER}/comments" --paginate \
-q "[.[] | select(.body | startswith(\"${marker}\"))][0].id // empty")"
if [[ -z "${existing}" ]]; then
gh api "repos/${GITHUB_REPOSITORY}/issues/${PR_NUMBER}/comments" -f body="${body}" > /dev/null
else
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/issues/comments/${existing}" -f body="${body}" > /dev/null
fi
2 changes: 1 addition & 1 deletion src/ingestion/scripts/bootstrap-db/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ Contributors whose physical table is not owned by dbt are covered too. `jira__ta
| `bootstrap-db.sh <config.yaml>` | Sources `pins.env` and `.env` (if present), runs `seed-connectors.sh`, runs all dbt models, runs `../apply-ch-migrations.sh`. |
| `run-dbt.sh [dbt args]` | Helper: generates a profiles.yml from the `CLICKHOUSE_*` variables and runs `dbt run` in `src/ingestion/dbt`. |
| `check-field-parity.py [--manifest PATH]` | Audits every staging contributor against its silver union target (column set, positional order, exact type) plus manifest-vs-warehouse coverage. Same `CLICKHOUSE_*` env contract as the other scripts. Non-zero exit on any finding. |
| `dump-ddl.sh` | Dumps `SHOW CREATE` for every `bronze_*` table, the `person`/`identity`/`silver`/`insight` databases (tables and views), and the gold-referenced `staging` tables into `../connectors-ddl/*.sql` — the committed snapshot that `../create-bronze-placeholders.sh` applies on fresh clusters. **Run it manually** after `bootstrap-db.sh` (see step above) whenever a schema changes, and commit the diff. `.github/workflows/connectors-ddl.yml` re-runs the whole pipeline on every same-repository PR and on every commit to `main`, and fails when the committed snapshot no longer matches — it validates, it never commits. |
| `dump-ddl.sh` | Dumps `SHOW CREATE` for every `bronze_*` table, the `person`/`identity`/`silver`/`insight` databases (tables and views), and the gold-referenced `staging` tables into `../connectors-ddl/*.sql` — the committed snapshot that `../create-bronze-placeholders.sh` applies on fresh clusters. **Run it manually** after `bootstrap-db.sh` (see step above) whenever a schema changes, and commit the diff. `.github/workflows/connectors-ddl.yml` re-runs the whole pipeline on every same-repository PR and on every commit to `main`, and fails when the committed snapshot no longer matches. On PR drift its `regen-pr` job opens a stacked PR against the PR's branch with the regenerated snapshot (and links it in a sticky comment) — review the DDL diff there and merge, no local regeneration or CRM credentials needed. Drift on `main` stays a red run. |

## Image pins (pins.env)

Expand Down
Loading