fix(docs): stop generated-reference drift from blocking the publish - #13056
fix(docs): stop generated-reference drift from blocking the publish#13056dagil-nvidia wants to merge 2 commits into
Conversation
api_freshness.py scopes the per-PR gate so a branch is only failed for drift it caused. That is right -- the alternative fails whoever is unlucky at merge time -- but it rests on the second half of its own docstring: "the next regeneration on main resolves it". Nothing regenerates main. Drift accumulates until a human notices, twice on 2026-08-11 alone (#12985, #13035), and the cost is not a red check: the strict check runs before the sync/publish steps in fern-docs.yml, so stale output skips the publish and docs.nvidia.com stops updating. This job supplies the missing half. It pushes a branch and stops there. Two other routes are closed on this repo, and I confirmed both by reading failing runs rather than assuming: * push to main -- GH013 "Changes must be made through a pull request". community-events-refresh.yml does this and has failed on every run; its commit has never landed. * peter-evans/create-pull-request -- "GitHub Actions is not permitted to create or approve pull requests". update-events.yml does this and is likewise failing every run. So this mirrors auto-dep-upgrade-trigger.yml, the one automation here that lands: push a branch with OPS_BOT_PAT, idempotent on branch existence, and let a human open the PR. Both dead patterns are named in the header so the next author does not re-derive them. The branch name keys on the main SHA that drifted, so re-running the same merge reuses the branch instead of stacking near-identical ones. Only docs/fern is staged. gen_kubernetes_api runs, but its first hop (make generate-api-docs over the Go types) needs Go and crd-ref-docs and stays a human's job -- running the renderer still catches renderer-side drift. Validation: YAML parses; the only uses: entries are the pinned checkout and setup-python actions; the new file is covered by the ci filter, so the coverage gate passes; a dry run of all four generators on current main produces no diff, so the job correctly exits without pushing when main is clean. Signed-off-by: Dan Gil <dagil@nvidia.com>
The freshness check sits third in the publish job, ahead of the docs-website checkout and every sync step, with no continue-on-error. When it fails the job aborts and nothing publishes. That is inverted. A skipped publish leaves docs.nvidia.com serving the PREVIOUS state, whose API pages are older than the ones being withheld, and it takes every unrelated docs change down with it. Blocking does not protect readers from stale API docs -- it guarantees they see strictly older ones, and withholds the rest of the site as well. The failure it prevents is milder than the failure it causes. It is also redundant. The same three --check invocations already run as the "Generated API References" job in pre-merge.yml, which is in pre-merge-status-check's needs, so drift is caught at merge time by the author who caused it. Enforcing it again at publish adds no signal, only blast radius. Now continue-on-error: the step still runs and still reports, so drift stays visible in the run, and the merge gate is untouched -- verified that api-docs carries no continue-on-error on any step and remains a required dependency. This happened twice on 2026-08-11 (#12985, #13035): both times a few stale generated pages stopped the entire site from updating. Validation: fern-docs.yml parses; the check is step 3 of 20 with continue-on-error true and "Checkout docs-website branch" still next; pre-merge.yml's api-docs job unchanged. Signed-off-by: Dan Gil <dagil@nvidia.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| - name: Report | ||
| if: steps.regen.outputs.drifted == 'true' && steps.branch.outputs.exists == 'false' | ||
| run: | | ||
| { | ||
| echo "### API references drifted" | ||
| echo | ||
| echo "Pushed \`${{ steps.branch.outputs.name }}\`. Open a PR from it to" | ||
| echo "restore \`main\`; until it merges, \`fern-docs.yml\` skips the publish" | ||
| echo "and docs.nvidia.com will not update." | ||
| } >> "$GITHUB_STEP_SUMMARY" |
There was a problem hiding this comment.
🟡 Automated drift notice tells maintainers the docs site has stopped updating, which is no longer true
The automated drift notice states that the documentation site will not update until the regeneration branch is merged (echo block at .github/workflows/regenerate-api-references.yml:117-126), even though the same change makes the publish continue past drift, so maintainers are told about a site outage that is not happening.
Impact: On-call maintainers get a false urgency signal and may scramble to fix a publish that is actually running normally.
Contradiction between the new non-blocking publish check and the notice text
.github/workflows/fern-docs.yml:138-139 now marks the generated-reference check continue-on-error: true, so drift no longer aborts the sync/publish job. The new workflow's header comment (.github/workflows/regenerate-api-references.yml:14-17) and its $GITHUB_STEP_SUMMARY output (.github/workflows/regenerate-api-references.yml:120-126) both still assert "the strict check runs BEFORE the sync/publish steps ... stale output skips the publish and docs.nvidia.com stops updating" / "until it merges, fern-docs.yml skips the publish and docs.nvidia.com will not update".
| - name: Report | |
| if: steps.regen.outputs.drifted == 'true' && steps.branch.outputs.exists == 'false' | |
| run: | | |
| { | |
| echo "### API references drifted" | |
| echo | |
| echo "Pushed \`${{ steps.branch.outputs.name }}\`. Open a PR from it to" | |
| echo "restore \`main\`; until it merges, \`fern-docs.yml\` skips the publish" | |
| echo "and docs.nvidia.com will not update." | |
| } >> "$GITHUB_STEP_SUMMARY" | |
| - name: Report | |
| if: steps.regen.outputs.drifted == 'true' && steps.branch.outputs.exists == 'false' | |
| run: | | |
| { | |
| echo "### API references drifted" | |
| echo | |
| echo "Pushed \`${{ steps.branch.outputs.name }}\`. Open a PR from it to" | |
| echo "restore \`main\`. The publish in \`fern-docs.yml\` still runs (its" | |
| echo "freshness check is non-blocking), but docs.nvidia.com will serve" | |
| echo "stale generated API pages until this merges." | |
| } >> "$GITHUB_STEP_SUMMARY" |
Was this helpful? React with 👍 or 👎 to provide feedback.
| run: | | ||
| # Keyed on the main SHA that drifted, so a re-run of the same merge | ||
| # reuses the branch instead of stacking near-identical ones. | ||
| BRANCH="docs/regen-api-references-${GITHUB_SHA::9}" | ||
| echo "name=$BRANCH" >> "$GITHUB_OUTPUT" | ||
| if git ls-remote --heads origin "$BRANCH" | grep -q "$BRANCH"; then | ||
| echo "Branch $BRANCH already exists on origin; skipping." | ||
| echo "exists=true" >> "$GITHUB_OUTPUT" | ||
| else | ||
| echo "exists=false" >> "$GITHUB_OUTPUT" | ||
| fi |
There was a problem hiding this comment.
🟡 Repeated regeneration branches pile up when the first one is not merged promptly
The automatic fix branch is named after the exact commit that drifted (BRANCH="docs/regen-api-references-${GITHUB_SHA::9}" at .github/workflows/regenerate-api-references.yml:92), so every later merge creates yet another branch with the same pending fix until one is merged.
Impact: The repository accumulates near-duplicate bot branches and each run's notice competes for attention, while the drift stays unfixed.
Idempotency is keyed on SHA, not on drift content
The comment at .github/workflows/regenerate-api-references.yml:90-91 says the key "reuses the branch instead of stacking near-identical ones", but this only dedupes re-runs of the same main SHA. Any subsequent push to main re-detects the same drift and produces a new branch (.github/workflows/regenerate-api-references.yml:94-99,101-115). Keying on a hash of the regenerated diff, or reusing/force-updating a single fixed branch name such as docs/regen-api-references, would make it truly idempotent. Additionally, when the branch already exists the job produces no summary at all (.github/workflows/regenerate-api-references.yml:118), so persistent drift becomes silent.
Was this helpful? React with 👍 or 👎 to provide feedback.
| - name: Install generator dependencies | ||
| run: python3 -m pip install 'griffe==2.1.0' pyyaml | ||
|
|
||
| - name: Regenerate | ||
| id: regen | ||
| run: | | ||
| set -euo pipefail | ||
| # gen_kubernetes_api.py re-renders a committed intermediate that | ||
| # `make generate-api-docs` produces from the operator Go types. That | ||
| # first hop needs Go and crd-ref-docs, so it stays a human's job; | ||
| # running the renderer here still catches renderer-side drift. | ||
| for g in gen_python_api gen_rust_api gen_kubernetes_api gen_llms_tables; do | ||
| python3 "docs/fern/scripts/${g}.py" | ||
| done |
There was a problem hiding this comment.
🔍 Any generator failure hard-fails the job on every main push
set -euo pipefail plus the loop means a generator that exits non-zero for an environmental reason (e.g. gen_kubernetes_api.py returning 2 because the committed intermediate produced by make generate-api-docs is absent/unparseable, or a missing runtime dependency beyond griffe/pyyaml) will fail this workflow on every push to main until a human intervenes. Worth confirming the dependency set matches what pre-merge.yml's api-docs job installs (.github/workflows/pre-merge.yml:112-124 also installs pytest/pydantic/fsspec, though those are test-only).
Was this helpful? React with 👍 or 👎 to provide feedback.
WalkthroughThe changes allow docs publication to continue after API reference drift and add a workflow that regenerates references on ChangesAPI reference automation
Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/regenerate-api-references.yml:
- Around line 53-58: Update the workflow after the “Checkout main” step to
capture the checked-out revision with git rev-parse HEAD, expose it as a step
output or environment value, and use that value instead of GITHUB_SHA for the
branch name and commit message in the later branch creation and commit steps.
- Around line 14-17: Update the stale-publication statements in
.github/workflows/regenerate-api-references.yml at lines 14-17 and 123-125:
state that fern-docs.yml reports generated-reference drift without blocking sync
or publication, and that publication may continue with stale generated pages
until the regeneration branch merges.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: a912fd92-fdad-4505-9955-05f64b2aeabb
📒 Files selected for processing (2)
.github/workflows/fern-docs.yml.github/workflows/regenerate-api-references.yml
| # Nothing regenerated main. Drift accumulated instead -- twice on 2026-08-11 | ||
| # alone (#12985, #13035) -- and the cost is not a red check: the strict check | ||
| # runs BEFORE the sync/publish steps in fern-docs.yml, so stale output skips | ||
| # the publish and docs.nvidia.com stops updating. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Update the stale publication behavior statements.
.github/workflows/fern-docs.yml now reports generated-reference drift without blocking the later sync and publish steps. These statements tell operators that drift prevents publication.
.github/workflows/regenerate-api-references.yml#L14-L17: Replace the statement that stale output skips publication..github/workflows/regenerate-api-references.yml#L123-L125: State that publication can continue with stale generated pages until the regeneration branch merges.
📍 Affects 1 file
.github/workflows/regenerate-api-references.yml#L14-L17(this comment).github/workflows/regenerate-api-references.yml#L123-L125
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/regenerate-api-references.yml around lines 14 - 17, Update
the stale-publication statements in
.github/workflows/regenerate-api-references.yml at lines 14-17 and 123-125:
state that fern-docs.yml reports generated-reference drift without blocking sync
or publication, and that publication may continue with stale generated pages
until the regeneration branch merges.
| - name: Checkout main | ||
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | ||
| with: | ||
| ref: main | ||
| token: ${{ secrets.OPS_BOT_PAT }} | ||
|
|
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Bind the branch identity to the checked-out revision.
Line 56 checks out mutable main, but Lines 92 and 113 use the triggering GITHUB_SHA. If this run waits behind the concurrency group, main can advance before checkout. The workflow then generates references from a newer revision but creates a branch named for an older revision. A later run can create another branch for the same generated state.
Record git rev-parse HEAD after checkout. Use that value for the branch name and commit message.
Proposed fix
- name: Checkout main
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
ref: main
token: ${{ secrets.OPS_BOT_PAT }}
+ - name: Record checked-out main revision
+ id: source
+ run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
+
- name: Compute branch name + idempotency check
if: steps.regen.outputs.drifted == 'true'
id: branch
run: |
- BRANCH="docs/regen-api-references-${GITHUB_SHA::9}"
+ SOURCE_SHA="${{ steps.source.outputs.sha }}"
+ BRANCH="docs/regen-api-references-${SOURCE_SHA::9}"
echo "name=$BRANCH" >> "$GITHUB_OUTPUT"Also applies to: 90-93, 111-115
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/regenerate-api-references.yml around lines 53 - 58, Update
the workflow after the “Checkout main” step to capture the checked-out revision
with git rev-parse HEAD, expose it as a step output or environment value, and
use that value instead of GITHUB_SHA for the branch name and commit message in
the later branch creation and commit steps.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: deb313daa0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| BRANCH="docs/regen-api-references-${GITHUB_SHA::9}" | ||
| echo "name=$BRANCH" >> "$GITHUB_OUTPUT" | ||
| if git ls-remote --heads origin "$BRANCH" | grep -q "$BRANCH"; then |
There was a problem hiding this comment.
When drift remains on main while the first generated branch awaits a human-created PR, every subsequent push receives a different GITHUB_SHA, bypasses this existence check, and pushes another near-identical regeneration branch. This can rapidly accumulate competing branches based on different snapshots; reuse or update one outstanding regeneration branch, or explicitly detect an existing regeneration branch before creating another.
Useful? React with 👍 / 👎.
| for g in gen_python_api gen_rust_api gen_kubernetes_api gen_llms_tables; do | ||
| python3 "docs/fern/scripts/${g}.py" |
There was a problem hiding this comment.
This new workflow invokes all four API generators, but .github/filters.yaml omits regenerate-api-references.yml from the api_docs filter even though it includes the other API workflow files. A future PR changing only this workflow will therefore skip the Generated API References pre-merge job, allowing invalid generator names, arguments, or dependency changes to surface only after merging when the main-push automation fails; add this workflow to that filter.
Useful? React with 👍 / 👎.
|
![dynamo-review-agent[bot]](https://avatars.githubusercontent.com/u/201626793?s=60&v=4){kind=small}
|
Closing — the gate stays blocking, by decision. The argument for this change was that blocking is inverted: a skipped publish leaves docs.nvidia.com serving pages older than the ones being withheld, and takes every unrelated docs change with it. That part still holds. What it missed is that the gate is also the only forcing function. Drift got fixed twice on 2026-08-11 precisely because the publish broke and someone felt it. Nothing here notifies on drift — The real fix is #13050: reconcile drift on main within minutes of the merge that caused it, so the gate stops firing on its own rather than being told not to. 🤖 Addressed by Claude Code |
1 participant
Summary
The generated-reference freshness check sits third in the publish job — ahead of the
docs-website checkout and every sync step — with no
continue-on-error. When it fails, the jobaborts and nothing publishes.
That is inverted:
362vs365symbols, an anchor at#L2591instead of#L2602). Everything else reaches users.Blocking does not protect readers from stale API docs. It guarantees they see strictly older ones,
and takes the rest of the site down with them. The failure it prevents is milder than the failure
it causes.
It is also redundant. The same three
--checkinvocations already run as theGenerated API Referencesjob inpre-merge.yml, which is inpre-merge-status-check'sneeds.Drift is caught at merge time, by the author who caused it. Enforcing it again at publish adds no
signal, only blast radius.
This bit twice on 2026-08-11 — #12985 and #13035 — where a handful of stale generated pages
stopped the entire site from updating.
Change
One line:
continue-on-error: trueon that step, plus a comment explaining why, so the nextperson does not "fix" it back.
The step still runs and still reports, so drift stays visible in the run.
Validation
fern-docs.ymlparses; the check is step 3 of 20,continue-on-error: true, andCheckout docs-website branchis still the next step.pre-merge.yml'sapi-docsjob is untouched — nocontinue-on-erroron any of its steps,still in
pre-merge-status-check'sneeds.Related
main, which is the other half of this.api_freshness.py, which is correctly scoped and should stay.🤖 Generated with Claude Code
Summary by CodeRabbit