Fix CI failures caused by docs build memory exhaustion#2580
Conversation
The Sphinx docs build had outgrown the memory available on pre-commit.ci runners and was failing consistently on every pull request, including ones that touched no documentation at all. The result was a red pre-commit.ci check on unrelated PRs, blocked merges, time lost to investigating failures that had nothing to do with the change under review, and an erosion of trust in required checks. Running the docs build on GitHub Actions restores that trust. PRs that do not touch docs or library sources now pass cleanly. PRs that do touch them get a reliable pass/fail signal with enough memory to actually complete the build, so warnings surface where they belong rather than being masked by OOM failures. pre-commit.ci is left to focus on what it is good at, fast lint and format checks, and reviewers and contributors get back the signal they rely on to ship changes confidently. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Revert before merging to main. This lets the workflow exercise on a push to this branch so we can confirm the Sphinx build succeeds on GitHub Actions before the fix reaches main. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Successful docs build GitHub actions run triggered from this branch: https://github.com/astronomer/astronomer-cosmos/actions/runs/24655247815/job/72087052273?pr=2580#step:5:331 Successful pre-commit.ci run from this PR: https://results.pre-commit.ci/run/github/577757880/1776672058.2nwJK3-bTuiR4xYgq_mZqA |
There was a problem hiding this comment.
Pull request overview
Moves the Sphinx docs build out of pre-commit.ci (where it was OOM’ing) into a dedicated GitHub Actions workflow so docs builds run with adequate resources and only when docs/library sources change.
Changes:
- Remove the local
check-docs-buildhook from.pre-commit-config.yaml. - Add a new
.github/workflows/docs-build.ymlworkflow to install docs deps and runsphinx-build. - Update
.github/workflows/test.ymlchanged-files filter to treatdocs-build.ymlas non-code (so it won’t trigger full test runs by itself).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
.pre-commit-config.yaml |
Removes the Sphinx build hook that was exhausting memory on pre-commit.ci. |
.github/workflows/test.yml |
Ensures edits to the new docs-build workflow don’t force the full test matrix to run. |
.github/workflows/docs-build.yml |
Introduces a dedicated CI docs build job for PRs/changes impacting docs or cosmos/. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2580 +/- ##
=======================================
Coverage 98.05% 98.05%
=======================================
Files 103 103
Lines 7601 7601
=======================================
Hits 7453 7453
Misses 148 148 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tatiana
left a comment
There was a problem hiding this comment.
Thanks for fixing this, @pankajkoti !
Bug Fixes * Fix ``ExecutionMode.WATCHER`` producer retry behaviour by @tatiana in #2559 * Prevent watcher producer skip propagating to downstream tasks via gateway task by @johnhoran and @tatiana in #2597 * Keep watcher sensor polling when producer is still running by @pankajkoti in #2592 * Fix circular import error in Cosmos plugin discovery under Astro Runtime by @tatiana in #2538 * Fix ``CosmosRichLogger`` crash on ``None`` log message by @tatiana in #2540 * Enable inlets and outlets using dbt Fusion on Airflow 3 by @ichirotakami in #2561 * Fix incorrectly skipped source downstream tasks in ``ExecutionMode.WATCHER`` by @pankajastro in #2563 * Fix duplicate logs in ``dbt build`` when source freshness is enabled by @pankajastro in #2564 * Warn and normalize when ``source_rendering_behavior=None`` is passed by @pankajastro in #2570 * Gracefully handle ``Variable.set()`` failures on Astro Remote Execution by @hkc-8010 in #2573 * Skip malformed YAML selectors instead of failing entirely by @YourRoyalLinus in #2577 Docs * Update watcher test behavior docs for Cosmos 1.14.0 by @tatiana in #2549 * Add redirect for moved partial-parsing docs page by @tatiana in #2550 * Document ``ExecutionMode.WATCHER`` and ``depends_on_past`` limitation by @tatiana in #2602 * Restore memory-optimised imports docs for Cosmos < 1.14.0 by @pankajkoti in #2604 Others * Speed up Airflow 3.1+ integration tests by caching InProcessExecutionAPI by @pankajkoti in #2547 * Improve stability of cache hash unit tests by @tatiana in #2539 * Fix mypy 1.20.0 type check failures by @pankajkoti in #2546 * Fix CI failures caused by docs build memory exhaustion by @pankajkoti in #2580 * Fix dbt Fusion broken integration tests by @tatiana in #2581 * Fix flaky ``cosmos_manifest_selectors_example`` DAG in CI by @pankajkoti in #2593 * Reduce pre-commit autoupdate frequency PRs by @tatiana in #2544 * Bump ``reviewdog/action-actionlint`` from 1.71.0 to 1.72.0 by @dependabot in #2542 * Skip watcher gateway test on Airflow 3.0 by @tatiana in #2607 closes: astronomer/oss-integrations-private#381
The Sphinx docs build had outgrown the memory available on
pre-commit.ci runners and was failing consistently on every pull
request, including ones that touched no documentation at all. The
result was a red pre-commit.ci check on unrelated PRs, blocked merges,
time lost to investigating failures that had nothing to do with the
change under review, and an erosion of trust in required checks.
Running the docs build on GitHub Actions restores that trust. PRs
that do not touch docs or library sources now pass cleanly. PRs that
do touch them get a reliable pass/fail signal with enough memory to
actually complete the build, so warnings surface where they belong
rather than being masked by OOM failures. pre-commit.ci is left to
focus on what it is good at, fast lint and format checks, and
reviewers and contributors get back the signal they rely on to ship
changes confidently.
related: #2578
related: #2579