docs: add Fern MDX transform scripts and CI integration - #7058
Closed
dagil-nvidia wants to merge 3 commits into
Closed
docs: add Fern MDX transform scripts and CI integration#7058dagil-nvidia wants to merge 3 commits into
dagil-nvidia wants to merge 3 commits into
Conversation
Add fernify transform scripts that convert generated API reference Markdown into Fern MDX components for the documentation site: - _fern_helpers.py: shared transforms (details->Accordion, tables->Cards, slugify, frontmatter injection) - fernify_python_api.py: transforms Python API reference - fernify_rust_api.py: transforms Rust API reference - fernify_k8s_api.py: transforms K8s CRD reference (wraps API groups in Tabs, resource lists in CardGroups, type defs in Accordions) CI integration (.github/workflows/fern-docs.yml): - Add Python setup + pip install for fernify dependencies - Run all 3 fernify scripts before syncing to docs-website - Pin Rust docs.rs links to release version on tag pushes Part 3 of 3 (split from #6989): 1. Docstrings (#7056) 2. API reference generators + navigation (#7057) 3. Fernify transform scripts + CI (this PR) Signed-off-by: Dan Gil <dagil@nvidia.com> Made-with: Cursor
github-actions
Bot
added
docs
documentation
Contributor
WalkthroughThis pull request introduces an API documentation generation pipeline for Fern, adding workflow steps to generate and transform API docs for Python, Rust, and Kubernetes, a shared helper module with utilities for MDX/JSX manipulation and content transformation, and language-specific fernification scripts that convert API reference markdown into Fern-compatible MDX. Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
docs/scripts/fernify_k8s_api.py (1)
260-265: Scan the full document in the idempotency guard.Checking only the first 50 lines makes reruns brittle. If the first
<Tabs>,<CardGroup>, or<Accordion>moves below Line 50, this will fernify an already-processed file again.Suggested fix
def _is_already_fernified(text: str) -> bool: """Detect if the file has already been processed (contains Fern MDX).""" - for line in text.split("\n")[:50]: - if "<Tabs>" in line or "<CardGroup" in line or "<Accordion" in line: - return True - return False + return any(marker in text for marker in ("<Tabs>", "<CardGroup", "<Accordion"))🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@docs/scripts/fernify_k8s_api.py` around lines 260 - 265, The idempotency guard in _is_already_fernified only scans the first 50 lines which is brittle; change it to scan the entire document (e.g., check the full text rather than text.split("\n")[:50]) and return True if any of the target markers ("<Tabs>", "<CardGroup", "<Accordion") appear anywhere; update _is_already_fernified to use a full-text search (or any(...) over all lines) so reruns correctly detect already-fernified files regardless of where the tags appear.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/fern-docs.yml:
- Around line 165-166: The workflow step in .github/workflows/fern-docs.yml that
runs "pip install -r source-checkout/docs/scripts/requirements-apidocs.txt"
points to a missing file and will fail; either remove that pip install step
entirely or add the referenced requirements-apidocs.txt containing the needed
third‑party packages. If you choose to remove it, delete the "Install API doc
dependencies" job/step (the run line) since the new fernify scripts
(fernify_python_api.py, fernify_rust_api.py, fernify_k8s_api.py) only use stdlib
and local modules; if you choose to keep it, create
source-checkout/docs/scripts/requirements-apidocs.txt and list any real
third-party dependencies required by the doc pipeline.
In `@docs/scripts/_fern_helpers.py`:
- Around line 16-17: render_details is passing the raw captured summary
(m.group(1)) into render_accordion which causes a second HTML-escape; instead
unescape the group before calling render_accordion so the summary is only
escaped once (e.g. use html.unescape(m.group(1)) or otherwise pass the original
unescaped summary to render_accordion). Update the calls in render_details and
the other similar sites (the blocks around lines 109-120 and 145-153) to
unescape the captured summary before invoking render_accordion.
- Around line 37-55: Card descriptions are emitted raw into <Card> bodies
without escaping and fernify_details_to_accordion is double-escaping summaries;
to fix, ensure you call escape_mdx_prose when rendering card descriptions (where
the <Card> body is constructed) so MDX-sensitive chars are escaped exactly once,
and change fernify_details_to_accordion so it does not pass an already-escaped
capture to render_accordion — either stop re-escaping the regex capture there or
adjust render_accordion to accept an already_escaped flag; specifically, locate
usages that construct Card content and add a single call to
escape_mdx_prose(text) and modify fernify_details_to_accordion/render_accordion
to perform escaping in one place only to prevent double-escaping.
In `@docs/scripts/fernify_k8s_api.py`:
- Around line 141-149: build_resource_cards' meta parameter is being ignored
because _resolve_meta always reads RESOURCE_META; update the code so callers'
custom metadata is honored: change _resolve_meta signature to accept an optional
meta dict (e.g., def _resolve_meta(name: str, full_text: str, meta:
dict[str,str]|None = None) -> dict[str,str]) and have it return meta[name] if
meta and name in meta, then fallback to RESOURCE_META, then to extracting
description and _discover_go_source; also update every call site (notably
build_resource_cards) to pass the incoming meta argument into _resolve_meta so
custom metadata is used.
---
Nitpick comments:
In `@docs/scripts/fernify_k8s_api.py`:
- Around line 260-265: The idempotency guard in _is_already_fernified only scans
the first 50 lines which is brittle; change it to scan the entire document
(e.g., check the full text rather than text.split("\n")[:50]) and return True if
any of the target markers ("<Tabs>", "<CardGroup", "<Accordion") appear
anywhere; update _is_already_fernified to use a full-text search (or any(...)
over all lines) so reruns correctly detect already-fernified files regardless of
where the tags appear.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 97e13d56-b189-4de1-bf4c-43974feca249
📒 Files selected for processing (5)
.github/workflows/fern-docs.ymldocs/scripts/_fern_helpers.pydocs/scripts/fernify_k8s_api.pydocs/scripts/fernify_python_api.pydocs/scripts/fernify_rust_api.py
Comment on lines
+16
to
+17
| import re | ||
| from pathlib import Path |
Contributor
There was a problem hiding this comment.
Don't double-escape <summary> text during details→accordion conversion.
render_details() already escapes the summary on the way into HTML. Passing m.group(1) straight into render_accordion() escapes the entities a second time, so titles containing &, <, > or quotes will render as &amp;, &lt;, etc. in the Python API path that uses this helper.
Suggested fix
+import html
import re
from pathlib import Path
@@
def fernify_details_to_accordion(text: str) -> str:
"""Convert all <details>/<summary> blocks to Fern <Accordion> components."""
def _replace(m: re.Match) -> str:
- title = m.group(1)
+ title = html.unescape(m.group(1))
body = m.group(2).strip()
return render_accordion(title, body)Also applies to: 109-120, 145-153
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/scripts/_fern_helpers.py` around lines 16 - 17, render_details is
passing the raw captured summary (m.group(1)) into render_accordion which causes
a second HTML-escape; instead unescape the group before calling render_accordion
so the summary is only escaped once (e.g. use html.unescape(m.group(1)) or
otherwise pass the original unescaped summary to render_accordion). Update the
calls in render_details and the other similar sites (the blocks around lines
109-120 and 145-153) to unescape the captured summary before invoking
render_accordion.
Signed-off-by: Dan Gil <dagil@nvidia.com> Made-with: Cursor
This was referenced Mar 7, 2026
Collaborator
Author
|
Closing as stale. This work has been untouched for ~2 months and is superseded or no longer prioritized. Reopen if we pick it back up. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_fern_helpers.py: shared transforms (HTML<details>→<Accordion>, Markdown tables →<CardGroup>/<Card>, slugify, frontmatter injection)fernify_python_api.py: transforms Python API reference outputfernify_rust_api.py: transforms Rust API reference outputfernify_k8s_api.py: transforms K8s CRD reference (wraps API groups in<Tabs>, resource lists in<CardGroup>, type defs in<Accordion>)Design
The fernify scripts run at CI time (not at commit time). The raw Markdown checked into
docs/is GitHub-compatible, and CI transforms it into Fern MDX before syncing to thedocs-websitebranch. This means:Known Limitations
The fernify transforms rely on regex-based Markdown parsing, which is inherently fragile. We chose this approach because:
We acknowledge this is a v1 approach and plan to iterate as edge cases are discovered. Unit tests for the transform helpers (
_fern_helpers.py) will follow in a fast-follow PR.Expected CI Failure
The
Preview or publish docscheck fails on this PR because the fernify scripts expectdocs/api/python/README.mdanddocs/api/rust/README.mdto exist — these files are created by PR #7057. Once #7057 is merged first, this check will pass.PR Series (split from #6989)
Merge in order:
Test Plan
python3 docs/scripts/fernify_python_api.pytransformsdocs/api/python/README.mdcorrectlypython3 docs/scripts/fernify_rust_api.pytransformsdocs/api/rust/README.mdcorrectlypython3 docs/scripts/fernify_k8s_api.pytransformsdocs/kubernetes/api-reference.mdcorrectlyFollow-Up
_fern_helpers.pytransform functions (fast-follow PR)