docs: add API reference generators and navigation - #7057
Closed
dagil-nvidia wants to merge 4 commits into
Closed
Conversation
Add static-analysis-based API reference generation for Python and Rust: - generate_python_api.py: uses griffe to parse Python source and produce a consolidated Markdown reference with module/class/function docs - generate_rust_api.py: generates a crate overview linking to docs.rs - requirements-apidocs.txt: pip dependencies for the generators Generated output: - docs/api/python/README.md (5940 lines, auto-generated) - docs/api/rust/README.md (crate table with docs.rs links) Navigation and config: - docs/index.yml: reorder API Reference above Documentation, hide Blog - fern/fern.config.json: bump SDK version 3.73.0 -> 4.15.0 - deploy/operator/docs/header.md: add sidebar-title and max-toc-depth - docs/kubernetes/api-reference.md: add frontmatter from header Part 2 of 3 (split from #6989): 1. Docstrings (#7056) 2. API reference generators + navigation (this PR) 3. Fernify transform scripts + CI Signed-off-by: Dan Gil <dagil@nvidia.com> Made-with: Cursor
github-actions
Bot
added
docs
documentation
Contributor
dagil-nvidia
added a commit
that referenced
this pull request
Mar 7, 2026
Add fernify transform scripts that convert generated API reference Markdown into Fern MDX components for the documentation site: - _fern_helpers.py: shared transforms (details->Accordion, tables->Cards, slugify, frontmatter injection) - fernify_python_api.py: transforms Python API reference - fernify_rust_api.py: transforms Rust API reference - fernify_k8s_api.py: transforms K8s CRD reference (wraps API groups in Tabs, resource lists in CardGroups, type defs in Accordions) CI integration (.github/workflows/fern-docs.yml): - Add Python setup + pip install for fernify dependencies - Run all 3 fernify scripts before syncing to docs-website - Pin Rust docs.rs links to release version on tag pushes Part 3 of 3 (split from #6989): 1. Docstrings (#7056) 2. API reference generators + navigation (#7057) 3. Fernify transform scripts + CI (this PR) Signed-off-by: Dan Gil <dagil@nvidia.com> Made-with: Cursor
7 tasks
Contributor
WalkthroughThis pull request adds new API documentation generation infrastructure with Python and Rust API scripts, reorganizes the documentation navigation structure, creates a Rust API reference guide, adds metadata to documentation front matter, updates a requirements file for API documentation dependencies, and bumps the Fern configuration version. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
docs/scripts/requirements-apidocs.txt (1)
4-5: Pin the API-doc generator toolchain.
griffe>=1.0makes the checked-in Python API output depend on whichever 1.x release a contributor or CI happens to install. Sincegenerate_python_api.pyrelies on Griffe's parsed-docstring APIs, this will eventually cause unnecessary doc churn or break generation unexpectedly. Please lock the validated version here, or pair this file with a constraints/lock file used by the docs generation job.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@docs/scripts/requirements-apidocs.txt` around lines 4 - 5, The requirements entry for the API docs generator in docs/scripts/requirements-apidocs.txt currently uses a loose pin ("griffe>=1.0") which can cause unpredictable doc churn; update that requirement to a fixed, validated version (or add a companion constraints/lock file used by your docs generation job) so generate_python_api.py always runs against the exact Griffe release you validated—modify the griffe line in requirements-apidocs.txt accordingly and ensure CI/docs scripts reference the lock/constraints file if you choose that approach.docs/scripts/generate_rust_api.py (1)
136-145: Consider glob pattern support for workspace members.While the current
Cargo.tomluses only concrete directory paths, Cargo supports glob patterns inworkspace.members(e.g.,crates/*). The current code treats all members as literal paths and would fail if a glob were added. This is a defensive programming improvement rather than a current issue.Suggested fix
- members = list(dict.fromkeys(ws["workspace"]["members"])) + members: list[str] = [] + for entry in ws["workspace"]["members"]: + if any(ch in entry for ch in "*?["): + members.extend( + str(path.parent.relative_to(root)) + for path in sorted(root.glob(f"{entry}/Cargo.toml")) + ) + else: + members.append(entry) + members = list(dict.fromkeys(members))🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@docs/scripts/generate_rust_api.py` around lines 136 - 145, Workspace member entries in members can be glob patterns but the current loop treats them as literal paths; update the expansion before calling _load_toml by detecting glob patterns in members (e.g., presence of '*', '?', '[') and expanding them with root.glob(m) to collect matching directories (filter to directories that contain a Cargo.toml), preserve order/uniqueness (similar to dict.fromkeys behavior) and then iterate the expanded, filtered list so pkg = _load_toml(root / matched / "Cargo.toml") works for both literal paths and globs.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/scripts/generate_python_api.py`:
- Around line 806-827: The header currently builds cards for every MODULE_ORDER
entry, producing dead anchor links when modules are not present; modify
_render_header to first call _discover_modules() (or accept its returned module
list) and filter MODULE_ORDER to only iterate over discovered modules so the
cards list comprehension uses "for m in MODULE_ORDER if m in
discovered_modules"; ensure you still use _module_display_name, slugify,
MODULE_ICONS, MODULE_DESCRIPTIONS, and render_markdown_table as before; apply
the same filtering change to the similar code block around the section that
corresponds to lines 850-855.
---
Nitpick comments:
In `@docs/scripts/generate_rust_api.py`:
- Around line 136-145: Workspace member entries in members can be glob patterns
but the current loop treats them as literal paths; update the expansion before
calling _load_toml by detecting glob patterns in members (e.g., presence of '*',
'?', '[') and expanding them with root.glob(m) to collect matching directories
(filter to directories that contain a Cargo.toml), preserve order/uniqueness
(similar to dict.fromkeys behavior) and then iterate the expanded, filtered list
so pkg = _load_toml(root / matched / "Cargo.toml") works for both literal paths
and globs.
In `@docs/scripts/requirements-apidocs.txt`:
- Around line 4-5: The requirements entry for the API docs generator in
docs/scripts/requirements-apidocs.txt currently uses a loose pin ("griffe>=1.0")
which can cause unpredictable doc churn; update that requirement to a fixed,
validated version (or add a companion constraints/lock file used by your docs
generation job) so generate_python_api.py always runs against the exact Griffe
release you validated—modify the griffe line in requirements-apidocs.txt
accordingly and ensure CI/docs scripts reference the lock/constraints file if
you choose that approach.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 6f78f682-15f8-415f-b138-bf79504a03cc
📒 Files selected for processing (9)
deploy/operator/docs/header.mddocs/api/python/README.mddocs/api/rust/README.mddocs/index.ymldocs/kubernetes/api-reference.mddocs/scripts/generate_python_api.pydocs/scripts/generate_rust_api.pydocs/scripts/requirements-apidocs.txtfern/fern.config.json
Signed-off-by: Dan Gil <dagil@nvidia.com> Made-with: Cursor
Move griffe dependency from standalone requirements-apidocs.txt into [dependency-groups] docs in pyproject.toml, following the project's standard pattern. Pin to ~=2.0 (compatible release) since the generator script uses griffe 2.0 APIs. Signed-off-by: Dan Gil <dagil@nvidia.com> Made-with: Cursor Signed-off-by: Dan Gil <dagil@nvidia.com> Made-with: Cursor
3 tasks
Contributor
|
This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
Collaborator
Author
|
Closing as stale. This work has been untouched for ~2 months and is superseded or no longer prioritized. Reopen if we pick it back up. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
generate_python_api.py) using griffe for static analysis of Python source — no import requiredgenerate_rust_api.py) producing a crate overview table with docs.rs linksdocs/api/python/README.md(5940 lines) anddocs/api/rust/README.mdsidebar-title,max-toc-depth)griffe>=2.0to[dependency-groups] docsinpyproject.toml(removed standalonerequirements-apidocs.txt)Python API Generation Methodology
The Python generator uses griffe's
GriffeLoader(docstring_parser="auto")to statically parse Python source files. It auto-detects docstring style (Google, NumPy, Sphinx) and renders a consolidated Markdown page with:Docstrings were generated using Claude Opus 4.6 by comparing Python binding stubs (
.pyi) against Rust source implementations — see PR #7056. This PR will produce output regardless of whether #7056 has merged, but the generated docs are richer with docstrings present.PR Series (split from #6989)
Merge in order:
Test Plan
python3 docs/scripts/generate_python_api.pyproduces valid Markdownpython3 docs/scripts/generate_rust_api.pyproduces valid Markdown