Docs: fern migration - #1435

Closed

lbliii wants to merge 163 commits into

NVIDIA-NeMo:mainfrom

lbliii:llane/fern-docs-migration

Commits on Sep 22, 2025

text curation updates
lbliii
committed
de2b136
concepts
lbliii
committed
c520dde
Merge branch 'main' into llane/text-curation-updates
lbliii
authored
0bb66b0
remove synthetic docs not for this release
lbliii
committed
62bf887
updates
lbliii
committed
faaf621
text concepts and getting started changes
lbliii
committed
faf801a
links, concepts
lbliii
committed
c6d85e7
crosslinks
lbliii
committed
f46e9ae
quality assessment updates
lbliii
committed
54af361
more cleanup
lbliii
committed
1e8da76
semdedup
lbliii
committed
342b14f
example import cleanup
lbliii
committed
5c8e46f
concepts
lbliii
committed
89fc14d

Commits on Sep 23, 2025

Merge branch 'main' into llane/text-curation-updates
lbliii
authored
2d18203

Commits on Sep 24, 2025

Merge branch 'main' into llane/text-curation-updates
lbliii
authored
c81a102
Update docs/about/concepts/text/data-acquisition-concepts.md

lbliii
and
praateekmahajan
authored
29fa656
feedback batch 1
lbliii
committed
17261a5
feedback batch 2
lbliii
committed
bcff339
file_paths="/path/to/jsonl_directory",
lbliii
committed
6f6c733
revert removal of xenna for common crawl executors
lbliii
committed
545104e
Merge branch 'main' into llane/text-curation-updates
lbliii
authored
29b0f6b
quickstart installation steps
lbliii
committed
9099950
Update docs/about/concepts/text/data-acquisition-concepts.md

lbliii
and
sarahyurick
authored
7775532
data loading concepts updates / simplification
lbliii
committed
2454ea8

Commits on Sep 25, 2025

Merge branch 'main' into llane/text-curation-updates
lbliii
authored
baeb2a7
data processing feedback
lbliii
committed
3e1be2b
read-existing pg updates
lbliii
committed
1c45863
add-id updates
lbliii
committed
98f1d0f
dedup updates
lbliii
committed
7946c4e
feedback
lbliii
committed
0c5e675
Merge branch 'main' into llane/text-curation-updates
lbliii
authored
67d163d
Merge llane/text-curation-updates
lbliii
committed
a528229
gerge branch 'lbliii-llane/text-curation-updates'
lbliii
committed
9cf24e0
Merge branch 'main' of github.com:NVIDIA/NeMo-Curator
lbliii
committed
c950aa9

Commits on Feb 19, 2026

ci: Test latest uv and no build group first install (#1117)

thomasdhc
authored and
lbliii
committed
8bc3e2e
Audio curation improvements (#1113)

authored and
lbliii
committed
9eb14b7
Add iv2 install guide and guard import (#1112)

authored and
lbliii
committed
935ee9c
Enhance RayClient shutdown process by resetting RAY_ADDRESS (#1101)

abhinavg4
authored and
lbliii
committed
a9dae11
video doc improvement (#1124)

suiyoubi
authored and
lbliii
committed
28421b9
Add torchvision as dependency for image (#1131)

authored and
lbliii
committed
28f6711
about section updates (#1122)
lbliii
committed
b300de3
Update cherry-pick workflow to use v0.63.0 (#1137)

pablo-garay
authored and
lbliii
committed
1f5da7e
changelog (#1141)
lbliii
committed
5d869c6
Llane/docs image updates (#1087)
lbliii
committed
6ed06a3
text curation review feedback (#1129)

committed
de3a32e
Fix auto download models for images (#1140)

authored and
lbliii
committed
5e9c249
Llane/docs video feedback (#1146)
lbliii
committed
bed9806
Fix InternVideo2 doc and replace NVIDIA/NeMo-Curator with NVIDIA-NeMo/Curator (#1150)

committed
e347af8
Raise informative error when on unsupported system (#1126)

authored and
lbliii
committed
c4b3b1a
Fixing versions1.json to support nested folders (#1157)

aschilling-nv
authored and
lbliii
committed
370bfbf
Resolve a few warnings in the docs build (#1120)

authored and
lbliii
committed
f96cceb
Context manager support for ``RayClient`` (#1155)

authored and
lbliii
committed
fc5e416
added hydra params as defaults (#1159)

authored and
lbliii
committed
648812f
Increase timeout for CPU tests (#1162)

sarahyurick
authored and
lbliii
committed
b07dd68
Llane/docs readme updates (#1163)

lbliii
and
sarahyurick
committed
1a42754
add_codecov_badge (#1172)

pablo-garay
authored and
lbliii
committed
6227e75
Support more flags when building docs (#1175)

ayushdg
authored and
lbliii
committed
1225061
Fix bug for running filters on empty batches (#1173)

sarahyurick
authored and
lbliii
committed
9274b8e
docs: pyrpoject toml classifiers section (#1176)
lbliii
committed
bf4d2cf
docs: migration guide and faq (#1167)

lbliii
and
sarahyurick
committed
04fe990
citation file (#1165)

committed
a398f84
add enhanced captioning to rn; bump versions on json files (#1154)
lbliii
committed
88e071d
Add large file splitting script (#1161)

jrbourbeau
authored and
lbliii
committed
a88829b
Fix ``JsonlWriter`` / ``ParquetWriter`` output directory typo (#1180)

jrbourbeau
authored and
lbliii
committed
23d8f07
new extension: rich metadata for SEO (#1182)

lbliii
and
Copilot
committed
b3206cb
Llama Nemotron Data Curation tutorial (#1063)

authored and
lbliii
committed
1fea56f
Fix flaky ``test_split_parquet_file_by_size`` failure (#1185)

authored and
lbliii
committed
23823ad
Switch from pynvml to nvidia-ml-py (#1186)

jrbourbeau
authored and
lbliii
committed
0d23825
Remove ftfy pin (#1189)

sarahyurick
authored and
lbliii
committed
89baaf3
Increase UV_HTTP_TIMEOUT to reduce sporadic test failures (#1196)

jrbourbeau
authored and
lbliii
committed
5f8166f
Link to documentation within text tutorials (#1190)

sarahyurick
authored and
lbliii
committed
dd929a7
Bump ray to the latest version (#1195)

ayushdg
authored and
lbliii
committed
d843fa8
remove max_seq_len_to_capture (#1200)

suiyoubi
authored and
lbliii
committed
8fc2cf5
GLiNER PII Redaction tutorial (#1208)

sarahyurick
authored and
lbliii
committed
919771a
SDG Pipeline (#1215)

suiyoubi
authored and
lbliii
committed
3909b89
ci: Resolve starlette cve (#1219)

thomasdhc
authored and
lbliii
committed
954222f
Adds initial benchmarking framework (#1197)

rlratzel
authored and
lbliii
committed
6af4735
Small feedback updates to video curation (#1222)

sarahyurick
authored and
lbliii
committed
602524e
Bump rapids to 25.10 (#1204)

ayushdg
authored and
lbliii
committed
da4ec43
Update user-facing variable names and add override checks for `name`, `resources`, and `batch_size` (#1223)

sarahyurick
authored and
lbliii
committed
4f224c2
Update field names for classifiers (#1220)

authored and
lbliii
committed
0f1b98f
Bug fix in Text Semantic Dedup Worklfow to allow output_filetype to be jsonl #1230

praateekmahajan
authored and
lbliii
committed
324f8b1
Adds optional filename arg to capture ray stdout/stderr (#1239)

rlratzel
authored and
lbliii
committed
63d3b98
Add ability to ignore_head_node for RayDataExecutor and RayActorPoolExecutor (#1209)

praateekmahajan
authored and
lbliii
committed
a8ba762
Fix ParquetReader and *Writer for Cloud I/O (#1249)

praateekmahajan
authored and
lbliii
committed
4a82ed7
Fix Pairwise IO and IdentifyDuplicates in SemDedup for Cloud I/O (#1253)

praateekmahajan
authored and
lbliii
committed
eb5f118
Add tests for cloud I/O changes (#1257)

praateekmahajan
authored and
lbliii
committed
f1a0577
``ScoreFilter`` ``score_fn=`` keyword typo in text filtering docs (#1258)

jrbourbeau
authored and
lbliii
committed
131011b
Nemotron-CC SDG stages (#1245)

authored and
lbliii
committed
24dbe1c
Review and update text curation documentation (#1241)

committed
7f9abc0
Update execution-backends.md (#1263)

arhamm1
and
lbliii
committed
f28f812
Update container-environments.md (#1262)

arhamm1
and
lbliii
committed
df2f5d9
Update get-started/video.md (#1261)

arhamm1
and
lbliii
committed
42d8a42
Update audio-tutorials-beginner.md (#1252)

arhamm1
and
lbliii
committed
94db01e
Docs - Update curate-audio/process-data/quality-assessment/index.md (#1244)

arhamm1
and
lbliii
committed
c3374a0
Update curate-audio/process-data/text-integration/index.md (#1247)

committed
c8442c0
Update duration-filtering.md (#1243)

arhamm1
and
lbliii
committed
f0966a0
Update get-started/ image.md (#1240)

committed
419b8c7
Update getting started - text quickstart.md (#1238)

committed
a2520f7
Update getting started - index.md (#1237)

committed
6a497d4
Nemotron-CC SDG section pipelines (#1268)

authored and
lbliii
committed
264573c
ci: Unbound transformers in default dep (#1267)

thomasdhc
authored and
lbliii
committed
0f968bc
feat: Add pre-commit hooks for uv lock and export (#1277)

pablo-garay
authored and
lbliii
committed
8d0b308
Adds option to set object-store size when starting Ray cluster, uses a process group for managing ray child processes (#1274)

authored and
lbliii
committed
7decb5d
feat: CI perf improvement (manage bottlenecks) (#1285)

pablo-garay
authored and
lbliii
committed
96e6fc5
Update Memory Management Guide (#1279)

sarahyurick
and
lbliii
committed
a8219ce
Adds FuzzyDedup identification and Removal benchmarks (#1233)

rlratzel
authored and
lbliii
committed
98e0277
ci: Create contrainst for ray for CVE (#1286)

authored and
lbliii
committed
cf449c2
Download Hugging Face datasets for text classification tutorials (#1288)

sarahyurick
authored and
lbliii
committed
b87379a
Update wer-filtering.md (#1246)

arhamm1
and
lbliii
committed
08453b9
Enhance ASR documentation and code examples (#1276)

abhinavg4
and
lbliii
committed
8d8d02e
ci: Revert cicd cancellation on failure (#1298)

thomasdhc
authored and
lbliii
committed
d1b6e36
bump urllib3 due to GHSA-gm62-xv2j-4w53 (#1306)

ayushdg
authored and
lbliii
committed
11388a5
ci: Disable uv caching for unit test (#1323)

thomasdhc
authored and
lbliii
committed
a597ddf
Add Cursor rules (#1294)

sarahyurick
authored and
lbliii
committed
be3634f
feat: Add dependabot workflow for periodic lock file updates v2 (#1307)

pablo-garay
authored and
lbliii
committed
0c788d8
ci: Update vllm and torch to address CVE (#1287)

thomasdhc
authored and
lbliii
committed
a9aa663
Inital tutorial for e2e fuzzy deduplication (#1242)

ayushdg
authored and
lbliii
committed
968732d
Collision warning code (#1325)

authored and
lbliii
committed
b9126eb
[Doc Review 25.09] Huy - Video Concepts + Text Advanced (#1270)

authored and
lbliii
committed
fffbace
feat: add install-test v3 (#1335)

pablo-garay
authored and
lbliii
committed
8efe410
ci: Skip video cuda12 and pip for extra all (#1338)

thomasdhc
authored and
lbliii
committed
d0d8c1c
Review Image Documentations (#1272)

authored and
lbliii
committed
e6a3caa
Megatron tokenization pipeline (#1259)

authored and
lbliii
committed
7fab03e
Fix ID generator to be non-blocking (#1344)

authored and
lbliii
committed
666e9ec
Specify `allow_module_level=True` in InternVideo tests (#1354)

sarahyurick
authored and
lbliii
committed
b4ac62f
ci: Always trigger NeMo CICD test stage (#1353)

thomasdhc
authored and
lbliii
committed
a14dbfb
[benchmarking] Updates to support multiple nightly runs, code cleanup (#1301)

authored and
lbliii
committed
1eebb85
Remove nvenc/dec for xenna 0.1.6 (#1202)

authored and
lbliii
committed
162a768
Isolate modality tests with PyTest markers (#1293)

sarahyurick
authored and
lbliii
committed
a5baf25
Add YAML support for text pipelines (#1212)

authored and
lbliii
committed
bc85ff2
feat: FFmpeg to 8.0.1 (#1362)

suiyoubi
authored and
lbliii
committed
d2b98c6
ci: Bump version to 1.1.0 (#1364)

thomasdhc
authored and
lbliii
committed
13be895
Adding one worker per partition to FilePartioningStage and URLGeneratorStage (#1350)

authored and
lbliii
committed
44864c6
Add workflow results (#1275)

praateekmahajan
authored and
lbliii
committed
4bf949a
Add vLLM and Sentence Transformers support for embedding generation (#1346)

praateekmahajan
authored and
lbliii
committed
e148326
[benchmarking] Adds image curation benchmark to nightly (#1341)

rlratzel
authored and
lbliii
committed
e22dbc4
[benchmarking] Adds audio curation benchmark to nightly (#1360)

rlratzel
authored and
lbliii
committed
c0df452
Fix bug in SDG example (#1370)

sarahyurick
authored and
lbliii
committed
fbde2cd
ferm docs init
lbliii
committed
a2b1863
broken link batch
lbliii
committed
299bcee
link fixes
lbliii
committed
209c4fa
updates
lbliii
committed
8df9853
update
lbliii
committed
6ae0906
update
lbliii
committed
ab145c2
update
lbliii
committed
9d774be
update
lbliii
committed
c4eb63a
style updates
lbliii
committed
aa2e33f

Commits on Mar 3, 2026

Commits on Mar 16, 2026

updates
lbliii
committed
9559c99

Commits on Mar 23, 2026

fix "/index" routes
lbliii
committed
7784a58
updates
lbliii
committed
aa55fae
updates
lbliii
committed
66ebe29
updates
lbliii
committed
3879af8
updates
lbliii
committed
0d65e51
fix
lbliii
committed
778b0c7
fix links
lbliii
committed
800ef64
card link fixes
lbliii
committed
3f8ec59
mass link fixes
lbliii
committed
12db1e5
broken link bash
lbliii
committed
d2bfb23