docs: add release notes and docs for jusText OOM fix (PR #1534)

fern/versions/v26.04/pages/about/release-notes/index.mdx

@@ -83,6 +83,10 @@ The data acquisition pipeline now uses a three-stage architecture instead of fou
 - **Improved Memory Efficiency**: The fused stage processes records inline instead of materializing intermediate DataFrames, reducing peak memory usage. With limited RAM (200 GB), the Common Crawl pipeline succeeds with 32 CPUs where the unfused pipeline ran out of memory even at 16 CPUs.
 - **Better Performance**: Benchmarks show faster runtimes across both the Ray Data and Xenna executors (e.g., ~6% faster with Ray Data, ~18% faster with Xenna).
 
+### Worker Recycling for JusText Extraction (PR #1534)
+
+Added `max_calls_per_worker` support to mitigate out-of-memory errors caused by lxml/libxml2 memory fragmentation in long-running jusText extraction jobs. `CommonCrawlDownloadExtractStage` now defaults to `extractor_max_calls_per_worker=2` for `JusTextExtractor`, automatically recycling workers to reclaim fragmented memory.
+
 ### Per-Stage Runtime Environments (PR #1623)
 
 Pipeline stages can now declare isolated Python dependencies using Ray's native `runtime_env` support. Each stage can specify a different set of pip or uv packages, and Ray creates a cached virtualenv per unique dependency set so that incompatible library versions coexist in the same pipeline:
@@ -168,6 +172,10 @@ Confirmed full GPU utilization for the AEGIS safety classifier when running on m
 
 ## Bug Fixes
 
+### JusText Extraction OOM (PR #1534)
+
+Fixed out-of-memory errors during long-running jusText extraction jobs caused by lxml/libxml2 C-heap memory fragmentation. Worker recycling through `max_calls_per_worker` now prevents unbounded RSS growth by restarting worker processes periodically.
+
 ### CUDA Fork Error with vLLM and RayDataExecutor (PR #1606)
 
 Fixed a `RuntimeError: Cannot re-initialize CUDA in forked subprocess` error that occurred when running vLLM stages with `RayDataExecutor`. The vLLM auto-detection for `spawn` versus `fork` multiprocessing only triggers inside Ray actors, not Ray tasks. The `RayDataExecutor.execute_setup_on_node` method dispatches `setup_on_node` as a remote task, so vLLM previously defaulted to `fork` and caused a CUDA reinitialization error. Fixed by setting `VLLM_WORKER_MULTIPROC_METHOD=spawn` in the remote task.

fern/versions/v26.04/pages/curate-text/load-data/common-crawl.mdx

@@ -119,6 +119,7 @@ pipeline.add_stage(writer)
 | `url_limit` | int \| None | Maximum number of WARC files to download (useful for testing) | None |
 | `record_limit` | int \| None | Maximum number of records to extract per WARC file | None |
 | `add_filename_column` | bool \| str | Whether to add source filename column to output; if str, uses it as the column name (default name: "file_name") | True |
+| `extractor_max_calls_per_worker` | int \| None | Restart Ray Data worker processes after this many tasks to mitigate memory fragmentation. Auto-set to `2` for `JusTextExtractor`. | Auto (`2` for jusText, `None` otherwise) |
 
 ## Output Format
 
@@ -228,7 +229,7 @@ For production workloads, consider these optimizations:
 ```python
 cc_stage = CommonCrawlDownloadExtractStage(
     start_snapshot="2020-50",
-    end_snapshot="2020-50", 
+    end_snapshot="2020-50",
     download_dir="/fast_storage/cc_downloads",
     use_aws_to_download=True,  # Faster S3 downloads
     verbose=False,  # Reduce logging overhead
@@ -237,3 +238,31 @@ cc_stage = CommonCrawlDownloadExtractStage(
     # record_limit=None
 )
 ```
+
+### Memory Management for Extraction
+
+JusText extraction relies on lxml/libxml2, which can cause C-heap memory fragmentation during long-running jobs. Over many WARC files, this fragmentation causes resident memory to grow until workers run out of memory.
+
+To mitigate this, Curator automatically sets `extractor_max_calls_per_worker=2` when using `JusTextExtractor`. This restarts Ray Data worker processes every two tasks, reclaiming fragmented memory. You can override this value:
+
+```python
+# Increase recycling frequency for very memory-constrained environments
+cc_stage = CommonCrawlDownloadExtractStage(
+    start_snapshot="2020-50",
+    end_snapshot="2020-50",
+    download_dir="./downloads",
+    extractor_max_calls_per_worker=1,  # Recycle after every task
+)
+
+# Disable worker recycling (not recommended for large jobs with jusText)
+cc_stage = CommonCrawlDownloadExtractStage(
+    start_snapshot="2020-50",
+    end_snapshot="2020-50",
+    download_dir="./downloads",
+    extractor_max_calls_per_worker=None,  # No recycling
+)
+```
+
+<Note>
+Worker recycling is only supported with the Ray Data executor and applies to task-based stages (not actor-based). For custom extraction stages that use C libraries prone to memory fragmentation, set `max_calls_per_worker` on `DocumentIterateExtractStage` directly.
+</Note>

fern/versions/v26.04/pages/reference/infrastructure/memory-management.mdx

@@ -232,3 +232,4 @@ NeMo Curator integrates with Prometheus and Grafana for pipeline monitoring. Ref
    - **Clean Up Large Objects**: When working with large datasets in custom stages, explicitly delete temporary objects (e.g., `del large_dataframe`) and consider calling `gc.collect()` after processing large batches to free memory immediately rather than waiting for automatic garbage collection.
    - **GPU Memory**: For GPU-based stages, PyTorch may cache GPU memory. If you encounter GPU out-of-memory errors despite having sufficient GPU capacity, try `torch.cuda.empty_cache()` between stages to clear the cache.
    - **Worker Lifecycle**: Xenna automatically recycles workers periodically (controlled by `worker_max_lifetime_m` and `worker_restart_interval_m` in stage configs) to prevent memory leaks from accumulating during long-running pipelines.
+   - **Worker Recycling (Ray Data)**: For stages that use C libraries prone to heap fragmentation (such as jusText/lxml for HTML extraction), set `max_calls_per_worker` on `DocumentIterateExtractStage` to restart worker processes after a fixed number of tasks. `CommonCrawlDownloadExtractStage` automatically sets this to `2` for jusText extraction. Refer to the [Common Crawl](/curate-text/load-data/common-crawl#memory-management-for-extraction) guide for details.