diff --git a/fern/versions/v26.04/pages/about/release-notes/index.mdx b/fern/versions/v26.04/pages/about/release-notes/index.mdx index 373a5c9783..d6d44a5564 100644 --- a/fern/versions/v26.04/pages/about/release-notes/index.mdx +++ b/fern/versions/v26.04/pages/about/release-notes/index.mdx @@ -24,6 +24,13 @@ Built-in LLM serving alongside curation pipelines using Ray Serve and vLLM: Learn more in the [Inference Server](/curate-text/synthetic/inference-server) documentation. +### RayDataExecutor Promoted from Experimental (PR #1619) + +Moved `RayDataExecutor` out of the experimental namespace to `nemo_curator.backends.ray_data`. The executor is no longer marked as experimental, and the startup warning has been removed. Import path changes: + +- **Before**: `from nemo_curator.backends.experimental.ray_data import RayDataExecutor` +- **After**: `from nemo_curator.backends.ray_data import RayDataExecutor` + ### Shared Tokenizer Support for Multiple Classifiers (PR #1528) Text classifiers that share the same base tokenizer can now reuse tokens across a pipeline, avoiding redundant tokenization. New parameters `keep_tokens` and `use_existing_tokens` on all distributed classifiers control this behavior. DeBERTa-based classifiers (DomainClassifier, MultilingualDomainClassifier, QualityClassifier, ContentTypeClassifier, FineWeb variants, PromptTaskComplexityClassifier) and LlamaGuard-based classifiers (AegisClassifier, InstructionDataGuardClassifier) each form a compatible tokenizer group. @@ -184,6 +191,7 @@ Fixed a race condition in `CaptionGenerationStage` and `CaptionEnhancementStage` - **`TextSemanticDeduplicationWorkflow` embedding backend**: The default embedding backend changed from SentenceTransformers to vLLM. The default model changed from `sentence-transformers/all-MiniLM-L6-v2` to `google/embeddinggemma-300m`. The parameters `embedding_model_inference_batch_size`, `embedding_pooling`, `embedding_padding_side`, and `embedding_max_seq_length` have been removed. Use `embedding_vllm_init_kwargs` to pass configuration to the vLLM backend instead. - **`Resources` API**: The `nvdecs`, `nvencs`, and `entire_gpu` fields have been removed from `Resources`. Stages that previously used `entire_gpu=True` should use `gpus=1` instead. Stages that used `nvdecs` or `nvencs` should use `gpus` for GPU allocation. +- **`RayDataExecutor` import path**: Moved from `nemo_curator.backends.experimental.ray_data` to `nemo_curator.backends.ray_data`. Update imports accordingly. - **`DocumentExtractStage` Removed**: The standalone `DocumentExtractStage` class has been removed. Use `DocumentIterateExtractStage` with an optional `extractor` parameter instead. The `DocumentExtractor` abstract base class is unchanged. - **`DocumentIterateStage` Renamed**: `DocumentIterateStage` has been replaced by `DocumentIterateExtractStage`. Update imports from `nemo_curator.stages.text.download.base.iterator`. - **Three-Stage Pipeline**: The data acquisition pipeline is now a three-step pattern (URL generation → download → iterate-extract) instead of four steps. diff --git a/fern/versions/v26.04/pages/api-reference/executors/experimental.mdx b/fern/versions/v26.04/pages/api-reference/executors/experimental.mdx index b3a43d075a..70182c042e 100644 --- a/fern/versions/v26.04/pages/api-reference/executors/experimental.mdx +++ b/fern/versions/v26.04/pages/api-reference/executors/experimental.mdx @@ -9,35 +9,9 @@ NeMo Curator provides experimental executors for alternative execution backends. Experimental executors are subject to change and may not have full feature parity with `XennaExecutor`. -## RayDataExecutor - -Uses Ray Data for distributed execution. - -### Import - -```python -from nemo_curator.backends.experimental import RayDataExecutor -``` - -### Usage - -```python -executor = RayDataExecutor( - config={ - "ignore_failures": False, - }, - ignore_head_node=True, # Exclude head node from execution -) - -results = pipeline.run(executor=executor) -``` - -### Configuration - -| Option | Type | Default | Description | -|--------|------|---------|-------------| -| `ignore_failures` | `bool` | `False` | Continue on task failures | -| `ignore_head_node` | `bool` | `False` | Exclude head node from execution | + +`RayDataExecutor` was promoted from experimental in 26.04. Import it from `nemo_curator.backends.ray_data`. See [Pipeline Execution Backends](/reference/infrastructure/execution-backends) for details. + ## RayActorPoolExecutor @@ -153,7 +127,7 @@ class MyCustomExecutor(BaseExecutor): | Executor | Best For | Considerations | |----------|----------|----------------| | `XennaExecutor` | Production workloads | Default choice, most stable | -| `RayDataExecutor` | Ray-native environments | Experimental | +| `RayDataExecutor` | Ray-native environments | Promoted from experimental in 26.04 | | `RayActorPoolExecutor` | Fine-grained actor control | Experimental | ## Source Code diff --git a/fern/versions/v26.04/pages/curate-text/process-data/quality-assessment/heuristic.mdx b/fern/versions/v26.04/pages/curate-text/process-data/quality-assessment/heuristic.mdx index 1eba38e531..92f22af988 100644 --- a/fern/versions/v26.04/pages/curate-text/process-data/quality-assessment/heuristic.mdx +++ b/fern/versions/v26.04/pages/curate-text/process-data/quality-assessment/heuristic.mdx @@ -427,7 +427,7 @@ pipeline.add_stage(JsonlWriter(path="filtered_output/")) pipeline.run() # Or use Ray for distributed processing (see Performance Tuning section) -# from nemo_curator.backends.experimental.ray_data import RayDataExecutor +# from nemo_curator.backends.ray_data import RayDataExecutor # pipeline.run(RayDataExecutor(ignore_head_node=True)) ``` @@ -457,11 +457,11 @@ results = pipeline.run(executor) If no executor is specified, `pipeline.run()` uses `XennaExecutor` with default settings. - + `RayDataExecutor` provides distributed processing using Ray Data. It has shown performance improvements for filtering workloads compared to the default executor. ```python -from nemo_curator.backends.experimental.ray_data import RayDataExecutor +from nemo_curator.backends.ray_data import RayDataExecutor executor = RayDataExecutor( config={"ignore_failures": False}, diff --git a/fern/versions/v26.04/pages/reference/infrastructure/execution-backends.mdx b/fern/versions/v26.04/pages/reference/infrastructure/execution-backends.mdx index c24221c826..7d1d04e7cd 100644 --- a/fern/versions/v26.04/pages/reference/infrastructure/execution-backends.mdx +++ b/fern/versions/v26.04/pages/reference/infrastructure/execution-backends.mdx @@ -169,12 +169,28 @@ For more details, refer to [Text Deduplication ](/curate-text/process-data/dedup - **Scalable transformations**: Efficient map-batch operations across distributed workers ```python -from nemo_curator.backends.experimental.ray_data import RayDataExecutor +from nemo_curator.backends.ray_data import RayDataExecutor -executor = RayDataExecutor() +executor = RayDataExecutor( + config={"ignore_failures": False}, + ignore_head_node=True, # Exclude head node from computation +) results = pipeline.run(executor) ``` +**Constructor Parameters**: + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `config` | `dict` | `{}` | Configuration dictionary for Ray Data execution (see config keys below) | +| `ignore_head_node` | `bool` | `False` | Exclude the Ray cluster's head node from execution | + +**Config Dictionary Keys** (passed via `config={...}`): + +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| `ignore_failures` | `bool` | `False` | If `True`, continue pipeline execution even when tasks fail | + ## Per-Stage Runtime Environments All three backends support per-stage runtime environments, which allow individual stages to declare isolated Python dependencies. When a stage sets a `runtime_env`, the backend forwards it to Ray so that each stage's workers run in a dedicated virtualenv. This enables pipelines where stages require incompatible library versions.