docs: document multi-user metrics isolation (PR #1523)

fern/versions/v26.04.yml

@@ -504,6 +504,9 @@ navigation:
               - page: Memory Management
                 path: ./v26.04/pages/reference/infrastructure/memory-management.mdx
                 slug: memory-management
+              - page: Monitoring
+                path: ./v26.04/pages/reference/infrastructure/monitoring.mdx
+                slug: monitoring
               - page: GPU Processing
                 path: ./v26.04/pages/reference/infrastructure/gpu-processing.mdx
                 slug: gpu-processing

fern/versions/v26.04/pages/about/release-notes/index.mdx

@@ -8,139 +8,18 @@ content_type: "reference"
 modality: "universal"
 ---
 
-# NeMo Curator Release Notes: 26.02
+# NeMo Curator Release Notes: 26.04
 
-## What's New in 26.02
+## What's New in 26.04
 
-### Benchmarking Infrastructure
+### Multi-User Metrics Isolation
 
-New comprehensive benchmarking framework for performance monitoring and optimization:
+Improved Prometheus and Grafana monitoring support for shared clusters:
 
-- **End-to-End Pipeline Benchmarking**: Automated benchmarks for all curation modalities (text, image, video, audio)
-- **Performance Tracking**: Integration with MLflow for metrics tracking and Slack for notifications
-- **Nightly Benchmarks**: Continuous performance monitoring across:
-  - Text pipelines: exact deduplication, fuzzy deduplication, semantic deduplication, score filters, modifiers
-  - Image curation workflows with DALI-based processing
-  - Video processing pipelines with scene detection and semantic deduplication
-  - Audio ASR inference and quality assessment
-- **Grafana Dashboards**: Real-time monitoring of pipeline performance and resource utilization
+- **Per-user metrics directories**: The default metrics path now includes the user ID (`/tmp/nemo_curator_metrics_{uid}`), which prevents conflicts when multiple users share a Ray cluster.
+- **`metrics_dir` parameter**: New parameter on `RayClient` and `start_prometheus_grafana.py` for explicit control of where metrics data, PID files, and configuration are stored.
+- **PID-file-based process tracking**: NeMo Curator tracks Prometheus and Grafana instances through PID files instead of process-name scanning, which enables multiple isolated monitoring instances per node.
+- **Automatic Ray dashboard generation**: The Grafana setup now auto-generates Ray default, data, serve, and serve-deployment dashboards from the built-in Ray dashboard factory.
+- **Graceful cleanup on shutdown**: `RayClient.stop()` now removes Ray service discovery entries from the Prometheus configuration automatically.
 
-### Ray Actor Pool Executor Improvements
-
-Enhanced features for the experimental Ray Actor Pool execution backend:
-
-- **Progress Bars**: New visual feedback for long-running actor pool operations, making it easier to monitor pipeline execution
-- **Improved Load Balancing**: Better worker distribution and task scheduling
-- **Enhanced Stability**: Continued refinements to the experimental executor
-
-Learn more in the [Execution Backends documentation](/reference/infra/execution-backends).
-
-### YAML Configuration Support
-
-Declarative pipeline configuration for text curation workflows:
-
-- **YAML-Based Pipelines**: Define entire curation pipelines in YAML configuration files
-- **Pre-Built Configurations**: Ready-to-use configs for common workflows:
-  - Code filtering, exact/fuzzy/semantic deduplication
-  - Heuristic filtering (English and non-English)
-  - FastText language identification
-- **Reproducible Workflows**: Version-controlled pipeline definitions for consistent results
-
-Example:
-```bash
-python -m nemo_curator.config.run --config_file heuristic_filter_english_pipeline.yaml
-```
-
-### Workflow Results API
-
-New API for tracking and analyzing pipeline execution:
-
-- **Performance Metrics**: Automatic tracking of processing time, throughput, and resource usage
-- **Better Debugging**: Detailed logs and error reporting for failed stages
-
-## Improvements from 25.09
-
-### Video Curation
-
-- **Model Updates**: Removed InternVideo2 dependency; updated to more performant alternatives
-- **vLLM 0.14.1**: Upgraded for better video captioning compatibility and performance
-- **FFmpeg 8.0.1**: Latest FFmpeg with improved codec support and performance
-- **Enhanced Tutorials**: Improved video processing examples with real-world scenarios
-
-### Audio Curation
-
-- **Enhanced Documentation**: Comprehensive ASR inference and quality assessment guides
-- **Improved WER Filtering**: Better guidance for Word Error Rate filtering thresholds
-- **Manifest Handling**: More robust JSONL manifest processing for large audio datasets
-
-### Image Curation
-
-- **Optimized Batch Sizes**: Reduced default batch sizes for better CPU memory usage (batch_size=50, num_threads=4)
-- **Memory Guidance**: Added troubleshooting documentation for out-of-memory errors
-- **Tutorial Improvements**: Updated examples optimized for typical GPU configurations
-
-### Text Curation
-
-- **Better Memory Management**: Improved handling of large-scale semantic deduplication
-- **Small Cluster Warnings**: Automatic warnings when n_clusters is too small for effective deduplication
-- **FilePartitioning Improvements**: One worker per partition for better parallelization
-
-### Deduplication Enhancements
-
-- **Cloud Storage Support**: Fixed ParquetReader/Writer and pairwise I/O for S3, GCS, and Azure Blob
-- **Non-Blocking ID Generation**: Improved ID generator performance for large datasets
-- **Empty Batch Handling**: Better error handling for filters processing empty data batches
-
-## Dependency Updates
-
-- **Transformers**: Pinned to 4.55.2 for stability and compatibility
-- **vLLM**: Updated to 0.14.1 with video pipeline compatibility fixes
-- **FFmpeg**: Upgraded to 8.0.1 for enhanced multimedia processing
-- **Security Patches**:
-  - Addressed CVEs in aiohttp, urllib3, python-multipart, setuptools
-  - Removed vulnerable thirdparty aiohttp file from Ray
-  - Updated to secure dependency versions
-
-## Bug Fixes
-
-- Fixed fasttext predict call compatibility with numpy>2
-- Fixed broken NeMo Framework documentation links
-- Fixed MegatronTokenizerWriter to download only necessary tokenizer files
-- Fixed ID generator blocking issues for large-scale processing
-- Fixed vLLM API compatibility with video captioning pipeline
-- Fixed Gliner tutorial examples and SDG workflow bugs
-- Improved semantic deduplication unit test reliability
-
-## Infrastructure & Developer Experience
-
-- **Secrets Detection**: Automated secret scanning in CI/CD workflows
-- **Dependabot Integration**: Automatic dependency update pull requests
-- **Enhanced Install Tests**: Comprehensive installation validation across environments
-- **AWS Runner Support**: CI/CD execution on AWS infrastructure
-- **Docker Optimization**: Improved layer caching and build times with uv
-- **Code Linting**: Standardized code quality checks with markdownlint and pre-commit hooks
-- **Cursor Rules**: Development guidelines and patterns for IDE assistance
-
-## Breaking Changes
-
-- **InternVideo2 Removed**: Video pipelines must use alternative embedding models (Cosmos-Embed1)
-
-## Documentation Improvements
-
-- **Heuristic Filter Guide**: Comprehensive documentation for language-specific filtering strategies
-- **Distributed Classifier**: Enhanced GPU memory optimization guidance with length-based sequence sorting
-- **Installation Guide**: Clearer instructions with troubleshooting for common issues
-- **Memory Management**: New guidance for handling CPU/GPU memory constraints
-- **AWS Integration**: Updated tutorials with correct AWS credentials setup
-
----
-
-## What's Next
-
-Future releases will focus on:
-
-- **Code Curation**: Specialized pipelines for curating code datasets
-- **Math Curation**: Mathematical reasoning and problem-solving data curation
-- **Generation Features**: Completing the Ray refactor for synthetic data generation
-- **PII Processing**: Enhanced privacy-preserving data curation with Ray backend
-- **Blending & Shuffling**: Large-scale multi-source dataset blending and shuffling operations
+Learn more in the [Monitoring](/reference/infra/monitoring) documentation.

fern/versions/v26.04/pages/reference/infrastructure/memory-management.mdx

@@ -212,63 +212,15 @@ If you encounter a `torch.OutOfMemoryError` during model classification, it is a
 
 ## Memory Monitoring
 
-### Why Monitor Memory?
+Monitoring memory is essential for production data curation pipelines, especially when processing large-scale datasets over extended periods. Without monitoring, you may encounter silent performance degradation, unexpected out-of-memory failures, resource waste, and difficult-to-debug crashes.
 
-Memory monitoring is essential for production data curation pipelines, especially when processing large-scale datasets over extended periods. Without monitoring, you may encounter:
-
-- **Silent Performance Degradation**: Memory leaks can gradually slow down processing without obvious errors
-- **Unexpected Failures**: Out-of-memory crashes can occur hours or days into long-running jobs
-- **Resource Waste**: Underutilized workers consume resources without contributing to throughput
-- **Difficult Debugging**: Without historical data, it's hard to identify which pipeline stage caused a memory issue
-
-Effective monitoring helps you:
-- **Detect Issues Early**: Identify memory leaks or inefficient stages before they cause failures
-- **Optimize Resource Allocation**: Adjust worker counts and batch sizes based on actual usage patterns
-- **Plan Capacity**: Understand resource requirements for scaling to larger datasets
-- **Debug Failures**: Investigate what happened leading up to a crash using historical metrics
-
-### Monitoring Stack: Prometheus and Grafana
-
-NeMo Curator supports integration with **Prometheus** and **Grafana**, the industry-standard open-source monitoring stack:
-
-**Prometheus** is a time-series database and monitoring system that:
-- Collects metrics from your pipeline at regular intervals (for example, every 15 seconds)
-- Stores metrics like CPU usage, GPU memory, worker counts, and task throughput
-- Provides a query language (PromQL) to aggregate and analyze metrics
-- Runs as a standalone service that "scrapes" metrics exposed by Curator workers
-
-**Grafana** is a visualization platform that:
-- Connects to Prometheus as a data source
-- Displays metrics in customizable dashboards with graphs, gauges, and alerts
-- Provides real-time views of your pipeline's health and performance
-- Allows you to set up alerts (for example, notify when GPU memory exceeds 90%)
-
-**How They Work Together**:
-1. Curator workers expose metrics in a format Prometheus understands
-2. Prometheus periodically scrapes these metrics and stores them
-3. Grafana queries Prometheus and displays the data in dashboards
-4. You view the dashboards to monitor your pipeline in real-time and historically
-
-### Key Metrics to Monitor
-
-When running production pipelines, track these critical metrics:
-
-- **CPU Memory Usage**: Total RAM consumption across workers to prevent out-of-memory errors
-- **GPU Memory Usage**: VRAM consumption per GPU for model-based stages (classifiers, embedders)
-- **Worker Count**: Number of active workers per stage to verify proper scaling
-- **Task Throughput**: Documents or batches processed per second to measure pipeline efficiency
-- **Stage Latency**: Time spent in each pipeline stage to identify bottlenecks
-- **Error Rates**: Failed tasks or worker crashes to detect stability issues
-
-### Setting Up Monitoring
-
-Refer to [NeMo Curator Metrics](https://github.com/NVIDIA-NeMo/Curator/tree/main/nemo_curator/metrics) for information about how to use Prometheus and Grafana with NeMo Curator.
+NeMo Curator integrates with Prometheus and Grafana for pipeline monitoring. Refer to the [Monitoring](/reference/infra/monitoring) page for setup instructions, key metrics to track, and multi-user cluster configuration.
 
 ## Best Practices
 
 1. **Monitor Memory Usage**
    - **During Development** Use system monitoring tools (`htop`, `nvidia-smi`, `watch -n 1 nvidia-smi`) to observe memory usage patterns as your pipeline runs. Start with small datasets to identify memory bottlenecks before scaling up.
-   - **In Production** Set up monitoring dashboards using Prometheus and Grafana (refer to [Memory Monitoring](#memory-monitoring) section above) to track CPU/GPU memory usage, worker utilization, and pipeline throughput over time.
+   - **In Production** Set up monitoring dashboards using Prometheus and Grafana (refer to [Monitoring](/reference/infra/monitoring)) to track CPU/GPU memory usage, worker utilization, and pipeline throughput over time.
    - **Ray Dashboard** If using Ray-based executors, access the Ray dashboard (typically at `http://localhost:8265`) to view real-time resource usage, task execution, and memory consumption across workers.
 
 2. **Optimize Data Loading**

fern/versions/v26.04/pages/reference/infrastructure/monitoring.mdx

@@ -0,0 +1,104 @@
+---
+description: "Set up Prometheus and Grafana to monitor NeMo Curator pipelines on single-user and multi-user Ray clusters"
+categories: ["reference"]
+tags: ["monitoring", "prometheus", "grafana", "ray", "observability", "metrics"]
+personas: ["mle-focused", "admin-focused"]
+difficulty: "intermediate"
+content_type: "reference"
+modality: "universal"
+---
+
+# Monitoring
+
+NeMo Curator supports integration with **Prometheus** and **Grafana**, the industry-standard open-source monitoring stack.
+
+**Prometheus** is a time-series database and monitoring system that:
+- Collects metrics from your pipeline at regular intervals (for example, every 15 seconds).
+- Stores metrics like CPU usage, GPU memory, worker counts, and task throughput.
+- Provides a query language (PromQL) to aggregate and analyze metrics.
+- Runs as a standalone service that scrapes metrics exposed by Curator workers.
+
+**Grafana** is a visualization platform that:
+- Connects to Prometheus as a data source.
+- Displays metrics in customizable dashboards with graphs, gauges, and alerts.
+- Provides real-time views of your pipeline's health and performance.
+- Allows you to set up alerts (for example, notify when GPU memory exceeds 90%).
+
+**How They Work Together**:
+1. Curator workers expose metrics in a format Prometheus understands.
+2. Prometheus periodically scrapes these metrics and stores them.
+3. Grafana queries Prometheus and displays the data in dashboards.
+4. You view the dashboards to monitor your pipeline in real time and historically.
+
+## Key Metrics to Monitor
+
+When running production pipelines, track these critical metrics:
+
+- **CPU Memory Usage**: Total RAM consumption across workers to prevent out-of-memory errors.
+- **GPU Memory Usage**: VRAM consumption per GPU for model-based stages (classifiers, embedders).
+- **Worker Count**: Number of active workers per stage to verify proper scaling.
+- **Task Throughput**: Documents or batches processed per second to measure pipeline efficiency.
+- **Stage Latency**: Time spent in each pipeline stage to identify bottlenecks.
+- **Error Rates**: Failed tasks or worker crashes to detect stability issues.
+
+## Setting Up Monitoring
+
+NeMo Curator provides a built-in script to download, configure, and start Prometheus and Grafana. When `RayClient` starts, it automatically registers Ray's metrics with the running Prometheus instance.
+
+### 1. Start Prometheus and Grafana
+
+On the head node, run the setup script before starting your pipeline:
+
+```bash
+python -m nemo_curator.metrics.start_prometheus_grafana
+```
+
+This downloads Prometheus and Grafana binaries, writes configuration files, starts both services, and auto-generates Ray dashboards (default, data, serve, and serve-deployment).
+
+By default, the script stores metrics data in a per-user directory (`/tmp/nemo_curator_metrics_{uid}`). To specify a custom location, use `--metrics_dir`:
+
+```bash
+python -m nemo_curator.metrics.start_prometheus_grafana --metrics_dir /shared/metrics/user_a
+```
+
+You can also customize ports:
+
+```bash
+python -m nemo_curator.metrics.start_prometheus_grafana \
+  --prometheus_web_port 9090 \
+  --grafana_web_port 3000 \
+  --metrics_dir /shared/metrics/user_a
+```
+
+### 2. Configure RayClient
+
+Pass the same `metrics_dir` to `RayClient` so it can locate the running Prometheus instance and register Ray's service discovery:
+
+```python
+from nemo_curator.core.client import RayClient
+
+ray_client = RayClient(
+    include_dashboard=True,
+    metrics_dir="/shared/metrics/user_a"  # Must match --metrics_dir from step 1
+)
+ray_client.start()
+```
+
+When `RayClient.start()` runs, it checks whether Prometheus and Grafana are running (through PID files in `metrics_dir`) and, if so, adds Ray's metrics service discovery to the Prometheus configuration. When `RayClient.stop()` runs, it removes the service discovery entry automatically.
+
+<Note>
+**SLURM clusters**: The `metrics_dir` passed to `RayClient` must match the `--metrics_dir` used when starting Prometheus and Grafana. Similarly, the `ray_temp_dir` must match the `temp_dir` used when starting the Ray cluster.
+</Note>
+
+### Multi-User Clusters
+
+On shared clusters, each user can run an isolated monitoring stack by specifying a unique `metrics_dir`. Each instance tracks its own Prometheus and Grafana processes through PID files, so multiple users can monitor their pipelines independently on the same node.
+
+### Stopping Monitoring Services
+
+To stop Prometheus and Grafana, use the PID files written to your metrics directory:
+
+```bash
+kill $(cat /shared/metrics/user_a/prometheus.pid)
+kill $(cat /shared/metrics/user_a/grafana.pid)
+```