Add RULERv2

docs/evaluation/long-context.md

@@ -26,6 +26,47 @@ Other supported options
   * **base**: evaluate base model with answer prefix.
   * **chat**: evaluate chat model including non-reasoning and reasoning model without answer prefix.
 
+### ruler2
+- Benchmark is defined in [`nemo_skills/dataset/ruler2/__init__.py`](https://github.com/NVIDIA-NeMo/Skills/blob/main/nemo_skills/dataset/ruler2/__init__.py)
+
+It's recommended to use [data_dir parameter](../evaluation/index.md#using-data-on-cluster) when running evaluation.
+Ruler2 also requires `setup`, `tokenizer_path` and `max_seq_length` to be specified. Example command to prepare data
+
+```bash
+ns prepare_data ruler2 \
+    --cluster=<cluster config> \
+    --data_dir=<mounted location to store data into> \
+    --setup=<typically MODEL_NAME-LENGTH but can be any string> \
+    --tokenizer_path=<model name, e.g. Qwen/Qwen3-1.7B> \
+    --max_seq_length=<length you want to evaluate, e.g. 131072>
+```
+
+Example evaluation command
+
+```bash
+ns eval \
+    --cluster=<cluster config> \
+    --data_dir=<must match prepare_data parameter> \
+    --output_dir=<any mounted output location> \
+    --benchmarks=ruler2.<what you used for prepare_data setup argument> \
+    --model=<model name, e.g. Qwen/Qwen3-1.7B> \
+    --server_nodes=1 \
+    --server_gpus=8 \
+    --server_type=vllm
+```
+
+Example scores
+
+| Model                                   | Avg  | 8192 | 16384 | 32768 | 65536 | 131072 | 262144 | 524288 | 1000000 |
+|-----------------------------------------|------|------|-------|-------|-------|--------|--------|--------|---------|
+| Gemini 2.5 Flash Think On               | 91.4 | 94.3 | 93.7  | 91.4  | 88.4  | 89.0   | -      | -      | -       |
+| Gemini 2.5 Flash Think Off              | 88.0 | 91.3 | 89.0  | 88.8  | 85.5  | 85.5   | 82.5   | 79.1   | 77.0    |
+| GPT 4.1                                 | 89.2 | 91.2 | 90.8  | 89.8  | 87.7  | 86.5   | 80.6   | 74.5   | 75.2    |
+| Qwen3-235B-A22B-Thinking-2507           | 85.2 | 92.9 | 91.3  | 85.3  | 80.6  | 75.7   | -      | -      | -       |
+| Qwen3-235B-A22B-Instruct-2507           | 83.7 | 87.3 | 85.8  | 84.5  | 82.5  | 78.2   | 65.3   | 53.0   | 36.1    |
+
+For more details see [https://github.com/NVIDIA/RULER/blob/rulerv2-ns](https://github.com/NVIDIA/RULER/blob/rulerv2-ns/)
+
 ### mrcr
 
 - Benchmark is defined in [`nemo_skills/dataset/mrcr/__init__.py`](https://github.com/NVIDIA-NeMo/Skills/blob/main/nemo_skills/dataset/mrcr/__init__.py)

nemo_skills/dataset/ruler2/__init__.py

@@ -0,0 +1,15 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+DATASET_GROUP = "long-context"