BIRD Benchmark (Text-to-SQL)

docs/evaluation/code.md

@@ -316,6 +316,48 @@ Due to variance between runs, you can automatically repeat the evaluation and av
 --benchmarks=livecodebench:3
 ```
 
+### BIRD
+
+The [BIRD benchmark](https://bird-bench.github.io/) is currently the only text-to-SQL benchmark that is supported. Evaluation is based on the SQL evaluation accuracy calculated in [this file](https://github.com/AlibabaResearch/DAMO-ConvAI/blob/main/bird/llm/src/evaluation.py) provided in the BIRD GitHub repository.
+
+#### Data Preparation
+
+
+First, the data must be downloaded and prepared, which you can do by running:
+```bash
+ns prepare_data birdbench
+```
+
+This will download and unpack a file into `<output_directory>/dev_20240627`, which contains the BIRD dev manifest, table information, and database schemas. By default, `output_directory` will be under `nemo_skills/dataset/birdbench/`, though this can be changed via command line argument.
+
+The script will also process the original manifest into `<output_directory>/dev.jsonl`, which will be the input for evaluation.
+
+#### Running the Evaluation
+
+The following command runs an evaluation of [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) on a Slurm cluster.
+
+```bash
+ns eval \
+     --cluster=<CLUSTER_NAME> \
+     --server_type='sglang' \
+     --server_gpus=8 \
+     --model=Qwen/Qwen3-8B \
+     --benchmarks=birdbench \
+     --output_dir=<OUTPUT_DIR> \
+     --installation_command="pip install func_timeout" \
+     ++inference.tokens_to_generate=10000 \
+     ++inference.temperature=0.6 \
+     ++inference.top_p=0.95 \
+     ++inference.top_k=20 \
+     ++max_concurrent_requests=1024 \
+     ++eval_config.dev_json_filepath=<DATA_DIR>/dev.json \
+     ++eval_config.db_path=<DATA_DIR>/dev_databases
+```
+!!! note
+    BIRD evaluation requires the `func_timeout` library, so you will need to add the installation argument `--installation_command` seen in this example if it is not included in your container.
+
+The arguments for evaluation also include paths for `++eval_config.dev_json_filepath` and `++eval_config.db_path`. These are used for SQL execution. You should point these to the unmodified `dev.json` file and `dev_databases` directory found in the original downloaded and unzipped data (i.e. `.../dev_20240627/dev.json`, etc.). If using a Slurm cluster, you may need to upload and mount the data directory first.
+
 ### livecodebench-cpp
 
 - Benchmark is defined in [`nemo_skills/dataset/livecodebench-cpp/__init__.py`](https://github.com/NVIDIA-NeMo/Skills/blob/main/nemo_skills/dataset/livecodebench-cpp/__init__.py)

docs/index.md

@@ -16,7 +16,7 @@ Here are some of the features we support:
     - Evaluate your models on many popular benchmarks.
         - [**Math (natural language**)](./evaluation/natural-math.md): e.g. [aime24](./evaluation/natural-math.md#aime24), [aime25](./evaluation/natural-math.md#aime25), [hmmt_feb25](./evaluation/natural-math.md#hmmt_feb25)
         - [**Math (formal language)**](./evaluation/formal-math.md): e.g. [minif2f](./evaluation/formal-math.md#minif2f), [proofnet](./evaluation/formal-math.md#proofnet), [putnam-bench](./evaluation/formal-math.md#putnam-bench)
-        - [**Code**](./evaluation/code.md): e.g. [swe-bench](./evaluation/code.md#swe-bench), [livecodebench](./evaluation/code.md#livecodebench)
+        - [**Code**](./evaluation/code.md): e.g. [swe-bench](./evaluation/code.md#swe-bench), [livecodebench](./evaluation/code.md#livecodebench), [bird](./evaluation/code.md#BIRD)
         - [**Scientific knowledge**](./evaluation/scientific-knowledge.md): e.g., [hle](./evaluation/scientific-knowledge.md#hle), [scicode](./evaluation/scientific-knowledge.md#scicode), [gpqa](./evaluation/scientific-knowledge.md#gpqa)
         - [**Instruction following**](./evaluation/instruction-following.md): e.g. [ifbench](./evaluation/instruction-following.md#ifbench), [ifeval](./evaluation/instruction-following.md#ifeval)
         - [**Long-context**](./evaluation/long-context.md): e.g. [ruler](./evaluation/long-context.md#ruler), [mrcr](./evaluation/long-context.md#mrcr)

nemo_skills/dataset/birdbench/__init__.py

@@ -0,0 +1,29 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# settings that define how evaluation should be done by default (all can be changed from cmdline)
+DATASET_GROUP = "code"
+METRICS_TYPE = "bird"
+EVAL_SPLIT = "dev"
+GENERATION_ARGS = (
+    "++prompt_config=generic/text_to_sql "
+    "++eval_type=bird "
+    "++inference.tokens_to_generate=10000 "
+    "++inference.temperature=0.6 "
+    "++inference.top_p=0.95 "
+    "++inference.top_k=20 "
+    "++max_concurrent_requests=1024 "
+    # Note: Override this path via ++eval_config.db_path
+    "++eval_config.db_path=/sql_data/BIRD/dev_databases"
+)

nemo_skills/dataset/birdbench/evaluation.py

@@ -0,0 +1,54 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Original code from https://github.com/AlibabaResearch/DAMO-ConvAI/blob/main/bird/llm/src/evaluation.py
+
+# Original license as follows:
+
+# MIT License
+#
+# Copyright (c) 2022 Alibaba Research
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+import sqlite3
+
+
+def execute_sql(predicted_sql, ground_truth, db_path):
+    # Connect to the database
+    with sqlite3.connect(db_path) as conn:
+        cursor = conn.cursor()
+        cursor.execute(predicted_sql)
+        predicted_res = cursor.fetchall()
+        cursor.execute(ground_truth)
+        ground_truth_res = cursor.fetchall()
+        res = 0
+        if set(predicted_res) == set(ground_truth_res):
+            res = 1
+    return res

nemo_skills/dataset/birdbench/prepare.py

@@ -0,0 +1,129 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import glob
+import json
+import os
+import re
+import sqlite3
+import zipfile
+from pathlib import Path
+
+import wget
+
+
+def download_data(output_dir):
+    # Download zip directly (HF Dataset is missing SQL files and table info)
+    print("Downloading and extracting data file...")
+    url = "https://bird-bench.oss-cn-beijing.aliyuncs.com/dev.zip"
+    filename = wget.download(url, out=output_dir)
+    with zipfile.ZipFile(Path(output_dir, filename), "r") as f_in:
+        f_in.extractall(output_dir)
+
+    # Expand tables zipfiles
+    print("Extracting databases...")
+    dev_dir = Path(output_dir, "dev_20240627/")
+    dbs_zipfile = Path(dev_dir, "dev_databases.zip")
+    with zipfile.ZipFile(dbs_zipfile, "r") as f_dbs:
+        f_dbs.extractall(dev_dir)
+
+    print("Extracted all data!")
+    return dev_dir
+
+
+def read_tables_file(base_dir):
+    """
+    Gets each db's information by using sqlite3 to get a table dump.
+    """
+    tables_info = {}
+    all_db_dirs = glob.glob("*", root_dir=os.path.join(base_dir, "dev_databases"))
+
+    for db_dir in all_db_dirs:
+        print(f"Reading database info from: {db_dir}")
+        table_info = ""
+
+        # Grab the db's sqlite file & read the dump
+        full_db_dir = os.path.join(base_dir, "dev_databases", db_dir)
+        sqlite_file = os.path.join(full_db_dir, db_dir + ".sqlite")
+        assert os.path.exists(sqlite_file)
+
+        with sqlite3.connect(os.path.join(full_db_dir, db_dir + ".sqlite")) as con:
+            con.text_factory = lambda b: b.decode(errors="ignore")
+            for line in con.iterdump():
+                if line[:6] == "INSERT":
+                    line = line.replace("\n", " ")
+                line = re.sub(" +", " ", line)
+                table_info += line + "\n"
+
+        # Time to truncate any long INSERT chains (allow 10 max at once)
+        insert_chain = r"((INSERT.*$\n){10})((INSERT.*\n)*)"
+        table_info = re.sub(insert_chain, r"\1\n...\n", table_info, flags=re.MULTILINE)
+
+        # Also get rid of any INSERT INTO * VALUES (...) <- lots of entries (>10)
+        many_values = r"(?:VALUES )(((\([^)]*)\)[,;]\s*)){10}(.*)(?:;)"
+        table_info = re.sub(many_values, r"...", table_info, flags=re.MULTILINE)
+
+        tables_info[db_dir] = table_info
+
+    return tables_info
+
+
+def format_entries(file_path, tables_info, out_file):
+    """
+    Combines the raw BIRD data entries with corresponding table info and
+    ground truth solution to form dev manifest
+    """
+    with open(out_file, "w") as f_out:
+        with open(file_path, "r") as f_in:
+            entries = json.load(f_in)
+
+        for i, entry in enumerate(entries):
+            new_entry = {
+                "question": entry["question"],
+                "gt_sql": entry["SQL"],
+                "sql_context": tables_info[entry["db_id"]],
+                "difficulty": entry["difficulty"],
+                "db_id": entry["db_id"],
+                "id": i,
+            }
+
+            f_out.write(json.dumps(new_entry))
+            f_out.write("\n")
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--output_dir", type=str, default=str(Path(__file__).parent))
+    args = parser.parse_args()
+
+    dev_dir = download_data(args.output_dir)
+    # If already downloaded: dev_dir = Path(args.output_dir, "dev_20240627/")
+    print(f"\nData downloaded to: {dev_dir}")
+
+    print("Starting processing...")
+
+    # First read tables data
+    tables_info = read_tables_file(dev_dir)
+    print("Finished reading tables.")
+
+    # Naming the input and output files the nearly same thing is likely
+    # confusing, but <split>.jsonl is the expected format so we'll just
+    # keep the result in the output directory.
+    format_entries(Path(dev_dir, "dev.json"), tables_info, Path(args.output_dir, "dev.jsonl"))
+    print("Finished formatting entries. All done!")
+
+
+if __name__ == "__main__":
+    main()

nemo_skills/evaluation/evaluator/__init__.py

@@ -18,6 +18,7 @@
 from nemo_skills.evaluation.evaluator.audio import AudioEvaluator
 from nemo_skills.evaluation.evaluator.base import BaseEvaluator
 from nemo_skills.evaluation.evaluator.bfcl import eval_bfcl
+from nemo_skills.evaluation.evaluator.bird import BirdEvaluator
 from nemo_skills.evaluation.evaluator.code import (
     CodeExecEvaluator,
     eval_bigcodebench,
@@ -67,6 +68,7 @@
     "ioi": IOIEvaluator,
     "icpc": ICPCEvaluator,
     "audio": AudioEvaluator,
+    "bird": BirdEvaluator,
 }
 
 # Validation: Ensure no overlap between class and function maps