Add Global PIQA benchmark

[naymaraq]

Add Global PIQA Benchmark

Summary

• Adds support for the Global PIQA multilingual benchmark, a physical commonsense reasoning dataset (https://arxiv.org/abs/2510.24081). The benchmark covers 116 languages.
• Implements prepare.py to download and format the dataset from HuggingFace.
• Allows users to optionally specify which languages to process via the --languages flag.
• Formats entries as two-choice questions (A/B) using a prompt template adapted from lm-evaluation-harness.

Summary by CodeRabbit

• New Features
  • Added Global PIQA multilingual support with dataset loading, MCQ prompt generation, answer extraction, and a CLI to produce per-language evaluation files.
• Documentation
  • Expanded multilingual evaluation guide: multi-language flags, per-language results, added MMMLU and Global PIQA benchmark pages with per-model examples and command snippets.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a Global PIQA dataset package: module constants, utilities to load/format the mrlbenchmarks/global-piqa-nonparallel dataset into MCQ prompts with extraction regexes, and a CLI to write per-language test.jsonl with expected answers and metadata.

Changes

Cohort / File(s)	Summary
Module Configuration nemo_skills/dataset/global_piqa/__init__.py	Adds exported constants METRICS_TYPE = "multichoice" and GENERATION_ARGS = "++prompt_config=generic/default ++eval_type=multichoice".
Dataset Utilities nemo_skills/dataset/global_piqa/global_piqa_utils.py	New utilities: supported_languages(), load_global_piqa_datasets(), digit_to_letter(), get_mcq_fields(), Schema class, QUERY_TEMPLATE, and answer-extraction regex constants for MCQ prompt generation and extraction.
Dataset Preparation CLI nemo_skills/dataset/global_piqa/prepare.py	New CLI script with format_entry(entry, language) and main(args): validates languages, loads datasets, formats entries (expected answer, extraction metadata, MCQ fields) and writes per-language test.jsonl.
Documentation docs/evaluation/multilingual.md	Expanded multilingual evaluation docs: switched --language → --languages, added MMMLU and Global PIQA sections, per-model examples, and translation metrics examples.

Sequence Diagram(s)

sequenceDiagram
    actor User
    participant Prepare as "prepare.py"
    participant Utils as "global_piqa_utils"
    participant HF as "HuggingFace\n(global-piqa-nonparallel)"
    participant FS as "FileSystem\n(test.jsonl)"

    User->>Prepare: run with --languages
    Prepare->>Utils: supported_languages()
    Utils->>HF: list dataset configs
    HF-->>Utils: available languages
    Utils-->>Prepare: supported languages

    Prepare->>Utils: load_global_piqa_datasets(languages, split="test")
    Utils->>HF: load each language split
    HF-->>Utils: dataset entries per language
    Utils-->>Prepare: dict(language -> entries)

    loop per language entry
        Prepare->>Utils: get_mcq_fields(entry)
        Utils-->>Prepare: MCQ prompt/options/labels
        Prepare->>Prepare: format_entry(entry, language)
        Prepare->>FS: append JSON line to test.jsonl
    end

    FS-->>User: test.jsonl ready

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main addition: a new Global PIQA benchmark implementation spanning utilities, dataset preparation, and documentation.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch dkaramyan/global_piqa

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Tip

You can enable review details to help with troubleshooting, context usage and more.

Enable the reviews.review_details setting to include review details such as the model used, the time taken for each step and more in the review comments.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@nemo_skills/dataset/global_piqa/prepare.py`:
- Around line 44-53: The current loop writes each formatted entry to output_file
as it's produced, risking partial writes if format_entry raises; instead, first
iterate through datasets (using load_global_piqa_datasets and format_entry) and
collect all formatted entries into a list (e.g., formatted_entries), then open
output_file (data_dir / "test.jsonl") once and write the precomputed entries to
the file; ensure you reference datasets, format_entry, and output_file in your
changes so you compute everything before any file I/O.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: ac8f52b0-5f48-44f2-bef8-9c8872479777

📥 Commits

Reviewing files that changed from the base of the PR and between fe92aec and 88698ba.

📒 Files selected for processing (3)

• nemo_skills/dataset/global_piqa/__init__.py
• nemo_skills/dataset/global_piqa/global_piqa_utils.py
• nemo_skills/dataset/global_piqa/prepare.py

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Compute all entries before writing to prevent data loss on failure.

Per coding guidelines, when adding new benchmarks, all computation should complete before file writes to prevent accidental data loss if code fails mid-way. Currently, if format_entry raises an exception during iteration, the file will contain partial data.

🛡️ Proposed fix to prevent partial writes

     datasets = load_global_piqa_datasets(args.languages)
 
+    # Compute all entries first to prevent partial writes on failure
+    all_entries = [
+        format_entry(entry=entry, language=language)
+        for language in datasets
+        for entry in datasets[language]
+    ]
+
     data_dir = Path(__file__).absolute().parent
     output_file = data_dir / "test.jsonl"
     with open(output_file, "wt", encoding="utf-8") as fout:
-        for language in datasets:
-            for entry in datasets[language]:
-                entry = format_entry(entry=entry, language=language)
-                json.dump(entry, fout, ensure_ascii=False)
-                fout.write("\n")
+        for entry in all_entries:
+            json.dump(entry, fout, ensure_ascii=False)
+            fout.write("\n")

As per coding guidelines: "When adding new benchmarks, avoid data loss by doing all computation before re-opening files for writing; ensure computation completes before file writes to prevent accidental data loss if code fails".

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@nemo_skills/dataset/global_piqa/prepare.py` around lines 44 - 53, The current
loop writes each formatted entry to output_file as it's produced, risking
partial writes if format_entry raises; instead, first iterate through datasets
(using load_global_piqa_datasets and format_entry) and collect all formatted
entries into a list (e.g., formatted_entries), then open output_file (data_dir /
"test.jsonl") once and write the precomputed entries to the file; ensure you
reference datasets, format_entry, and output_file in your changes so you compute
everything before any file I/O.

[shuoyangd]

Just one nit question, otherwise looking good to me.

[coderabbitai]

Actionable comments posted: 2

♻️ Duplicate comments (1)

nemo_skills/dataset/global_piqa/prepare.py (1)

48-53: ⚠️ Potential issue | 🟡 Minor

Precompute the rows before truncating test.jsonl.

If format_entry() fails mid-loop, the file has already been opened in write mode and will be left partially written. Build the formatted records first, then open the file once to emit them.

🛡️ Proposed fix

     datasets = load_global_piqa_datasets(args.languages)
 
+    formatted_entries = [
+        format_entry(entry=entry, language=language)
+        for language, dataset in datasets.items()
+        for entry in dataset
+    ]
+
     data_dir = Path(__file__).absolute().parent
     output_file = data_dir / "test.jsonl"
     with open(output_file, "wt", encoding="utf-8") as fout:
-        for language in datasets:
-            for entry in datasets[language]:
-                entry = format_entry(entry=entry, language=language)
-                json.dump(entry, fout, ensure_ascii=False)
-                fout.write("\n")
+        for entry in formatted_entries:
+            json.dump(entry, fout, ensure_ascii=False)
+            fout.write("\n")

As per coding guidelines: "When adding new benchmarks, avoid data loss by doing all computation before re-opening files for writing; ensure computation completes before file writes to prevent accidental data loss if code fails".

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@nemo_skills/dataset/global_piqa/prepare.py` around lines 48 - 53, The current
loop opens output_file for writing before calling format_entry, risking
truncation if format_entry raises; instead, first iterate over datasets and
collect formatted entries by calling format_entry(entry, language) into a list
(e.g., formatted = []), handling or propagating errors during formatting, and
only after all entries are successfully formatted open output_file once and
write the JSON lines from that list using json.dump and fout.write("\n");
reference format_entry, datasets, and output_file to locate and modify the code.

🧹 Nitpick comments (1)

nemo_skills/dataset/global_piqa/prepare.py (1)

40-45: Avoid duplicate supported_languages() lookup by deferring default to main().

The function is called during parser setup (line 60) before args are parsed, then again in main() (line 41) for validation. Using default=None and handling the default in main() with languages = args.languages or supported_languages() eliminates the unnecessary early lookup on the no-flag path while keeping the validation logic clean.

♻️ Proposed refactor

 def main(args):
-    invalid = set(args.languages) - set(supported_languages())
+    all_languages = supported_languages()
+    languages = args.languages or all_languages
+    invalid = set(languages) - set(all_languages)
     if invalid:
         raise ValueError(f"Unsupported languages: {invalid}")
-    datasets = load_global_piqa_datasets(args.languages)
+    datasets = load_global_piqa_datasets(languages)
 ...
     parser.add_argument(
         "--languages",
-        default=supported_languages(),
+        default=None,
         nargs="+",
-        help="Languages to process.",
+        help="Languages to process. Defaults to all supported languages.",
     )

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@nemo_skills/dataset/global_piqa/prepare.py` around lines 40 - 45, Change the
argument parser to use default=None for the --languages flag and in main() avoid
calling supported_languages() twice by doing: languages = args.languages or
supported_languages(); then validate using invalid = set(languages) -
set(supported_languages()) and pass languages to load_global_piqa_datasets.
Update references from args.languages to the local languages variable so the
initial parser setup no longer triggers supported_languages() on the common
(no-flag) path.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@nemo_skills/dataset/global_piqa/global_piqa_utils.py`:
- Around line 26-27: The function digit_to_letter currently maps any integer to
letters via chr(ord("A") + digit) which can produce labels beyond "A"/"B";
update digit_to_letter to validate the input digit (e.g., ensure it's 0 or 1 or
at least within 0..1) and raise a ValueError for out-of-range values, leaving
the mapping of 0->"A" and 1->"B" intact; reference the function name
digit_to_letter when making this change so callers receive a clear exception
instead of invalid labels.
- Around line 54-55: Update ANSWER_REGEX1 and ANSWER_REGEX2 to use word-boundary
or lookaround so captured A/B are not part of larger words (e.g., replace the
[^A-B]*([A-B]) postfix with a pattern that ensures the captured letter is not
preceded or followed by word characters, using constructs like
(?<![A-Za-z0-9_])([AB])(?![A-Za-z0-9_]) or \b([AB])\b so "maybe" or "probably"
won't match); then add bounds checking in digit_to_letter(digit: int) to raise
ValueError when digit is not 0 or 1 (e.g., if digit not in (0,1): raise
ValueError(...)) so invalid inputs fail fast.

---

Duplicate comments:
In `@nemo_skills/dataset/global_piqa/prepare.py`:
- Around line 48-53: The current loop opens output_file for writing before
calling format_entry, risking truncation if format_entry raises; instead, first
iterate over datasets and collect formatted entries by calling
format_entry(entry, language) into a list (e.g., formatted = []), handling or
propagating errors during formatting, and only after all entries are
successfully formatted open output_file once and write the JSON lines from that
list using json.dump and fout.write("\n"); reference format_entry, datasets, and
output_file to locate and modify the code.

---

Nitpick comments:
In `@nemo_skills/dataset/global_piqa/prepare.py`:
- Around line 40-45: Change the argument parser to use default=None for the
--languages flag and in main() avoid calling supported_languages() twice by
doing: languages = args.languages or supported_languages(); then validate using
invalid = set(languages) - set(supported_languages()) and pass languages to
load_global_piqa_datasets. Update references from args.languages to the local
languages variable so the initial parser setup no longer triggers
supported_languages() on the common (no-flag) path.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 979e869e-f766-4632-b6de-8d0a078dcfb7

📥 Commits

Reviewing files that changed from the base of the PR and between 36f080f and 49956cf.

📒 Files selected for processing (3)

• nemo_skills/dataset/global_piqa/__init__.py
• nemo_skills/dataset/global_piqa/global_piqa_utils.py
• nemo_skills/dataset/global_piqa/prepare.py

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Fail fast on labels outside A/B.

This is a two-choice benchmark, but chr(ord("A") + digit) will happily emit C, D, etc. if the source label is ever unexpected. Raising here is safer than writing impossible expected_answer values.

🛡️ Proposed fix

 def digit_to_letter(digit: int) -> str:
+    if digit not in (0, 1):
+        raise ValueError(f"Unsupported Global PIQA label: {digit}")
     return chr(ord("A") + digit)

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	def digit_to_letter(digit: int) -> str:
	return chr(ord("A") + digit)
	def digit_to_letter(digit: int) -> str:
	if digit not in (0, 1):
	raise ValueError(f"Unsupported Global PIQA label: {digit}")
	return chr(ord("A") + digit)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@nemo_skills/dataset/global_piqa/global_piqa_utils.py` around lines 26 - 27,
The function digit_to_letter currently maps any integer to letters via
chr(ord("A") + digit) which can produce labels beyond "A"/"B"; update
digit_to_letter to validate the input digit (e.g., ensure it's 0 or 1 or at
least within 0..1) and raise a ValueError for out-of-range values, leaving the
mapping of 0->"A" and 1->"B" intact; reference the function name digit_to_letter
when making this change so callers receive a clear exception instead of invalid
labels.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
python - <<'PY'
import re

pattern = re.compile(r"(?i)[Tt]he (?:[Bb]est [Aa]nswer|[Ff]inal [Aa]nswer|[Aa]nswer)[^A-B]*([A-B])")
for sample in [
    "The answer is maybe A",
    "The answer is probably B",
    "The final answer is A",
]:
    match = pattern.search(sample)
    print(f"{sample!r} -> {match.group(1) if match else None}")
PY

Repository: NVIDIA-NeMo/Skills

Length of output: 151

---

🏁 Script executed:

fd "global_piqa_utils.py" --type f

Repository: NVIDIA-NeMo/Skills

Length of output: 114

---

🏁 Script executed:

cat -n nemo_skills/dataset/global_piqa/global_piqa_utils.py | head -100

Repository: NVIDIA-NeMo/Skills

Length of output: 3055

---

🏁 Script executed:

rg "digit_to_letter" nemo_skills/dataset/global_piqa/global_piqa_utils.py -A 2 -B 2

Repository: NVIDIA-NeMo/Skills

Length of output: 411

---

🏁 Script executed:

rg "EXTRACT_REGEX" nemo_skills/dataset/global_piqa/ -A 3 -B 2

Repository: NVIDIA-NeMo/Skills

Length of output: 1484

---

🏁 Script executed:

rg "digit_to_letter" nemo_skills/dataset/global_piqa/ -B 2 -A 2

Repository: NVIDIA-NeMo/Skills

Length of output: 1689

---

🏁 Script executed:

cat nemo_skills/dataset/global_piqa/prepare.py | grep -A 10 "Schema.LABEL"

Repository: NVIDIA-NeMo/Skills

Length of output: 389

---

🏁 Script executed:

python3 <<'EOF'
import re

# Test the proposed fix from the review comment
proposed_regex1 = r"(?i)\bthe (?:best answer|final answer|answer)\b(?:\s+is)?\s*[:\-]?\s*(?:option\s*)?\(?([AB])\)?(?=[^\w]|$)"
proposed_regex2 = r"(?i)\banswer\s*:\s*(?:option\s*)?\(?([AB])\)?(?=[^\w]|$)"

# Also test a simpler fix: use [^A-Ba-b]
simple_regex1 = r"(?i)[Tt]he (?:[Bb]est [Aa]nswer|[Ff]inal [Aa]nswer|[Aa]nswer)[^A-Ba-b]*([A-B])"

test_cases = [
    "The answer is maybe A",
    "The answer is probably B",
    "The final answer is A",
    "The best answer is A",
    "Answer: B",
]

print("=== ORIGINAL REGEX (BROKEN) ===")
original_regex1 = re.compile(r"(?i)[Tt]he (?:[Bb]est [Aa]nswer|[Ff]inal [Aa]nswer|[Aa]nswer)[^A-B]*([A-B])")
for test in test_cases:
    match = original_regex1.search(test)
    print(f"{test!r} -> {match.group(1) if match else None}")

print("\n=== PROPOSED FIX (COMPLEX) ===")
proposed_pattern1 = re.compile(proposed_regex1)
for test in test_cases:
    match = proposed_pattern1.search(test)
    print(f"{test!r} -> {match.group(1) if match else None}")

print("\n=== SIMPLE FIX [^A-Ba-b] ===")
simple_pattern1 = re.compile(simple_regex1)
for test in test_cases:
    match = simple_pattern1.search(test)
    print(f"{test!r} -> {match.group(1) if match else None}")
EOF

Repository: NVIDIA-NeMo/Skills

Length of output: 575

---

🏁 Script executed:

python3 <<'EOF'
import re

# Test alternative fixes
word_boundary_regex1 = r"(?i)[Tt]he (?:[Bb]est [Aa]nswer|[Ff]inal [Aa]nswer|[Aa]nswer)[\s:=\-]*\b([A-B])\b"
word_boundary_regex2 = r"(?i)[Aa]nswer\s*:[\s]*\b([A-B])\b"

# Negative lookbehind/lookahead approach
negative_lb_regex1 = r"(?i)[Tt]he (?:[Bb]est [Aa]nswer|[Ff]inal [Aa]nswer|[Aa]nswer)[^A-B]*(?<![a-z])([A-B])(?![a-z])"

test_cases = [
    "The answer is maybe A",  # Invalid LLM output
    "The answer is probably B",  # Invalid LLM output
    "The final answer is A",  # Valid
    "The best answer is A",  # Valid
    "The best answer is: A",  # Valid
    "Answer: B",  # Valid
    "The answer is B",  # Valid
]

print("=== WORD BOUNDARY FIX ===")
pattern1 = re.compile(word_boundary_regex1)
pattern2 = re.compile(word_boundary_regex2)
for test in test_cases:
    match = pattern1.search(test)
    if not match:
        match = pattern2.search(test)
    print(f"{test!r} -> {match.group(1) if match else None}")

print("\n=== NEGATIVE LOOKBEHIND/LOOKAHEAD FIX ===")
pattern_lb = re.compile(negative_lb_regex1)
for test in test_cases:
    match = pattern_lb.search(test)
    print(f"{test!r} -> {match.group(1) if match else None}")
EOF

Repository: NVIDIA-NeMo/Skills

Length of output: 531

---

🏁 Script executed:

python3 <<'EOF'
import re

# Test ANSWER_REGEX2 with the same approach
answer_regex2_original = r"(?i)[Aa]nswer\s*:[^A-B]*([A-B])"
answer_regex2_fixed = r"(?i)[Aa]nswer\s*:(?<![a-z])[^A-B]*(?<![a-z])([A-B])(?![a-z])"

test_cases = [
    "answer: maybe A",
    "answer: probably B",
    "Answer: A",
    "Answer: B",
]

print("=== ANSWER_REGEX2 ORIGINAL ===")
pattern = re.compile(answer_regex2_original)
for test in test_cases:
    match = pattern.search(test)
    print(f"{test!r} -> {match.group(1) if match else None}")

print("\n=== ANSWER_REGEX2 WITH NEGATIVE LOOKBEHIND/LOOKAHEAD ===")
pattern_fixed = re.compile(answer_regex2_fixed)
for test in test_cases:
    match = pattern_fixed.search(test)
    print(f"{test!r} -> {match.group(1) if match else None}")
EOF

Repository: NVIDIA-NeMo/Skills

Length of output: 322

---

Fix regex patterns to reject letters inside words like "maybe" and "probably".

With [^A-B]*([A-B]), the regex captures the b in words like maybe and probably, marking correct outputs as B. Use negative lookbehind/lookahead instead to ensure the captured letter is not part of a word:

🔧 Proposed fix

-ANSWER_REGEX1 = r"(?i)[Tt]he (?:[Bb]est [Aa]nswer|[Ff]inal [Aa]nswer|[Aa]nswer)[^A-B]*([A-B])"
-ANSWER_REGEX2 = r"(?i)[Aa]nswer\s*:[^A-B]*([A-B])"
+ANSWER_REGEX1 = r"(?i)[Tt]he (?:[Bb]est [Aa]nswer|[Ff]inal [Aa]nswer|[Aa]nswer)[^A-B]*(?<![a-z])([A-B])(?![a-z])"
+ANSWER_REGEX2 = r"(?i)[Aa]nswer\s*:[^A-B]*(?<![a-z])([A-B])(?![a-z])"

Additionally, add bounds checking to digit_to_letter() to fail fast if passed values outside [0, 1]:

def digit_to_letter(digit: int) -> str:
    if digit not in (0, 1):
        raise ValueError(f"digit must be 0 or 1, got {digit}")
    return chr(ord("A") + digit)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@nemo_skills/dataset/global_piqa/global_piqa_utils.py` around lines 54 - 55,
Update ANSWER_REGEX1 and ANSWER_REGEX2 to use word-boundary or lookaround so
captured A/B are not part of larger words (e.g., replace the [^A-B]*([A-B])
postfix with a pattern that ensures the captured letter is not preceded or
followed by word characters, using constructs like
(?<![A-Za-z0-9_])([AB])(?![A-Za-z0-9_]) or \b([AB])\b so "maybe" or "probably"
won't match); then add bounds checking in digit_to_letter(digit: int) to raise
ValueError when digit is not 0 or 1 (e.g., if digit not in (0,1): raise
ValueError(...)) so invalid inputs fail fast.

[coderabbitai]

♻️ Duplicate comments (1)

nemo_skills/dataset/global_piqa/global_piqa_utils.py (1)

54-55: ⚠️ Potential issue | 🟠 Major

Tighten ANSWER_REGEX1/2 so they only capture standalone option letters.

[^A-B]*([A-B]) can still grab the a/b inside words like maybe or probably, so extract_answer() may return a choice before a real A/B token is reached. Make these two regexes strict and let GREEDY_REGEX remain the fallback for looser outputs.

🔧 Proposed fix

-ANSWER_REGEX1 = r"(?i)[Tt]he (?:[Bb]est [Aa]nswer|[Ff]inal [Aa]nswer|[Aa]nswer)[^A-B]*([A-B])"
-ANSWER_REGEX2 = r"(?i)[Aa]nswer\s*:[^A-B]*([A-B])"
+ANSWER_REGEX1 = r"(?i)\bthe (?:best answer|final answer|answer)\b(?:\s+is)?\s*[:\-]?\s*(?:option\s*)?\(?([A-B])\)?(?=[^\w]|$)"
+ANSWER_REGEX2 = r"(?i)\banswer\s*:\s*(?:option\s*)?\(?([A-B])\)?(?=[^\w]|$)"

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@nemo_skills/dataset/global_piqa/global_piqa_utils.py` around lines 54 - 55,
The current ANSWER_REGEX1 and ANSWER_REGEX2 allow matching letters inside words
(e.g., "maybe"), so update those two patterns used by extract_answer() to only
capture standalone choice tokens A or B (use word boundaries or
negative/positive lookarounds so the captured group is an isolated letter
possibly followed/preceded by punctuation or whitespace, and keep GREEDY_REGEX
as the fallback). Locate ANSWER_REGEX1, ANSWER_REGEX2 and the extract_answer()
usage in global_piqa_utils.py and replace the loose [^A-B]*([A-B]) capture with
a stricter pattern that enforces boundaries around the captured A/B while
preserving case-insensitive matching.

🧹 Nitpick comments (1)

nemo_skills/dataset/global_piqa/global_piqa_utils.py (1)

22-23: Return Dataset directly and fix the annotation.

This helper advertises dict[str, list[dict]], but the datasets API returns a single Dataset when split is provided, and a DatasetDict only when it is omitted. Switching to load_dataset(..., split=split) makes the contract accurate and removes the extra indexing hop. (huggingface.co)

♻️ Proposed fix

-from datasets import get_dataset_config_names, load_dataset
+from datasets import Dataset, get_dataset_config_names, load_dataset
@@
-def load_global_piqa_datasets(languages: list[str], split: str = "test") -> dict[str, list[dict]]:
-    return {lang: load_dataset("mrlbenchmarks/global-piqa-nonparallel", lang)[split] for lang in languages}
+def load_global_piqa_datasets(languages: list[str], split: str = "test") -> dict[str, Dataset]:
+    return {
+        lang: load_dataset("mrlbenchmarks/global-piqa-nonparallel", lang, split=split)
+        for lang in languages
+    }

As per coding guidelines, "Keep code simple and elegant; reuse/extend existing functionality when possible, minimize conditional checks, use self-explanatory code over comments, avoid complicated type interfaces with unions, and keep naming consistent with existing conventions"

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@nemo_skills/dataset/global_piqa/global_piqa_utils.py` around lines 22 - 23,
The function load_global_piqa_datasets currently uses
load_dataset("mrlbenchmarks/global-piqa-nonparallel", lang)[split] and declares
return type dict[str, list[dict]]; change it to call
load_dataset("mrlbenchmarks/global-piqa-nonparallel", lang, split=split) so each
value is a datasets.Dataset, update the return type annotation to dict[str,
Dataset], and add the necessary import for Dataset from the datasets package;
keep the mapping behavior (one entry per language) so callers still get a
Dataset per language without the extra indexing hop.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@nemo_skills/dataset/global_piqa/global_piqa_utils.py`:
- Around line 54-55: The current ANSWER_REGEX1 and ANSWER_REGEX2 allow matching
letters inside words (e.g., "maybe"), so update those two patterns used by
extract_answer() to only capture standalone choice tokens A or B (use word
boundaries or negative/positive lookarounds so the captured group is an isolated
letter possibly followed/preceded by punctuation or whitespace, and keep
GREEDY_REGEX as the fallback). Locate ANSWER_REGEX1, ANSWER_REGEX2 and the
extract_answer() usage in global_piqa_utils.py and replace the loose
[^A-B]*([A-B]) capture with a stricter pattern that enforces boundaries around
the captured A/B while preserving case-insensitive matching.

---

Nitpick comments:
In `@nemo_skills/dataset/global_piqa/global_piqa_utils.py`:
- Around line 22-23: The function load_global_piqa_datasets currently uses
load_dataset("mrlbenchmarks/global-piqa-nonparallel", lang)[split] and declares
return type dict[str, list[dict]]; change it to call
load_dataset("mrlbenchmarks/global-piqa-nonparallel", lang, split=split) so each
value is a datasets.Dataset, update the return type annotation to dict[str,
Dataset], and add the necessary import for Dataset from the datasets package;
keep the mapping behavior (one entry per language) so callers still get a
Dataset per language without the extra indexing hop.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 6a6669c8-6bf0-4020-8d56-492fb5fc13cd

📥 Commits

Reviewing files that changed from the base of the PR and between 49956cf and 3b362e5.

📒 Files selected for processing (1)

• nemo_skills/dataset/global_piqa/global_piqa_utils.py

[Kipok]

@naymaraq could you add this to the docs? Should be good to merge after that

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@docs/evaluation/multilingual.md`:
- Around line 227-229: Replace the fenced markdown code blocks with indented
code blocks for the two command snippets (the one containing "ns prepare_data
mmmlu --languages <lang1> <lang2> ... --include_english" and the later snippet
around line 300) by removing the triple backticks and indenting each line of the
snippet with four spaces so the document matches the repository's MD046 style;
update both occurrences (the mmmlu prepare_data snippet and the other snippet
referenced) to use the indented format.
- Line 223: Replace the generic link text "here" with descriptive link text for
both instances in docs/evaluation/multilingual.md (lines referencing the
original benchmark and the other occurrence) to satisfy MD059; for example
change "[here](https://huggingface.co/datasets/openai/MMMLU)" to "[OpenAI MMMLU
dataset on Hugging Face](https://huggingface.co/datasets/openai/MMMLU)" (and
make a similarly descriptive replacement for the second occurrence) so the links
convey their destination and purpose.
- Around line 313-325: The benchmark flag in the command block uses the wrong
value; replace the --benchmarks mmmlu entry in the shown CLI snippet with the
correct benchmark name for Global PIQA (e.g., --benchmarks global_piqa) so the
invocation runs the intended Global PIQA benchmark; update the relevant fenced
code block where the CLI options are defined (the line containing "--benchmarks
mmmlu") to use the correct benchmark name.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: d68cb002-64ab-4c44-87ce-e3b3a2bf9e59

📥 Commits

Reviewing files that changed from the base of the PR and between 3b362e5 and 3d44942.

📒 Files selected for processing (1)

• docs/evaluation/multilingual.md

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Use descriptive link text instead of “here”.

Line 223 and Line 296 use generic link text, which triggers MD059 and is less accessible in rendered docs.

Suggested fix

-- Original benchmark source is [here](https://huggingface.co/datasets/openai/MMMLU).
+- Original benchmark source is [OpenAI MMMLU dataset](https://huggingface.co/datasets/openai/MMMLU).

-- Original benchmark source is [here](https://huggingface.co/datasets/mrlbenchmarks/global-piqa-nonparallel).
+- Original benchmark source is [Global PIQA nonparallel dataset](https://huggingface.co/datasets/mrlbenchmarks/global-piqa-nonparallel).

Also applies to: 296-296

🧰 Tools

🪛 markdownlint-cli2 (0.21.0)

[warning] 223-223: Link text should be descriptive

(MD059, descriptive-link-text)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@docs/evaluation/multilingual.md` at line 223, Replace the generic link text
"here" with descriptive link text for both instances in
docs/evaluation/multilingual.md (lines referencing the original benchmark and
the other occurrence) to satisfy MD059; for example change
"[here](https://huggingface.co/datasets/openai/MMMLU)" to "[OpenAI MMMLU dataset
on Hugging Face](https://huggingface.co/datasets/openai/MMMLU)" (and make a
similarly descriptive replacement for the second occurrence) so the links convey
their destination and purpose.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Match repository markdown code-block style (MD046).

Line 227 and Line 300 use fenced blocks, but lint expects indented blocks in this doc style. Please convert these two snippets to the expected indentation format.

Also applies to: 300-302

🧰 Tools

🪛 markdownlint-cli2 (0.21.0)

[warning] 227-227: Code block style
Expected: indented; Actual: fenced

(MD046, code-block-style)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@docs/evaluation/multilingual.md` around lines 227 - 229, Replace the fenced
markdown code blocks with indented code blocks for the two command snippets (the
one containing "ns prepare_data mmmlu --languages <lang1> <lang2> ...
--include_english" and the later snippet around line 300) by removing the triple
backticks and indenting each line of the snippet with four spaces so the
document matches the repository's MD046 style; update both occurrences (the
mmmlu prepare_data snippet and the other snippet referenced) to use the indented
format.