PyPy3 execution support for LiveCodeBench evaluation

[wasiahmad]

In this PR, we are attempting to include LiveCodeBench score calculation into nemo-skills. We include all supporting scripts at nemo-skills/evaluation/evaluator/livecodebench. And updated the evaluation function at nemo_skills/evaluation/evaluator/livecodebench.py.

Summary by CodeRabbit

• New Features

  • New LiveCodeBench evaluator: async sandboxed runs with Python/PyPy3 support, dependency setup, per-file processing, and persisted graded outputs.
  • livecodebench-pro post-processing tool to normalize evaluation input/output fields.
  • Data prep CLI adds --keep_all_columns; prepared outputs now include subset_for_metrics and release_version.

• Documentation

  • Added end-to-end guides for data preparation, running evaluations (Python/PyPy3), verifying results, and averaging runs.

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

docs/evaluation/code.md (1)

185-249: Fix fenced code block languages to satisfy markdownlint.

markdownlint (MD040) is flagging each of these LiveCodeBench snippets because the fences lack a language hint. Tagging them with the appropriate language (bash/json) will unblock the lint job and keeps syntax highlighting consistent.

Apply this diff to cover all occurrences:

-```
+```bash
 ns prepare_data livecodebench --release_version v6 --start_date 2024-08 --end_date 2025-05

@@
- +bash
ns prepare_data livecodebench --release_version v6 --start_date 2024-08 --end_date 2025-05 --keep_all_columns --cluster=<CLUSTER_NAME> --data_dir=<DATA_DIR>

@@
-```
+```bash
ns eval \
    --cluster=<CLUSTER_NAME> \
    --model=nvidia/OpenReasoning-Nemotron-32B \
@@
    ++inference.tokens_to_generate=65536

@@
- +bash
--extra_eval_args="++eval_config.interpreter=pypy3 ++eval_config.test_file=<DATA_DIR>/livecodebench/test_v6_2408_2505.jsonl"

@@
-```
+```json
{
  "livecodebench": { "pass@1": { "accuracy": 71.14 ... }},
  "livecodebench-easy": { "pass@1": { "accuracy": 99.09 ... }},
  "livecodebench-hard": { "pass@1": { "accuracy": 46.30 ... }},
  "livecodebench-medium": { "pass@1": { "accuracy": 85.10 ... }}
}

@@
- +bash
--benchmarks=livecodebench:3

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 1ed6bb1 and 0ccde9f.

📒 Files selected for processing (1)

• docs/evaluation/code.md (1 hunks)

🧰 Additional context used

🪛 markdownlint-cli2 (0.18.1)

docs/evaluation/code.md

185-185: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

192-192: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

204-204: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

226-226: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

234-234: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

247-247: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)

• GitHub Check: unit-tests
• GitHub Check: pre-commit

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

docs/evaluation/code.md (1)

186-248: Add info strings to new code fences.

All newly added code blocks (commands, JSON output) lack language specifiers, which triggers MD040 and hurts syntax highlighting. Please annotate them with something like shell for CLI commands and json for metrics snippets.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 0ccde9f and 4970d6e.

📒 Files selected for processing (2)

• docs/evaluation/code.md (1 hunks)
• nemo_skills/evaluation/evaluator/__init__.py (1 hunks)

🚧 Files skipped from review as they are similar to previous changes (1)

• nemo_skills/evaluation/evaluator/init.py

🧰 Additional context used

🪛 markdownlint-cli2 (0.18.1)

docs/evaluation/code.md

185-185: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

192-192: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

204-204: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

226-226: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

234-234: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

247-247: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)

• GitHub Check: unit-tests
• GitHub Check: pre-commit

[coderabbitai]

Actionable comments posted: 1

♻️ Duplicate comments (2)

nemo_skills/evaluation/evaluator/livecodebench.py (2)

40-71: Install packages in the same sandbox session used for evaluation.

install_packages() (lines 40-57) creates its own sandbox context, installs the LiveCodeBench library, then closes that sandbox. The actual evaluation at line 71 opens a new sandbox session. For isolated sandbox types (Docker, SSH, remote HTTP), the second sandbox starts clean and will not have access to the packages installed in the first session, causing the from livecodebench.evaluate import evaluate statement to fail at runtime.

Move the package installation inside the evaluation sandbox session:

-    if not await install_packages(eval_config):
-        return
-
     async with sandbox_context(eval_config.sandbox) as sandbox:
+        if not await install_packages(eval_config, sandbox):
+            return
+
         for jsonl_file in unroll_files(cfg.input_files):

And update install_packages to accept an active sandbox:

-async def install_packages(eval_config: LiveCodeBenchEvaluatorConfig) -> bool:
+async def install_packages(eval_config: LiveCodeBenchEvaluatorConfig, sandbox) -> bool:
     """
     Installs required packages in a temporary sandbox.
     Returns True on success, False on failure.
     """
-    async with sandbox_context(eval_config.sandbox) as sandbox:
-        LOG.info(f"Installing livecodebench with {eval_config.interpreter}...")
-        pip_cmd = "pip" if eval_config.interpreter == "python" else "pypy3 -m pip"
-        git_url = LIVECODEBENCH_PYTHON_GIT_URL if eval_config.interpreter == "python" else LIVECODEBENCH_PYPY3_GIT_URL
-        cmd = f"{pip_cmd} install {git_url}"
-
-        result, _ = await sandbox.execute_code(cmd, language="shell", timeout=300)
-        if result.get("process_status") != "completed":
-            LOG.warning(f"Failed to install livecodebench: {result.get('stderr', 'Unknown error')}")
-            return False
-
-        LOG.info("Successfully installed livecodebench.")
-        return True
+    LOG.info(f"Installing livecodebench with {eval_config.interpreter}...")
+    pip_cmd = "pip" if eval_config.interpreter == "python" else "pypy3 -m pip"
+    git_url = LIVECODEBENCH_PYTHON_GIT_URL if eval_config.interpreter == "python" else LIVECODEBENCH_PYPY3_GIT_URL
+    cmd = f"{pip_cmd} install {git_url}"
+
+    result, _ = await sandbox.execute_code(cmd, language="shell", timeout=300)
+    if result.get("process_status") != "completed":
+        LOG.warning(f"Failed to install livecodebench: {result.get('stderr', 'Unknown error')}")
+        return False
+
+    LOG.info("Successfully installed livecodebench.")
+    return True

---

127-129: Avoid asyncio.run when an event loop may already be running.

asyncio.run() will raise RuntimeError: asyncio.run() cannot be called from a running event loop if the caller is async-aware (e.g., Jupyter notebooks, async CLI frameworks, Triton job managers). This breaks the function in those environments.

Replace the wrapper with a version that checks for an active loop:

 def eval_livecodebench(cfg):
-    """Synchronous wrapper to run the async evaluation."""
-    asyncio.run(eval_livecodebench_async(cfg))
+    """Run the async evaluation, reusing an existing loop when present."""
+    try:
+        loop = asyncio.get_running_loop()
+    except RuntimeError:
+        asyncio.run(eval_livecodebench_async(cfg))
+    else:
+        return loop.create_task(eval_livecodebench_async(cfg))

🧹 Nitpick comments (4)

nemo_skills/dataset/livecodebench/prepare.py (1)

154-157: Consider extracting the exception message.

The static analysis tool flags a long exception message inline. While this is a minor style issue, extracting it to a module-level constant improves maintainability.

Apply this diff:

+CUSTOM_SPLIT_ERROR_MSG = (
+    "If preparing a custom split, you must specify all "
+    "--release_version, --start_date, and --end_date arguments."
+)
+
 ...
         if args.release_version == "all" or args.start_date == "all" or args.end_date == "all":
-            raise ValueError(
-                "If preparing a custom split, you must specify all "
-                "--release_version, --start_date, and --end_date arguments."
-            )
+            raise ValueError(CUSTOM_SPLIT_ERROR_MSG)

nemo_skills/evaluation/evaluator/code.py (1)

152-163: Clarify the purpose and scope of eval_livecodebench_pro.

This function only transforms field names (task_id → problem_id, completion → text_response) and sets response_meta to None. It does not invoke any external evaluation library or compute correctness. The name eval_livecodebench_pro implies evaluation, but this is strictly a post-processing step.

Consider renaming to postprocess_livecodebench_pro or adding a docstring that clarifies this is a schema transformation, not an evaluation workflow:

 def eval_livecodebench_pro(cfg):
+    """Post-process LiveCodeBench-Pro samples: rename fields and add response_meta."""
     for jsonl_file in unroll_files(cfg.input_files):

Alternatively, if this function is intended to be called by a separate evaluation harness, document that expectation.

docs/evaluation/code.md (1)

186-248: Add language specifiers to fenced code blocks.

The markdown linter flags that fenced code blocks at lines 186, 193, 205, 227, 235, and 248 are missing language specifiers. Adding bash or shell identifiers improves syntax highlighting and readability.

Apply this diff:

-```
+```bash
 ns prepare_data livecodebench --release_version v6 --start_date 2024-08 --end_date 2025-05

...

- +bash
ns prepare_data livecodebench --release_version v6 --start_date 2024-08 --end_date 2025-05 --keep_all_columns --cluster=<CLUSTER_NAME> --data_dir=<DATA_DIR>

...

-```
+```bash
ns eval \
    ...

...

- +bash
--extra_eval_args="++eval_config.interpreter=pypy3 ++eval_config.test_file=<DATA_DIR>/livecodebench/test_v6_2408_2505.jsonl"

...

-```
+```json
{
  "livecodebench": { "pass@1": { "accuracy": 71.14 ... }},
  ...
}

...

- +bash
--benchmarks=livecodebench:3

nemo_skills/evaluation/evaluator/livecodebench.py (1)

64-80: Extract long exception messages to improve maintainability.

Static analysis flags lines 64, 66, and 80 for embedding long exception messages inline. Extracting them to module-level constants improves readability and maintainability.

Apply this diff:

+INVALID_PYTHON_INTERPRETER_MSG = "Python interpreter must be 'python' or 'pypy3'."
+CPP_REQUIRES_TEST_FILE_MSG = "C++ evaluation requires a test_file."
+MIXED_RELEASE_VERSIONS_MSG = "All samples should have the same release version. Found: {versions}"
+
 ...
     if eval_config.language == "python" and eval_config.interpreter not in ["python", "pypy3"]:
-        raise ValueError("Python interpreter must be 'python' or 'pypy3'.")
+        raise ValueError(INVALID_PYTHON_INTERPRETER_MSG)
     if eval_config.language == "cpp" and eval_config.test_file is None:
-        raise ValueError("C++ evaluation requires a test_file.")
+        raise ValueError(CPP_REQUIRES_TEST_FILE_MSG)
 
     ...
 
             versions = {s["release_version"] for s in samples}
             if len(versions) > 1:
-                raise ValueError(f"All samples should have the same release version. Found: {versions}")
+                raise ValueError(MIXED_RELEASE_VERSIONS_MSG.format(versions=versions))

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 4970d6e and 4042286.

📒 Files selected for processing (5)

• docs/evaluation/code.md (1 hunks)
• nemo_skills/dataset/livecodebench/prepare.py (6 hunks)
• nemo_skills/evaluation/evaluator/__init__.py (1 hunks)
• nemo_skills/evaluation/evaluator/code.py (2 hunks)
• nemo_skills/evaluation/evaluator/livecodebench.py (1 hunks)

🚧 Files skipped from review as they are similar to previous changes (1)

• nemo_skills/evaluation/evaluator/init.py

🧰 Additional context used

🧬 Code graph analysis (2)

nemo_skills/evaluation/evaluator/code.py (2)

nemo_skills/file_utils.py (1)

• unroll_files (21-32)

nemo_skills/utils.py (1)

• get_logger_name (130-131)

nemo_skills/evaluation/evaluator/livecodebench.py (3)

nemo_skills/code_execution/sandbox.py (2)

• get_sandbox (419-422)
• close (77-79)

nemo_skills/evaluation/evaluator/code.py (1)

• preprocess_code (36-92)

nemo_skills/utils.py (2)

• get_logger_name (130-131)
• nested_dataclass (49-82)

🪛 markdownlint-cli2 (0.18.1)

docs/evaluation/code.md

186-186: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

193-193: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

205-205: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

227-227: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

235-235: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

248-248: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🪛 Ruff (0.13.1)

nemo_skills/dataset/livecodebench/prepare.py

154-157: Avoid specifying long messages outside the exception class

(TRY003)

nemo_skills/evaluation/evaluator/livecodebench.py

64-64: Avoid specifying long messages outside the exception class

(TRY003)

---

66-66: Avoid specifying long messages outside the exception class

(TRY003)

---

80-80: Avoid specifying long messages outside the exception class

(TRY003)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: unit-tests

🔇 Additional comments (4)

nemo_skills/evaluation/evaluator/code.py (1)

26-27: Import fix confirmed.

The import of unroll_files from nemo_skills.file_utils is now correct and resolves the ImportError flagged in the previous review.

nemo_skills/evaluation/evaluator/livecodebench.py (1)

103-103: LGTM: Safe command construction.

Using shlex.quote to escape the Python code passed to -c prevents shell injection and correctly handles special characters in the evaluation script.

nemo_skills/dataset/livecodebench/prepare.py (2)

122-125: No downstream JSONL readers detected—output format change is self-contained.
All JSONL handling lives in prepare.py; no consumer code parses these files expecting a fixed schema.

---

96-97: Cast columns only when they are present.

Lines 96-97 unconditionally cast public_test_cases and private_test_cases to large_string, but when keep_all_columns=False, these columns are removed at line 98. The datasets library will raise a KeyError when attempting to cast a column that has already been removed from the dataset.

Move the casting before the removal, or conditionally cast only when keep_all_columns=True:

+    if keep_all_columns:
+        dataset = dataset.cast_column("public_test_cases", Value("large_string"))
+        dataset = dataset.cast_column("private_test_cases", Value("large_string"))
     remove_columns = []
     if not keep_all_columns:
         remove_columns = [
             "question_title",
             "contest_id",
             "metadata",
             "question_content",
             "platform",
             "question_id",
             "starter_code",
             "public_test_cases",
             "private_test_cases",
         ]
-    dataset = dataset.cast_column("public_test_cases", Value("large_string"))
-    dataset = dataset.cast_column("private_test_cases", Value("large_string"))
     dataset = dataset.map(map_fn, remove_columns=remove_columns)

Likely an incorrect or invalid review comment.

[coderabbitai]

Actionable comments posted: 5

🧹 Nitpick comments (2)

docs/evaluation/code.md (2)

223-229: Replace PyPy3 snippet with a runnable command and use bash fence.

A full command avoids ambiguity and is immediately executable.

-##### Pypy3 Evaluation
+##### PyPy3 Evaluation
@@
-To run with the Pypy3 interpreter, modify the `--extra_eval_args` flag as shown below.
-```
---extra_eval_args="++eval_config.interpreter=pypy3 ++eval_config.test_file=<DATA_DIR>/livecodebench/test_v6_2408_2505.jsonl"
-```
+To run with the PyPy3 interpreter, use:
+```bash
+ns eval \
+    --cluster=<CLUSTER_NAME> \
+    --model=nvidia/OpenReasoning-Nemotron-32B \
+    --server_type=vllm \
+    --server_args="--async-scheduling" \
+    --server_nodes=1 \
+    --server_gpus=8 \
+    --benchmarks=livecodebench \
+    --split=test_v6_2408_2505 \
+    --data_dir=<DATA_DIR> \
+    --output_dir=<OUTPUT_DIR> \
+    --extra_eval_args="++eval_config.interpreter=pypy3 ++eval_config.test_file=<DATA_DIR>/livecodebench/test_v6_2408_2505.jsonl" \
+    --with_sandbox \
+    ++inference.temperature=0.6 \
+    ++inference.top_p=0.95 \
+    ++inference.tokens_to_generate=65536
+```

---

196-221: Optional: add a local (non‑Slurm) example.

Many users run locally; adding a minimal local command improves usability.

Tip: For local runs, omit --cluster and use a smaller --server_gpus (e.g., 1). Example:
```bash
ns eval \
  --model=nvidia/OpenReasoning-Nemotron-32B \
  --server_type=vllm \
  --benchmarks=livecodebench \
  --split=test_v6_2408_2505 \
  --data_dir=<DATA_DIR> \
  --output_dir=<OUTPUT_DIR> \
  --extra_eval_args="++eval_config.interpreter=python" \
  --with_sandbox

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used**: CodeRabbit UI

**Review profile**: CHILL

**Plan**: Pro

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 404228692685c29e5298af14b5707a0cbd0df366 and 27fde4f274ab1f6f5104a59a8a6e53bb4ed26164.

</details>

<details>
<summary>📒 Files selected for processing (1)</summary>

* `docs/evaluation/code.md` (1 hunks)

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🪛 markdownlint-cli2 (0.18.1)</summary>

<details>
<summary>docs/evaluation/code.md</summary>

185-185: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

192-192: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

204-204: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

226-226: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

234-234: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

247-247: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

</details>

</details>

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)</summary>

* GitHub Check: unit-tests

</details>

<details>
<summary>🔇 Additional comments (1)</summary><blockquote>

<details>
<summary>docs/evaluation/code.md (1)</summary><blockquote>

`205-221`: **Confirm vLLM flag support and test file path**

- `--async-scheduling` is forwarded verbatim by the CLI; manually verify your vLLM server image supports this flag.  
- The evaluator config keys `interpreter` (used for Python vs. PyPy) and `test_file` (required for C++ runs) match those in `livecodebench.py`.  
- Ensure `<DATA_DIR>/livecodebench/test_v6_2408_2505.jsonl` (i.e. `{data_dir}/livecodebench/{split}.jsonl`) aligns with the output of `ns prepare_data --keep_all_columns`.

</blockquote></details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->

[Kipok]

thanks @wasiahmad! Just a small comment, but otherwise looks good

[coderabbitai]

Actionable comments posted: 1

♻️ Duplicate comments (6)

docs/evaluation/code.md (6)

185-187: Add bash fence to the data prep snippet.

Without a language tag markdownlint (MD040) keeps failing; please mark this command as bash.

-```
+```bash
 ns prepare_data livecodebench --release_version v6 --start_date 2024-08 --end_date 2025-05

---

189-194: Fix PyPy3 casing and add bash fence.

Line 189 should read “PyPy3”, and the shell example needs a bash fence to satisfy MD040.

-##### For Pypy3 Evaluation:
-If you plan to evaluate using the Pypy3 interpreter, you must add the `--keep_all_columns` flag during data preparation. This will download a larger dataset (~1.9GB) containing the necessary test cases. So, we recommend downloading the dataset into a Slurm cluster location.
-```
+##### For PyPy3 Evaluation:
+If you plan to evaluate using the PyPy3 interpreter, you must add the `--keep_all_columns` flag during data preparation. This will download a larger dataset (~1.9GB) containing the necessary test cases. So, we recommend downloading the dataset into a Slurm cluster location.
+```bash
 ns prepare_data livecodebench --release_version v6 --start_date 2024-08 --end_date 2025-05 --keep_all_columns --cluster=<CLUSTER_NAME> --data_dir=<DATA_DIR>

---

204-221: Mark the evaluation command as bash.

The primary eval command also needs a bash fence for linting and readability.

-```
+```bash
 ns eval \
     --cluster=<CLUSTER_NAME> \
     --model=nvidia/OpenReasoning-Nemotron-32B \
@@
     ++inference.top_p=0.95 \
     ++inference.tokens_to_generate=65536

---

227-228: Add bash fence to the PyPy3 override snippet.

This single-line CLI flag should be fenced as bash to clear MD040.

-```
+```bash
 --extra_eval_args="++eval_config.interpreter=pypy3 ++eval_config.test_file=<DATA_DIR>/livecodebench/test_v6_2408_2505.jsonl"

---

234-253: Tag the metrics block as text (lint fix).

The metrics excerpt needs a language on the fence; text works well here.

-```
+```text
 -------------------------- livecodebench --------------------------
 evaluation_mode | num_entries | avg_tokens | gen_seconds | accuracy
 pass@1          | 454         | 15995      | 2188        | 71.15%
@@
 ----------------------- livecodebench-medium ----------------------
 evaluation_mode | num_entries | avg_tokens | gen_seconds | accuracy
 pass@1          | 141         | 14178      | 1889        | 85.11%

---

259-261: Add bash fence to the repeats snippet.

Marking this CLI flag as bash resolves MD040 and improves copy/paste.

-```
+```bash
 --benchmarks=livecodebench:3

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 27fde4f and 29dad0f.

📒 Files selected for processing (1)

• docs/evaluation/code.md (1 hunks)

🧰 Additional context used

🪛 markdownlint-cli2 (0.18.1)

docs/evaluation/code.md

185-185: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

192-192: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

204-204: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

226-226: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

234-234: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

259-259: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)

• GitHub Check: pre-commit
• GitHub Check: unit-tests

[coderabbitai]

Actionable comments posted: 2

♻️ Duplicate comments (4)

docs/evaluation/code.md (4)

185-187: Add a bash fence to the prepare command.

Apply the markdownlint fix so the snippet lint-checks cleanly and gets syntax highlighting.

-```
+```bash
 ns prepare_data livecodebench --release_version v6 --start_date 2024-08 --end_date 2025-05
-```
+```

---

189-194: Fix PyPy3 casing and tag the command as bash.

Consistent branding and a language-tagged fence keep the docs polished.

-##### For Pypy3 Evaluation:
+##### For PyPy3 Evaluation:
@@
-```
+```bash
 ns prepare_data livecodebench --release_version v6 --start_date 2024-08 --end_date 2025-05 --keep_all_columns --cluster=<CLUSTER_NAME> --data_dir=<DATA_DIR>
-```
+```

---

205-221: Annotate the full eval command as bash.

Needed for markdownlint MD040 and better UX.

-```
+```bash
 ns eval \
     --cluster=<CLUSTER_NAME> \
@@
     ++inference.top_p=0.95 \
     ++inference.tokens_to_generate=65536
-```
+```

---

260-261: Add a bash fence to the repeat-run flag.

Keeps lint happy and clarifies that this is a CLI flag.

-```
+```bash
 --benchmarks=livecodebench:3
-```
+```

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 29dad0f and d050d9f.

📒 Files selected for processing (1)

• docs/evaluation/code.md (1 hunks)

🧰 Additional context used

🪛 markdownlint-cli2 (0.18.1)

docs/evaluation/code.md

185-185: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

192-192: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

204-204: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

226-226: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

234-234: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

---

259-259: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: unit-tests

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Tag the interpreter override snippet as bash.

Prevents MD040 lint failures.

-```
+```bash
 --extra_eval_args="++eval_config.interpreter=pypy3 ++eval_config.test_file=<DATA_DIR>/livecodebench/test_v6_2408_2505.jsonl"
-```
+```

🤖 Prompt for AI Agents

In docs/evaluation/code.md around lines 227-228, the code fence containing the
--extra_eval_args snippet is not tagged with a language which triggers MD040;
update the opening fence to ```bash so the snippet is explicitly labeled as bash
and ensure the closing ``` fence remains present and correctly placed.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Declare the metrics block as plain text.

The ASCII table isn’t JSON; marking it as text satisfies MD040 and keeps formatting intact.

-```
+```text
 -------------------------- livecodebench --------------------------
 evaluation_mode | num_entries | avg_tokens | gen_seconds | accuracy
@@
 pass@1          | 141         | 14178      | 1889        | 85.11%
-```
+```

🤖 Prompt for AI Agents

In docs/evaluation/code.md around lines 235 to 253, the ASCII metrics table is
currently in a fenced code block without a language, triggering MD040; change
the fence to declare the block as plain text by adding "text" after the opening
triple backticks (i.e., use ```text) and keep the closing triple backticks
unchanged so the table renders as plain text and preserves formatting.