LCB generic prompting

[wasiahmad]

Summary by CodeRabbit

• New Features

  • Unified "default reasoning" prompt applied across LiveCodeBench Python/C++ variants.
  • Dataset records now include explicit formatting hints and preserved starter code fields.

• Refactor

  • Prompt templates reorganized to use labeled Question/Format/Answer sections and require step‑by‑step reasoning.
  • Starter code is stored as formatted code blocks and question assembly moved to distinct fields.

[coderabbitai]

📝 Walkthrough

Walkthrough

Consolidates livecodebench prompt templates to a new default_reasoning prompt and refactors dataset preparation to emit structured fields (question_content, formatting_message, starter_code) instead of assembling a composite question string. Several language-specific prompt YAMLs were removed or updated.

Changes

Cohort / File(s)	Summary
Dataset generation args nemo_skills/dataset/livecodebench/__init__.py, nemo_skills/dataset/livecodebench-cpp/__init__.py, nemo_skills/dataset/livecodebench-pro/__init__.py	Switched GENERATION_ARGS to use eval/livecodebench/default_reasoning (replaced language-specific prompt configs). C++ variant changed from a string to a single-element tuple.
Dataset preparation mapping nemo_skills/dataset/livecodebench/prepare.py, nemo_skills/dataset/livecodebench-cpp/prepare.py, nemo_skills/dataset/livecodebench-pro/prepare.py	Refactored map logic to set formatting_message and a fenced starter_code field (language-specific fences). Removed construction of composite question; retained question_content and no longer drop question_content/starter_code from outputs.
Prompt files removed nemo_skills/prompt/config/eval/livecodebench/cpp_codegen.yaml, .../cpp_codegen_reasoning.yaml, .../python_codegen.yaml, .../python_codegen_reasoning.yaml	Deleted language-specific and prior reasoning prompt templates.
Prompt files added/updated nemo_skills/prompt/config/eval/livecodebench/default.yaml, .../default_reasoning.yaml, nemo_skills/prompt/config/gpt-oss/livecodebench.yaml	Added default_reasoning.yaml (step-by-step reasoning template using question_content, formatting_message, starter_code) and updated other templates to use the new structured placeholders and explicit answer formatting.
Docs docs/releases/openreasoning/training.md	Replaced references from python_codegen_reasoning to default_reasoning in documentation.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• Prompting fix to improve LCB score for GPTOSS #1169: Modifies the same gpt-oss livecodebench prompt template with overlapping structural/format changes.
• Add LCB Prompts, fix regex bug in robust_eval, remove CR, make summarize_robustness generic for more benchmarks, update docstrings. #1079: Related LiveCodeBench prompt/config updates and placeholder restructuring.

Suggested labels

enhancement

Suggested reviewers

• titu1994
• Kipok

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	❓ Inconclusive	The title 'LCB generic prompting' is vague and lacks specificity about what changes are being made to the LiveCodeBench prompting system.	Use a more descriptive title that clarifies the main objective, such as 'Generalize LiveCodeBench prompt templates to support multiple languages' or 'Refactor LiveCodeBench prompts to use unified default_reasoning template'.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch lcb_prompt_generalization

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[greptile-apps]

_{1 file reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Additional Comments (1)

nemo_skills/dataset/livecodebench/prepare.py
Invalid cast/map order
clean_data() removes public_test_cases/private_test_cases when keep_all_columns=False (they’re in remove_columns), but then calls dataset.cast_column("public_test_cases", ...) / cast_column("private_test_cases", ...) before map(remove_columns=...). This will raise at runtime because those columns don’t exist in the dataset after the map step (and the cast is pointless if the columns are about to be removed anyway). Either keep these columns (remove them from remove_columns) or move the casts into the keep_all_columns branch where the columns are retained.

[coderabbitai]

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

docs/releases/openreasoning/training.md (1)

98-108: ⚠️ Potential issue | 🔴 Critical

Key mismatch will cause runtime failure: prompt template requires question_content, formatting_message, and starter_code fields.

The apply_format function passes {'question': question} to prompt.fill() at line 102, but the default_reasoning.yaml template uses {question_content}, {formatting_message}, and {starter_code} as placeholders. Since prompt.fill() uses Python's str.format() method, it will raise a KeyError when trying to fill these missing placeholders.

Update the apply_format function to provide all required fields:

elem['input'] = prompt.fill({'question_content': question, 'formatting_message': '...', 'starter_code': '...'}, format_as_string=True)

Alternatively, use a prompt template that matches the provided input (e.g., one that only requires {question}).

🧹 Nitpick comments (1)

nemo_skills/dataset/livecodebench-pro/prepare.py (1)

72-73: Consider re-raising the exception after logging.

Per coding guidelines, exceptions that aren't expected to be normally raised should cause clear failures rather than being silently handled. The current code prints the error but continues processing, which could result in partial data generation without a clear failure signal.

♻️ Proposed fix to fail clearly on error

         except Exception as e:
             print(f"    Error processing split {split}: {e}")
+            raise

Alternatively, if partial success is intentional, consider tracking failures and returning a non-zero exit code at the end.

[greptile-apps]

_{2 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Additional Comments (2)

nemo_skills/prompt/config/eval/aai/livecodebench.yaml
Prompt placeholder mismatch

This prompt template uses {question}, but the LiveCodeBench prepare scripts in this PR stopped emitting a question field (they now emit question_content, formatting_message, and starter_code). Prompt.fill() calls str.format(**input_dict) (see nemo_skills/prompt/utils.py), so this will raise KeyError: 'question' at runtime for AAI LiveCodeBench evals unless the template or dataset fields are updated to match.

---

docs/releases/openreasoning/training.md
Docs example will break

This snippet uses prompt.fill({'question': question}), but the new eval/livecodebench/default_reasoning config expects question_content / formatting_message / starter_code placeholders. As written, users following the docs will hit KeyError during str.format() when {question_content} etc. are missing.

[Kipok]

is this a breaking change?

[wasiahmad]

No, because we reduced prompt configs and set the defaults accordingly.

[Kipok]

do i understand correctly that all lcb jobs will fail unless prepare data is redone?

[wasiahmad]

You are right @Kipok

[greptile-apps]

_{1 file reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Additional Comments (1)

nemo_skills/prompt/config/robustness/code_prompts/ns_python_codegen.yaml
Stale prompt reference

This file still points to the deleted eval/livecodebench/python_codegen.yaml (see header on lines 1-2). As-is, anyone relying on this robustness prompt config will be using an out-of-date template/fields ({question}) that no longer matches the LiveCodeBench dataset outputs in this PR ({question_content}, formatting_message, starter_code). Update this robustness prompt (or remove the misleading header) to reference the new unified LiveCodeBench prompt and expected fields.

[greptile-apps]

_{No files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{3 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{No files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

nemo_skills/dataset/livecodebench-cpp/prepare.py (1)

76-80: ⚠️ Potential issue | 🟡 Minor

The keep_all_columns parameter in clean_data is inaccessible from prepare in the C++ variant.

The clean_data function accepts keep_all_columns (line 46), but the prepare function (line 76) does not expose this parameter and never passes it through—it always defaults to False. The Python variant's prepare function correctly accepts and forwards this parameter (see Python variant line 98 and 107), allowing column filtering to be controlled by the caller. The C++ variant is missing this feature.

🧹 Nitpick comments (1)

nemo_skills/dataset/livecodebench/prepare.py (1)

25-28: PromptConstants is duplicated across livecodebench and livecodebench-cpp.

Both prepare.py files define nearly identical PromptConstants classes, differing only in the language name ("python" vs "c++"). Consider extracting a shared base into a common module, parameterized by language, to avoid drift between the two copies.