diff --git a/README.md b/README.md index f3d64d5ef4..e5aa340ab9 100644 --- a/README.md +++ b/README.md @@ -450,7 +450,7 @@ source .venv/bin/activate pytest ``` -At some point, you will want to actually add data that can be used to query your server. Please follow the instructions for [How To: Prepare and validate data for PR submission or RL training](#how-to-prepare-and-validate-data-for-pr-submission-or-rl-training). +At some point, you will want to actually add data that can be used to query your server. Please follow the instructions for [How To: Prepare and validate data for PR submission or RL training](#how-to-prepare-and-validate-data-for-pr-submission-or-rl-training). If you need some dataset preprocessing or formatting scripts, please place them your resources server directory e.g. `resources_servers/simple_weather/my_preprocess_script.py`. @@ -501,7 +501,7 @@ Gitlab uses MLFlow to interface with its model artifact registry. You will need: 2. The URI will look something like `https://gitlab-master.nvidia.com/api/v4/projects/191584/ml/mlflow/` 2. Your Gitlab token. Your Gitlab token must have the `api` and `read_api` scopes. -Provide your MLFlow credentials in `env.yaml`. +Provide your MLFlow credentials in `env.yaml`. ```yaml mlflow_tracking_uri: {your NeMo Gym Gitlab URI} mlflow_tracking_token: {your Gitlab PAT} @@ -700,7 +700,7 @@ More often than node, the SHA256 displayed by Github (SHA256:xxxx) should be the For developers that sign commits via SSH keys, this is configuration so that VSCode source control is able to sign commits properly! ```bash -git config gpg.format ssh +git config gpg.format ssh git config user.signingkey ~/.ssh/id_ed25519.pub ``` @@ -724,35 +724,35 @@ Tying back to NeMo Gym, NeMo gym can be used to create synthetic data for SFT tr # FAQ: Why NeMo Gym? -NeMo Gym is a large-scale collection of high-quality verifier environments for multi-verifier RL training. +NeMo Gym is a large-scale collection of high-quality verifier environments for multi-verifier RL training. To enable this, NeMo Gym provides infra support for the rollout server that runs 100+ verifiers in parallel. The document below details why we designed NeMo Gym the way we did. It also includes a direct comparative study that clearly differentiates NeMo Gym from other environment frameworks. \[Banghua\] As of Thu Aug 21: -1. Gym is completely different from any of the alternatives above in terms of data **coverage, quantity and quality.** For example, for math only, gym contains 1M+ high-quality math verifiable dataset curated by our internal team, with great math verify \+ LLM-as-a-judge support. In contrast, SkyRL and verifiers above only have a small train subset of GSM8K and AIME. We also have close to 10k SWE development, which require both high quality data curation efforts and good infra support. In contrast, Aviary only focuses on scientific knowledge environment. **None of the existing frameworks support general multi-turn tool-use agent, with tools like search, code execution, and other synthetic tools.** -2. We will be a **superset** of all existing gym environments. We are already a super-set of Sky RL Lab Gym and verifiers. We have integrated all GEM environments. We’re working with Aviary to incorporate them as well. +1. Gym is completely different from any of the alternatives above in terms of data **coverage, quantity and quality.** For example, for math only, gym contains 1M+ high-quality math verifiable dataset curated by our internal team, with great math verify \+ LLM-as-a-judge support. In contrast, SkyRL and verifiers above only have a small train subset of GSM8K and AIME. We also have close to 10k SWE development, which require both high quality data curation efforts and good infra support. In contrast, Aviary only focuses on scientific knowledge environment. **None of the existing frameworks support general multi-turn tool-use agent, with tools like search, code execution, and other synthetic tools.** +2. We will be a **superset** of all existing gym environments. We are already a super-set of Sky RL Lab Gym and verifiers. We have integrated all GEM environments. We’re working with Aviary to incorporate them as well. 3. As is shown from Brian’s comparison below, we have much **better infra support for scaling**. And the plan is to use NeMo Gym for 500B+ model training for quality improvement. This will make nemo gym battle tested in frontier model training, while the other gyms are mostly for smaller-scale experiments. Key use case requirements to avoid training environment scale, complexity, and diversity limitations: -1. Can I easily build my environment without worrying about a training framework? -2. Can I easily call my model using OpenAI Responses and not worry about reasoning parsing? -3. Can I easily use your environment framework to build an agent application product? -4. Can I easily use your environment framework to build a simple multi-agent system? -5. Can I easily run individual SWE-bench task Docker containers? -6. Can I easily add an agent built with any agent framework? -7. Can I easily add any environment framework? -8. Can I easily simultaneously use math-verify==0.7.0 and math-verify==0.8.0 in 2 different environments? +1. Can I easily build my environment without worrying about a training framework? +2. Can I easily call my model using OpenAI Responses and not worry about reasoning parsing? +3. Can I easily use your environment framework to build an agent application product? +4. Can I easily use your environment framework to build a simple multi-agent system? +5. Can I easily run individual SWE-bench task Docker containers? +6. Can I easily add an agent built with any agent framework? +7. Can I easily add any environment framework? +8. Can I easily simultaneously use math-verify==0.7.0 and math-verify==0.8.0 in 2 different environments? 9. Can I easily spin up multiple environments at once? Key principles -1. \[Reqs 1, 2\] Decoupled from training framework -2. \[Reqs 2, 3, 4, 6, 7\] Standardized behind OpenAI Responses -3. \[Reqs 3, 4, 6\] Explicit Agent vs model abstraction -4. \[Reqs 3, 4, 5, 6, 7\] REST environment servers and container compatible +1. \[Reqs 1, 2\] Decoupled from training framework +2. \[Reqs 2, 3, 4, 6, 7\] Standardized behind OpenAI Responses +3. \[Reqs 3, 4, 6\] Explicit Agent vs model abstraction +4. \[Reqs 3, 4, 5, 6, 7\] REST environment servers and container compatible 5. \[Reqs 8, 9\] Separate Python env per server at runtime \[Brian note\] There are some rows yet to be filled in here. diff --git a/pyproject.toml b/pyproject.toml index a15ecb43a6..b35dd43f8d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -64,10 +64,10 @@ dependencies = [ # 1. Why this dependency is here in NeMo Gym # 2. When this dependency was last updated # 3. The license of the dependencies. - # + # # If you are adding or removing dependencies, please do your due diligience to update this information. PRs to main that modify dependencies will not be accepted unless this information is provided. # The licenses of the below dependencies include: Apache 2.0, MIT, and BSD 3-Clause - # + # # By design, most (if not all) dependencies are unfrozen here to be easier to consume. The core pieces we need are server infra like FastAPI, etc. ######################################## diff --git a/resources_servers/comp_coding/README.md b/resources_servers/comp_coding/README.md new file mode 100644 index 0000000000..ae045ad645 --- /dev/null +++ b/resources_servers/comp_coding/README.md @@ -0,0 +1,104 @@ +# Competitive Coding Resources Server + +### Overview +Verifies competitive programming solutions by executing submitted code against unit tests. The server consumes agent trajectories and returns a reward based on whether the assistant's code produces the correct outputs for given test inputs. +Data source: [Filtered competitive programming dataset](https://huggingface.co/datasets/Nexusflow/comp_prog_filtered_no_function); split=`train` + +### Input schema +- `responses_create_params`: OpenAI Responses create params + - Use only a user message with the problem statement and instructions (e.g., "You are an expert competitive programmer..."). + - `verifier_metadata` (required): + - `unit_tests` (required): dict with `inputs` and `outputs` arrays containing test cases. + - `inputs`: list of strings representing stdin input for each test case + - `outputs`: list of strings representing expected stdout output for each test case + - `problem_id` (optional): unique identifier for the problem + +**Notes** +- All test cases must pass for a solution to receive a reward of 1.0 +- Failed test cases result in a reward of 0.0 with detailed error information + +### Test execution (for now) +- Code is executed using Python's `exec()` function in a controlled environment +- Each test case runs with redirected stdin/stdout: + - `stdin` is populated with the test input + - `stdout` is captured for comparison with expected output +- Available built-ins include common functions: `input`, `print`, `range`, `len`, `int`, `str`, `list`, etc. +- Newlines in test data are properly handled (converts `\\n` to actual newlines) + +### Example dataset row +```json +{ + "responses_create_params": { + "input": [ + { + "role": "user", + "content": "You are an expert competitive programmer. You will be given a problem statement and must output a complete Python solution that reads from stdin and writes to stdout.\n\nPolycarp has $n$ different binary words. A word called binary if it contains only characters '0' and '1'. For example, these words are binary: \"0001\", \"11\", \"0\" and \"0011100\".\n\nPolycarp wants to offer his set of $n$ binary words to play a game \"words\". In this game, players name words and each next word (starting from the second) must start with the last character of the previous word. The first word can be any. For example, these sequence of words can be named during the game: \"0101\", \"1\", \"10\", \"00\", \"00001\".\n\nWord reversal is the operation of reversing the order of the characters. For example, the word \"0111\" after the reversal becomes \"1110\", the word \"11010\" after the reversal becomes \"01011\".\n\nProbably, Polycarp has such a set of words that there is no way to put them in the order correspondent to the game rules. In this situation, he wants to reverse some words from his set so that: the final set of $n$ words still contains different words (i.e. all words are unique); there is a way to put all words of the final set of words in the order so that the final sequence of $n$ words is consistent with the game rules. \n\nPolycarp wants to reverse minimal number of words. Please, help him.\n\n\n-----Input-----\n\nThe first line of the input contains one integer $t$ ($1 \\le t \\le 10^4$) — the number of test cases in the input. Then $t$ test cases follow.\n\nThe first line of a test case contains one integer $n$ ($1 \\le n \\le 2\\cdot10^5$) — the number of words in the Polycarp's set. Next $n$ lines contain these words. All of $n$ words aren't empty and contains only characters '0' and '1'. The sum of word lengths doesn't exceed $4\\cdot10^6$. All words are different.\n\nGuaranteed, that the sum of $n$ for all test cases in the input doesn't exceed $2\\cdot10^5$. Also, guaranteed that the sum of word lengths for all test cases in the input doesn't exceed $4\\cdot10^6$.\n\n\n-----Output-----\n\nPrint answer for all of $t$ test cases in the order they appear.\n\nIf there is no answer for the test case, print -1. Otherwise, the first line of the output should contain $k$ ($0 \\le k \\le n$) — the minimal number of words in the set which should be reversed. The second line of the output should contain $k$ distinct integers — the indexes of the words in the set which should be reversed. Words are numerated from $1$ to $n$ in the order they appear. If $k=0$ you can skip this line (or you can print an empty line). If there are many answers you can print any of them.\n\n\n-----Example-----\nInput\n4\n4\n0001\n1000\n0011\n0111\n3\n010\n101\n0\n2\n00000\n00001\n4\n01\n001\n0001\n00001\n\nOutput\n1\n3 \n-1\n0\n\n2\n1 2" + } + ] + }, + "verifier_metadata": { + "problem_id": "c69268d8bdb4da0685d7b187c88296c1", + "unit_tests": { + "inputs": ["4\n4\n0001\n1000\n0011\n0111\n3\n010\n101\n0\n2\n00000\n00001\n4\n01\n001\n0001\n00001\n"], + "outputs": ["1\n3 \n-1\n0\n\n2\n1 2 \n"] + } + } +} +``` + +### Example of rollouts and usage + +```bash +config_paths="responses_api_agents/simple_agent/configs/simple_agent.yaml,\ +responses_api_models/openai_model/configs/openai_model.yaml,\ +resources_servers/comp_coding/configs/comp_coding.yaml" + +# Running the server +ng_run "+config_paths=[$config_paths]" \ + +simple_agent.responses_api_agents.simple_agent.resources_server.name=comp_coding + +# Prepare example data for validation +ng_prepare_data "+config_paths=[$config_paths]" \ + +output_dirpath=resources_servers/comp_coding/data/ \ + +mode=example_validation + +# Download train data from gitlab model registry +ng_download_dataset_from_gitlab \ + +dataset_name=comp_coding \ + +version=0.0.1 \ + +run_id=5a1167ef-3533-486f-9c0e-49d1e97fc887 \ + +artifact_fpath=train.jsonl \ + +output_fpath=resources_servers/comp_coding/data/train.jsonl + +# Collect rollouts from example problems +ng_collect_rollouts +agent_name=comp_coding_simple_agent \ + +input_jsonl_fpath=resources_servers/comp_coding/data/example.jsonl \ + +output_jsonl_fpath=resources_servers/comp_coding/data/example_rollouts.jsonl \ + +limit=null +``` + +### Optional data preperation/validation scripts + +```bash +# Build training dataset from collected examples +uv run python resources_servers/comp_coding/scripts/build_examples.py \ + --out resources_servers/comp_coding/data/train.jsonl \ + --split train[:5000] + +# Validate and pre-process train dataset +uv run python resources_servers/comp_coding/scripts/validate_dataset.py \ + --in data/comp_coding/train.jsonl --fail-fast +``` + +### Error handling +The server provides specific error messages for different failure modes: +- `Empty model output`: No text found in the response +- `Missing verifier_metadata.unit_tests`: Required test data not provided +- `Invalid unit_tests`: Malformed test case data +- `Could not extract code`: No valid Python code found in response +- `INVALID_TEST_FORMAT`: Test inputs/outputs length mismatch or empty +- `TEST_CASE_N_FAILED`: Specific test case failed with expected vs actual output +- `TEST_CASE_N_ERROR`: Runtime error during test execution + +## Licensing information +TODO: @kbhardwaj to confirm data/code licensing information w Vahid and team diff --git a/resources_servers/comp_coding/app.py b/resources_servers/comp_coding/app.py new file mode 100644 index 0000000000..925ace87d4 --- /dev/null +++ b/resources_servers/comp_coding/app.py @@ -0,0 +1,228 @@ +# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import io +import re +import sys +from typing import Any, ClassVar, List, Optional, Pattern, Tuple + +from fastapi import FastAPI, HTTPException +from pydantic import BaseModel + +from nemo_gym.base_resources_server import ( + BaseResourcesServerConfig, + BaseRunRequest, + BaseVerifyRequest, + BaseVerifyResponse, + SimpleResourcesServer, +) + + +# ---------------------------- +# Config +# ---------------------------- +class CompCodingResourcesServerConfig(BaseResourcesServerConfig): + pass + + +# ---------------------------- +# Schemas +# ---------------------------- +class UnitTests(BaseModel): + inputs: List[str] + outputs: List[str] + + +class CompCodingRunRequest(BaseRunRequest): + uuid: Optional[str] = None + metadata: Optional[dict[str, Any]] = None + + +class CompCodingVerifyRequest(CompCodingRunRequest, BaseVerifyRequest): + verifier_metadata: Optional[dict[str, Any]] = None + + +class CompCodingVerifyResponse(BaseVerifyResponse): + reason: Optional[str] = None + + +# ------------ helpers ------------ +CODE_BLOCK_RE: ClassVar[Pattern[str]] = re.compile(r"```(?:python)?\s*(.*?)```", re.DOTALL | re.IGNORECASE) + + +def _extract_code(text: str) -> Optional[str]: + # We allow two kinds of responses: + # 1. Code inside a fenced block (```python ... ``` or ``` ... ```) + # 2. Raw code returned without any fences + if not text: + return None + m = CODE_BLOCK_RE.search(text) + if m: + return m.group(1).strip() + return text.strip() + + +def _parse_unit_tests(ut_dict: dict) -> UnitTests: + return UnitTests.model_validate(ut_dict) + + +def _run_code_against_tests(code: str, tests: UnitTests) -> Tuple[bool, str]: + """ + Executes `code` with in-process exec(), redirecting stdin/stdout per test. + Assumes dataset pre-processing has already validated test shapes (non-empty, + equal-length inputs/outputs). + """ + for i, (test_input, expected_output) in enumerate(zip(tests.inputs, tests.outputs), start=1): + # capture originals + orig_stdin, orig_stdout = sys.stdin, sys.stdout + try: + sys.stdin = io.StringIO(test_input.replace("\\n", "\n")) + sys.stdout = io.StringIO() + + exec_globals = { + "__builtins__": __builtins__, + "input": input, + "print": print, + "range": range, + "len": len, + "list": list, + "int": int, + "str": str, + "float": float, + "bool": bool, + "enumerate": enumerate, + "zip": zip, + "sum": sum, + "max": max, + "min": min, + "abs": abs, + "round": round, + "sorted": sorted, + "reversed": reversed, + "all": all, + "any": any, + "set": set, + "dict": dict, + "tuple": tuple, + "map": map, + "filter": filter, + } + + exec(code, exec_globals) + + actual_output = sys.stdout.getvalue() + exp = expected_output.replace("\\n", "\n") + if actual_output.rstrip() != exp.rstrip(): + return ( + False, + f"TEST_CASE_{i}_FAILED: Expected {repr(exp)} got {repr(actual_output)}", + ) + except SystemExit as e: + # Handle sys.exit() calls in the executed code + return False, f"TEST_CASE_{i}_ERROR: Code called sys.exit({e.code})" + except Exception as e: + return False, f"TEST_CASE_{i}_ERROR: {e}" + finally: + sys.stdin, sys.stdout = orig_stdin, orig_stdout + + return True, f"SUCCESS: All {len(tests.inputs)} test cases passed" + + +def _extract_text_from_response(response_obj) -> Optional[str]: + """ + Extract the assistant's output_text string from a NeMoGymResponse-like object: + response.output -> list of messages + message.content -> list of blocks with {"type": "output_text", "text": "..."} + """ + try: + if not response_obj: + return None + output_list = getattr(response_obj, "output", None) + if not output_list: + return None + for msg in output_list: + content = msg.get("content") if isinstance(msg, dict) else getattr(msg, "content", None) + if not content: + continue + + # content may be a list of blocks or a bare string (be tolerant) + if isinstance(content, str) and content.strip(): + return content + + for block in content: + btype = block.get("type") if isinstance(block, dict) else getattr(block, "type", None) + if btype in ("output_text", "text"): + text = block.get("text") if isinstance(block, dict) else getattr(block, "text", None) + if isinstance(text, str) and text.strip(): + return text + return None + except Exception: + return None + + +# ---------------------------- +# Server +# ---------------------------- +class CompCodingResourcesServer(SimpleResourcesServer): + config: CompCodingResourcesServerConfig + + def setup_webserver(self) -> FastAPI: + app = super().setup_webserver() + + # (optional) a simple health route + @app.get("/health") + async def health(): + return {"ok": True, "server": "comp_coding"} + + return app + + # ------------ verifier ------------ + async def verify(self, body: CompCodingVerifyRequest) -> CompCodingVerifyResponse: + # Enforce a single source of truth for model output: the Responses API object + response_obj = getattr(body, "response", None) + if not response_obj: + # Treat absence of a response as an input/contract error + raise HTTPException(status_code=422, detail="Missing response") + + model_out = _extract_text_from_response(response_obj) + if not model_out or not model_out.strip(): + # A response existed but had no usable text -> model failure + return CompCodingVerifyResponse(**body.model_dump(), reward=0.0, reason="Empty model output") + + # 2) unit tests (must be present & valid BEFORE runtime; otherwise raise) + if not body.verifier_metadata or "unit_tests" not in body.verifier_metadata: + raise HTTPException(status_code=422, detail="Missing verifier_metadata.unit_tests") + try: + tests = _parse_unit_tests(body.verifier_metadata["unit_tests"]) + except Exception as e: + # Treat bad inputs as an input error, not a model failure. + raise HTTPException(status_code=422, detail=f"Invalid unit_tests: {e}") + + # 3) extract code (code fence or raw) + code = _extract_code(model_out) + if not code: + return CompCodingVerifyResponse(**body.model_dump(), reward=0.0, reason="Could not extract code") + + # 4) run (no sandbox) + ok, msg = _run_code_against_tests(code, tests) + + return CompCodingVerifyResponse( + **body.model_dump(), + reward=1.0 if ok else 0.0, + reason=msg, # always include the reason message + ) + + +if __name__ == "__main__": + CompCodingResourcesServer.run_webserver() diff --git a/resources_servers/comp_coding/configs/comp_coding.yaml b/resources_servers/comp_coding/configs/comp_coding.yaml new file mode 100644 index 0000000000..3be2b90b51 --- /dev/null +++ b/resources_servers/comp_coding/configs/comp_coding.yaml @@ -0,0 +1,27 @@ +comp_coding: + resources_servers: + comp_coding: + entrypoint: app.py + +comp_coding_simple_agent: + responses_api_agents: + simple_agent: + entrypoint: app.py + resources_server: + type: resources_servers + name: comp_coding + model_server: + type: responses_api_models + name: openai_model + datasets: + - name: train + type: train + jsonl_fpath: resources_servers/comp_coding/data/train.jsonl + gitlab_identifier: + dataset_name: comp_coding + version: 0.0.1 + artifact_fpath: train.jsonl + license: Apache 2.0 + - name: example + type: example + jsonl_fpath: resources_servers/comp_coding/data/example.jsonl diff --git a/resources_servers/comp_coding/data/.gitignore b/resources_servers/comp_coding/data/.gitignore new file mode 100644 index 0000000000..4424b6fde0 --- /dev/null +++ b/resources_servers/comp_coding/data/.gitignore @@ -0,0 +1,5 @@ +*train.jsonl +*validation.jsonl +*train_prepare.jsonl +*validation_prepare.jsonl +*example_prepare.jsonl diff --git a/resources_servers/comp_coding/data/example.jsonl b/resources_servers/comp_coding/data/example.jsonl new file mode 100644 index 0000000000..495860d880 --- /dev/null +++ b/resources_servers/comp_coding/data/example.jsonl @@ -0,0 +1,5 @@ +{"responses_create_params": {"input": [{"role": "user", "content": "You are an expert competitive programmer. You will be given a problem statement and must output a complete Python solution that reads from stdin and writes to stdout.\n\nPolycarp has $n$ different binary words. A word called binary if it contains only characters '0' and '1'. For example, these words are binary: \"0001\", \"11\", \"0\" and \"0011100\".\n\nPolycarp wants to offer his set of $n$ binary words to play a game \"words\". In this game, players name words and each next word (starting from the second) must start with the last character of the previous word. The first word can be any. For example, these sequence of words can be named during the game: \"0101\", \"1\", \"10\", \"00\", \"00001\".\n\nWord reversal is the operation of reversing the order of the characters. For example, the word \"0111\" after the reversal becomes \"1110\", the word \"11010\" after the reversal becomes \"01011\".\n\nProbably, Polycarp has such a set of words that there is no way to put them in the order correspondent to the game rules. In this situation, he wants to reverse some words from his set so that: the final set of $n$ words still contains different words (i.e. all words are unique); there is a way to put all words of the final set of words in the order so that the final sequence of $n$ words is consistent with the game rules. \n\nPolycarp wants to reverse minimal number of words. Please, help him.\n\n\n-----Input-----\n\nThe first line of the input contains one integer $t$ ($1 \\le t \\le 10^4$) — the number of test cases in the input. Then $t$ test cases follow.\n\nThe first line of a test case contains one integer $n$ ($1 \\le n \\le 2\\cdot10^5$) — the number of words in the Polycarp's set. Next $n$ lines contain these words. All of $n$ words aren't empty and contains only characters '0' and '1'. The sum of word lengths doesn't exceed $4\\cdot10^6$. All words are different.\n\nGuaranteed, that the sum of $n$ for all test cases in the input doesn't exceed $2\\cdot10^5$. Also, guaranteed that the sum of word lengths for all test cases in the input doesn't exceed $4\\cdot10^6$.\n\n\n-----Output-----\n\nPrint answer for all of $t$ test cases in the order they appear.\n\nIf there is no answer for the test case, print -1. Otherwise, the first line of the output should contain $k$ ($0 \\le k \\le n$) — the minimal number of words in the set which should be reversed. The second line of the output should contain $k$ distinct integers — the indexes of the words in the set which should be reversed. Words are numerated from $1$ to $n$ in the order they appear. If $k=0$ you can skip this line (or you can print an empty line). If there are many answers you can print any of them.\n\n\n-----Example-----\nInput\n4\n4\n0001\n1000\n0011\n0111\n3\n010\n101\n0\n2\n00000\n00001\n4\n01\n001\n0001\n00001\n\nOutput\n1\n3 \n-1\n0\n\n2\n1 2"}]}, "verifier_metadata": {"problem_id": "c69268d8bdb4da0685d7b187c88296c1", "unit_tests": {"inputs": ["4\n4\n0001\n1000\n0011\n0111\n3\n010\n101\n0\n2\n00000\n00001\n4\n01\n001\n0001\n00001\n"], "outputs": ["1\n3 \n-1\n0\n\n2\n1 2 \n"]}}} +{"responses_create_params": {"input": [{"role": "user", "content": "You are an expert competitive programmer. You will be given a problem statement and must output a complete Python solution that reads from stdin and writes to stdout.\n\nMikhail walks on a Cartesian plane. He starts at the point $(0, 0)$, and in one move he can go to any of eight adjacent points. For example, if Mikhail is currently at the point $(0, 0)$, he can go to any of the following points in one move: $(1, 0)$; $(1, 1)$; $(0, 1)$; $(-1, 1)$; $(-1, 0)$; $(-1, -1)$; $(0, -1)$; $(1, -1)$. \n\nIf Mikhail goes from the point $(x1, y1)$ to the point $(x2, y2)$ in one move, and $x1 \\ne x2$ and $y1 \\ne y2$, then such a move is called a diagonal move.\n\nMikhail has $q$ queries. For the $i$-th query Mikhail's target is to go to the point $(n_i, m_i)$ from the point $(0, 0)$ in exactly $k_i$ moves. Among all possible movements he want to choose one with the maximum number of diagonal moves. Your task is to find the maximum number of diagonal moves or find that it is impossible to go from the point $(0, 0)$ to the point $(n_i, m_i)$ in $k_i$ moves.\n\nNote that Mikhail can visit any point any number of times (even the destination point!).\n\n\n-----Input-----\n\nThe first line of the input contains one integer $q$ ($1 \\le q \\le 10^4$) — the number of queries.\n\nThen $q$ lines follow. The $i$-th of these $q$ lines contains three integers $n_i$, $m_i$ and $k_i$ ($1 \\le n_i, m_i, k_i \\le 10^{18}$) — $x$-coordinate of the destination point of the query, $y$-coordinate of the destination point of the query and the number of moves in the query, correspondingly.\n\n\n-----Output-----\n\nPrint $q$ integers. The $i$-th integer should be equal to -1 if Mikhail cannot go from the point $(0, 0)$ to the point $(n_i, m_i)$ in exactly $k_i$ moves described above. Otherwise the $i$-th integer should be equal to the the maximum number of diagonal moves among all possible movements.\n\n\n-----Example-----\nInput\n3\n2 2 3\n4 3 7\n10 1 9\n\nOutput\n1\n6\n-1\n\n\n\n-----Note-----\n\nOne of the possible answers to the first test case: $(0, 0) \\to (1, 0) \\to (1, 1) \\to (2, 2)$.\n\nOne of the possible answers to the second test case: $(0, 0) \\to (0, 1) \\to (1, 2) \\to (0, 3) \\to (1, 4) \\to (2, 3) \\to (3, 2) \\to (4, 3)$.\n\nIn the third test case Mikhail cannot reach the point $(10, 1)$ in 9 moves."}]}, "verifier_metadata": {"problem_id": "570676e22bd21c28e452486b0668ec74", "unit_tests": {"inputs": ["3\n2 2 3\n4 3 7\n10 1 9\n", "3\n2 2 3\n4 3 7\n7 1 9\n", "3\n2 2 3\n4 3 9\n7 1 9\n", "3\n2 2 3\n6 1 9\n7 1 1\n", "3\n2 1 3\n6 1 9\n7 0 2\n", "3\n2 0 3\n3 1 9\n13 0 2\n", "3\n4 0 3\n3 1 9\n13 0 2\n", "3\n7 0 0\n3 1 5\n22 0 2\n", "3\n1 0 0\n5 0 5\n22 0 2\n", "3\n1 0 1\n5 0 5\n22 0 2\n", "3\n1 0 1\n9 0 5\n35 0 1\n", "3\n2 2 3\n4 3 6\n10 1 9\n", "3\n4 2 3\n4 3 7\n7 1 9\n", "3\n2 2 3\n6 3 9\n7 2 9\n", "3\n2 2 3\n6 3 9\n7 1 12\n", "3\n3 2 3\n6 1 9\n7 1 9\n", "3\n2 2 3\n6 1 12\n7 1 1\n", "3\n2 2 3\n6 2 9\n7 0 1\n", "3\n5 0 3\n3 2 9\n13 0 2\n", "3\n7 0 0\n3 1 0\n22 0 2\n", "3\n1 0 1\n5 1 5\n22 0 3\n", "3\n1 1 1\n9 -1 5\n35 0 1\n", "3\n4 2 3\n4 3 7\n4 1 9\n", "3\n2 2 1\n6 3 9\n7 2 9\n", "3\n2 2 3\n1 3 9\n7 1 12\n", "3\n3 2 1\n6 1 9\n7 1 9\n", "3\n2 0 4\n6 1 9\n7 0 2\n", "3\n2 1 3\n6 2 9\n25 0 2\n", "3\n1 1 0\n3 1 3\n22 0 2\n", "3\n2 2 1\n1 3 9\n7 1 12\n", "3\n2 2 3\n6 2 18\n10 0 1\n", "3\n2 1 0\n6 2 9\n25 0 2\n", "3\n2 2 3\n6 3 9\n7 1 9\n", "3\n2 2 3\n6 1 9\n7 1 9\n", "3\n2 2 3\n6 1 9\n7 0 1\n", "3\n2 2 3\n6 1 9\n7 0 2\n", "3\n2 1 3\n6 1 9\n13 0 2\n", "3\n2 0 3\n6 1 9\n13 0 2\n", "3\n5 0 3\n3 1 9\n13 0 2\n", "3\n5 0 0\n3 1 9\n13 0 2\n", "3\n5 0 0\n3 1 9\n22 0 2\n", "3\n7 0 0\n3 1 9\n22 0 2\n", "3\n1 0 0\n3 1 5\n22 0 2\n", "3\n1 0 0\n5 1 5\n22 0 2\n", "3\n1 0 0\n5 0 5\n22 0 1\n", "3\n1 0 0\n5 0 5\n22 1 1\n", "3\n1 0 1\n5 0 5\n22 0 3\n", "3\n1 0 1\n5 0 5\n22 0 1\n", "3\n1 0 1\n5 0 5\n35 0 1\n", "3\n1 0 1\n9 -1 5\n35 0 1\n", "3\n1 0 1\n9 0 5\n35 -1 1\n", "3\n2 0 3\n6 1 9\n7 0 2\n", "3\n2 1 3\n6 1 9\n10 0 2\n", "3\n2 1 3\n6 1 9\n25 0 2\n", "3\n2 0 3\n6 1 9\n13 -1 2\n", "3\n2 0 1\n3 1 9\n13 0 2\n", "3\n4 0 3\n1 1 9\n13 0 2\n", "3\n5 0 0\n3 1 9\n13 0 0\n", "3\n5 -1 0\n3 1 9\n22 0 2\n", "3\n1 1 0\n3 1 5\n22 0 2\n", "3\n1 0 0\n5 1 5\n38 0 2\n", "3\n1 0 0\n5 0 5\n22 0 0\n", "3\n1 0 0\n5 0 5\n22 2 1\n", "3\n1 0 1\n5 0 5\n22 1 1\n", "3\n1 0 1\n9 0 5\n15 0 1\n", "3\n1 0 1\n9 0 5\n35 -1 0\n", "3\n2 0 3\n4 3 6\n10 1 9\n", "3\n2 2 3\n6 1 12\n5 1 1\n", "3\n2 2 3\n6 2 9\n10 0 1\n", "3\n2 2 3\n6 1 9\n10 0 2\n", "3\n3 0 3\n6 1 9\n13 0 2\n", "3\n2 0 1\n3 1 9\n20 0 2\n", "3\n4 0 3\n1 1 9\n13 0 1\n", "3\n5 0 3\n3 2 9\n13 0 3\n", "3\n5 0 0\n5 1 9\n13 0 0\n", "3\n3 -1 0\n3 1 9\n22 0 2\n", "3\n7 0 0\n3 1 1\n22 0 2\n", "3\n1 0 0\n5 1 5\n38 0 3\n", "3\n1 0 0\n5 0 5\n22 2 0\n", "3\n1 0 1\n5 1 5\n17 0 3\n", "3\n1 0 1\n5 1 5\n22 1 1\n", "3\n1 0 1\n9 0 0\n15 0 1\n", "3\n1 1 1\n9 0 5\n35 0 1\n", "3\n1 0 1\n9 0 5\n52 -1 0\n", "3\n2 0 3\n7 3 6\n10 1 9\n", "3\n4 2 3\n4 1 7\n4 1 9\n", "3\n2 2 1\n4 3 9\n7 2 9\n", "3\n3 2 1\n6 1 9\n7 1 2\n", "3\n2 2 3\n6 1 12\n9 1 1\n", "3\n2 0 0\n6 1 9\n7 0 2\n", "3\n2 2 3\n6 2 9\n10 0 2\n", "3\n2 0 3\n1 1 9\n13 0 2\n", "3\n2 0 1\n3 0 9\n20 0 2\n", "3\n4 0 1\n1 1 9\n13 0 1\n", "3\n5 0 3\n3 2 9\n13 0 4\n", "3\n5 0 0\n5 1 9\n22 0 0\n", "3\n2 2 3\n4 3 7\n10 1 9\n"], "outputs": ["1\n6\n-1\n", "1\n6\n9\n", "1\n8\n9\n", "1\n8\n-1\n", "2\n8\n-1\n", "1\n9\n-1\n", "-1\n9\n-1\n", "-1\n5\n-1\n", "-1\n4\n-1\n", "0\n4\n-1\n", "0\n-1\n-1\n", "1\n5\n-1\n", "-1\n6\n9\n", "1\n8\n8\n", "1\n8\n10\n", "2\n8\n9\n", "1\n11\n-1\n", "1\n7\n-1\n", "-1\n8\n-1\n", "-1\n-1\n-1\n", "0\n5\n-1\n", "1\n-1\n-1\n", "-1\n6\n8\n", "-1\n8\n8\n", "1\n9\n10\n", "-1\n8\n9\n", "4\n8\n-1\n", "2\n7\n-1\n", "-1\n3\n-1\n", "-1\n9\n10\n", "1\n18\n-1\n", "-1\n7\n-1\n", "1\n8\n9\n", "1\n8\n9\n", "1\n8\n-1\n", "1\n8\n-1\n", "2\n8\n-1\n", "1\n8\n-1\n", "-1\n9\n-1\n", "-1\n9\n-1\n", "-1\n9\n-1\n", "-1\n9\n-1\n", "-1\n5\n-1\n", "-1\n5\n-1\n", "-1\n4\n-1\n", "-1\n4\n-1\n", "0\n4\n-1\n", "0\n4\n-1\n", "0\n4\n-1\n", "0\n-1\n-1\n", "0\n-1\n-1\n", "1\n8\n-1\n", "2\n8\n-1\n", "2\n8\n-1\n", "1\n8\n-1\n", "-1\n9\n-1\n", "-1\n9\n-1\n", "-1\n9\n-1\n", "-1\n9\n-1\n", "-1\n5\n-1\n", "-1\n5\n-1\n", "-1\n4\n-1\n", "-1\n4\n-1\n", "0\n4\n-1\n", "0\n-1\n-1\n", "0\n-1\n-1\n", "1\n5\n-1\n", "1\n11\n-1\n", "1\n7\n-1\n", "1\n8\n-1\n", "2\n8\n-1\n", "-1\n9\n-1\n", "-1\n9\n-1\n", "-1\n8\n-1\n", "-1\n9\n-1\n", "-1\n9\n-1\n", "-1\n-1\n-1\n", "-1\n5\n-1\n", "-1\n4\n-1\n", "0\n5\n-1\n", "0\n5\n-1\n", "0\n-1\n-1\n", "1\n-1\n-1\n", "0\n-1\n-1\n", "1\n-1\n-1\n", "-1\n6\n8\n", "-1\n8\n8\n", "-1\n8\n-1\n", "1\n11\n-1\n", "-1\n8\n-1\n", "1\n7\n-1\n", "1\n9\n-1\n", "-1\n8\n-1\n", "-1\n9\n-1\n", "-1\n8\n-1\n", "-1\n9\n-1\n", "1\n6\n-1\n"]}}} +{"responses_create_params": {"input": [{"role": "user", "content": "You are an expert competitive programmer. You will be given a problem statement and must output a complete Python solution that reads from stdin and writes to stdout.\n\nYou are given three sequences: $a_1, a_2, \\ldots, a_n$; $b_1, b_2, \\ldots, b_n$; $c_1, c_2, \\ldots, c_n$.\n\nFor each $i$, $a_i \\neq b_i$, $a_i \\neq c_i$, $b_i \\neq c_i$.\n\nFind a sequence $p_1, p_2, \\ldots, p_n$, that satisfy the following conditions:\n\n\n\n $p_i \\in \\{a_i, b_i, c_i\\}$\n\n $p_i \\neq p_{(i \\mod n) + 1}$.\n\nIn other words, for each element, you need to choose one of the three possible values, such that no two adjacent elements (where we consider elements $i,i+1$ adjacent for $i=2.20.0 diff --git a/resources_servers/comp_coding/scripts/build_examples.py b/resources_servers/comp_coding/scripts/build_examples.py new file mode 100644 index 0000000000..4d2fc1590e --- /dev/null +++ b/resources_servers/comp_coding/scripts/build_examples.py @@ -0,0 +1,220 @@ +# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +r""" +Build N rows of train/validation/example data from the HF dataset: + Nexusflow/comp_prog_filtered_no_function + +Each output row conforms to NeMo-Gym dataset requirements: +- responses_create_params: OpenAI Responses-compatible input +- verifier_metadata.unit_tests: {inputs: [...], outputs: [...]} (strings) + +Usage: + uv run python resources_servers/comp_coding/scripts/build_examples.py \ + --out resources_servers/comp_coding/data/example.jsonl \ + --count 5000 + + # Also write a human-readable sample of the first 10 rows: + uv run python resources_servers/comp_coding/scripts/build_examples.py \ + --out resources_servers/comp_coding/data/example.jsonl \ + --count 5000 --pretty-sample resources_servers/comp_coding/data/sample.json \ + --pretty-k 10 + + # Gzip the jsonl: + uv run python resources_servers/comp_coding/scripts/build_examples.py \ + --out resources_servers/comp_coding/data/example.jsonl.gz \ + --count 5000 + +Sanity checks after writing: + # Confirm total rows written + jq -c . resources_servers/comp_coding/data/example.jsonl | wc -l + + # Inspect first few rows in pretty form + head -n 3 resources_servers/comp_coding/data/example.jsonl | jq . + + # Ensure each row is single-line JSON (no embedded real newlines) + awk 'NR==1{print; exit}' resources_servers/comp_coding/data/example.jsonl | tr -d '\n' | wc -c + + # Detect raw Unicode LS/PS (should be NONE after this script) + # grep (PCRE) variant: + grep -nP '\x{2028}|\x{2029}' resources_servers/comp_coding/data/example.jsonl || echo "No LS/PS found" + # or python: + python - <<'PY' +import sys, re +p = re.compile('[\u2028\u2029]') +fn = 'resources_servers/comp_coding/data/example.jsonl' +bad = [] +with open(fn, 'r', encoding='utf-8') as f: + for i, line in enumerate(f, 1): + if p.search(line): + bad.append(i) +print("LS/PS lines:", bad if bad else "None") +PY + +Notes: + - We escape U+2028 (LS) and U+2029 (PS) as \\u2028 / \\u2029 so JSON parsers (esp. JS) won’t choke. +""" + +import argparse +import ast +import gzip +import json +import re +from itertools import islice +from typing import Any, Dict, Iterable, List, Optional + +from datasets import load_dataset +from tqdm import tqdm + + +SYSTEM_PREFIX = ( + "You are an expert competitive programmer. You will be given a problem statement " + "and must output a complete Python solution that reads from stdin and writes to stdout." +) + +CODEFENCE_RE = re.compile(r"^```(?:\w+)?\s*|\s*```$", re.MULTILINE) + + +def _strip_codefences(s: str) -> str: + return CODEFENCE_RE.sub("", s).strip() + + +def _normalize_scalar(s: Any) -> str: + s = "" if s is None else str(s) + s = s.replace("\r\n", "\n").replace("\r", "\n") + return s.rstrip("\n").strip() + + +def _normalize_list(lst: Any) -> List[str]: + if lst is None: + return [] + if isinstance(lst, (str, bytes)): + lst = [lst] + if not isinstance(lst, list): + return [] + out = [] + for v in lst: + sv = _normalize_scalar(v) + sv = _strip_codefences(sv) + out.append(sv) + return out + + +def _safe_literal_eval(s: str) -> Any: + try: + return ast.literal_eval(s) + except Exception: + return None + + +def _parse_unit_tests(raw: Any) -> Dict[str, List[str]]: + parsed: Dict[str, Any] = {} + if isinstance(raw, dict): + parsed = raw + elif isinstance(raw, str) and raw.strip(): + try: + parsed = json.loads(raw) + except Exception: + maybe = _safe_literal_eval(raw) + if isinstance(maybe, dict): + parsed = maybe + else: + parsed = {} + else: + parsed = {} + + return { + "inputs": _normalize_list(parsed.get("inputs", [])), + "outputs": _normalize_list(parsed.get("outputs", [])), + } + + +def make_row(q: str, unit_tests: Dict[str, List[str]], problem_id: Optional[str] = None) -> dict: + q_norm = _normalize_scalar(q) + return { + "responses_create_params": {"input": [{"role": "user", "content": f"{SYSTEM_PREFIX}\n\n{q_norm}"}]}, + "verifier_metadata": { + "problem_id": _normalize_scalar(problem_id) if problem_id is not None else None, + "unit_tests": { + "inputs": unit_tests.get("inputs", []), + "outputs": unit_tests.get("outputs", []), + }, + }, + } + + +def _open_out(path: str): + if path.endswith(".gz"): + return gzip.open(path, "wt", encoding="utf-8") + return open(path, "w", encoding="utf-8") + + +def json_safe_dumps(obj: dict) -> str: + """ + Dump JSON compactly and escape problematic Unicode line separators. + Ensures no raw U+2028/U+2029 appear in output (they become \\u2028/\\u2029). + """ + s = json.dumps(obj, ensure_ascii=False, separators=(",", ":")) + # Escape LS (U+2028) and PS (U+2029) + return s.replace("\u2028", "\\u2028").replace("\u2029", "\\u2029") + + +def stream_dataset(ds_name: str, split: str = "train") -> Iterable[dict]: + ds = load_dataset(ds_name, split=split, streaming=True) + for ex in ds: + yield ex + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--out", required=True, help="Output .jsonl or .jsonl.gz") + ap.add_argument("--count", type=int, default=5000, help="Number of rows to write") + ap.add_argument("--split", default="train", help="HF split name (default: train)") + ap.add_argument("--pretty-sample", default=None, help="Optional pretty JSON of first K rows") + ap.add_argument("--pretty-k", type=int, default=10, help="How many rows to pretty-print") + ap.add_argument("--ds-name", default="Nexusflow/comp_prog_filtered_no_function") + args = ap.parse_args() + + rows_for_pretty = [] + total = 0 + + with _open_out(args.out) as f: + for ex in tqdm( + islice(stream_dataset(args.ds_name, args.split), args.count), + total=args.count, + ): + q = ex.get("question", "") + raw_ut = ex.get("unit_tests", {}) or {} + ut = _parse_unit_tests(raw_ut) + pid = ex.get("hash_id") + + row = make_row(q, ut, pid) + f.write(json_safe_dumps(row) + "\n") + + if args.pretty_sample and len(rows_for_pretty) < args.pretty_k: + rows_for_pretty.append(row) + + total += 1 + + if args.pretty_sample and rows_for_pretty: + with open(args.pretty_sample, "w", encoding="utf-8") as ps: + json.dump(rows_for_pretty, ps, ensure_ascii=False, indent=2) + + print(f"wrote {total} rows to {args.out}") + if args.pretty_sample: + print(f"wrote pretty sample ({len(rows_for_pretty)} rows) to {args.pretty_sample}") + + +if __name__ == "__main__": + main() diff --git a/resources_servers/comp_coding/scripts/validate_dataset.py b/resources_servers/comp_coding/scripts/validate_dataset.py new file mode 100644 index 0000000000..bcd3280c23 --- /dev/null +++ b/resources_servers/comp_coding/scripts/validate_dataset.py @@ -0,0 +1,236 @@ +# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +r""" +Validate and (optionally) normalize comp_coding JSONL datasets **before** runtime. + +What it does: +- Ensures each row has: + - responses_create_params.input (non-empty) + - verifier_metadata.unit_tests with: + - "inputs": list[str] (non-empty) + - "outputs": list[str] (same length as inputs) +- Optionally coerces stringified unit_tests into a dict (e.g., when stored as JSON string) +- Optionally normalizes newlines by converting literal "\\n" to "\n" (inputs/outputs) +- Can write out a cleaned JSONL (dropping bad rows or failing fast) + +Usage: + uv run python resources_servers/comp_coding/scripts/validate_dataset.py \ + --in data/comp_coding/train.jsonl --fail-fast + + uv run python resources_servers/comp_coding/scripts/validate_dataset.py \ + --in data/comp_coding/train.jsonl \ + --out data/comp_coding/train.cleaned.jsonl \ + --autofix --normalize-newlines --drop-bad + +CLI flags: + --in PATH [--in PATH ...] One or more JSONL files to validate + --out PATH Where to write a cleaned JSONL (optional) + --autofix Try to parse stringified unit_tests to dict + --normalize-newlines Replace literal "\\n" with "\n" in tests + --fail-fast Stop at first error (default: keep scanning) + --drop-bad When --out is set, skip invalid rows instead of failing + --pretty-sample PATH Write a small pretty-printed sample (first 100 ok rows) + +Exit codes: + 0 on success (or successful write when --out provided) + 1 on validation error (unless --drop-bad used and output written) +""" + +import argparse +import json +from pathlib import Path +from typing import Any, Dict, List, Tuple + + +def _is_list_of_str(x: Any) -> bool: + return isinstance(x, list) and all(isinstance(s, str) for s in x) + + +def _maybe_parse_unit_tests(ut: Any, autofix: bool) -> Dict[str, Any]: + if isinstance(ut, dict): + return ut + if isinstance(ut, str) and autofix: + # Try strict JSON first + try: + parsed = json.loads(ut) + if isinstance(parsed, dict): + return parsed + except Exception: + pass + # Try lenient: strip whitespace and single quotes + try: + s = ut.strip().replace("'", '"') + parsed = json.loads(s) + if isinstance(parsed, dict): + return parsed + except Exception: + pass + raise ValueError("unit_tests must be a dict (or a JSON string if --autofix).") + + +def _normalize_newlines_in_tests(ut: Dict[str, Any]) -> Dict[str, Any]: + def fix(s: str) -> str: + return s.replace("\\n", "\n") + + inputs = ut.get("inputs", []) + outputs = ut.get("outputs", []) + if isinstance(inputs, list): + inputs = [fix(s) if isinstance(s, str) else s for s in inputs] + if isinstance(outputs, list): + outputs = [fix(s) if isinstance(s, str) else s for s in outputs] + ut["inputs"] = inputs + ut["outputs"] = outputs + return ut + + +def _validate_unit_tests(ut: Dict[str, Any]) -> Tuple[bool, str]: + inputs = ut.get("inputs") + outputs = ut.get("outputs") + if not _is_list_of_str(inputs): + return False, "unit_tests.inputs must be list[str] and non-empty" + if not _is_list_of_str(outputs): + return False, "unit_tests.outputs must be list[str]" + if len(inputs) == 0: + return False, "unit_tests.inputs cannot be empty" + if len(inputs) != len(outputs): + return False, f"inputs/outputs length mismatch: {len(inputs)} vs {len(outputs)}" + return True, "ok" + + +def _validate_row( + row: Dict[str, Any], idx: int, autofix: bool, normalize_newlines: bool +) -> Tuple[bool, Dict[str, Any], str]: + # responses_create_params sanity + rcp = row.get("responses_create_params") + if not isinstance(rcp, dict): + return False, row, "missing responses_create_params" + input_blocks = rcp.get("input") + if not isinstance(input_blocks, list) or len(input_blocks) == 0: + return False, row, "responses_create_params.input must be a non-empty list" + + # unit_tests presence + structure + vm = row.get("verifier_metadata") + if not isinstance(vm, dict): + return False, row, "missing verifier_metadata" + if "unit_tests" not in vm: + return False, row, "missing verifier_metadata.unit_tests" + + try: + ut = _maybe_parse_unit_tests(vm["unit_tests"], autofix=autofix) + except Exception as e: + return False, row, f"unit_tests parse error: {e}" + + if normalize_newlines: + ut = _normalize_newlines_in_tests(ut) + + ok, msg = _validate_unit_tests(ut) + if not ok: + return False, row, msg + + # If we fixed ut, write it back normalized + vm["unit_tests"] = {"inputs": ut["inputs"], "outputs": ut["outputs"]} + row["verifier_metadata"] = vm + return True, row, "ok" + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--in", dest="inputs", nargs="+", required=True) + ap.add_argument("--out", dest="out", type=str, default=None) + ap.add_argument("--autofix", action="store_true") + ap.add_argument("--normalize-newlines", action="store_true") + ap.add_argument("--fail-fast", action="store_true") + ap.add_argument("--drop-bad", action="store_true") + ap.add_argument("--pretty-sample", type=str, default=None) + args = ap.parse_args() + + in_paths = [Path(p) for p in args.inputs] + out_path = Path(args.out) if args.out else None + sample_path = Path(args.pretty_sample) if args.pretty_sample else None + + if out_path: + out_path.parent.mkdir(parents=True, exist_ok=True) + if sample_path: + sample_path.parent.mkdir(parents=True, exist_ok=True) + + total = 0 + ok_count = 0 + bad_count = 0 + written = 0 + sample: List[Dict[str, Any]] = [] + + out_f = open(out_path, "w", encoding="utf-8") if out_path else None + try: + for in_file in in_paths: + with open(in_file, "r", encoding="utf-8") as f: + for line_idx, line in enumerate(f, start=1): + line = line.strip() + if not line: + continue + total += 1 + try: + row = json.loads(line) + except Exception as e: + bad_count += 1 + msg = f"{in_file}:{line_idx} invalid JSON: {e}" + if args.fail_fast: + raise SystemExit(msg) + else: + print("ERROR:", msg) + continue + + ok, fixed, msg = _validate_row( + row, + total, + autofix=args.autofix, + normalize_newlines=args.normalize_newlines, + ) + if ok: + ok_count += 1 + if out_f: + out_f.write(json.dumps(fixed, ensure_ascii=False) + "\n") + written += 1 + if len(sample) < 100 and sample_path: + sample.append(fixed) + else: + bad_count += 1 + if args.fail_fast and not args.drop_bad: + raise SystemExit(f"{in_file}:{line_idx} {msg}") + print("ERROR:", f"{in_file}:{line_idx}", msg) + if out_f and args.drop_bad: + # skip writing this row + pass + elif out_f and not args.drop_bad: + # fail the whole run if we plan to produce a cleaned file but don’t drop bad rows + raise SystemExit( + f"Refusing to write invalid row without --drop-bad: {in_file}:{line_idx} {msg}" + ) + + if sample_path and sample: + with open(sample_path, "w", encoding="utf-8") as s: + json.dump(sample, s, ensure_ascii=False, indent=2) + + print(f"Scanned rows: {total} | OK: {ok_count} | Bad: {bad_count}") + if out_f: + print(f"Wrote cleaned rows: {written} -> {out_path}") + if bad_count and not (out_f and args.drop_bad): + raise SystemExit(1) + finally: + if out_f: + out_f.close() + + +if __name__ == "__main__": + main() diff --git a/resources_servers/comp_coding/tests/__init__.py b/resources_servers/comp_coding/tests/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/resources_servers/comp_coding/tests/test_app.py b/resources_servers/comp_coding/tests/test_app.py new file mode 100644 index 0000000000..887d5c2e48 --- /dev/null +++ b/resources_servers/comp_coding/tests/test_app.py @@ -0,0 +1,257 @@ +# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from unittest.mock import MagicMock + +import pytest +from app import ( + CompCodingResourcesServer, + CompCodingResourcesServerConfig, + CompCodingVerifyRequest, +) +from pydantic import ValidationError + +from nemo_gym.openai_utils import NeMoGymResponse +from nemo_gym.server_utils import ServerClient + + +class TestApp: + def test_sanity(self) -> None: + cfg = CompCodingResourcesServerConfig(host="0.0.0.0", port=8080, entrypoint="") + CompCodingResourcesServer(config=cfg, server_client=MagicMock(spec=ServerClient)) + + async def test_verify_pass_via_response(self) -> None: + # Assistant returns a python code block that squares the input + response = NeMoGymResponse( + id="resp_ok", + created_at=0.0, + model="dummy", + object="response", + output=[ + { + "id": "msg_ok", + "content": [ + { + "annotations": [], + "text": "```python\nn=int(input())\nprint(n*n)\n```", + "type": "output_text", + } + ], + "role": "assistant", + "status": "completed", + "type": "message", + } + ], + parallel_tool_calls=False, + tool_choice="auto", + tools=[], + ) + + server = CompCodingResourcesServer( + config=CompCodingResourcesServerConfig(host="0.0.0.0", port=8080, entrypoint=""), + server_client=MagicMock(spec=ServerClient), + ) + + verify_req = CompCodingVerifyRequest( + responses_create_params={ + "input": [{"role": "user", "content": "Read n and print n^2."}], + "temperature": 0, + "parallel_tool_calls": False, + }, + response=response, + verifier_metadata={"unit_tests": {"inputs": ["2\n", "5\n"], "outputs": ["4", "25"]}}, + ) + + res = await server.verify(verify_req) + assert res.reward == 1.0, res.reason + + async def test_verify_fail_wrong_answer(self) -> None: + # Assistant prints n+1 instead of n*n + response_bad = NeMoGymResponse( + id="resp_bad", + created_at=0.0, + model="dummy", + object="response", + output=[ + { + "id": "msg_bad", + "content": [ + { + "annotations": [], + "text": "```python\nn=int(input())\nprint(n+1)\n```", + "type": "output_text", + } + ], + "role": "assistant", + "status": "completed", + "type": "message", + } + ], + parallel_tool_calls=False, + tool_choice="auto", + tools=[], + ) + + server = CompCodingResourcesServer( + config=CompCodingResourcesServerConfig(host="0.0.0.0", port=8080, entrypoint=""), + server_client=MagicMock(spec=ServerClient), + ) + + verify_req_bad = CompCodingVerifyRequest( + responses_create_params={"input": [{"role": "user", "content": "square n"}]}, + response=response_bad, + verifier_metadata={"unit_tests": {"inputs": ["3\n"], "outputs": ["9"]}}, + ) + + res2 = await server.verify(verify_req_bad) + assert res2.reward == 0.0 and "FAILED" in res2.reason + + async def test_verify_missing_response_validation_error(self) -> None: + """Omitting `response` should fail request validation (schema requires it).""" + _ = CompCodingResourcesServer( + config=CompCodingResourcesServerConfig(host="0.0.0.0", port=8080, entrypoint=""), + server_client=MagicMock(spec=ServerClient), + ) + + with pytest.raises(ValidationError): + _ = CompCodingVerifyRequest( + responses_create_params={"input": [{"role": "user", "content": "anything"}]}, + # response is intentionally omitted + verifier_metadata={"unit_tests": {"inputs": ["1\n"], "outputs": ["1"]}}, + ) + + async def test_verify_no_code_block(self) -> None: + """Test when response contains no code block - should extract raw text""" + response = NeMoGymResponse( + id="resp_no_block", + created_at=0.0, + model="dummy", + object="response", + output=[ + { + "id": "msg_no_block", + "content": [ + { + "annotations": [], + "text": "n=int(input())\nprint(n*n)", # No ```python``` wrapper + "type": "output_text", + } + ], + "role": "assistant", + "status": "completed", + "type": "message", + } + ], + parallel_tool_calls=False, + tool_choice="auto", + tools=[], + ) + + server = CompCodingResourcesServer( + config=CompCodingResourcesServerConfig(host="0.0.0.0", port=8080, entrypoint=""), + server_client=MagicMock(spec=ServerClient), + ) + + verify_req = CompCodingVerifyRequest( + responses_create_params={ + "input": [{"role": "user", "content": "Read n and print n^2."}], + }, + response=response, + verifier_metadata={"unit_tests": {"inputs": ["2\n"], "outputs": ["4"]}}, + ) + + res = await server.verify(verify_req) + assert res.reward == 1.0, res.reason + + async def test_verify_syntax_error(self) -> None: + """Code has a syntax error -> should report ERROR and reward 0.0""" + server = CompCodingResourcesServer( + config=CompCodingResourcesServerConfig(host="0.0.0.0", port=8080, entrypoint=""), + server_client=MagicMock(spec=ServerClient), + ) + + response = NeMoGymResponse( + id="resp_syntax_error", + created_at=0.0, + model="dummy", + object="response", + output=[ + { + "id": "msg_bad_syntax", + "content": [ + { + "annotations": [], + "text": "```python\nprint('hello' # Missing closing parenthesis\n```", + "type": "output_text", + } + ], + "role": "assistant", + "status": "completed", + "type": "message", + } + ], + parallel_tool_calls=False, + tool_choice="auto", + tools=[], + ) + + verify_req = CompCodingVerifyRequest( + responses_create_params={"input": [{"role": "user", "content": "Print hello"}]}, + response=response, + verifier_metadata={"unit_tests": {"inputs": ["\n"], "outputs": ["hello"]}}, + ) + + res = await server.verify(verify_req) + assert res.reward == 0.0 and "ERROR" in res.reason + + async def test_verify_runtime_error(self) -> None: + server = CompCodingResourcesServer( + config=CompCodingResourcesServerConfig(host="0.0.0.0", port=8080, entrypoint=""), + server_client=MagicMock(spec=ServerClient), + ) + + response = NeMoGymResponse( + id="resp_runtime_error", + created_at=0.0, + model="dummy", + object="response", + output=[ + { + "id": "msg_runtime_error", + "content": [ + { + "annotations": [], + "text": "```python\nn=int(input())\nprint(n/0)\n```", + "type": "output_text", + } + ], + "role": "assistant", + "status": "completed", + "type": "message", + } + ], + parallel_tool_calls=False, + tool_choice="auto", + tools=[], + ) + + verify_req = CompCodingVerifyRequest( + responses_create_params={"input": [{"role": "user", "content": "Divide by zero"}]}, + response=response, + verifier_metadata={"unit_tests": {"inputs": ["5\n"], "outputs": ["error"]}}, + ) + + res = await server.verify(verify_req) + assert res.reward == 0.0 and "ERROR" in res.reason