diff --git a/docs/about/concepts/configuration-system.md b/docs/about/concepts/configuration-system.md deleted file mode 100644 index a37816441..000000000 --- a/docs/about/concepts/configuration-system.md +++ /dev/null @@ -1,210 +0,0 @@ -(configuration-management)= - -# Configuration Management - -NeMo Gym uses a powerful configuration system with three sources that are resolved in this order: - -``` -Server YAML Config Files < env.yaml < Command Line Arguments - (lowest priority) (highest priority) -``` - -This allows for: -- Base configuration in YAML files (shared settings) -- Secrets and environment-specific values in `env.yaml` -- Runtime overrides via command line arguments - -::::{tab-set} - -:::{tab-item} 1. Server YAML Config Files - -These base configurations define server structures and default values, with later files overriding earlier ones. - -Example: Multi-Server Configuration -```bash -# Define which config files to load -config_paths="responses_api_models/openai_model/configs/openai_model.yaml,\ -resources_servers/example_simple_weather/configs/simple_weather.yaml,\ -responses_api_agents/simple_agent/configs/simple_agent.yaml" - -ng_run "+config_paths=[${config_paths}]" -``` - -Every config file defines **server instances** with this hierarchy: - -```yaml -# Server ID - unique name used in requests and references -simple_weather_simple_agent: - # Server type - must be one of: responses_api_models, resources_servers, responses_api_agents - # These match the 3 top-level folders in NeMo-Gym - responses_api_agents: - # Implementation type - must match a folder name inside responses_api_agents/ - simple_agent: - # Entrypoint - Python file to run (relative to implementation folder) - entrypoint: app.py - # Server-specific configuration (varies by implementation) - resources_server: - type: resources_servers # What type of server to reference - name: simple_weather # Which specific server instance - model_server: - type: responses_api_models - name: policy_model # References the model server -``` - -::: -:::{tab-item} 2. env.yaml - -Your `env.yaml` file contains **secrets and environment-specific values** that should never be committed to version control. - -### Basic env.yaml - -```yaml -# API credentials (never commit these!) -policy_base_url: https://api.openai.com/v1 -policy_api_key: sk-your-actual-api-key-here -policy_model_name: gpt-4o-2024-11-20 -``` - -### Advanced env.yaml with Config Paths - -```yaml -# Store complex config paths for convenience -simple_weather_config_paths: - - responses_api_models/openai_model/configs/openai_model.yaml - - resources_servers/example_simple_weather/configs/simple_weather.yaml - -# Different environments -dev_model_name: gpt-4o-mini -prod_model_name: gpt-4o-2024-11-20 - -# Custom server settings -custom_host: 0.0.0.0 -custom_port: 8080 -``` - -**Usage with stored config paths**: -```bash -ng_run '+config_paths=${simple_weather_config_paths}' -``` - -::: - -:::{tab-item} 3. Command Line Arguments - -**Runtime overrides** using Hydra syntax for maximum flexibility. These command line arguments have the highest priority and can override any settings from config.yaml or env.yaml files. - -Basic Overrides -```bash -# Override a specific model -ng_run "+config_paths=[config.yaml]" \ - +policy_model.responses_api_models.openai_model.openai_model=gpt-4o-mini - -# Point agent to different resource server -ng_run "+config_paths=[config.yaml]" \ - +simple_agent.responses_api_agents.simple_agent.resources_server.name=different_weather -``` - -Advanced Overrides -```bash -# Multiple overrides for testing -ng_run "+config_paths=[${config_paths}]" \ - +policy_model_name=gpt-4o-mini \ - +simple_weather.resources_servers.simple_weather.host=localhost \ - +simple_weather.resources_servers.simple_weather.port=9090 -``` - -::: -:::: - -## Special Policy Model Variables - -NeMo Gym provides standard placeholders for the {term}`Policy Model` being trained: - -```yaml -# These variables are available in any config file -policy_base_url: https://api.openai.com/v1 # Model API endpoint -policy_api_key: sk-your-key # Authentication -policy_model_name: gpt-4o-2024-11-20 # Model identifier -``` - -**Why these exist**: When training agents, you need consistent references to "the model being trained" across different resource servers and agents. - -**Usage in config files**: -```yaml -policy_model: - responses_api_models: - openai_model: - openai_base_url: ${policy_base_url} # Resolves from env.yaml - openai_api_key: ${policy_api_key} # Resolves from env.yaml - openai_model: ${policy_model_name} # Resolves from env.yaml -``` - - -## Troubleshooting - -NeMo Gym validates your configuration and provides helpful error messages. - -:::{dropdown} "Missing mandatory value" -``` -omegaconf.errors.MissingMandatoryValue: Missing mandatory value: policy_api_key -``` -**Fix**: Add the missing value to `env.yaml` or command line. -::: - -:::{dropdown} "Could not find X in the list of available servers" -``` -AssertionError: Could not find type='resources_servers' name='typo_weather' -in the list of available servers: [simple_weather, math_with_judge, ...] -``` -**Fix**: Check your server name spelling and ensure the config is loaded. -::: - -:::{dropdown} "Almost-Servers Detected" -Example: -```bash -═══════════════════════════════════════════════════ -Configuration Warnings: Almost-Servers Detected -═══════════════════════════════════════════════════ - - Almost-Server Detected: 'example_simple_agent' - This server configuration failed validation: - -- ResourcesServerInstanceConfig -> resources_servers -> example_server -> domain: Input should be 'math', 'coding', 'agent', 'knowledge', 'instruction_following', 'long_context', 'safety', 'games', 'e2e' or 'other' - - This server will NOT be started. -``` -**What this means**: Your server configuration has the correct structure (entrypoint, server type, etc.) but contains invalid values that prevent it from starting. - -**Common causes**: -- Invalid `license` enum values in datasets (must be one of the allowed options). - - see the `license` field in `DatasetConfig` in `config_types.py`. -- Missing or invalid `domain` field for resources servers (math, coding, agent, knowledge, etc.) - - see the `Domain` class in `config_types.py`. -- Malformed server references (wrong type or name) - -**Fix**: Update the configuration based on the validation errors shown. The warning will detail exactly which fields are problematic. - -#### Strict Validation Mode - -By default, invalid servers will throw an error. You can bypass strict validation and just show a warning: - -**In env.yaml:** -```yaml -error_on_almost_servers: false # Will not error on invalid config -``` - -**Via command line:** -```bash -ng_run "+config_paths=[config.yaml]" +error_on_almost_servers=false -``` - -**Default behavior** (`error_on_almost_servers=true`): -- All configuration issues are detected and warnings are printed -- NeMo Gym exits with an error, preventing servers from starting with invalid configs - -**When disabled** (`error_on_almost_servers=false`): -- All configuration issues are still detected and warnings are printed -- NeMo Gym continues execution despite the invalid configurations -- Invalid servers are skipped, and valid servers will attempt to start - -::: diff --git a/docs/about/concepts/configuration.md b/docs/about/concepts/configuration.md new file mode 100644 index 000000000..85ce55b32 --- /dev/null +++ b/docs/about/concepts/configuration.md @@ -0,0 +1,67 @@ +(configuration-concepts)= + +# Configuration System + +NeMo Gym uses YAML configuration files to define [Model, Resources, and Agent servers](./core-components.md). Each server gets its own configuration block, providing modular control over the entire training environment. + +## How Servers Connect + +A training environment typically includes all three server types working together. The Agent server config specifies which Model and Resources servers to use by referencing their server IDs. These references connect each training environment together — the Agent knows which Model to call and which Resources to use. + +## Config File Locations + +Each server type has a dedicated directory with its implementations and their configs: + +```text +# Model Server Config +responses_api_models/ + └── openai_model/ + └── configs/openai_model.yaml + +# Resources Server Config +resources_servers/ + └── example_simple_weather/ + └── configs/simple_weather.yaml + +# Agent Server Config +responses_api_agents/ + └── simple_agent/ + └── configs/simple_agent.yaml +``` + +## Server Block Structure + +Each config file defines a server using this structure: +```yaml +server_id: # Your unique name for this server + server_type: # responses_api_models | resources_servers | responses_api_agents + implementation: # Directory name inside the server type directory + entrypoint: app.py # Python file to run + # ... additional fields vary by server type +``` + +Different server types have additional required fields (e.g., `domain` for resources servers, `resources_server` and `model_server` for agents). See {doc}`/reference/configuration` for complete field specifications. + +Config files in NeMo Gym often use the same name for both server ID and implementation: + +```yaml +example_simple_weather: # ← Server ID + resources_servers: + example_simple_weather: # ← Implementation +``` + +These serve different purposes: + +- **Server ID** (`example_simple_weather` on line 1): Your chosen identifier for this server instance. Used in API requests and when other servers reference it. You could name it `my_weather` or `weather_prod` instead. + +- **Implementation** (`example_simple_weather` on line 3): Must match the folder `resources_servers/example_simple_weather/`. This tells NeMo Gym which code to run. + +Examples often use matching names for simplicity, but the two values are independent choices. + +--- + +:::{seealso} +- {doc}`/reference/configuration` for complete syntax and field specifications +- {doc}`/troubleshooting/configuration` for troubleshooting configuration related errors +::: + diff --git a/docs/about/concepts/index.md b/docs/about/concepts/index.md index f8ccce4e5..75f67af47 100644 --- a/docs/about/concepts/index.md +++ b/docs/about/concepts/index.md @@ -26,12 +26,6 @@ Each explainer below covers one foundational idea and links to deeper material. Understand how Models, Resources, and Agents remain decoupled yet coordinated as independent HTTP services, including which endpoints each component exposes. ::: -:::{grid-item-card} {octicon}`gear;1.5em;sd-mr-1` Configuration System -:link: configuration-management -:link-type: ref -Learn how NeMo Gym's three-tier configuration system (YAML → env.yaml → CLI) enables secure secrets management and flexible multi-environment deployments. -::: - :::{grid-item-card} {octicon}`check-circle;1.5em;sd-mr-1` Task Verification :link: task-verification :link-type: ref @@ -44,6 +38,12 @@ Explore how resource servers score agent outputs with `verify()` implementations Essential vocabulary for agent training, RL workflows, and NeMo Gym. This glossary defines terms you'll encounter throughout the tutorials and documentation. ::: +:::{grid-item-card} {octicon}`gear;1.5em;sd-mr-1` Configuration System +:link: configuration-concepts +:link-type: ref +Understand the three-level config pattern and why server IDs and implementations are independent choices. +::: + :::: --- @@ -53,7 +53,7 @@ Essential vocabulary for agent training, RL workflows, and NeMo Gym. This glossa :maxdepth: 1 Core Components -Configuration System +Configuration System Task Verification Key Terminology ``` diff --git a/docs/get-started/setup-installation.md b/docs/get-started/setup-installation.md index 1bffefa0b..63c5c493f 100644 --- a/docs/get-started/setup-installation.md +++ b/docs/get-started/setup-installation.md @@ -123,6 +123,8 @@ Replace `sk-your-actual-openai-api-key-here` with your real OpenAI API key. This - The model must support function calling (most GPT-4 models do) - Refer to [OpenAI's models documentation](https://platform.openai.com/docs/models) 🔗 for available models +Refer to {doc}`/reference/configuration` for additional `env.yaml` options. + **Why GPT-4.1?** We use GPT-4.1 for getting started because it provides low latency (no reasoning step) and reliable function calling support out-of-the-box, letting you focus on learning NeMo Gym without configuration complexity. **Can I use my own model?** Yes! NeMo Gym works with any OpenAI-compatible inference server that supports function calling: @@ -130,7 +132,6 @@ Replace `sk-your-actual-openai-api-key-here` with your real OpenAI API key. This - **Other providers**: Any inference server that implements the OpenAI API specification Simply update `policy_base_url`, `policy_api_key`, and `policy_model_name` in your `env.yaml` to point to your chosen endpoint. - ::: :::{dropdown} Optional: Validate your API key before proceeding diff --git a/docs/index.md b/docs/index.md index 290cbc8b8..01e0c3443 100644 --- a/docs/index.md +++ b/docs/index.md @@ -172,8 +172,18 @@ reference/cli-commands.md ``` ```{toctree} -:caption: Development +:caption: Reference :hidden: +:maxdepth: 1 +Configuration apidocs/index.rst ``` + +```{toctree} +:caption: Troubleshooting +:hidden: +:maxdepth: 1 + +troubleshooting/configuration.md +``` diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md new file mode 100644 index 000000000..f48155c72 --- /dev/null +++ b/docs/reference/configuration.md @@ -0,0 +1,172 @@ +(configuration-reference)= + +# Configuration Reference + +Complete syntax and field specifications for NeMo Gym configuration files. + +:::{seealso} +{doc}`/about/concepts/configuration` for understanding how configs are structured and how servers connect. +::: + +## File Locations + +| File | Location | Version Control | +|------|----------|-----------------| +| Server configs | `//configs/*.yaml` | ✅ Committed | +| env.yaml | Repository root (`./env.yaml`) | ❌ Gitignored (user creates) | + +--- + +## Server Configuration + +All servers share this structure: + +```yaml +server_id: # Your unique name for this server + server_type: # responses_api_models | resources_servers | responses_api_agents + implementation: # Directory name inside the server type directory + entrypoint: app.py # Python file to run + # ... additional fields vary by server type +``` + +### Model Server Fields + +```yaml +policy_model: # Server ID (use "policy_model" — agent configs expect this name) + responses_api_models: # Server type (must be "responses_api_models" for model servers) + openai_model: # Implementation (use "openai_model", "vllm_model", or "azure_openai_model") + entrypoint: app.py # Python file to run + openai_base_url: ${policy_base_url} # API endpoint URL + openai_api_key: ${policy_api_key} # Authentication key + openai_model: ${policy_model_name} # Model identifier +``` + +:::{tip} +Keep the server ID as `policy_model` — agent configs reference this name by default. The `${policy_base_url}`, `${policy_api_key}`, and `${policy_model_name}` placeholders should be defined in `env.yaml` at the repository root, allowing you to change model settings in one place. +::: + +### Resources Server Fields + +```yaml +my_resource: # Server ID (your choice — agents reference this name) + resources_servers: # Server type (must be "resources_servers" for resources servers) + example_simple_weather: # Implementation (must match a directory in resources_servers/) + entrypoint: app.py # Python file to run + domain: agent # Server category (see values below) + verified: false # Passed reward profiling and training checks (default: false) + description: "Short description" # Server description + value: "What this improves" # Training value provided +``` + +**Domain values:** `math`, `coding`, `agent`, `knowledge`, `instruction_following`, `long_context`, `safety`, `games`, `e2e`, `other` (see {py:class}`~nemo_gym.config_types.Domain`) + +### Agent Server Fields + +Agent servers must include both a `resources_server` and `model_server` block to specify which servers to use. + +```yaml +my_agent: # Server ID (your choice — used in API requests) + responses_api_agents: # Server type (must be "responses_api_agents" for agent servers) + simple_agent: # Implementation (must match a directory in responses_api_agents/) + entrypoint: app.py # Python file to run + resources_server: # Specifies which resources server to use + type: resources_servers # Always "resources_servers" + name: my_resource # Server ID of the resources server + model_server: # Specifies which model server to use + type: responses_api_models # Always "responses_api_models" + name: policy_model # Server ID of the model server + datasets: # Optional: define for training workflows + - name: train # Dataset identifier + type: train # example | train | validation + jsonl_fpath: path/to/data.jsonl # Path to data file + license: Apache 2.0 # Required for train/validation +``` + +#### Dataset Configuration + +Define datasets associated with agent servers for training and evaluation. + +```yaml +datasets: + - name: my_dataset + type: train + jsonl_fpath: path/to/data.jsonl + license: Apache 2.0 + num_repeats: 1 +``` + +| Field | Required | Description | +|-------|----------|-------------| +| `name` | Yes | Dataset identifier | +| `type` | Yes | `example`, `train`, or `validation` | +| `jsonl_fpath` | Yes | Path to data file | +| `license` | For train/validation | License identifier (see values below) | +| `num_repeats` | No | Repeat dataset n times (default: `1`) | + +**Dataset types:** +- `example` — For testing and development +- `train` — Training data (requires `license`) +- `validation` — Evaluation data (requires `license`) + +**License values:** `Apache 2.0`, `MIT`, `Creative Commons Attribution 4.0 International`, `Creative Commons Attribution-ShareAlike 4.0 International`, `TBD` (see {py:attr}`~nemo_gym.config_types.DatasetConfig.license`) + +--- + +## Local Configuration (env.yaml) + +Store secrets and local settings at the repository root. This file is gitignored. + +```yaml +# Policy model (required for most setups) +# Reference these variables in server configs using `${variable_name}` syntax (e.g., `${policy_base_url}`) +policy_base_url: https://api.openai.com/v1 +policy_api_key: sk-your-api-key +policy_model_name: gpt-4o-2024-11-20 + +# Optional: store config paths for reuse +my_config_paths: + - responses_api_models/openai_model/configs/openai_model.yaml + - resources_servers/example_simple_weather/configs/simple_weather.yaml + +# Optional: validation behavior +error_on_almost_servers: true # Exit on invalid configs (default: true) +``` + +--- + +## Command Line Usage + +Run servers using `ng_run`, NeMo Gym uses [Hydra](https://hydra.cc/) for configuration management." + +### Loading Configs + +```bash +# Load one or more config files +ng_run "+config_paths=[config1.yaml,config2.yaml]" + +# Use paths stored in env.yaml +ng_run "+config_paths=${my_config_paths}" +``` + +### Overriding Values + +```bash +# Override nested values (use dot notation after server ID) +ng_run "+config_paths=[config.yaml]" \ + +my_server.resources_servers.my_impl.domain=coding + +# Override policy model +ng_run "+config_paths=[config.yaml]" \ + +policy_model_name=gpt-4o-mini + +# Disable strict validation +ng_run "+config_paths=[config.yaml]" +error_on_almost_servers=false +``` + +--- + +## Troubleshooting + +:::{seealso} +{doc}`/troubleshooting/configuration` for common configuration errors and solutions. +::: diff --git a/docs/reference/index.md b/docs/reference/index.md new file mode 100644 index 000000000..bb9ce4ae3 --- /dev/null +++ b/docs/reference/index.md @@ -0,0 +1,9 @@ +--- +orphan: true +--- + +(reference)= + +# Reference + +Quick lookup for configuration syntax, options, and specifications. diff --git a/docs/troubleshooting/configuration.md b/docs/troubleshooting/configuration.md new file mode 100644 index 000000000..a443b784c --- /dev/null +++ b/docs/troubleshooting/configuration.md @@ -0,0 +1,103 @@ +(troubleshooting-configuration)= + +# Configuration Errors + +These errors appear when running `ng_run` or `ng_collect_rollouts`. NeMo Gym validates configuration at startup before launching servers. + +:::{seealso} +{doc}`/reference/configuration` for complete configuration syntax and options. +::: + +--- + +## Startup Errors + +Errors that prevent servers from starting. + +### Missing Mandatory Value + +``` +omegaconf.errors.MissingMandatoryValue: Missing mandatory value: policy_api_key +``` + +**When**: A required variable (usually in `env.yaml`) is not defined. + +**Fix**: Add the missing value to `env.yaml`: + +```yaml +policy_api_key: sk-your-api-key +``` + +Or pass via command line: + +```bash +ng_run "+config_paths=[config.yaml]" +policy_api_key=sk-your-api-key +``` + +### Server Reference Not Found + +``` +AssertionError: Could not find ResourcesServerRef(type='resources_servers', name='typo_weather') +in the list of available servers: [ResourcesServerRef(...), ...] +``` + +**When**: A server config references another server that doesn't exist or isn't loaded. + +**Common causes**: +- Typo in the server name +- Referenced server's config file not included in `+config_paths` +- Server defined in a different config file that wasn't loaded + +**Fix**: +1. Check server name spelling in your config +2. Ensure all required config files are in `+config_paths`: + +```bash +ng_run "+config_paths=[model.yaml,resource.yaml,agent.yaml]" +``` + +--- + +## Validation Errors + +Errors where config structure is correct but values are invalid. + +### Almost-Servers Detected + +``` +Configuration Warnings: Almost-Servers Detected + + Almost-Server Detected: 'example_simple_agent' + This server configuration failed validation: + +- ResourcesServerInstanceConfig -> resources_servers -> example_server -> domain: + Input should be 'math', 'coding', 'agent', 'knowledge', 'instruction_following', + 'long_context', 'safety', 'games', 'e2e' or 'other' +``` + +**When**: Config has the right structure (server type, entrypoint) but contains invalid field values. + +**Common causes**: +- Invalid `domain` value for resources servers +- Invalid `license` value in dataset configs +- Missing required fields for the server type + +**Fix**: Check the validation error path (for example, `resources_servers -> example_server -> domain`) and update the field with a valid value. Refer to {ref}`configuration-reference` for valid field values. + +#### Bypass Strict Validation + +To continue with invalid configs (invalid servers will be skipped): + +```yaml +# In env.yaml +error_on_almost_servers: false +``` + +```bash +# Or via command line +ng_run "+config_paths=[config.yaml]" +error_on_almost_servers=false +``` + +:::{warning} +Bypassing validation means invalid servers won't start. Use this only for debugging, not production. +::: diff --git a/docs/troubleshooting/index.md b/docs/troubleshooting/index.md new file mode 100644 index 000000000..8253e9ea0 --- /dev/null +++ b/docs/troubleshooting/index.md @@ -0,0 +1,11 @@ +--- +orphan: true +--- + +(troubleshooting)= + +# Troubleshooting + +Solutions for common errors and issues. + +