diff --git a/docs/about/concepts/configuration.md b/docs/about/concepts/configuration.md new file mode 100644 index 000000000..c4c72620b --- /dev/null +++ b/docs/about/concepts/configuration.md @@ -0,0 +1,60 @@ +(configuration-concepts)= + +# Configuration System + +NeMo Gym uses YAML configuration files to define which servers to run. Understanding the config structure helps you avoid common confusion when setting up training environments. + +## The Three-Level Pattern + +Every server definition follows this pattern: + +```yaml +server_id: # What YOU call it (your choice) + server_type: # What KIND of server (matches a folder) + implementation: # Which CODE to run (matches a subfolder) + entrypoint: ... +``` + +**Why three levels?** Each level serves a different purpose: + +| Level | You Control | Must Match | +|-------|-------------|------------| +| **Server ID** | ✅ Yes - name it anything | Nothing - it's your identifier | +| **Server Type** | ❌ Pick from 3 options | `responses_api_models`, `resources_servers`, or `responses_api_agents` | +| **Implementation** | ❌ Pick existing implementation | A folder inside that server type | + +### Understanding the Naming Pattern + +In many examples, you'll see the same name appear twice: + +```yaml +example_simple_weather: # ← Server ID + resources_servers: + example_simple_weather: # ← Implementation +``` + +These serve different purposes: + +- **Server ID** (`example_simple_weather` on line 1): Your chosen identifier for this server instance. Used in API requests and when other servers reference it. You could name it `my_weather` or `weather_prod` instead. + +- **Implementation** (`example_simple_weather` on line 3): Must match the folder `resources_servers/example_simple_weather/`. This tells NeMo Gym which code to run. + +Examples often use matching names for simplicity, but the two values are independent choices. + +## Policy Model Variables + +NeMo Gym provides three standard placeholders for "the model being trained": + +- `policy_base_url` - Model API endpoint +- `policy_api_key` - Authentication key +- `policy_model_name` - Model identifier + +These let you reference the training model consistently across configs without hardcoding values. Define them once in `env.yaml`, then use `${variable_name}` syntax anywhere you need them. + +--- + +:::{seealso} +- {doc}`/reference/configuration` for complete syntax and field specifications +- {doc}`/troubleshooting/configuration` for common errors +::: + diff --git a/docs/about/concepts/index.md b/docs/about/concepts/index.md index 088ae11c0..964d4beaf 100644 --- a/docs/about/concepts/index.md +++ b/docs/about/concepts/index.md @@ -38,6 +38,12 @@ Explore how resource servers score agent outputs with `verify()` implementations Essential vocabulary for agent training, RL workflows, and NeMo Gym. This glossary defines terms you'll encounter throughout the tutorials and documentation. ::: +:::{grid-item-card} {octicon}`gear;1.5em;sd-mr-1` Configuration System +:link: configuration-concepts +:link-type: ref +Understand the three-level config pattern and why server IDs and implementations are independent choices. +::: + :::: --- @@ -49,4 +55,5 @@ Essential vocabulary for agent training, RL workflows, and NeMo Gym. This glossa Core Abstractions Task Verification Key Terminology +Configuration System ``` diff --git a/docs/about/configuration-system.md b/docs/about/configuration-system.md deleted file mode 100644 index bcb614f3a..000000000 --- a/docs/about/configuration-system.md +++ /dev/null @@ -1,210 +0,0 @@ -(configuration-management)= - -# Configuration Management - -NeMo Gym uses a powerful configuration system with three sources that are resolved in this order: - -``` -Server YAML Config Files < env.yaml < Command Line Arguments - (lowest priority) (highest priority) -``` - -This allows for: -- Base configuration in YAML files (shared settings) -- Secrets and environment-specific values in `env.yaml` -- Runtime overrides via command line arguments - -::::{tab-set} - -:::{tab-item} 1. Server YAML Config Files - -These are your base configurations that define server structures and default values. Later files override earlier files. - -Example: Multi-Server Configuration -```bash -# Define which config files to load -config_paths="responses_api_models/openai_model/configs/openai_model.yaml,\ -resources_servers/example_simple_weather/configs/simple_weather.yaml,\ -responses_api_agents/simple_agent/configs/simple_agent.yaml" - -ng_run "+config_paths=[${config_paths}]" -``` - -Every config file defines **server instances** with this hierarchy: - -```yaml -# Server ID - unique name used in requests and references -simple_weather_simple_agent: - # Server type - must be one of: responses_api_models, resources_servers, responses_api_agents - # These match the 3 top-level folders in NeMo-Gym - responses_api_agents: - # Implementation type - must match a folder name inside responses_api_agents/ - simple_agent: - # Entrypoint - Python file to run (relative to implementation folder) - entrypoint: app.py - # Server-specific configuration (varies by implementation) - resources_server: - type: resources_servers # What type of server to reference - name: simple_weather # Which specific server instance - model_server: - type: responses_api_models - name: policy_model # References the model server -``` - -::: -:::{tab-item} 2. env.yaml - -Your `env.yaml` file contains **secrets and environment-specific values** that should never be committed to version control. - -### Basic env.yaml - -```yaml -# API credentials (never commit these!) -policy_base_url: https://api.openai.com/v1 -policy_api_key: sk-your-actual-api-key-here -policy_model_name: gpt-4o-2024-11-20 -``` - -### Advanced env.yaml with Config Paths - -```yaml -# Store complex config paths for convenience -simple_weather_config_paths: - - responses_api_models/openai_model/configs/openai_model.yaml - - resources_servers/example_simple_weather/configs/simple_weather.yaml - -# Different environments -dev_model_name: gpt-4o-mini -prod_model_name: gpt-4o-2024-11-20 - -# Custom server settings -custom_host: 0.0.0.0 -custom_port: 8080 -``` - -**Usage with stored config paths**: -```bash -ng_run '+config_paths=${simple_weather_config_paths}' -``` - -::: - -:::{tab-item} 3. Command Line Arguments - -**Runtime overrides** using Hydra syntax for maximum flexibility. These runtime command line have the highest priority, meaning they can override any previous setting set in the config.yaml or env.yaml files. - -Basic Overrides -```bash -# Override a specific model -ng_run "+config_paths=[config.yaml]" \ - +policy_model.responses_api_models.openai_model.openai_model=gpt-4o-mini - -# Point agent to different resource server -ng_run "+config_paths=[config.yaml]" \ - +simple_agent.responses_api_agents.simple_agent.resources_server.name=different_weather -``` - -Advanced Overrides -```bash -# Multiple overrides for testing -ng_run "+config_paths=[${config_paths}]" \ - +policy_model_name=gpt-4o-mini \ - +simple_weather.resources_servers.simple_weather.host=localhost \ - +simple_weather.resources_servers.simple_weather.port=9090 -``` - -::: -:::: - -## Special Policy Model Variables - -NeMo Gym provides standard placeholders for the model being trained: - -```yaml -# These variables are available in any config file -policy_base_url: https://api.openai.com/v1 # Model API endpoint -policy_api_key: sk-your-key # Authentication -policy_model_name: gpt-4o-2024-11-20 # Model identifier -``` - -**Why these exist**: When training agents, you need consistent references to "the model being trained" across different resource servers and agents. - -**Usage in config files**: -```yaml -policy_model: - responses_api_models: - openai_model: - openai_base_url: ${policy_base_url} # Resolves from env.yaml - openai_api_key: ${policy_api_key} # Resolves from env.yaml - openai_model: ${policy_model_name} # Resolves from env.yaml -``` - - -## Troubleshooting - -NeMo Gym validates your configuration and provides helpful error messages. - -:::{dropdown} "Missing mandatory value" -``` -omegaconf.errors.MissingMandatoryValue: Missing mandatory value: policy_api_key -``` -**Fix**: Add the missing value to `env.yaml` or command line. -::: - -:::{dropdown} "Could not find X in the list of available servers" -``` -AssertionError: Could not find type='resources_servers' name='typo_weather' -in the list of available servers: [simple_weather, math_with_judge, ...] -``` -**Fix**: Check your server name spelling and ensure the config is loaded. -::: - -:::{dropdown} "Almost-Servers Detected" -Example: -```bash -═══════════════════════════════════════════════════ -Configuration Warnings: Almost-Servers Detected -═══════════════════════════════════════════════════ - - Almost-Server Detected: 'example_simple_agent' - This server configuration failed validation: - -- ResourcesServerInstanceConfig -> resources_servers -> example_server -> domain: Input should be 'math', 'coding', 'agent', 'knowledge', 'instruction_following', 'long_context', 'safety', 'games', 'e2e' or 'other' - - This server will NOT be started. -``` -**What this means**: Your server configuration has the correct structure (entrypoint, server type, etc.) but contains invalid values that prevent it from starting. - -**Common causes**: -- Invalid `license` enum values in datasets (must be one of the allowed options). - - see the `license` field in `DatasetConfig` in `config_types.py`. -- Missing or invalid `domain` field for resources servers (math, coding, agent, knowledge, etc.) - - see the `Domain` class in `config_types.py`. -- Malformed server references (wrong type or name) - -**Fix**: Update the configuration based on the validation errors shown. The warning will detail exactly which fields are problematic. - -#### Strict Validation Mode - -By default, invalid servers will throw an error. You can bypass strict validation and just show a warning: - -**In env.yaml:** -```yaml -error_on_almost_servers: false # Will not error on invalid config -``` - -**Via command line:** -```bash -ng_run "+config_paths=[config.yaml]" +error_on_almost_servers=false -``` - -**Default behavior** (`error_on_almost_servers=true`): -- All configuration issues are detected and warnings are printed -- NeMo Gym exits with an error, preventing servers from starting with invalid configs - -**When disabled** (`error_on_almost_servers=false`): -- All configuration issues are still detected and warnings are printed -- NeMo Gym continues execution despite the invalid configurations -- Invalid servers are skipped, and valid servers will attempt to start - -::: diff --git a/docs/about/index.md b/docs/about/index.md index 75867be16..80d7babdf 100644 --- a/docs/about/index.md +++ b/docs/about/index.md @@ -15,9 +15,3 @@ Three components work together to generate and evaluate agent interactions: - **Models**: LLM inference endpoints (OpenAI-compatible or vLLM). Handle single-turn text generation and tool-calling decisions. - **Resources**: Provide tools (functions agents call) + verifiers (logic to score performance). Examples: math environments, code sandboxes, web search. -```{toctree} -:hidden: -:maxdepth: 1 - -Configuration System -``` diff --git a/docs/get-started/setup-installation.md b/docs/get-started/setup-installation.md index 076db6997..ca77abef7 100644 --- a/docs/get-started/setup-installation.md +++ b/docs/get-started/setup-installation.md @@ -123,6 +123,7 @@ Replace `sk-your-actual-openai-api-key-here` with your real OpenAI API key. This - The model must support function calling (most GPT-4 models do) - Refer to [OpenAI's models documentation](https://platform.openai.com/docs/models) 🔗 for available models +Refer to {doc}`/reference/configuration` for additional `env.yaml` options. ::: :::{dropdown} Optional: Validate your API key before proceeding diff --git a/docs/index.md b/docs/index.md index 1d0e29c55..e8f45a286 100644 --- a/docs/index.md +++ b/docs/index.md @@ -112,7 +112,6 @@ Home about/index.md Concepts Ecosystem -Configuration System ``` ```{toctree} @@ -139,8 +138,18 @@ how-to-faq.md ```{toctree} -:caption: Development +:caption: Reference :hidden: +:maxdepth: 1 +Configuration apidocs/index.rst ``` + +```{toctree} +:caption: Troubleshooting +:hidden: +:maxdepth: 1 + +troubleshooting/configuration.md +``` diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md new file mode 100644 index 000000000..165fe3943 --- /dev/null +++ b/docs/reference/configuration.md @@ -0,0 +1,202 @@ +(configuration-reference)= + +# Configuration Reference + +Complete syntax and options for NeMo Gym configuration. + +:::{seealso} +{doc}`/about/concepts/configuration` for understanding the three-level config pattern. +::: + +## File Locations + +| File | Location | Version Control | +|------|----------|-----------------| +| Server configs | `//configs/*.yaml` | ✅ Committed | +| env.yaml | Repository root (`./env.yaml`) | ❌ Gitignored (user creates) | + +**Example paths**: + +```text +resources_servers/example_simple_weather/configs/simple_weather.yaml +responses_api_models/openai_model/configs/openai_model.yaml +responses_api_agents/simple_agent/configs/simple_agent.yaml +./env.yaml ← you create this +``` + +--- + +## Config File Structure + +Each config file defines one or more server instances using three-level nesting: + +```yaml +server_id: # Level 1: Your chosen name (used in API requests) + server_type: # Level 2: resources_servers | responses_api_models | responses_api_agents + implementation: # Level 3: Must match a folder inside the server type directory + entrypoint: app.py # Required: Python file to run + # Additional fields vary by server type... +``` + +**Example**: + +```yaml +my_agent: # Server ID (you choose this) + responses_api_agents: # Server type + simple_agent: # Implementation (must match folder: responses_api_agents/simple_agent/) + entrypoint: app.py + # ... +``` + +:::{tip} +The server ID and implementation name are independent. Use descriptive server IDs that reflect your use case (for example, `math_tutor_agent`), while the implementation name must match an existing folder. +::: + +### Server Types + +| Type | Purpose | Required Fields | +|------|---------|-----------------| +| `responses_api_models` | LLM inference endpoints | `entrypoint` | +| `resources_servers` | Tools and verification logic | `entrypoint`, `domain` | +| `responses_api_agents` | Orchestration between models and resources | `entrypoint`, `resources_server`, `model_server` | + +### Resources Server Fields + +```yaml +my_resource: + resources_servers: + my_implementation: + entrypoint: app.py + domain: math # Required (see domain values below) + verified: false # Optional: marks server as production-ready + description: "Short description" # Optional +``` + +**Domain values**: `math`, `coding`, `agent`, `knowledge`, `instruction_following`, `long_context`, `safety`, `games`, `e2e`, `other` + +### Agent Server Fields + +Agent servers reference other servers using the `type` and `name` pattern: + +```yaml +my_agent: + responses_api_agents: + simple_agent: + entrypoint: app.py + resources_server: + type: resources_servers + name: my_resource # Must match a loaded server ID + model_server: + type: responses_api_models + name: policy_model # Must match a loaded server ID + datasets: # Optional: define associated datasets + - name: example + type: example # example | train | validation + jsonl_fpath: path/to/data.jsonl +``` + +### Model Server Fields + +```yaml +policy_model: + responses_api_models: + openai_model: + entrypoint: app.py + openai_base_url: ${policy_base_url} + openai_api_key: ${policy_api_key} + openai_model: ${policy_model_name} +``` + +--- + +## env.yaml Options + +Store secrets and environment-specific values that should not be committed to version control. + +```yaml +# Required for OpenAI-compatible models +policy_base_url: https://api.openai.com/v1 +policy_api_key: sk-your-api-key +policy_model_name: gpt-4o-2024-11-20 + +# Optional: store config paths for reuse +my_config_paths: + - responses_api_models/openai_model/configs/openai_model.yaml + - resources_servers/example_simple_weather/configs/simple_weather.yaml + +# Optional: validation behavior +error_on_almost_servers: true # Default: true (exit on invalid configs) +``` + +--- + +## Command Line Overrides + +Override any configuration value at runtime using Hydra syntax. Command line arguments have the highest priority. + +```bash +# Load config files +ng_run "+config_paths=[config1.yaml,config2.yaml]" + +# Override nested values (use dot notation after server ID) +ng_run "+config_paths=[config.yaml]" \ + +my_server.resources_servers.my_impl.domain=coding + +# Override model selection +ng_run "+config_paths=[config.yaml]" \ + +policy_model_name=gpt-4o-mini + +# Use paths stored in env.yaml +ng_run '+config_paths=${my_config_paths}' + +# Disable strict validation +ng_run "+config_paths=[config.yaml]" +error_on_almost_servers=false +``` + +--- + +## Policy Model Variables + +Standard placeholders resolved from `env.yaml` for consistent model references across configurations. + +| Variable | Description | Example | +|----------|-------------|---------| +| `policy_base_url` | Model API endpoint | `https://api.openai.com/v1` | +| `policy_api_key` | Authentication key | `sk-...` | +| `policy_model_name` | Model identifier | `gpt-4o-2024-11-20` | + +Reference in config files using `${variable_name}`: + +```yaml +openai_base_url: ${policy_base_url} +openai_api_key: ${policy_api_key} +openai_model: ${policy_model_name} +``` + +--- + +## Dataset Configuration + +Define datasets associated with servers for training data collection. + +```yaml +datasets: + - name: my_dataset + type: example # example | train | validation + jsonl_fpath: path/to/data.jsonl + num_repeats: 1 # Optional: repeat dataset (default: 1) + license: "Apache 2.0" # Required for train/validation types +``` + +**License values**: `Apache 2.0`, `MIT`, `Creative Commons Attribution 4.0 International`, `Creative Commons Attribution-ShareAlike 4.0 International`, `TBD` + +**Dataset types**: +- `example`: For testing and development +- `train`: Requires `license` and GitLab identifier +- `validation`: Requires `license` and GitLab identifier + +--- + +:::{seealso} +{doc}`/troubleshooting/configuration` for common configuration errors and solutions. +::: diff --git a/docs/reference/index.md b/docs/reference/index.md new file mode 100644 index 000000000..bb9ce4ae3 --- /dev/null +++ b/docs/reference/index.md @@ -0,0 +1,9 @@ +--- +orphan: true +--- + +(reference)= + +# Reference + +Quick lookup for configuration syntax, options, and specifications. diff --git a/docs/troubleshooting/configuration.md b/docs/troubleshooting/configuration.md new file mode 100644 index 000000000..a443b784c --- /dev/null +++ b/docs/troubleshooting/configuration.md @@ -0,0 +1,103 @@ +(troubleshooting-configuration)= + +# Configuration Errors + +These errors appear when running `ng_run` or `ng_collect_rollouts`. NeMo Gym validates configuration at startup before launching servers. + +:::{seealso} +{doc}`/reference/configuration` for complete configuration syntax and options. +::: + +--- + +## Startup Errors + +Errors that prevent servers from starting. + +### Missing Mandatory Value + +``` +omegaconf.errors.MissingMandatoryValue: Missing mandatory value: policy_api_key +``` + +**When**: A required variable (usually in `env.yaml`) is not defined. + +**Fix**: Add the missing value to `env.yaml`: + +```yaml +policy_api_key: sk-your-api-key +``` + +Or pass via command line: + +```bash +ng_run "+config_paths=[config.yaml]" +policy_api_key=sk-your-api-key +``` + +### Server Reference Not Found + +``` +AssertionError: Could not find ResourcesServerRef(type='resources_servers', name='typo_weather') +in the list of available servers: [ResourcesServerRef(...), ...] +``` + +**When**: A server config references another server that doesn't exist or isn't loaded. + +**Common causes**: +- Typo in the server name +- Referenced server's config file not included in `+config_paths` +- Server defined in a different config file that wasn't loaded + +**Fix**: +1. Check server name spelling in your config +2. Ensure all required config files are in `+config_paths`: + +```bash +ng_run "+config_paths=[model.yaml,resource.yaml,agent.yaml]" +``` + +--- + +## Validation Errors + +Errors where config structure is correct but values are invalid. + +### Almost-Servers Detected + +``` +Configuration Warnings: Almost-Servers Detected + + Almost-Server Detected: 'example_simple_agent' + This server configuration failed validation: + +- ResourcesServerInstanceConfig -> resources_servers -> example_server -> domain: + Input should be 'math', 'coding', 'agent', 'knowledge', 'instruction_following', + 'long_context', 'safety', 'games', 'e2e' or 'other' +``` + +**When**: Config has the right structure (server type, entrypoint) but contains invalid field values. + +**Common causes**: +- Invalid `domain` value for resources servers +- Invalid `license` value in dataset configs +- Missing required fields for the server type + +**Fix**: Check the validation error path (for example, `resources_servers -> example_server -> domain`) and update the field with a valid value. Refer to {ref}`configuration-reference` for valid field values. + +#### Bypass Strict Validation + +To continue with invalid configs (invalid servers will be skipped): + +```yaml +# In env.yaml +error_on_almost_servers: false +``` + +```bash +# Or via command line +ng_run "+config_paths=[config.yaml]" +error_on_almost_servers=false +``` + +:::{warning} +Bypassing validation means invalid servers won't start. Use this only for debugging, not production. +::: diff --git a/docs/troubleshooting/index.md b/docs/troubleshooting/index.md new file mode 100644 index 000000000..8253e9ea0 --- /dev/null +++ b/docs/troubleshooting/index.md @@ -0,0 +1,11 @@ +--- +orphan: true +--- + +(troubleshooting)= + +# Troubleshooting + +Solutions for common errors and issues. + +