diff --git a/docs/agents/insight-driven-optimization.mdx b/docs/agents/insight-driven-optimization.mdx new file mode 100644 index 0000000000..ba446ccd16 --- /dev/null +++ b/docs/agents/insight-driven-optimization.mdx @@ -0,0 +1,600 @@ +--- +title: "Insight-Driven Optimization" +description: "" +--- + + +Insight-driven optimization automates the agent improvement loop. It turns +production traces and evaluation results into autonomous experimentation that +continuously improves your application agent. Where Intake gives you +observability and Experiments gives you a comparison surface, the optimization +agents are the layer that reasons over that telemetry and acts on it. + +This is deliberately *not* an implementation of a single optimization +technique. It is a common front door through which many optimization +techniques can be distributed over time, all built on the shared flywheel +components the platform already provides. + +The system introduces one first-class entity and a family of agents that +operate on it: + +- The **Insight** is a named, persistent description of a recurring problem in + the agent under test (AUT), backed by the traces that evidence it. The + Insight is the unit of action across the whole loop. An experiment or an + evaluation-suite change can always be traced back to the Insight that + motivated it. +- The **Analyst** turns raw traces into actionable Insights. +- The **Experimenter** turns a single Insight into an empirically validated + pull request. +- The **Eval Author** builds and extends evaluation suites in response to + real-world usage. + +## Before You Start + +### Who This Is For + +The optimization agents are for engineers who own and operate an agent and +want to automate portions of the agent improvement loop. + +### Requirements + +- A NeMo Platform deployment (local or remote) with Intake enabled, so the + Analyst has traces to read. +- The optimization plugins installed in the same Python environment as the + NeMo CLI. From a source checkout, `uv sync` installs all three through the + default `enabled-plugins` group. The Experimenter and Eval Author + plugins require Python earlier than 3.14. +- Access to the code base for the agent under test. The Experimenter needs + a change surface (source, prompts, tool definitions, model selection, or + runtime config) and a way to run or evaluate the agent reproducibly. +- Traces in Intake for the agent under test. The Analyst diagnoses only what + it can observe, so an agent with no telemetry cannot be analyzed. +- Model access for the optimization agents themselves. The Analyst and the + Experimenter each drive their own LLMs, which is separate from the model + access your AUT needs at runtime. You can run any LLM you want here. +- Train and validation datasets in a Harbor-compatible layout, plus a task + template and an output directory for artifacts. This is what the Eval Author + works from and what the Experimenter uses to validate changes. + +To install only one side of the loop, use the convenience groups: + +```bash +uv sync --group insights # Analyst only +uv sync --group experimentalist # Experimenter and Eval Author +``` + +### Dependencies + +The optimization agents build on the following NeMo Platform service +dependencies: + +- **NeMo Intake** is the first-party trace store that fuels the loop. The + Analyst reads execution traces, evaluator scores, and user feedback from + Intake. +- **Harbor** so the Experimenter can run candidates through an evaluation + pipeline and construct Insight-specific suites. +- **NeMo Experiments** acts as the primary review and persistence surface for + candidate evaluation runs. +- **Platform entity store (Postgres)** is the durable storage for Insights, + analysis configs, run status, experiment runs, and candidates. + +### Compatibility and Access + +- All Insights service endpoints are workspace-scoped under + `/apis/insights/v2/workspaces/{workspace}/` and depend on platform auth. The + service checks workspace access before reading or writing. +- The plugins register themselves with the platform through entry points + (CLI, service, controller, job, SDK, skills). For a fully local platform, + restart `nemo services run` after installing so the platform discovers them. + +### The Shared Profile + +Both halves of the loop read a shared per-agent profile, `optimizer.yaml`, +discovered by walking up from the current directory. The Analyst consumes only +its analysis subset (`agent`, `agent_spec`, `workspace`); the Experimenter +validates the full schema: + +```yaml +agent: research-agent +agent_spec: AGENT-SPEC.md # optional; falls back to AGENT-SPEC.md, then README.md +workspace: default # optional; defaults to "default" +agent_source: . # local dir or git URL with optional @ref +task_template: ./task-template +datasets: + train: ./harbor_eval/dataset/train + validation: ./harbor_eval/dataset/validation +``` + +Relative paths resolve against the profile. An adjacent `.env` is loaded when +a profile is found, without replacing variables already set in the shell. +`NMP_BASE_URL` is the base-URL environment variable for this workflow, and +`--base-url` takes precedence over it. + +## How It Works + +### The Loop + +The optimization agents automate the four steps of the agent iteration loop: + +1. **Observe.** The AUT is exercised and emits traces (and optional evaluator + scores or feedback) into Intake. +2. **Diagnose.** The Analyst crawls those traces, clusters failures, and + writes or updates Insights. +3. **Experiment.** The Experimenter picks up a single Insight, hypothesizes + root causes, and generates candidate changes. +4. **Validate.** Candidates are scored on a training split and validated + against a held-out split. The winner becomes a draft PR, and the + evaluation runs are recorded in NeMo Experiments. + +### When to Use It + +Use the optimization agents when you want to: + +- **Diagnose real usage automatically.** Convert a pile of traces and + evaluator scores into a ranked, evidence-backed set of failure patterns. +- **Turn a diagnosis into a validated change.** Hand an Insight to the + Experimenter and get back a candidate PR with evaluation results, rather + than a hand-written ticket. +- **Run the loop continuously.** Opt an agent into periodic analysis so + Insights stay current as new traffic arrives. +- **Keep evaluation suites relevant.** Grow datasets and metrics in response + to the failures your agent actually exhibits. + +Production traffic is the best fuel. New agents can start with staged replays +or generated cases and improve coverage as real traffic lands. + +### Core Concepts and Data Model + +**The Insight.** + +The first-class entity of the loop. An Insight is a +persistent, named description of a recurring problem, stored in the platform +entity store with these fields: + +| Field | Type | Meaning | +|-------|------|---------| +| `title` | string | A short, human-readable sentence naming the core issue common to the linked traces. | +| `description` | string | A paragraph describing the problem statement and the situations in which the failure pattern is observed. | +| `agent` | string | The registered agent name the Insight is about. | +| `status` | enum | `open` (default), `resolved`, or `deleted`. An Insight starts open; you resolve it when fixed, or delete it if it isn't a real problem. | +| `trace_refs` | list[string] | Intake trace IDs the Analyst cited as evidence. Drives the evidence view in the UI and lets the loop find similar traces. | + +The store assigns `id`, `created_at`, and `updated_at`. The Analyst aims for at +least three representative traces as evidence before filing a new Insight, and +when it finds more evidence for an existing Insight it *appends* trace refs +rather than restating the problem. + +**Insight persistence: platform versus local file.** + +When no profile is +discovered, the Analyst reads and writes Insights through the Insights API. +When a profile governs the run, the Analyst reads and writes the shared local +file at `/.nemo-optimizer/insights.yaml` instead, which is the +same default the Experimenter reads. Pass `--insights-file-output ` +to point at a different file explicitly. Trace and feedback reads still hit +the live platform at `--base-url` either way, so the local file also covers +deployments that host Intake data but do not have the Insights plugin +installed. Each run merges into the file (de-duplicating trace refs) rather +than overwriting it. + +**Telemetry hierarchy.** + +The loop inherits Intake's model: a span is one timed +operation (LLM call, tool call, and so on), a trace is one end-to-end run, and +a session groups related traces. Insight evidence is cited at the trace level. + +**Experiment entities.** + +An optimization run is tracked as an `ExperimentRun` +(agent, insight, config snapshot, status, rounds completed, winner, summary). + +## The Agents + +### Analyst + +The Analyst reads telemetry from Intake and emits Insights. It runs as a +single reasoning agent with a set of read-only tools over Intake: + +- `fetch_spans` — survey spans, either grouped (for example by `session_id`, + to fan out across many runs) or flat (to drill into one session). Filters + include agent, status, span kind, model, provider, tool name, dataset, and + time range. +- `get_span` — fetch a single span by ID. +- `fetch_scores` — read evaluator results (verifier and judge outputs) + attached to a span. +- `fetch_annotations` and `get_annotation` — read feedback, labels, notes, and + metadata. Negative feedback is the strongest starting signal. +- `list_insights` — read existing Insights so findings are de-duplicated + against what is already filed. + +Its method is to survey sessions broadly, gather evidence (starting from +negative feedback and error spans), cluster similar failures across many +sessions, check for existing Insights, then emit a single result containing new +Insights and evidence appended to existing ones. Give the Analyst an optional +agent spec (`--agent-spec AGENT-SPEC.md`) so it can flag divergence from +intended behavior. + +### Experimenter + +The Experimenter turns an Insight into an empirically validated candidate. +Internally it is an evolutionary optimization loop that runs in rounds: + +1. **Baseline** — build the baseline agent (`agent-0`) and evaluate it on the + validation split; build an initial goal tree (a weighted capability rubric + used for trajectory scoring). +2. **Analyze** — read the target Insight and perform root cause analysis. +3. **Propose** — generate a small number of candidate improvements targeting + those root causes, each tagged with an optimization type. +4. **Implement** — a coding agent applies each proposed change to a copy of + the agent and runs an integration smoke test with a bounded repair loop. +5. **Validate** — score new candidates on the held-out validation split, + optionally adding a qualitative trajectory score against the goal tree. +6. **Select and continue** — keep a diverse Pareto front of survivors and + iterate until a budget or convergence condition is met, then pick the + winner. + +**Insight mode.** + +The Experimenter starts from a single Insight. By +default it reads the local `.nemo-optimizer/insights.yaml` beside the profile; +`--insight` names another local file or a platform Insight ID. When a local +file holds multiple Insights, `--insight-id` selects one by exact ID, exact +title, or zero-based index. The agent referenced by the Insight is used unless +`--agent` overrides it. The Eval Author step builds an Insight-specific +evaluation suite before optimization begins, which requires a task template. + +**Dataset mode.** + +Pass `--no-insight` to bypass both an explicit Insight and +the profile-local default and optimize directly against a dataset. + +**Train and validation isolation.** + +The validation split is *hidden* during +candidate generation. At run start the validation data is moved into a +held-out directory, and the bash tool the coding and analysis agents use +blocks reads of that path. Candidates therefore cannot be tuned against the +data they are later scored on. Validation data is temporarily restored only +when validation scoring runs. + +**Evaluation and rewards.** + +Candidates are scored with the Harbor evaluator. +Each trial yields metrics from the verifier, and the aggregate reward is the +mean across trials, with failed trials counted as zero. Metrics are normalized +to `[0.0, 1.0]`, where 1.0 is perfect. Live and production datasets often carry +no verifiable reward, so validation on those relies on curated metrics and +trajectory scoring rather than a ground-truth verifier. + +**Output.** + +Everything lands under `/eval-and-optimize/`: the +run record, per-candidate agent code and metadata, per-round analysis and goal +trees, and evaluator results. When configured with a git source, the +Experimenter can archive candidate branches and open a draft PR or MR for +the winning candidate against the baseline ref. + +### Eval Author + +The Eval Author builds and maintains the evaluation suites the loop depends +on. Given an Insight and its evidence traces, it creates an Insight-specific +evaluation suite that can be used to validate optimized candidates aimed at +resolving that Insight. It runs as a library-only plugin that the +Experimenter invokes in Insight mode; it is configured through the +`eval_author` section of the experiment config. + +## Get Started + +### Set Up + +Point the CLI at a platform and set model access for the optimization agents. +The examples use a local platform; replace the base URL for a remote +deployment. + +```bash +export NMP_BASE_URL=http://localhost:8080 +export WORKSPACE=default +export AGENT= # Must have traces in Intake + +# The Analyst's model reads a gateway virtual key: +export INFERENCE_API_KEY=sk-... + +# The Experimenter and Eval Author drive their own endpoint: +export EXPERIMENTALIST_API_BASE=https://inference-api.nvidia.com/v1 +export EXPERIMENTALIST_API_KEY=sk-... +``` + +Confirm the plugins are installed and discoverable: + +```bash +nemo insights --help +nemo experimentalist --help +``` + +From an agent directory with an `optimizer.yaml` profile, check that the +effective inputs and credentials resolve: + +```bash +nemo insights doctor +nemo experimentalist doctor +``` + +**Write traces to Intake.** + +The Analyst depends on Intake as its +observability store, so your agent must have traces in Intake before it can +generate Insights. + +### Generate Insights + +Run the Analyst against the target agent's traces: + +```bash +nemo insights analyze \ + --agent "$AGENT" \ + --workspace "$WORKSPACE" \ + --base-url "$NMP_BASE_URL" +``` + +Useful flags: + +- `--agent-spec AGENT-SPEC.md` — append a spec so the Analyst can flag + divergence from intended behavior. +- `--insights-file-output tmp/insights.yaml` — read and write Insights from a + specific local YAML file. Trace reads still hit `--base-url`. +- `--verbose` or `-v` — stream the Analyst's tool calls and reasoning to + stderr. + +**Confirm it worked.** + +List the Insights the Analyst filed through the API: + +```bash +curl "$NMP_BASE_URL/apis/insights/v2/workspaces/$WORKSPACE/insights?agent=$AGENT&page=1&page_size=20" +``` + +Or view them in Studio at +`http://localhost:8080/studio/workspaces/default/optimizer`. + +**What good looks like:** + +at least one Insight appears for the agent, each with +a clear `title`, an actionable `description`, and `trace_refs` pointing at real +Intake traces. + +### Run an Experiment + +**Convert evaluations to Harbor.** + +The Experimenter validates its work by +running evaluations against your agent so that it only keeps empirically +validated optimizations. The only supported evaluation backend today is +[Harbor](https://www.harborframework.com/). + +Before you can run the Experimenter, express your agent's evaluation as +Harbor tasks. This is more than just a scorer: it is a container to run the +agent, an input, and a verifier that scores the results. A +[Harbor task directory](https://www.harborframework.com/docs/tasks) looks like +this: + +```text +/ + task.toml # resources, timeouts, and env passthrough (e.g. INFERENCE_API_KEY) + instruction.md # the prompt given to the agent, incl. where to write output + environment/ + Dockerfile # a container that can run your agent (and the verifier) + tests/ + test.sh # verifier entry point -> writes /logs/verifier/reward.json + score.py # your scorer(s) — e.g. LLM-as-judge +``` + +The Experimenter relies on a classic train/test split for validation, so +create two separate directories of Harbor tasks, one for train and one for +validation. + +Two things to know about the Harbor integration: + +- *The agent-in-container contract.* Harbor runs your agent through + `WrappedAgent(BaseAgent)`. Its `setup()` uploads your agent directory to + `/app` and runs `uv sync` on it. Its `run()` executes the agent with the task + instruction as `--prompt`. +- *How eval scores reach Intake.* The Analyst and the Experimenter both + read scores from Intake, but neither Harbor nor the verifier pushes scores + there; the verifier's reward is written to local disk by default. The + Experimenter uploads scores to Intake after the Harbor run completes, and + each verifier result is stored as an `evaluation_results` record attached to + the trace in Intake. Pushing results to Intake requires a platform client, + so pass `--base-url` and `--workspace`. + +**Kick off the Experimenter.** + +Hand it a single Insight: + +```bash +nemo experimentalist run \ + --insight \ + --agent \ + --train-dataset ./harbor_eval/dataset/train \ + --validation-dataset ./harbor_eval/dataset/validation \ + --task-template ./task-template \ + --config \ + --experiment-dir tmp/experiment \ + --workspace default \ + --base-url http://localhost:8080 +``` + +From an agent directory with an `optimizer.yaml` profile, most of those flags +come from the profile: + +```bash +nemo experimentalist run +``` + +The Experimenter's own models are configured through environment variables +(see [Models](#models)). Train and validation datasets are required, and for +the Harbor evaluator they must be local paths. The loop runs a baseline eval, +performs root cause analysis, implements changes, then validates and picks a +winner. + +**Review the results.** + +View the experiment in Studio at +`http://localhost:8080/studio/workspaces/default/experiment`. There you can see +the originating Insight and compare evaluation runs across all the candidates. + +The Experimenter also writes intermediate artifacts along the way. Those +land under `/eval-and-optimize/`, which defaults to +`/.nemo-optimizer/experiments/` when a profile governs +the run, and `./tmp` otherwise: + +- `OPTIMIZATION.md` — the report with per-agent breakdowns and round-by-round + root causes. +- `agents/agent-0` — the baseline, with `agents/agent-N` as the candidates. + Each holds full agent source. +- `results/` — the per-agent evaluation outputs. + +## Additional Workflows + +### Opt an Agent into Periodic Analysis + +Instead of running `analyze` by hand, opt an agent in and let the platform run +the Analyst on a schedule: + +```bash +nemo insights analysis enable --agent "$AGENT" --workspace "$WORKSPACE" +nemo insights analysis status --workspace "$WORKSPACE" # omit --agent to list all +nemo insights analysis disable --agent "$AGENT" --workspace "$WORKSPACE" +``` + +A framework controller (`insights-analysis`) reconciles on a fixed 60-second +loop, finds every agent opted in across workspaces, and submits one analyze job +per agent that is *due* and has enough new telemetry. Runs are incremental: +each successful run records a cursor (`last_successful_run_at`), and subsequent +runs only consider traces newer than that cursor. + +### Configure the Analysis Schedule + +The global schedule lives in the Insights plugin config (or environment). +Defaults shown: + +```yaml +insights: + analyst: + enabled: true # master switch for the periodic controller + frequency: daily # daily (default) or weekly + run_at_hour: 0 # local hour-of-day, 0-23 + run_on_weekday: monday # only used when frequency: weekly + timezone: UTC # IANA name; e.g. America/Denver + job_profile: default # jobs execution profile for scheduled runs +``` + +`run_at_hour` is interpreted in `timezone` (an IANA name) and converted to the +server clock at evaluation time, so runs fire at the intended local hour even +across daylight-saving transitions. The controller still reconciles every 60 +seconds and submits a run once the scheduled daily or weekly window is reached. + +## Command Reference + +The Analyst lives under `nemo insights` and the Experimenter lives under +`nemo experimentalist`. They are separate command namespaces that share the +`optimizer.yaml` profile, and each validates its own part of it. + +### `nemo insights analyze` + +Run the Analyst once against an agent's traces. + +| Flag | Required | Default | Description | +|------|----------|---------|-------------| +| `--agent` | yes, unless a profile supplies it | profile `agent` | Agent under test the Analyst should focus on. | +| `--agent-spec` | no | profile `agent_spec`, else `AGENT-SPEC.md` or `README.md` beside the profile | Path to a markdown spec for the AUT. | +| `--workspace` | no | profile `workspace`, else `default` | Workspace to operate in. | +| `--base-url` | no | `NMP_BASE_URL`, else `http://localhost:8080` | Running platform instance the Analyst's tools call. | +| `--profile` | no | discovered by walking up from cwd | Path to `optimizer.yaml`. | +| `--insights-file-output` | no | `/.nemo-optimizer/insights.yaml` when a profile is found, else the Insights API | Read and write Insights from a local YAML file. | +| `--verbose` / `-v` | no | off | Stream tool calls and reasoning to stderr. | + +### `nemo insights analysis enable | disable | status` + +Manage per-agent opt-in for periodic analysis. `enable` and `disable` require +`--agent`; `status` takes an optional `--agent` (omit it to list all configs in +the workspace). All three accept `--workspace` and `--base-url`. + +### `nemo insights doctor` + +Check whether the current profile is ready for analysis. Exits non-zero when a +required check fails. + +### `nemo experimentalist run` + +Run the local Experimenter loop. + +| Flag | Required | Default | Description | +|------|----------|---------|-------------| +| `--insight` | no | local `.nemo-optimizer/insights.yaml` | The Insight to optimize against: a local Insight file or a platform Insight ID. | +| `--insight-id` | no | — | Select an exact ID, exact title, or zero-based index from a local multi-Insight file. | +| `--no-insight` | no | off | Run against a dataset directly rather than guided by an Insight. | +| `--agent` | no | profile `agent_source` | Baseline agent override: a local directory or a git URL with optional ref (`...repo.git@main`). A git source records provenance and enables opening a draft PR for the winner. | +| `--agent-spec` | no | profile `agent_spec` | URI of a markdown file describing the AUT. | +| `--train-dataset` | yes, unless the profile supplies it | profile `datasets.train` | Train dataset. Local path for the Harbor evaluator. | +| `--validation-dataset` | yes, unless the profile supplies it | profile `datasets.validation` | Validation dataset. Local path for the Harbor evaluator. | +| `--task-template` | required with an Insight | profile `task_template` | Evaluator-specific task-template URI, used to build the Insight-specific evaluation suite. | +| `--experiment-dir` / `-o` | no | `/.nemo-optimizer/experiments/`, else `./tmp` | Local experiment directory; writes `eval-and-optimize/` here. | +| `--framework-skills` | no | profile `framework_skills` | Directory of framework skills to load into the optimization agents. Repeatable. | +| `--mode` | no | `local` | `local` or `remote`. Only `local` is implemented today. | +| `--profile` | no | discovered by walking up from cwd | Path to `optimizer.yaml`. | +| `--workspace` | no | profile `workspace` | Workspace for traces and run/candidate metadata. | +| `--base-url` | no | `NMP_BASE_URL`, else `http://localhost:8080` | Running platform instance. | +| `--config` | no | profile `experiment_config` | YAML or JSON configuration for the run. | + +### `nemo experimentalist doctor` + +Diagnose the Experimenter setup: profile, credentials, Insight resolution, +datasets, and the experiment plan. + +### Models + +The optimization agents drive their own LLMs, configured by environment +variable: + +| Variable | Used by | Default and notes | +|----------|---------|-------------------| +| `INFERENCE_API_KEY` | Analyst | Required. API key for the Analyst model, reached through the NVIDIA Inference Gateway. | +| `EXPERIMENTALIST_API_BASE` | Experimenter, Eval Author | Required. OpenAI-compatible model API base URL. | +| `EXPERIMENTALIST_API_KEY` | Experimenter, Eval Author | Required. Model API key. On the gateway, `INFERENCE_API_KEY` fills this. | +| `EXPERIMENTALIST_SMART_MODEL_NAME` | Experimenter, Eval Author | High-capability model for analysis, proposing, coding, and curation. Defaults to a GPT-5-class model. | +| `EXPERIMENTALIST_MID_MODEL_NAME` | Experimenter, Eval Author | Mid-tier model. Defaults to a Gemini Flash-class model. | +| `EXPERIMENTALIST_FAST_MODEL_NAME` | Experimenter, Eval Author | Low-latency model for lightweight steps such as termination checks and summarization. Defaults to a GPT-5-mini-class model. | + +## Troubleshooting + +**`analyze` returns no Insights or aborts on the trace floor.** + +The agent has +too few (or no) traces in Intake for the target workspace. Confirm telemetry is +flowing and that you are filtering by the right `agent` name. + +**Periodic analysis never fires.** + +Check that the agent is enabled +(`nemo insights analysis status`), that `insights.analyst.enabled` is true, +that the scheduled window has passed in the configured `timezone`, and that at +least 10 new sessions have landed since the last run. + +**Insights are not in the platform.** + +A discovered `optimizer.yaml` profile +routes Insights to `.nemo-optimizer/insights.yaml` instead of the Insights API, +as does an explicit `--insights-file-output`. Check that file, or run without a +profile to write through the API. + +**The experiment cannot score, or rewards are zero.** + +Confirm datasets are +local Harbor-compatible paths. Live and production datasets may lack verifiable +rewards by design; rely on curated metrics and trajectory scoring in that case. + +**No PR was opened for the winner.** + +Publishing requires a git agent source +(`--agent @`) and publishing enabled in the run config. A +local-directory source cannot open a PR. diff --git a/docs/fern/gated-nav.yml b/docs/fern/gated-nav.yml index de0f479ca9..aeb41b79e2 100644 --- a/docs/fern/gated-nav.yml +++ b/docs/fern/gated-nav.yml @@ -111,3 +111,8 @@ contents: - page: Cluster Setup path: ../../troubleshooting/cluster-setup.mdx +# Publish inside the Agents section of versions/latest.yml, after "Optimize Agents". +- section: Agents + contents: + - page: Insight-Driven Optimization + path: ../../agents/insight-driven-optimization.mdx diff --git a/docs/fern/versions/latest.yml b/docs/fern/versions/latest.yml index 888de0b3ce..2d01d7eb74 100644 --- a/docs/fern/versions/latest.yml +++ b/docs/fern/versions/latest.yml @@ -263,6 +263,11 @@ navigation: path: ../../agents/deploy-agents.mdx - page: Optimize Agents path: ../../agents/optimization.mdx + # Gated until Insight-Driven Optimization ships. Uncomment the two + # lines below to publish it here, then drop the matching Agents block + # from gated-nav.yml and re-add inbound links from Optimize Agents. + # - page: Insight-Driven Optimization + # path: ../../agents/insight-driven-optimization.mdx - page: Secure Agents path: ../../agents/security.mdx - page: Plugins and Skills