diff --git a/plugins/nemo-insights/README.md b/plugins/nemo-insights/README.md index 2ee4dcee60..baf2355011 100644 --- a/plugins/nemo-insights/README.md +++ b/plugins/nemo-insights/README.md @@ -58,8 +58,7 @@ those conventions. ATIF maps its required `agent.name` automatically. ### Where insights are written -Insights always go to the platform, through the Insights plugin API. There is no -mode that keeps them off it. +Insights always go to the platform, through the Insights plugin API. Pass `--insights-file-output ` to also keep a local copy: the platform is written first and the file mirrors what it stored, platform ids included, so a @@ -107,13 +106,50 @@ The plugin SDK is available as `client.insights`, including: ## Configuration -Periodic analysis settings use the `NEMO_INSIGHTS_` environment prefix. For example: +Periodic analysis is a *deployment* setting, not a per-run one. Like every other +NeMo plugin it is a `NemoConfig`, so it can be set either in the `insights:` +section of the platform config file or through the environment, and the +environment wins. All settings live under `analyst`, with the +`NEMO_INSIGHTS_` environment prefix. + +| Variable | Config key | Default | Meaning | +|---|---|---|---| +| `NEMO_INSIGHTS_ANALYST_ENABLED` | `analyst.enabled` | `true` | Whether the periodic analysis controller runs at all. | +| `NEMO_INSIGHTS_ANALYST_FREQUENCY` | `analyst.frequency` | `daily` | Cadence for each opted-in agent: `daily` or `weekly`. | +| `NEMO_INSIGHTS_ANALYST_TIMEZONE` | `analyst.timezone` | `UTC` | IANA name (e.g. `America/Denver`) the schedule is interpreted in. Converted to the server clock at evaluation time, so runs hold their local hour across DST. An unknown name fails validation. | +| — (see below) | `analyst.run_at_hour` | `0` | Local hour-of-day, 0–23, that scheduled runs fire. | +| — (see below) | `analyst.run_on_weekday` | `monday` | Day scheduled runs fire. Used only when frequency is `weekly`. | +| — (see below) | `analyst.job_profile` | `default` | Jobs execution profile for scheduled analyst jobs. | +| — (see below) | `analyst.base_url` | unset | Platform base URL passed to analyst jobs. When unset, jobs use their active platform context. | +| — (see below) | `analyst.inference_api_key_secret_name` | unset | Platform secret whose value is exposed to analyst jobs as `INFERENCE_API_KEY`. Temporary until FP-202 moves analyst model execution to platform-registered models. | ```bash -export NEMO_INSIGHTS_ANALYST_FREQUENCY=daily +export NEMO_INSIGHTS_ANALYST_FREQUENCY=weekly export NEMO_INSIGHTS_ANALYST_TIMEZONE=America/Denver ``` +### Setting the underscored fields + +Only single-word fields — `enabled`, `frequency`, `timezone` — have a working +flat environment variable. The plugin sets `env_nested_delimiter="_"`, so a +name like `NEMO_INSIGHTS_ANALYST_RUN_AT_HOUR` is parsed as the nested path +`analyst.run.at.hour`, which does not exist. **The variable is ignored +silently: no error, and the default stays in effect.** Prefer the config file +for these. To set them from the environment anyway, assign the whole `analyst` +object as JSON — unlisted keys keep their defaults: + +```bash +export NEMO_INSIGHTS_ANALYST='{"run_at_hour": 17, "run_on_weekday": "friday", "job_profile": "gpu"}' +``` + +### Analyst self-observability + +`NEMO_INSIGHTS_ANALYST_OBSERVABILITY` is read directly from the environment +rather than through `NemoConfig`, and is off unless set to one of `1`, `true`, +`yes`, or `on` (case-insensitive). When enabled, the analyst exports its own +traces to Intake's workspace-scoped OTLP endpoint. The endpoint must be HTTPS +unless it is loopback. + ## Development ```bash @@ -125,3 +161,25 @@ uv run ruff check plugins/nemo-insights The analyst-only testbed is in [`testbed/`](testbed/). It can replay pinned Intake traces or run Tau2 benchmarks before invoking `nemo agents analyst run`. + +## What consumes an Insight + +Insights is the analysis half of a two-plugin loop. The +[NeMo Experimentalist](../nemo-experimentalist/README.md) plugin consumes what +the analyst produces and uses it to improve the agent against Harbor-compatible +train and validation datasets: + +```text +nemo agents analyst run → Platform Insight ID (or --insights-file-output mirror) + → nemo agents experimentalist doctor + → nemo agents experimentalist run +``` + +The Experimentalist accepts either a Platform Insight ID or a local mirror +file, so `--insights-file-output` is the option to reach for when you want a +run that does not have to resolve an ID against Platform. Insights does not +propose or evaluate agent changes itself; the Experimentalist does not analyze +traces or host an Insight API. + +[Insight-driven optimization](../../docs/agents/insight-driven-optimization.mdx) +walks the full loop end to end.