diff --git a/resources_servers/math_with_judge/configs/math_with_judge_kilocode_agent.yaml b/resources_servers/math_with_judge/configs/math_with_judge_kilocode_agent.yaml index fc069e7be1..94555fbe1a 100644 --- a/resources_servers/math_with_judge/configs/math_with_judge_kilocode_agent.yaml +++ b/resources_servers/math_with_judge/configs/math_with_judge_kilocode_agent.yaml @@ -1,6 +1,6 @@ # kilocode_agent.yaml stays the single source of truth for the harness (command, model, kilo_version, -# permissions, provider); _inherit_from pulls it in so this file only sets what the math environment -# changes: the resources server it talks to, the prompt, and the dataset. +# permissions, model server); _inherit_from pulls it in so this file only sets what the math +# environment changes: the resources server it talks to, the prompt, and the dataset. config_paths: - responses_api_agents/kilocode_agent/configs/kilocode_agent.yaml diff --git a/responses_api_agents/kilocode_agent/README.md b/responses_api_agents/kilocode_agent/README.md index 4921399170..0393bed87c 100644 --- a/responses_api_agents/kilocode_agent/README.md +++ b/responses_api_agents/kilocode_agent/README.md @@ -4,14 +4,13 @@ Runs the [Kilo Code](https://kilo.ai) CLI (`kilo run`). Kilo Code is a fork of O agent mirrors the `opencode_agent`: Kilo runs its own tools internally, and its JSON event stream (`--format json`) is parsed into Gym format and verified by the resources server. -Minimal, meant to be extended, and currently eval-only. Token IDs and logprobs are not wired up and -it does not use a Gym model server yet. +Minimal, meant to be extended, and currently eval-only: token IDs and logprobs are not wired up. ## Quick start Kilo must be on PATH (auto-installed on first start, or `npm install -g @kilocode/cli`). Set -`policy_base_url`, `policy_api_key`, and `policy_model_name` in `env.yaml` — Kilo calls that endpoint -directly, so the config needs nothing else to pick up your model. +`policy_base_url`, `policy_api_key`, and `policy_model_name` in `env.yaml`; the model server started +by `--model-type` serves that backend and Kilo calls the model server. ```bash gym env start \ @@ -31,15 +30,43 @@ runs don't share state and the global `~/.config/kilo` never bleeds in. `--pure` external plugins, so codebase indexing never starts. The project `kilo.json` written into the run dir supplies the provider and permissions. -## Model id +## Model server -`model` is `/`, where the provider is a label defined in `kilo_config` (written -to `kilo.json`) rather than a service. The shipped config declares one generic OpenAI-compatible -provider called `policy` aimed at `env.yaml`, so `policy/gpt-4.1` and `policy/Qwen/Qwen3-8B` both work -against whatever `policy_base_url` points at — OpenAI, an NVIDIA endpoint, or a Gym-served vLLM. This -bypasses the Kilo Gateway, so no Kilo account is needed. +With `model_server` set (the shipped default), Kilo's model calls go to that Gym model server rather +than to a provider directly. That is what makes requests and responses show up in Gym's capture, and +it means one config runs against vLLM, OpenAI, or an inference provider by swapping `--model-type`. +The agent writes a `nemo` provider into `kilo.json` pointed at the server's URL and passes +`-m nemo/`, so `model` is the bare model name: ```yaml +model_server: {type: responses_api_models, name: policy_model} +model: ${policy_model_name} +``` + +`context_window`, `max_output_tokens`, and `reasoning_field` describe the served model to Kilo and +apply only on this path (see Config fields). During a `/run` the base URL carries the per-rollout +`/ng-rollout/` prefix, so captured model calls are attributable to the rollout that made them. + +### Sizing the output budget + +The shipped `context_window` (32768) and `max_output_tokens` (8192) assume a 32k-window model server. +Both need to match whatever you actually serve, and `max_output_tokens` is the one that bites: Kilo's +system prompt and tool definitions run to roughly 10k tokens, so a large output budget pushes +`prompt + max_tokens` past `max_model_len`. vLLM rejects that with a 400, which the Gym model server +converts into an empty completion with `finish_reason: length` rather than an error. The run then +produces no assistant message and scores zero, with nothing in the CLI's own output to say why. The +agent logs a warning when it sees that shape; the fix is to lower `max_output_tokens` (or serve a +larger window), not to raise it. + +### Calling a provider directly + +Set `model_server: null` and declare the provider yourself in `kilo_config`. `model` is then +`/`, where the provider is a label defined in `kilo_config` rather than a +service. This bypasses both Gym and the Kilo Gateway, so no Kilo account is needed and no model calls +are captured: + +```yaml +model_server: null model: policy/${policy_model_name} kilo_config: provider: @@ -50,18 +77,20 @@ kilo_config: apiKey: ${policy_api_key} ``` -Kilo rejects a model that is not listed in its provider's `models` map (`Model not found: …`), so the -agent registers `model` there when it writes `kilo.json`. Only add `models` entries by hand if you need -per-model options; note that the config merge is struct-mode, so a config that uses `_inherit_from` -cannot add new keys to `models`. +Kilo splits `-m` on the first `/`, so `policy/Qwen/Qwen3-8B` is the model `Qwen/Qwen3-8B` under the +provider `policy`. It rejects a model that is not listed in its provider's `models` map +(`Model not found: …`), so the agent registers `model` there when it writes `kilo.json`. Only add +`models` entries by hand if you need per-model options; note that the config merge is struct-mode, so +a config that uses `_inherit_from` cannot add new keys to `models`. ## Config fields +- `model_server`: Gym model server Kilo calls; `null` to call a provider directly (see Model server) - `concurrency`: max simultaneous `run()` calls - `command`: the Kilo command, split on spaces so a multi-word launcher works (e.g. `npx kilo`) -- `model`: `/` (see Model id) -- `openai_api_key`: passed to the subprocess as `OPENAI_API_KEY` -- `openai_base_url`: passed to the subprocess as `OPENAI_BASE_URL` +- `model`: the model name, or `/` without a model server (see Model server) +- `openai_api_key`: passed to the subprocess as `OPENAI_API_KEY`; ignored when `model_server` is set +- `openai_base_url`: passed to the subprocess as `OPENAI_BASE_URL`; ignored when `model_server` is set - `env`: extra env vars for the subprocess - `workspace_root`: where per-request run dirs are created and deleted - `repo_dir`: optional persistent project dir to run in (default: ephemeral per-request dir) @@ -71,6 +100,16 @@ cannot add new keys to `models`. - `timeout`: seconds for the `kilo run` call (the only runaway bound — Kilo has no `--max-turns`) - `extra_args`: extra flags appended to `kilo run` - `kilo_config`: written to `kilo.json` in the run dir (OpenCode-compatible schema) +- `context_window`: the served model's context window. Kilo measures the session against it, but only + auto-compacts when `kilo_config` also sets `compaction.threshold_percent`; `0` turns the accounting + off entirely. `model_server` only. +- `max_output_tokens`: per-request output budget. Kilo asks for `min(this, 32000)`, its own + `OUTPUT_TOKEN_MAX`, so values above 32000 have no effect. Setting it too high fails silently — see + Sizing the output budget. `model_server` only. +- `reasoning_field`: response field carrying reasoning text, written as `interleaved.field`. Gym model + servers emit `reasoning_content`; Kilo turns interleaved reasoning off for custom OpenAI-compatible + providers unless the field is named, so without this the reasoning channel is dropped. `null` leaves + Kilo's default. `model_server` only. - `kilo_version`: `@kilocode/cli` npm version installed on a clean machine (shipped pinned to `7.4.15`; the parser was validated against it, so treat a bump as a deliberate change — raise it, re-run the tests and the live eval, then commit). `null` installs `@latest`. diff --git a/responses_api_agents/kilocode_agent/app.py b/responses_api_agents/kilocode_agent/app.py index c416645aef..1429dc4be6 100644 --- a/responses_api_agents/kilocode_agent/app.py +++ b/responses_api_agents/kilocode_agent/app.py @@ -36,7 +36,7 @@ Body, SimpleResponsesAPIAgent, ) -from nemo_gym.config_types import ResourcesServerRef +from nemo_gym.config_types import ModelServerRef, ResourcesServerRef from nemo_gym.openai_utils import ( NeMoGymEasyInputMessage, NeMoGymFunctionCallOutput, @@ -85,6 +85,7 @@ def parse_kilo_events(stdout: str) -> tuple[list[Any], dict[str, int]]: # two copies are identical for every event type we parse, so first-wins dedup by part id counts # each tool call, message, and token total once. Verified against @kilocode/cli 7.4.15. seen_part_ids: set[str] = set() + warned_empty_length = False for line in stdout.splitlines(): line = line.strip() @@ -112,7 +113,17 @@ def parse_kilo_events(stdout: str) -> tuple[list[Any], dict[str, int]]: tokens = part.get("tokens") or {} cache = tokens.get("cache") or {} input_tokens += int(tokens.get("input") or 0) + int(cache.get("read") or 0) - output_tokens += int(tokens.get("output") or 0) + step_output = int(tokens.get("output") or 0) + output_tokens += step_output + # Stopping on "length" with nothing generated is what an over-window request looks like by + # the time it reaches kilo; see KiloCodeAgentConfig.max_output_tokens. Warn once: the run + # is otherwise silent about why it produced no answer. + if part.get("reason") == "length" and not step_output and not warned_empty_length: + warned_empty_length = True + LOG.warning( + "kilo step stopped on 'length' with no output tokens; max_output_tokens is likely " + "too large for the model server's context window" + ) elif etype == "reasoning": text = (part.get("text") or "").strip() @@ -201,6 +212,10 @@ def _extract_instruction(body_input) -> tuple[str, Optional[str]]: class KiloCodeAgentConfig(BaseResponsesAPIAgentConfig): resources_server: ResourcesServerRef + # When set, kilo's model calls go through this Gym model server instead of straight to a provider, + # so they are captured. `model` is then the bare model name; the agent registers it under a `nemo` + # provider pointed at the server (see _build_kilo_config). + model_server: Optional[ModelServerRef] = None concurrency: int = 8 command: str = "kilo" model: str = "openai/gpt-4o-mini" @@ -217,6 +232,29 @@ class KiloCodeAgentConfig(BaseResponsesAPIAgentConfig): extra_args: list[str] = [] # written to kilo.json in the run dir (OpenCode-compatible config schema) kilo_config: dict[str, Any] = Field(default_factory=dict) + # The three fields below describe the served model to kilo. They apply only to the `model_server` + # path, where the agent owns the `nemo` model entry; a provider declared by hand in `kilo_config` + # keeps whatever it declares. They are config fields rather than `kilo_config` keys because the + # config merge is struct-mode, so a config using `_inherit_from` cannot add keys under `models`. + # + # `context_window` is what kilo measures a session against. It only drives auto-compaction when + # `kilo_config` also sets `compaction.threshold_percent`; 0 disables that accounting entirely. + context_window: int = 32768 + # `max_output_tokens` is the per-request output budget; kilo asks the model server for + # min(this, its own OUTPUT_TOKEN_MAX of 32000), so values above 32000 have no effect. + # + # Setting it too high fails silently, which is why the default is conservative. Kilo's system + # prompt and tool definitions run to ~10k tokens, and vLLM rejects prompt + max_tokens > + # max_model_len with a 400 that the Gym model server turns into an empty completion with + # finish_reason "length" (vllm_model/app.py `is_out_of_context_length`) rather than an error, so + # the run yields no assistant message at all. Raise it only alongside a model server whose window + # has room for it. Verified against @kilocode/cli 7.4.15 and vLLM serving a 32768-token window. + max_output_tokens: int = 8192 + # Response field carrying reasoning text, for kilo's `interleaved.field`. Gym model servers write + # `reasoning_content` (vLLM >= 0.16 also sends `reasoning`, which is why this is configurable). + # Kilo turns interleaved reasoning off for custom openai-compatible providers unless the field is + # named, so without this the reasoning channel is dropped. Null leaves kilo's default in place. + reasoning_field: Optional[str] = "reasoning_content" kilo_version: Optional[str] = None @property @@ -238,8 +276,8 @@ class KiloCodeAgent(SimpleResponsesAPIAgent): """Runs the Kilo Code CLI (``kilo run --auto --format json``). Kilo runs its own tools internally; we parse its JSON event stream into Gym format and use the - resources server to verify. Eval-only: token IDs and logprobs are not wired up and it does not - use a Gym model server yet. + resources server to verify. Kilo's model calls go through ``model_server`` when one is set — see + that field. Eval-only either way: token IDs and logprobs are not wired up. """ config: KiloCodeAgentConfig @@ -279,22 +317,56 @@ def _repo_dir(self, fallback: Path) -> Path: root.mkdir(parents=True, exist_ok=True) return root - def _write_kilo_config(self, work_dir: Path) -> None: - if not self.config.kilo_config: - return + def _resolve_model_base_url(self, rollout_id: Optional[str] = None) -> str: + """The Gym model server's ``/v1`` URL with the per-rollout capture prefix, "" if unconfigured.""" + if self.config.model_server is None: + return "" + return self.resolve_model_base_url(self.config.model_server.name, rollout_id) + + def _effective_model(self) -> str: + """The ``-m`` argument: ``model`` lives under the agent-owned `nemo` provider when one is set.""" + return f"nemo/{self.config.model}" if self.config.model_server else self.config.model + + def _build_kilo_config(self, model_base_url: str = "") -> dict[str, Any]: + """Assemble the kilo.json contents from `kilo_config` plus the Gym model server wiring. + + Kilo resolves `-m /` against that provider's `models` map (splitting on the + first `/`, so a slashed model name stays intact) and fails the run with "Model not found" if + the name is absent. Either branch below registers `model` there, so `model` is the only place + a config names it; without that, every config has to repeat it under + kilo_config.provider..models, which downstream overrides cannot add to (the config + merge is struct-mode, so a new model key is rejected). + """ config = self._deep_merge({}, copy.deepcopy(self.config.kilo_config)) - # Kilo resolves `-m /` against that provider's `models` map and fails the run - # with "Model not found" if the name is absent. Register it here so `model` is the only place a - # config names the model; without this, every config has to repeat it under - # kilo_config.provider..models, which downstream overrides cannot add to (the config - # merge is struct-mode, so a new model key is rejected). - provider_id, _, model_name = self.config.model.partition("/") - provider = (config.get("provider") or {}).get(provider_id) - if model_name and isinstance(provider, dict): - provider.setdefault("models", {}).setdefault(model_name, {}) + + if self.config.model_server: + # A generic openai-compatible provider aimed at the Gym model server. setdefault + # throughout so a config that wants to override any of this still can; only baseURL is + # forced, since it is resolved per run and a stale one would silently bypass Gym. + provider = config.setdefault("provider", {}).setdefault("nemo", {}) + provider.setdefault("npm", "@ai-sdk/openai-compatible") + options = provider.setdefault("options", {}) + options.setdefault("apiKey", "EMPTY") # pragma: allowlist secret + options["baseURL"] = model_base_url + model = provider.setdefault("models", {}).setdefault(self.config.model, {}) + model.setdefault("name", self.config.model) + model.setdefault("limit", {"context": self.config.context_window, "output": self.config.max_output_tokens}) + if self.config.reasoning_field: + model.setdefault("interleaved", {"field": self.config.reasoning_field}) + else: + provider_id, _, model_name = self.config.model.partition("/") + provider = (config.get("provider") or {}).get(provider_id) + if model_name and isinstance(provider, dict): + provider.setdefault("models", {}).setdefault(model_name, {}) + return config + + def _write_kilo_config(self, work_dir: Path, model_base_url: str = "") -> None: + config = self._build_kilo_config(model_base_url) + if not config: + return (work_dir / "kilo.json").write_text(json.dumps(config, indent=2)) - def _env(self, data_home: str, config_home: str) -> dict[str, str]: + def _env(self, data_home: str, config_home: str, model_base_url: str = "") -> dict[str, str]: # Per-run isolation. KILO_NO_DAEMON=1 forces a fresh embedded server (kilo's daemon client is # enabled iff this is unset), so concurrent rollouts share no daemon db/state. KILO_DB=:memory: # keeps sessions ephemeral (we read stdout, not the db). XDG_DATA_HOME/XDG_CONFIG_HOME point @@ -308,10 +380,14 @@ def _env(self, data_home: str, config_home: str) -> dict[str, str]: "XDG_DATA_HOME": data_home, "XDG_CONFIG_HOME": config_home, } - if self.config.openai_base_url: - env["OPENAI_BASE_URL"] = self.config.openai_base_url - if self.config.openai_api_key: - env["OPENAI_API_KEY"] = self.config.openai_api_key + # A Gym model server wins over openai_base_url, so a config carrying both does not point the + # subprocess environment at a provider the `nemo` provider is meant to replace. + base_url = model_base_url or self.config.openai_base_url + api_key = "EMPTY" if model_base_url else self.config.openai_api_key # pragma: allowlist secret + if base_url: + env["OPENAI_BASE_URL"] = base_url + if api_key: + env["OPENAI_API_KEY"] = api_key env.update({k: v for k, v in self.config.env.items() if v}) return env @@ -326,7 +402,7 @@ def _build_command(self, project_dir: Path, prompt: str) -> list[str]: "--format", "json", "-m", - self.config.model, + self._effective_model(), "--dir", str(project_dir), ] @@ -346,7 +422,9 @@ def _kill_process_group(proc: "asyncio.subprocess.Process") -> None: except (ProcessLookupError, PermissionError): proc.kill() - async def _run_kilo(self, instruction: str, system_prompt: Optional[str]) -> tuple[list[Any], dict[str, int], str]: + async def _run_kilo( + self, instruction: str, system_prompt: Optional[str], rollout_id: Optional[str] = None + ) -> tuple[list[Any], dict[str, int], str]: """Run one headless kilo run. Returns (output_items, usage, model_name).""" prompt = instruction if not system_prompt else f"{system_prompt}\n\n{instruction}" work_dir = self._workspace_root() @@ -355,8 +433,9 @@ async def _run_kilo(self, instruction: str, system_prompt: Optional[str]) -> tup config_home = work_dir / ".kilo-config" data_home.mkdir(parents=True, exist_ok=True) config_home.mkdir(parents=True, exist_ok=True) - self._write_kilo_config(project_dir) - env = self._env(str(data_home), str(config_home)) + model_base_url = self._resolve_model_base_url(rollout_id) + self._write_kilo_config(project_dir, model_base_url) + env = self._env(str(data_home), str(config_home), model_base_url) cmd = self._build_command(project_dir, prompt) try: @@ -397,7 +476,10 @@ async def responses( system_parts = [p for p in [self.config.system_prompt, input_system] if p] system_prompt = "\n\n".join(system_parts) if system_parts else None - output_items, usage, model_name = await self._run_kilo(user_message, system_prompt) + # run() reaches this handler through the /ng-rollout/ self-call route, which carries the + # correlation id kilo's model calls are tagged with. Absent on a direct /v1/responses call. + rollout_id = request.path_params.get("rollout_id") + output_items, usage, model_name = await self._run_kilo(user_message, system_prompt, rollout_id) if not any( getattr(item, "type", None) == "message" and getattr(item, "role", None) == "assistant" @@ -450,7 +532,7 @@ async def run(self, request: Request, body: KiloCodeAgentRunRequest) -> KiloCode agent_resp = await self.server_client.post( server_name=self.config.name, - url_path="/v1/responses", + url_path=self.url_path_for_run("/v1/responses", body), json=body.responses_create_params, cookies=cookies, ) diff --git a/responses_api_agents/kilocode_agent/configs/kilocode_agent.yaml b/responses_api_agents/kilocode_agent/configs/kilocode_agent.yaml index 8483e3fbb1..5d4a2b6599 100644 --- a/responses_api_agents/kilocode_agent/configs/kilocode_agent.yaml +++ b/responses_api_agents/kilocode_agent/configs/kilocode_agent.yaml @@ -5,11 +5,18 @@ kilocode_agent: resources_server: type: resources_servers name: ??? + # Kilo's model calls go through this Gym model server, so they are captured and the backend is + # swappable (vLLM, OpenAI, an inference provider) without touching this config. The agent + # registers `model` under a `nemo` provider pointed at the server. To call a provider directly + # instead, set this to null and declare the provider in kilo_config (see README). + model_server: + type: responses_api_models + name: policy_model # Environment-specific; declared here so configs that _inherit_from this one can override it. datasets: null concurrency: 8 command: kilo - model: policy/${policy_model_name} + model: ${policy_model_name} openai_api_key: "" openai_base_url: null env: {} @@ -26,14 +33,10 @@ kilocode_agent: bash: allow edit: allow webfetch: allow - # Kilo calls the model endpoint itself instead of going through a Gym model server, so it gets - # a generic OpenAI-compatible provider aimed at whatever env.yaml points to (OpenAI, an NVIDIA - # endpoint, a Gym-served vLLM). `policy` is only a local label and must match the prefix in - # `model`; `models` is filled in from `model` when kilo.json is written. - provider: - policy: - npm: "@ai-sdk/openai-compatible" - options: - baseURL: ${policy_base_url} - apiKey: ${policy_api_key} + # Sized for a 32k-window model server. Raise both together for a larger one; raising + # max_output_tokens past what the window leaves after kilo's ~10k-token system prompt makes runs + # come back empty (see README). + context_window: 32768 + max_output_tokens: 8192 + reasoning_field: reasoning_content kilo_version: 7.4.15 diff --git a/responses_api_agents/kilocode_agent/tests/test_app.py b/responses_api_agents/kilocode_agent/tests/test_app.py index 89a7e18f26..8d12c089c6 100644 --- a/responses_api_agents/kilocode_agent/tests/test_app.py +++ b/responses_api_agents/kilocode_agent/tests/test_app.py @@ -15,12 +15,15 @@ import asyncio import json +import logging +from functools import partial from pathlib import Path -from unittest.mock import MagicMock, patch +from unittest.mock import AsyncMock, MagicMock, patch import yaml +from omegaconf import OmegaConf -from nemo_gym.config_types import ResourcesServerRef +from nemo_gym.config_types import ModelServerRef, ResourcesServerRef from nemo_gym.openai_utils import ( NeMoGymEasyInputMessage, NeMoGymFunctionCallOutput, @@ -54,6 +57,20 @@ def _make_agent(**kwargs) -> KiloCodeAgent: return agent +def _make_model_server_agent(**kwargs) -> KiloCodeAgent: + """An agent wired to a Gym model server, with just enough server client to resolve its URL. + + The mocked client carries no global config, so the model server entry and the real base-URL + builder are attached to it; base-URL resolution itself runs unmocked. + """ + agent = _make_agent(model_server=ModelServerRef(type="responses_api_models", name="policy_model"), **kwargs) + agent.server_client.global_config_dict = OmegaConf.create( + {"policy_model": {"responses_api_models": {"vllm_model": {"host": "model-host", "port": 9000}}}} + ) + agent.server_client._build_server_base_url = partial(ServerClient._build_server_base_url, agent.server_client) + return agent + + def _events(*objs) -> str: """Serialize a list of event dicts into the JSONL stream kilo run --format json emits.""" return "\n".join(json.dumps(o) for o in objs) @@ -166,6 +183,22 @@ def test_step_finish_usage(self) -> None: assert usage["input_tokens"] == 105 assert usage["output_tokens"] == 20 + def test_empty_length_stop_warns(self, caplog) -> None: + # The signature of an output budget that does not fit the model server's context window. + event = _step_finish_event({"input": 10368, "output": 0}) + event["part"]["reason"] = "length" + with caplog.at_level(logging.WARNING): + _, usage = parse_kilo_events(_events(event)) + assert usage["output_tokens"] == 0 + assert "max_output_tokens" in caplog.text + + def test_length_stop_with_output_does_not_warn(self, caplog) -> None: + event = _step_finish_event({"input": 10368, "output": 8192}) + event["part"]["reason"] = "length" + with caplog.at_level(logging.WARNING): + parse_kilo_events(_events(event)) + assert "max_output_tokens" not in caplog.text + def test_reasoning_prepended_to_next_text(self) -> None: stream = _events(_reasoning_event("let me think"), _text_event("final answer")) items, _ = parse_kilo_events(stream) @@ -282,6 +315,96 @@ def test_unknown_provider_left_alone(self, tmp_path) -> None: assert written["provider"] == {"policy": {}} +class TestModelServer: + def test_effective_model_prefixed_only_with_model_server(self) -> None: + assert _make_agent(model="policy/m")._effective_model() == "policy/m" + assert _make_model_server_agent(model="Qwen/Qwen3-8B")._effective_model() == "nemo/Qwen/Qwen3-8B" + + def test_base_url_resolution(self) -> None: + assert _make_agent()._resolve_model_base_url() == "" + agent = _make_model_server_agent() + assert agent._resolve_model_base_url() == "http://model-host:9000/v1" + assert agent._resolve_model_base_url("r1") == "http://model-host:9000/ng-rollout/r1/v1" + + def test_nemo_provider_written(self, tmp_path) -> None: + # A slashed model name stays whole: kilo splits `-m` on the first `/` only, so the provider is + # `nemo` and the model is `Qwen/Qwen3-8B`, which is the key it looks up in `models`. + agent = _make_model_server_agent(model="Qwen/Qwen3-8B") + agent._write_kilo_config(tmp_path, agent._resolve_model_base_url()) + provider = json.loads((tmp_path / "kilo.json").read_text())["provider"]["nemo"] + assert provider["npm"] == "@ai-sdk/openai-compatible" + assert provider["options"] == { + "apiKey": "EMPTY", # pragma: allowlist secret + "baseURL": "http://model-host:9000/v1", + } + assert provider["models"] == { + "Qwen/Qwen3-8B": { + "name": "Qwen/Qwen3-8B", + "limit": {"context": 32768, "output": 8192}, + "interleaved": {"field": "reasoning_content"}, + } + } + + def test_model_server_config_written_without_kilo_config(self, tmp_path) -> None: + # An empty kilo_config used to short-circuit before kilo.json was written, which would leave a + # model-server-only run with no provider at all. + agent = _make_model_server_agent(model="m") + agent._write_kilo_config(tmp_path, agent._resolve_model_base_url()) + assert json.loads((tmp_path / "kilo.json").read_text())["provider"]["nemo"]["models"].keys() == {"m"} + + def test_kilo_config_merged_and_overridable(self, tmp_path) -> None: + agent = _make_model_server_agent( + model="m", + kilo_config={"permission": {"bash": "allow"}, "provider": {"nemo": {"name": "mine"}}}, + ) + agent._write_kilo_config(tmp_path, agent._resolve_model_base_url()) + written = json.loads((tmp_path / "kilo.json").read_text()) + assert written["permission"]["bash"] == "allow" + assert written["provider"]["nemo"]["name"] == "mine" + + def test_limits_and_reasoning_field_configurable(self, tmp_path) -> None: + agent = _make_model_server_agent( + model="m", context_window=262144, max_output_tokens=16384, reasoning_field=None + ) + agent._write_kilo_config(tmp_path, agent._resolve_model_base_url()) + model = json.loads((tmp_path / "kilo.json").read_text())["provider"]["nemo"]["models"]["m"] + assert model["limit"] == {"context": 262144, "output": 16384} + assert "interleaved" not in model + + def test_rollout_prefix_applied_to_base_url(self, tmp_path) -> None: + agent = _make_model_server_agent(model="m") + agent._write_kilo_config(tmp_path, agent._resolve_model_base_url("task0-rollout1")) + written = json.loads((tmp_path / "kilo.json").read_text()) + assert written["provider"]["nemo"]["options"]["baseURL"].endswith("/ng-rollout/task0-rollout1/v1") + + def test_command_uses_effective_model(self) -> None: + agent = _make_model_server_agent(model="Qwen/Qwen3-8B") + cmd = agent._build_command(Path("/tmp/ws"), "hi") + assert cmd[cmd.index("-m") + 1] == "nemo/Qwen/Qwen3-8B" + + def test_run_kilo_threads_resolved_url_into_config_and_env(self, tmp_path) -> None: + # The seam _run_kilo owns: resolve once, then hand the same URL to kilo.json and the env. + # repo_dir is the project dir, and unlike the workspace it survives the run's cleanup. + agent = _make_model_server_agent(model="m", repo_dir=str(tmp_path)) + proc = MagicMock() + proc.returncode = 0 + proc.communicate = AsyncMock(return_value=(b"", b"")) + with patch("responses_api_agents.kilocode_agent.app.asyncio.create_subprocess_exec") as spawn: + spawn.return_value = proc + asyncio.run(agent._run_kilo("hi", None, "task0-rollout1")) + + expected = "http://model-host:9000/ng-rollout/task0-rollout1/v1" + written = json.loads((tmp_path / "kilo.json").read_text()) + assert written["provider"]["nemo"]["options"]["baseURL"] == expected + assert spawn.call_args.kwargs["env"]["OPENAI_BASE_URL"] == expected + + def test_env_prefers_model_server_over_openai_base_url(self) -> None: + agent = _make_model_server_agent(openai_api_key="k", openai_base_url="https://api.openai.com/v1") + env = agent._env("/tmp/data", "/tmp/config", agent._resolve_model_base_url("r1")) + assert env["OPENAI_BASE_URL"] == "http://model-host:9000/ng-rollout/r1/v1" + assert env["OPENAI_API_KEY"] == "EMPTY" # pragma: allowlist secret + + class TestConfigYaml: def test_module_parses(self) -> None: app_path = Path(__file__).resolve().parent.parent / "app.py" @@ -295,5 +418,7 @@ def test_config_yaml_parses(self) -> None: assert inner["entrypoint"] == "app.py" assert inner["concurrency"] == 8 assert inner["command"] == "kilo" - # kilo resolves `model` as /, so the prefix has to name a declared provider. - assert inner["model"].split("/")[0] in inner["kilo_config"]["provider"] + # The shipped config routes model calls through a Gym model server, so `model` is the bare + # name and the agent supplies the provider; kilo_config must not declare one to collide with. + assert inner["model_server"] == {"type": "responses_api_models", "name": "policy_model"} + assert "provider" not in inner["kilo_config"]