cp: feat: support kilocode model calls through a Gym model server (2319) into r0.5.0 - #2326
Merged
Conversation
## Summary
The kilocode agent called its model provider directly, so its requests
and responses never reached Gym. Model calls went uncaptured, the
provider block had to be rewritten per backend, and there was no path to
token IDs or logprobs.
This adds `model_server` to `KiloCodeAgentConfig`. When set, the agent
writes a `nemo` provider into `kilo.json` pointed at that server and
passes `-m nemo/<model>`, so one config runs against vLLM, OpenAI, or an
inference provider by swapping `--model-type`. Setting `model_server:
null` keeps the previous behaviour of calling a provider declared in
`kilo_config`.
## What changed
- `model_server` wiring: `_write_kilo_config` becomes build-then-write.
It used to return early when `kilo_config` was empty, which would have
left a model-server-only run with no provider at all. `_build_command`
and `_env` now prefer the resolved URL, so a config carrying both a
model server and `openai_base_url` does not point the subprocess
environment at the provider the `nemo` provider is meant to replace.
- Per-rollout capture: During `/run` the base URL carries the
`/ng-rollout/<id>` prefix, using the base class's `url_path_for_run` and
`resolve_model_base_url` rather than a local reimplementation. Captured
model calls are attributable to the rollout that made them.
## On the model limits
`context_window` and `max_output_tokens` set kilo's `limit`. The
defaults are sized for a 32k-window model server rather than copied from
`opencode_agent`.
Kilo's system prompt and tool definitions run to ~10k tokens, so a large
output budget pushes `prompt + max_tokens` past `max_model_len`. vLLM
rejects that with a 400, which the Gym model server converts into an
empty completion with `finish_reason: length` rather than an error
(`vllm_model/app.py`, `is_out_of_context_length`). The run then produces
no assistant message and scores zero, with nothing in the CLI's output
to say why.
Two related notes:
- There is no truncating default to guard against. An unlisted model
gets `limit.output: 0`, and kilo's `min(limit.output, 32000) || 32000`
falls back to 32000. The hazard is a budget that is too *large*, not too
small.
- `reasoning_field` sets `interleaved.field`. Kilo turns interleaved
reasoning off for custom openai-compatible providers unless the field is
named (its built-in default only applies it to model ids containing
"deepseek"), so without this the reasoning channel is dropped. It
defaults to `reasoning_content`, which is what Gym model servers write;
vLLM >= 0.16 also sends `reasoning`, which is why it is configurable
rather than hardcoded.
Kilo splits `-m` on the first `/` only (`let [A, ...L] = H.split("/")`),
so a slashed model name such as `nemo/Qwen/Qwen2.5-7B-Instruct` resolves
to the model `Qwen/Qwen2.5-7B-Instruct` under the provider `nemo`.
Verified against the shipped binary and live.
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Author
|
/ok to test b22ef0c |
ananthsub
approved these changes
Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
beep boop [🤖]: Hi @ananthsub 👋,