feat(code,evals): GLM-5.2 harness profile

[Nick Hollon (nick-hollon-lc)]

Deep Agents Code now applies focused execution guidance when an agent is built with GLM-5.2 on Fireworks, OpenRouter, or Baseten. The guidance encourages earlier artifact creation, exact source preservation, and result-based verification. Headless Fireworks runs also get one bounded recovery attempt when GLM-5.2 reaches its output limit without calling a tool.

---

Terminal-Bench exposed a recurring GLM-5.2 failure mode: the model could spend an entire response planning or explaining, reach its output limit without calling a tool, and leave no deliverable. The same runs showed weaker discipline around exact-output tasks, including changing source material beyond the requested transformation, treating a successful command as proof of correctness, and delaying artifact creation until too late in the run.

This PR addresses those model-specific behaviors without changing the default prompt or runtime behavior for other models.

GLM-5.2 execution profile

Dcode registers a prompt suffix for the exact GLM-5.2 model specs exposed through Fireworks, OpenRouter, and Baseten. Registration is scoped to dcode rather than added as a global SDK default. If a user-provided or future built-in profile already owns the prompt suffix for one of those specs, dcode leaves it unchanged.

The suffix asks GLM-5.2 to:

• identify every required output path and translate exact, ordered, ranged, and prohibited requirements into a short checklist;
• create a valid artifact early, keep it valid while refining it, and reserve the end of the run for verification;
• use the exact requested source, version, tokenizer, date, or revision rather than relying on memory or a nearby substitute;
• preserve source-of-truth input except for transformations the task explicitly requests;
• verify the actual result, including boundary, negative, and cleanup behavior, rather than treating exit code 0 as sufficient;
• preserve correctness and input bytes before optimizing, then compare repeated measurements with enough margin for noise;
• keep unsupported media out of this text-only model's context by extracting needed text, metadata, or frames with a shell utility or script instead of calling read_file on images, PDFs, audio, or video;
• retry an unavailable command once after correcting it, then change approaches instead of repeating the same failure; and
• stop when the requested artifacts and assertions are complete instead of adding speculative work.

The suffix is selected when the agent graph is assembled. It contains guidance only; it does not filter read_file tool results at runtime.

Headless terminal-stall recovery

Non-interactive dcode stacks install a recovery middleware that checks the effective model on each call. For the exact Fireworks GLM-5.2 endpoint, a response that ends with finish_reason="length" and no tool call is retried once with reasoning disabled, a forced tool choice, and a short instruction to create the smallest valid artifact and run one discriminating check.

Recovery is deliberately limited to one retry and to the measured Fireworks endpoint. It does not activate for interactive sessions, OpenRouter, Baseten, structured responses, or responses that already contain a tool call. Dcode installs it on the main agent and the declarative subagents it builds for headless execution.

Benchmark harness fidelity

The Harbor graph now constructs the production headless dcode prompt instead of replacing it with a benchmark-only prompt. It populates the prompt's model identity, context limit, and unsupported-modality fields from the selected model, disables ask_user because Harbor has no resume loop, and defaults the exact GLM-5.2 specs to high reasoning effort unless the caller supplies a provider-native override.

Harbor session IDs are normalized to dcode's filesystem-safe 64-character assistant ID limit. A hash of the original ID preserves disambiguation when two trial IDs normalize to the same text or share a long prefix.

The reusable Harbor workflow also accepts multiple newline-separated package overrides and installs them in one uv resolution. This allows an evaluation to pin Harbor core and harbor-langsmith to the same branch so the plugin resolves against that checkout instead of the PyPI release.

[open-swe]

Open SWE Review found 1 potential issue.

Open in Web • View Open SWE trace