Add Harmony (GPT-OSS) Tool Calling Support in FL

[Sayan Shaw (sayanshaw24)]

Add Harmony (GPT-OSS) Tool Calling Support in FL

Summary

Adds data-driven tool calling support for GPT-OSS (Harmony) models, which use a fundamentally different protocol from ChatML-style models (Phi, Qwen). No model-specific code branches — all format knowledge is captured in a ToolCallConfig struct that the accumulator interprets at runtime.

Problem

ChatML models emit tool calls as:

<tool_call>{"name": "func", "arguments": {...}}</tool_call>

GPT-OSS models use header-based routing:

<|start|>assistant to=functions.get_weather<|channel|>default<|message|>{"city": "Seattle"}<|call|>

Key differences:

• Function name is in the header (to=functions.X), not in the JSON body
• Arguments are the raw body (no wrapping {"name": ..., "arguments": ...} object)
• <|call|> is both EOT and EOS (generation stops naturally)
• Parallel calls use <|end|> for intermediate calls, <|call|> for the final one

Changes

New: tool_call_config.h

• ToolCallConfig struct with Mode::kSimple (ChatML) and Mode::kHeaderInspection (Harmony)
• Harmony() preset: message_token, header_regex, channel_token, end_tokens
• FromModelType(model_type) factory: "gptoss" → Harmony, everything else → Simple

Modified: tool_call_stream_accumulator.h

• New constructor accepting ToolCallConfig
• Header-inspection state machine: Idle → InHeader → InBody → Emit
• Regex-based function name extraction from header
• Multiple end tokens supported (<|end|> + <|call|> for parallel calls)
• Simple mode path completely unchanged

Modified: chat_session.cc

• Both accumulator creation sites (Responses API + ChatCompletions) now detect model type via model_.GetGenAIConfig().model->type and pass the appropriate ToolCallConfig

New tests: tool_call_stream_accumulator_test.cc

• HarmonyBasicToolCall — single tool call in one chunk
• HarmonyToolCallSplitAcrossChunks — streaming token-by-token
• HarmonyRegularTextNotToolCall — header without to=functions.X is visible text
• HarmonyParallelToolCalls — two calls, first ends with <|end|>, second with <|call|>
• HarmonyTextBeforeToolCall — preamble text preserved
• HarmonyFlushUnterminatedHeader — graceful EOS mid-header
• HarmonyFlushUnterminatedBody — graceful EOS mid-body

Design Principles

• Data-driven: ToolCallConfig captures all protocol knowledge; no if/else on model name in the accumulator
• Backward compatible: existing models use Mode::kSimple (default) — zero behavior change
• Forward compatible: adding a new protocol = new Mode enum + preset; adding a new model with existing protocol = one line in FromModelType
• No EOS suppression needed: Harmony's <|end|> (not EOS) vs <|call|> (EOS) design naturally supports parallel calls

Depends On

• FL PR Read Tool & Reasoning Tokens from GenAI API instead of Catalog Metadata #838 (tool/reasoning metadata migration) — for bot_token_id/eot_token_id resolution from genai_config.json
• GenAI 0.15.1+ with bot/eot APIs

Testing

• 18/18 accumulator unit tests pass (11 existing + 7 new Harmony tests)
• E2E validation with gpt-oss-20b-generic-cpu model pending

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Ready	Preview	Aug 11, 2026 12:02am