[Proposal] Standardize Discovery of Tool Output Schemas #12FactorTools

[Elijas]

Tip

tldr²: (1) llm should be able to know exactly what kind of data of what type will be returned before poking around with the api, this faciliates planning. (2) Let tools return larger outputs, and let the llms prewrite jq (or similar) filtering/extract query based on the output schema. I.e. LLM could in one step do list_top_users(count=10) AND jq '.[] | .email' on the output, and efficiently retrieve only the data it actually wants

Tip

TL;DR: Building on Issue #21 (mandating structured results like {"name": "John"} instead of "The name is John"):

• Problem: Clients (LLMs, frameworks) lack a standard way to know the schema of tool results beforehand (unlike discoverable input schemas).
• Proposal: Introduce a standard mechanism for discovering tool output schemas (e.g., JSON Schema).
• Key LLM Benefit: Enables better planning and token optimization (Factor 3). Knowing the schema allows the LLM to request specific data subsets (e.g., extracting just result.name) instead of ingesting large, verbose results into the context window.
• Other Benefits: Framework validation, developer tooling (code-gen, type safety), complete tool contracts.

This proposal is part of a series. Check out the rest here.

---

Introduction

First off, thanks for the excellent work on the 12-Factor Agents guide! It provides a really valuable set of principles for building robust and maintainable agentic systems.

Building upon previous discussions, I wanted to raise a point regarding the discoverability of tool output structures.

Context

• Issue [Proposal] Explicitly Recommend Structured Data for Tool Results #12FactorTools #21 proposes that tools must return structured data as their canonical result (e.g., {"structured_result": {"product": 15}}), potentially with an optional text summary.
• Existing principles/Factor 4 (often implemented via protocols like MCP) already emphasize standardized discovery for tool input parameters, typically using JSON Schema via methods like tools/list.
• This creates an asymmetry: we can programmatically discover how to call a tool, but not what structure to expect back.

The Gap / Suggestion

Currently, there's no standard mechanism recommended within the 12-Factor principles for a client (be it the agent framework, an LLM, or a developer tool) to programmatically determine the expected schema of the structured data returned by a tool.

I propose that the guide should recommend or incorporate a standardized mechanism for discovering the output schema of tools, complementing the existing practices for input schema discovery.

Reasoning / Value Proposition:

Knowing the output schema beforehand provides significant advantages:

1. Enhanced LLM Planning: An LLM can better plan multi-step workflows if it knows the precise structure of data it will receive from intermediate tool calls.
2. LLM Self-Optimization / Token Efficiency: As discussed previously, an LLM aware of the output schema could potentially request that the framework call the tool but only return specific fields from the result (e.g., using a JSONPath or jq-like expression derived from the schema). This drastically reduces tokens added back into the context window when dealing with large/verbose tool outputs.
3. Framework/Client-Side Validation: The agent framework or client can validate that the tool's actual response conforms to the advertised output schema, improving robustness and error detection.
4. Developer Tooling & Type Safety: Output schemas enable automatic generation of client-side data classes/types, improving developer experience and code safety, especially in statically-typed languages.
5. Complete Tool Contract: Standardizing both input and output schema discovery provides a full, machine-readable contract for each tool, enhancing system predictability and maintainability.
6. Improved Debugging: Makes it easier to compare expected vs. actual tool outputs during development and troubleshooting.

Potential Implementation Approaches:

A standard mechanism could take different forms (the specific choice is open for discussion):

• Extend Existing Discovery: The response format for discovering tools (e.g., MCP's tools/list) could be extended to include an outputSchema (e.g., JSON Schema) alongside the input parameters for each tool.
• New Discovery Method: A new standard method could be introduced (e.g., tools/getOutputSchema(tool_name: string)) that returns the output schema for a specific tool on demand.

The key is the presence of some standard way to achieve this discovery.

Proposed Action:

Consider incorporating the principle of output schema discoverability into the guide, potentially by:

• Adding discussion to factors related to tools and their interaction (Factor 3, Factor 4, Factor 8).
• Introducing a new factor or appendix note dedicated to the importance of complete, discoverable tool contracts (including output schemas).

Standardizing output schema discovery feels like a crucial step towards fully robust, introspectable, and efficient tool usage in agentic systems, building directly on the foundation laid by structured results (Issue #21) and standardized interfaces (like MCP, Issue #20).

Would love to hear your thoughts on this!

Thanks again for the great resource.