fix(llm): use JSON object structured output

[IceCodeNew]

Summary

• send OpenAI-compatible completion requests with response_format={"type":"json_object"}
• include the existing Pydantic JSON Schema in the final user message and retain strict response validation
• reject pinned any-llm completion providers that do not use the supported OpenAI-compatible JSON Object path
• exhaustively classify every provider exposed by pinned any-llm 1.21.0

Root cause

The production OpenCode Go endpoint accepts the configured base URL and custom User-Agent, but rejects OpenAI json_schema structured-output requests. The same endpoint succeeds with JSON Object mode. The prior SDK-native structured-output call therefore failed upstream even though routing and headers were correct.

Scope

This does not change provider priority, add runtime configuration, modify README or .env examples, or upgrade any-llm.

Validation

• prek run --all-files
• uv run --with pytest --with pytest-cov -- pytest --cov --cov-branch --cov-report=xml (1255 passed)
• production primary-provider probe against the configured OpenCode Go endpoint

Summary by CodeRabbit

• New Features
  • Improved structured LLM responses by enforcing JSON Object mode with strict, schema-based validation.
  • Provider compatibility is now validated for required JSON Object output capability (including fallback), with clearer error messages when unsupported.
• Documentation
  • Updated LLM design documentation to reflect the standardized JSON Object request/validation flow and compatibility rules.
• Tests
  • Expanded coverage for JSON Object transport, outbound request formatting, provider classification, and configuration error paths.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 60bf3ffd-4a48-4adb-b08c-2122518834cb

📥 Commits

Reviewing files that changed from the base of the PR and between 6e69abb and 14c9097.

📒 Files selected for processing (2)

• tests/test_any_llm_provider.py
• weather_briefing/llm/any_llm.py

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/test_any_llm_provider.py

---

📝 Walkthrough

Walkthrough

Changes

The any-llm integration now uses JSON Object response formatting with Pydantic schemas embedded in the final user message. Provider compatibility is tracked centrally, unsupported providers are rejected by configuration and the adapter factory, and tests cover transport, provider classification, and error paths.

JSON Object Output Compatibility

Layer / File(s)	Summary
Provider compatibility contract weather_briefing/data/any_llm_compatibility.py, tests/test_any_llm_compatibility.py	Defines and verifies the provider set that lacks required JSON Object output support.
Schema-backed adapter transport weather_briefing/llm/any_llm.py, tests/test_any_llm_provider.py, tests/test_llm.py, docs/design.md	Builds JSON Object requests, appends Pydantic schemas to the final user message, disables streaming for AnyLLM calls, and validates structured transport behavior.
Configuration capability validation weather_briefing/config/settings.py, tests/test_config.py, tests/test_cli.py	Rejects unsupported primary and fallback providers and updates provider-forwarding and configuration error-path coverage.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant AnyLLMStructuredProvider
  participant PydanticModel
  participant LLMCompletionClient
  participant AnyLLM
  AnyLLMStructuredProvider->>PydanticModel: Generate JSON schema
  AnyLLMStructuredProvider->>LLMCompletionClient: Send schema and json_object response format
  LLMCompletionClient->>AnyLLM: Request completion with streaming disabled
  AnyLLM-->>LLMCompletionClient: Return JSON Object response

Loading

Possibly related PRs

• IceCodeNew/weather-briefing#96: Introduced the adapter and provider factory extended by this change.
• IceCodeNew/weather-briefing#114: Uses the structured-output behavior updated here.
• IceCodeNew/weather-briefing#118: Also updates provider factory capability validation.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: switching LLM structured output to OpenAI-compatible JSON object mode.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/fix-llm-json-object

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.86%. Comparing base (7c874a6) to head (14c9097).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #121   +/-   ##
=======================================
  Coverage   99.86%   99.86%           
=======================================
  Files         115      115           
  Lines       12239    12327   +88     
  Branches      730      736    +6     
=======================================
+ Hits        12222    12310   +88     
  Misses         12       12           
  Partials        5        5           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[qodo-code-review]

PR Summary by Qodo

Fix structured LLM output by using OpenAI JSON Object mode + prompt schema

🐞 Bug fix 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Switch structured completions to OpenAI-compatible json_object response format.
• Embed Pydantic JSON Schema into the final user message and re-validate strictly.
• Reject any-llm providers that cannot support required JSON Object output.

Diagram

graph TD
  A["Settings.from_env"] --> B["Provider validation"] --> C["any_llm.create_any_llm_provider"] --> D["AnyLLMStructuredProvider"] --> E["any-llm SDK"] --> F["OpenAI-compatible endpoint"]
  B --> G["Compatibility metadata"]
  H["Test suite"] --> D --> I["_structured_output_request"]
  J["Design docs"] --> D

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Provider-specific structured output branching

• ➕ Keeps native json_schema structured output where supported
• ➕ Avoids embedding schemas into prompts for providers that support schema natively
• ➖ Adds multiple request-format code paths and higher long-term maintenance
• ➖ Makes behavior/provider parity harder to test and reason about

2. Use tool/function-calling instead of JSON Mode

• ➕ Can be more robust than prompt-only constraints for some providers
• ➕ May provide better error reporting and schema enforcement upstream
• ➖ Not consistently supported across OpenAI-compatible gateways/providers
• ➖ Requires larger contract changes in both request shape and response decoding

3. Runtime capability detection (feature probing) vs pinned blacklist

• ➕ Reduces manual maintenance when any-llm adds/removes provider support
• ➕ Can adapt to upstream capability changes without code updates
• ➖ Requires extra startup calls (latency/cost) and more failure modes
• ➖ Capability probes can be flaky and environment-dependent

Recommendation: Keep the PR’s single-path json_object approach with schema-in-final-user-message: it directly addresses OpenAI-compatible endpoints that reject json_schema while preserving strict Pydantic validation. The pinned blacklist + exhaustive provider classification tests are a pragmatic guardrail; consider capability probing only if provider churn becomes frequent.

Files changed (9) +308 / -70

Bug fix (1) +46 / -5

any_llm.pySwitch structured output to JSON Object with schema appended to user prompt +46/-5

Switch structured output to JSON Object with schema appended to user prompt

• Replaces SDK-native structured-output usage with an OpenAI-compatible JSON Object request ('response_format={"type":"json_object"}') while embedding the Pydantic JSON Schema in the final user message. Adds strict preconditions for the final user message, keeps explicit SDK boundary types, and rejects providers in 'UNSUPPORTED_JSON_OBJECT_PROVIDERS' in the factory.

weather_briefing/llm/any_llm.py

Tests (5) +229 / -59

test_any_llm_compatibility.pyAdd pinned any-llm JSON Object compatibility verification +23/-1

Add pinned any-llm JSON Object compatibility verification

• Introduces a test that derives the set of completion providers that are OpenAI-provider subclasses and asserts the remainder matches 'UNSUPPORTED_JSON_OBJECT_PROVIDERS'. Ensures the compatibility metadata stays aligned with the pinned any-llm SDK behavior.

tests/test_any_llm_compatibility.py

test_any_llm_provider.pyUpdate LLM adapter and factory tests for JSON Object + schema-in-prompt +176/-49

Update LLM adapter and factory tests for JSON Object + schema-in-prompt

• Reworks structured-output assertions to expect 'response_format={"type":"json_object"}' and validates that the final user message includes the JSON Schema. Adds negative tests for missing/invalid final user message content and expands factory tests to classify every any-llm provider, rejecting those without completion or JSON Object support.

tests/test_any_llm_provider.py

test_cli.pyAdjust CLI forwarding test to use a supported any-llm provider +3/-3

Adjust CLI forwarding test to use a supported any-llm provider

• Renames and updates the CLI test to forward a known-supported provider ('openrouter') rather than a now-rejected provider. Keeps the CLI/provider wiring expectations consistent with the stricter provider validation.

tests/test_cli.py

test_config.pyAdd config error coverage for providers lacking JSON Object support +26/-5

Add config error coverage for providers lacking JSON Object support

• Updates provider env fixtures to use supported providers and adds explicit error-path tests for primary and fallback providers that do not support required JSON Object output. Ensures settings validation fails early with clear messages.

tests/test_config.py

test_llm.pyRelax test protocol typing for JSON Object response_format +1/-1

Relax test protocol typing for JSON Object response_format

• Updates the completion stub signature to accept either a Pydantic model type or a JSON Object dict, matching the adapter’s new request shape.

tests/test_llm.py

Documentation (1) +1 / -1

design.mdDocument JSON Object transport and provider blacklist strategy +1/-1

Document JSON Object transport and provider blacklist strategy

• Updates the LLM design section to explain using OpenAI JSON Object mode and embedding the Pydantic JSON Schema in the final user message. Clarifies that strict Pydantic re-validation remains and that unsupported providers are rejected via a pinned compatibility blacklist.

docs/design.md

Other (2) +32 / -5

settings.pyValidate JSON Object provider support during settings load +9/-5

Validate JSON Object provider support during settings load

• Adds 'UNSUPPORTED_JSON_OBJECT_PROVIDERS' checks to '_validate_llm_provider' so primary and fallback providers are rejected if they cannot use required JSON Object output. Also slightly reorders header parsing/validation for the primary provider to run immediately after reading 'LLM_PROVIDER'.

weather_briefing/config/settings.py

any_llm_compatibility.pyIntroduce pinned blacklist of providers without JSON Object support +23/-0

Introduce pinned blacklist of providers without JSON Object support

• Adds 'UNSUPPORTED_JSON_OBJECT_PROVIDERS' to codify which any-llm providers cannot use the OpenAI-compatible JSON Object structured output path. This metadata is used by both settings validation and the adapter factory.

weather_briefing/data/any_llm_compatibility.py

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 46 rules

✅ REVIEW.md

1. Missing content triggers KeyError ✓ Resolved 🐞 Bug ☼ Reliability

Description

_structured_output_request() checks only that the last message role is "user", then directly indexes
messages[-1]["content"], so a malformed message dict can raise KeyError before request-error
normalization runs. This yields an unnormalized exception and skips the API call context wrapper,
making failures harder to interpret.

Code

weather_briefing/llm/any_llm.py[R351-367]

+def _structured_output_request(
+    messages: list[dict[str, str]],
+    response_format: type[BaseModel],
+) -> tuple[list[dict[str, str]], ResponseFormat]:
+    """Prepare prompt-constrained JSON Object transport."""
+    if not messages or messages[-1].get("role") != "user":
+        raise ValueError("JSON Object structured output requires a final user message")
+    schema = json.dumps(response_format.model_json_schema(), ensure_ascii=False, separators=(",", ":"))
+    final_message = {
+        **messages[-1],
+        "content": (
+            f"{messages[-1]['content']}\n\n"
+            "Return only a JSON object matching this JSON Schema exactly. "
+            "Do not wrap it in Markdown fences.\n"
+            f"{schema}"
+        ),
+    }

Relevance

⭐⭐⭐ High

Team often hardens helpers to avoid unexpected exceptions escaping normalization; add defensive
content check is consistent.

PR-#120

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The helper validates role but not content presence, and messages[-1]['content'] will raise
KeyError if absent. Also, _complete() invokes the helper before entering the error-normalization
context, so this KeyError bypasses the intended normalization path.

weather_briefing/llm/any_llm.py[102-118]
weather_briefing/llm/any_llm.py[351-367]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`_structured_output_request()` assumes the final user message contains a `content` key. If an internal caller ever passes a final `{"role": "user"}` message without `content`, the adapter throws `KeyError` outside `_normalize_request_errors()`.

## Issue Context
`AnyLLMStructuredProvider._complete()` calls `_structured_output_request()` before entering the context managers that normalize exceptions and attach request context.

## Fix Focus Areas
- weather_briefing/llm/any_llm.py[102-137]
- weather_briefing/llm/any_llm.py[351-368]

Suggested implementation:
- In `_structured_output_request()`, validate:
 - `content` exists
 - `content` is a non-empty `str` (or at least a `str`)
- Raise `ValueError` with a clear message (e.g., "final user message must include string content") instead of allowing `KeyError`.
- (Optional) Move the `_structured_output_request()` call inside the `with (...)` block if you want these validation errors to be wrapped/attributed consistently.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Validation order hides provider ✗ Dismissed 🐞 Bug ◔ Observability

Description

Settings.from_env() validates LLM_EXTRA_HEADERS against provider header support before validating
that the provider supports the required JSON Object output, so when both are misconfigured the
thrown error can misleadingly point at headers first. This can cause iterative config churn
(removing headers) while the real blocker remains the provider’s lack of JSON Object support.

Code

weather_briefing/config/settings.py[R159-163]

        llm_provider = clean_env(os.getenv("LLM_PROVIDER", "deepseek"))
+        llm_extra_headers = headers_from_env("LLM_EXTRA_HEADERS")
+        _validate_llm_headers_provider("LLM_EXTRA_HEADERS", "LLM_PROVIDER", llm_provider, llm_extra_headers)
        _validate_llm_provider("LLM_PROVIDER", llm_provider)
        llm_model = clean_env(os.getenv("LLM_MODEL"))

Relevance

⭐⭐ Medium

Ordering affects error clarity; team likes fail-fast config validation, but specific
precedence/order policy not established.

PR-#118

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new code validates headers first (so header errors can be raised before provider capability
checks), while JSON Object support is only rejected inside _validate_llm_provider(). This ordering
makes the first raised ConfigurationError dependent on which validation runs first rather than which
incompatibility is fundamental.

weather_briefing/config/settings.py[159-163]
weather_briefing/config/settings.py[305-323]
PR-#118

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`Settings.from_env()` currently validates custom header compatibility before validating the provider’s completion + JSON Object requirements. When multiple env vars are wrong at once (e.g., unsupported provider for JSON Object + headers configured), the user may see a header-related `ConfigurationError` first, which is not the primary incompatibility.

## Issue Context
Provider validation (including JSON Object support) lives in `_validate_llm_provider()`, while header compatibility is checked in `_validate_llm_headers_provider()`. The call order in `from_env()` determines which error message “wins” when multiple conditions fail.

## Fix Focus Areas
- weather_briefing/config/settings.py[159-187]
- weather_briefing/config/settings.py[305-323]

Suggested implementation:
- Call `_validate_llm_provider("LLM_PROVIDER", llm_provider)` before `_validate_llm_headers_provider(...)` (or enhance `_validate_llm_headers_provider` to short-circuit when the provider is unsupported for JSON Object).
- Apply the same ordering/logic for fallback provider validation as well.
- Add/adjust a small test that sets `LLM_PROVIDER=mistral` *and* `LLM_EXTRA_HEADERS` to assert the primary error reported is the JSON Object incompatibility (if that’s the desired policy).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Schema rebuilt per request ✗ Dismissed 🐞 Bug ➹ Performance

Description

Each completion call regenerates and JSON-serializes the full Pydantic schema inside
_structured_output_request(), repeating non-trivial work on every request. This adds avoidable
CPU/allocation overhead on the hot path even though the schema is static per model class.

Code

weather_briefing/llm/any_llm.py[358]

+    schema = json.dumps(response_format.model_json_schema(), ensure_ascii=False, separators=(",", ":"))

Relevance

⭐⭐ Medium

Caching schema could help hot-path CPU, but adds complexity; no clear precedent for caching Pydantic
schemas here.

PR-#86

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

_structured_output_request() constructs the JSON schema string inline via model_json_schema()
and json.dumps(), and _complete() calls _structured_output_request() for every request, making
this repeated work unavoidable without caching.

weather_briefing/llm/any_llm.py[102-113]
weather_briefing/llm/any_llm.py[351-368]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`response_format.model_json_schema()` + `json.dumps(...)` are executed for every completion request, even though the schema for a given `response_format` model is invariant.

## Issue Context
The schema is appended into the final user message for JSON Object mode, so it is safe to precompute per response model class if the schema does not change at runtime.

## Fix Focus Areas
- weather_briefing/llm/any_llm.py[351-368]

Suggested implementation:
- Add a small helper like:
 - `@functools.lru_cache(maxsize=None)`
 - `def _schema_json(model: type[BaseModel]) -> str: ...`
- Use the cached string in `_structured_output_request()`.
- Keep `ensure_ascii=False` and compact separators to preserve current prompt behavior.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[qodo-code-review]

Code review by qodo was updated up to the latest commit 14c9097