fix(agent): retry title generation with json_object on HTTP 400

[chelsealong]

What does this PR do?

generate_title() hardcodes a strict response_format: json_schema on every
title-generation call, with no fallback. Providers that reject json_schema
get a 400 that lands in the generic except bucket as non-retryable — the
session keeps its truncated derived title forever, with no user-visible
error.

This adds a single retry: if the json_schema attempt fails with an HTTP
400, retry once with response_format: json_object, which these providers
accept and which the existing loose-JSON extraction (_extract_title_text)
already tolerates. Detection is via status_code (set by the openai SDK)
with a text fallback for the "Error code: 400 - ..." message shape, so it
also catches rejections that never mention response_format by name — e.g.
vLLM translating json_schema into an uncompilable guided_grammar and
400ing with a compile_grammar_error: No module named 'xgrammar' message.

Related Issue

Fixes #82816

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

• agent/title_generator.py: added _response_format_rejected(); generate_title() now retries the title call with json_object when the json_schema attempt 400s, instead of giving up.
• tests/agent/test_title_generator.py: added tests for the SDK-status_code-400 case, the vLLM guided_grammar/xgrammar message shape (no status_code attribute, no mention of response_format), and a regression guard that an unrelated failure (e.g. a 402) does not retry.

How to Test

1. Configure auxiliary.title_generation to point at an OpenAI-compatible endpoint that rejects response_format: json_schema (any vLLM gateway without xgrammar, or DeepSeek/Kimi).
2. Before the fix: every session's title stays the truncated first-message text; agent.log logs Title generation failed: Error code: 400 - ... on every new session.
3. After the fix: the title call retries with json_object and the session gets an LLM-generated title.

Test output (in a local venv with the project's pinned core deps + pytest):

$ python -m pytest tests/agent/test_title_generator.py -q
....................................                                     [100%]
36 passed in 1.24s

Confirmed the two new tests fail without the fix (git checkout HEAD~1 -- agent/title_generator.py — the commit before this one — then rerun):

FAILED tests/agent/test_title_generator.py::TestGenerateTitle::test_retries_with_json_object_on_http_400_status_code - AssertionError: assert None == 'Fix login button'
FAILED tests/agent/test_title_generator.py::TestGenerateTitle::test_retries_on_vllm_guided_grammar_rejection - AssertionError: assert None == 'Debug xgrammar error'
2 failed, 1 passed, 33 deselected in 0.39s

Also ran the neighboring suite to check for regressions:

$ python -m pytest tests/agent/test_auxiliary_client.py -q
180 passed in 7.41s

ruff check agent/title_generator.py tests/agent/test_title_generator.py — all checks passed.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):)
• I searched for existing PRs to make sure this isn't a duplicate — fix(aux): downgrade json_schema response_format for json_object-only providers #82372, fix(title): fall back to json_object response_format on provider rejection #82751, fix(titles): fall back when JSON schema is unavailable #82073, and fix(aux): strip response_format from Anthropic extra_body passthrough #82307 all touch the same response_format fallback problem, but each is scoped to a specific provider (DeepSeek/Kimi hardcoded list, or Anthropic's extra_body passthrough) or gates the retry on the error message containing the literal string "response_format". None of those cover this issue's vLLM guided_grammar/xgrammar failure, whose error text never mentions response_format. This PR's status_code-based detection is provider-agnostic and covers that case.
• My PR contains only changes related to this fix/feature
• I've run the test suite and all tests pass
• I've added tests for my changes
• I've tested on my platform — tested via a scratch venv (pinned core deps + pytest 9.1.1) on Linux; no platform-specific code touched

Documentation & Housekeeping

• N/A — no config keys, docs, or architecture changed

AI assistance disclosure

This PR was prepared with AI assistance (Claude, Anthropic) under human supervision: the agent read the issue and the four related open PRs, identified the gap (none of them catch the vLLM guided_grammar/xgrammar 400, since its message never says response_format), wrote the minimal fix and tests, and verified the tests fail on the pre-fix code and pass after.

[alt-glitch]

This was generated by AI during triage.

Related: #82073 uses a narrower response-format rejection predicate, while #82372 uses a static provider-aware downgrade. This PR's HTTP-400 fallback also covers the vLLM guided_grammar case from #82816; maintainers should choose the retry scope.

[teknium1]

Thanks @chelsealong for working on the title-generation 400 on json_schema. This landed on main through #89589 and #113966 (5cc8177), which covers the same symptom on the primary and fallback auxiliary paths and omits the field up front for providers known to reject it. Closing as superseded by the landed fix — the tracking issue (#83390 cluster) is closed with the same references.