feat: add pre-flight LLM validation endpoint (POST /api/profiles/{name}/validate) by neubig · Pull Request #4422 · OpenHands/software-agent-sdk

neubig · 2026-08-07T22:54:27Z

HUMAN:
Validated the endpoint behavior with the listed focused pytest cases.

Summary

Adds a POST /api/profiles/{name}/validate endpoint to the agent-server that fires a minimal 1-token LLM completion call to catch misconfigurations before a profile is saved.

This resolves the backend half of OpenHands/OpenHands#16416.

AGENT:

Why

PostHog error tracking shows thousands of recurring exceptions from LLM misconfigurations — invalid model names, missing provider prefixes, bad base URLs, and invalid API keys — that only surface when a conversation starts and the first LLM call fails. All of these could be caught at save time.

Summary

New POST /api/profiles/{name}/validate endpoint in profiles_router.py
Accepts an LLM config (same shape as SaveProfileRequest.llm)
Decrypts incoming secrets (same as save_profile)
Fires llm.completion(messages=[ping], max_tokens=1)
Returns ValidateProfileResponse with {valid, error?: {type, message}}
Uses the SDK typed exception hierarchy (LLMAuthenticationError, LLMBadRequestError, etc.)
Transient errors (LLMRateLimitError, LLMTimeoutError) are non-blocking

REST API contract changes

Compared with base OpenAPI c7e270aae43a for public /api/** paths.

--- base public OpenAPI
+++ head public OpenAPI
@@ -93,0 +94 @@
+operation POST /api/file/create_directory operationId=create_directory_api_file_create_directory_post
@@ -108,0 +110 @@
+operation POST /api/profiles/{name}/validate operationId=validate_profile_api_profiles__name__validate_post
@@ -228,0 +231 @@
+parameter POST /api/file/create_directory query:path required=true schema=type="string"
@@ -234,0 +238 @@
+parameter POST /api/profiles/{name}/validate path:name required=true schema=type="string" minLength=1 maxLength=64 pattern="^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$"
@@ -272,0 +277 @@
+requestBody POST /api/profiles/{name}/validate application/json required=true schema=ValidateProfileRequest
@@ -487,0 +493,2 @@
+response POST /api/file/create_directory 200 application/json schema=Success
+response POST /api/file/create_directory 422 application/json schema=HTTPValidationError
@@ -518,0 +526,2 @@
+response POST /api/profiles/{name}/validate 200 application/json schema=ValidateProfileResponse
+response POST /api/profiles/{name}/validate 422 application/json schema=HTTPValidationError
@@ -2565,0 +2575,8 @@
+schema ValidateProfileError property message required schema=type="string"
+schema ValidateProfileError property type required schema=type="string"
+schema ValidateProfileError type="object"
+schema ValidateProfileRequest property llm required schema=LLM-Input
+schema ValidateProfileRequest type="object"
+schema ValidateProfileResponse property error optional schema=anyOf=[ValidateProfileError,type="null"]
+schema ValidateProfileResponse property valid required schema=type="boolean"
+schema ValidateProfileResponse type="object"

How to Test

pytest tests/agent_server/test_profiles_router.py -k validate_profile — all 6 pass
Frontend PR (OpenHands/OpenHands) will wire this into the save flow

This PR was created by an AI agent (OpenHands) on behalf of the user.

🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant	Architectures	Base Image	Docs / Tags
java	amd64, arm64	`eclipse-temurin:17-jdk`	Link
python	amd64, arm64	`nikolaik/python-nodejs:python3.13-nodejs22-slim`	Link
golang	amd64, arm64	`golang:1.21-bookworm`	Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:03dab92-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-03dab92-python \
  ghcr.io/openhands/agent-server:03dab92-python

All tags pushed for this build

ghcr.io/openhands/agent-server:03dab92-golang-amd64
ghcr.io/openhands/agent-server:03dab921876a8c5aef27f202a1c39eeb24924ca2-golang-amd64
ghcr.io/openhands/agent-server:feat-llm-preflight-validation-golang-amd64
ghcr.io/openhands/agent-server:03dab92-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:03dab92-golang-arm64
ghcr.io/openhands/agent-server:03dab921876a8c5aef27f202a1c39eeb24924ca2-golang-arm64
ghcr.io/openhands/agent-server:feat-llm-preflight-validation-golang-arm64
ghcr.io/openhands/agent-server:03dab92-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:03dab92-java-amd64
ghcr.io/openhands/agent-server:03dab921876a8c5aef27f202a1c39eeb24924ca2-java-amd64
ghcr.io/openhands/agent-server:feat-llm-preflight-validation-java-amd64
ghcr.io/openhands/agent-server:03dab92-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:03dab92-java-arm64
ghcr.io/openhands/agent-server:03dab921876a8c5aef27f202a1c39eeb24924ca2-java-arm64
ghcr.io/openhands/agent-server:feat-llm-preflight-validation-java-arm64
ghcr.io/openhands/agent-server:03dab92-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:03dab92-python-amd64
ghcr.io/openhands/agent-server:03dab921876a8c5aef27f202a1c39eeb24924ca2-python-amd64
ghcr.io/openhands/agent-server:feat-llm-preflight-validation-python-amd64
ghcr.io/openhands/agent-server:03dab92-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim-amd64
ghcr.io/openhands/agent-server:03dab92-python-arm64
ghcr.io/openhands/agent-server:03dab921876a8c5aef27f202a1c39eeb24924ca2-python-arm64
ghcr.io/openhands/agent-server:feat-llm-preflight-validation-python-arm64
ghcr.io/openhands/agent-server:03dab92-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim-arm64
ghcr.io/openhands/agent-server:03dab92-golang
ghcr.io/openhands/agent-server:03dab921876a8c5aef27f202a1c39eeb24924ca2-golang
ghcr.io/openhands/agent-server:feat-llm-preflight-validation-golang
ghcr.io/openhands/agent-server:03dab92-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:03dab92-java
ghcr.io/openhands/agent-server:03dab921876a8c5aef27f202a1c39eeb24924ca2-java
ghcr.io/openhands/agent-server:feat-llm-preflight-validation-java
ghcr.io/openhands/agent-server:03dab92-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:03dab92-python
ghcr.io/openhands/agent-server:03dab921876a8c5aef27f202a1c39eeb24924ca2-python
ghcr.io/openhands/agent-server:feat-llm-preflight-validation-python
ghcr.io/openhands/agent-server:03dab92-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim

About Multi-Architecture Support

Each variant tag (e.g., 03dab92-python) is a multi-arch manifest supporting both amd64 and arm64
Docker automatically pulls the correct architecture for your platform
Individual architecture tags (e.g., 03dab92-python-amd64) are also available if needed

Closes #4429

Add a validation endpoint that fires a minimal 1-token LLM completion to catch misconfigurations (invalid model names, missing provider prefixes, bad base URLs, invalid API keys) before a profile is saved. Transient errors (rate limits, timeouts) are non-blocking. The endpoint returns a structured {valid, error?: {type, message}} response using the SDK's typed exception hierarchy. Resolves OpenHands/OpenHands#16416 Co-authored-by: openhands <openhands@all-hands.dev>

github-actions · 2026-08-07T22:54:55Z

Python API breakage checks — ✅ PASSED

Result: ✅ PASSED

Action log

github-actions · 2026-08-07T22:55:23Z

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

github-actions · 2026-08-07T22:58:10Z

Coverage Report •

File	Stmts	Miss	Cover	Missing
openhands-agent-server/openhands/agent_server
profiles_router.py	177	6	97%	441–446
TOTAL	40745	17206	58%

all-hands-bot · 2026-08-07T23:34:26Z

👋 This PR needs a couple of things fixed before OpenHands can review it:

the PR description's HUMAN: section needs at least 20 characters describing what you tested, not just the template placeholder

Push an update once this is addressed and this check re-runs automatically.

This is an automated check - no AI was used to generate this comment.

all-hands-bot · 2026-08-10T19:56:16Z

🚦 CI is currently failing on this PR's latest commit.

Please fix the failing checks before OpenHands reviews it - this is re-checked automatically once you push a new commit. (A maintainer can also request @all-hands-bot as a reviewer to have it reviewed regardless of CI status.)

This is an automated check - no AI was used to generate this comment.

all-hands-bot · 2026-08-13T02:10:32Z

✅ Review complete.

This review was performed through OpenHands Cloud Automation. You can log in and view the conversation here.

all-hands-bot

🟡 Taste Rating: Acceptable — the endpoint shape is straightforward, but the validation call currently uses the wrong execution path for the server/runtime.

[CRITICAL ISSUES]

[openhands-agent-server/openhands/agent_server/profiles_router.py, Line 281] Correctness / async safety: The endpoint calls synchronous LLM.completion() directly inside an async FastAPI route. That can block the server event loop during provider I/O/retry waits, and it validates the chat-completions path even for profiles that runtime conversations will send through the Responses API.

[TESTING GAPS]

[tests/agent_server/test_profiles_router.py, Line 1261] Add coverage for the runtime API path selection, especially api_mode="responses" / uses_responses_api(), so this endpoint fails if it stops validating the same path conversations actually use.

[RISK ASSESSMENT]

[Overall PR] ⚠️ Risk Assessment: 🟡 MEDIUM
This is an additive REST endpoint, but it sits on the profile save path and performs live LLM provider I/O from the agent-server. The main risk is operational/correctness: a validation request can pin the event loop and can report a profile as valid/invalid based on a different LLM API path than normal conversations use.

VERDICT:
❌ Needs rework: Use the async runtime-equivalent LLM path (aresponses vs acompletion, or the shared agent helper) and add a test that covers Responses API profiles before a human maintainer relies on this endpoint.

KEY INSIGHT:
Preflight validation only buys safety if it exercises the same non-blocking provider path the agent-server will use in real conversations.

Improve this review? If any feedback above seems incorrect or irrelevant to this repository, you can teach the reviewer to do better:

Add a .agents/skills/custom-codereview-guide.md file to your branch (or edit it if one already exists) with the /codereview trigger and the context the reviewer is missing (e.g., "Security concerns about X do not apply here because Y"). See the customization docs for the required frontmatter format.

Re-request a review - the reviewer reads guidelines from the PR branch, so your changes take effect immediately.

When your PR is merged, the guideline file goes through normal code review by repository maintainers.

Resolve with AI? Install the iterate skill in your agent and run /iterate to automatically drive this PR through CI, review, and QA until it is merge-ready.

Was this review helpful? React with 👍 or 👎 to give feedback.

This review was generated by an AI agent (OpenHands) on behalf of the user through OpenHands Automation. View conversation

…mpletion) Co-authored-by: openhands <openhands@all-hands.dev>

neubig · 2026-08-14T23:40:10Z

@all-hands-bot I've pushed f06f8799c addressing both review threads:

Async + runtime-path dispatch (profiles_router.py): the validate endpoint now branches to await llm.aresponses(...) when llm.uses_responses_api() is true, else await llm.acompletion(...) — mirroring the runtime dispatch (amake_llm_completion) and staying async so provider I/O doesn't pin the FastAPI event loop.
Tests (test_profiles_router.py): added test_validate_profile_responses_api covering the Responses API path; failure-path tests now patch acompletion with the chat mode.

Both inline review threads are resolved. Full agent-server profiles test file passes (85 tests). Requesting re-review.

all-hands-bot · 2026-08-14T23:41:28Z

✅ Review complete.

This review was performed through OpenHands Cloud Automation. You can log in and view the conversation here.

all-hands-bot

🟡 Acceptable — Clean, minimal solution that solves a documented real problem (PostHog showing thousands of LLM misconfiguration exceptions). The exception hierarchy is used correctly, the async dispatch mirrors the runtime path, and the 7 tests cover the important cases. Two minor improvement opportunities below.

[IMPROVEMENT OPPORTUNITIES]

[profiles_router.py, Line 287] Observability: Transient error is caught without binding (except _TRANSIENT_ERROR_TYPES:), so the log message can't include which error type was swallowed. In a rate-limit scenario vs. a timeout, these are meaningfully different signals. Bind it so the log is actionable.
[profiles_router.py, Line 296] Transient mis-classification: LLMServiceUnavailableError is treated as a blocking config error, but a 503 from the provider is indistinguishable from a provider outage vs. a wrong base URL. During a provider outage, a user with a perfectly valid config will see valid=false and be confused. Consider moving it to _TRANSIENT_ERROR_TYPES or documenting the deliberate trade-off in a comment so the next reader understands the choice.

[RISK ASSESSMENT]

[Overall PR] ⚠️ Risk Assessment: 🟡 MEDIUM
New endpoint opens a real LLM I/O path from an HTTP handler: any caller authenticated to the agent server can fire arbitrary 1-token completions against any provider + API key they supply. This is consistent with what save_profile already allows and presumably sits behind the same auth middleware, but it's worth noting the endpoint can be used as a low-friction key oracle (confirm validity of any key in one call). The ProfileName path-parameter validation (422 on invalid names) is a nice defensive touch.

VERDICT:
✅ Worth merging — Functionally correct, solves a documented pain point, 7 focused tests. The LLMServiceUnavailableError classification is the one call worth revisiting before the frontend wires this into the save flow.

KEY INSIGHT:
The dispatch correctly mirrors amake_llm_completion's uses_responses_api() branching — good future-proofing — but LLMServiceUnavailableError in the blocking handler will produce false negatives during provider outages.

Improve this review? If any feedback above seems incorrect or irrelevant to this repository, you can teach the reviewer to do better:

Add a .agents/skills/custom-codereview-guide.md file to your branch (or edit it if one already exists) with the /codereview trigger and the context the reviewer is missing (e.g., "Security concerns about X do not apply here because Y"). See the customization docs for the required frontmatter format.

Re-request a review - the reviewer reads guidelines from the PR branch, so your changes take effect immediately.

When your PR is merged, the guideline file goes through normal code review by repository maintainers.

Resolve with AI? Install the iterate skill in your agent and run /iterate to automatically drive this PR through CI, review, and QA until it's merge-ready.

Was this review helpful? React with 👍 or 👎 to give feedback.

This review was generated by an AI agent (OpenHands) on behalf of the user through OpenHands Automation. View conversation

…or trade-off

all-hands-bot · 2026-08-14T23:52:01Z

✅ Review complete.

This review was performed through OpenHands Cloud Automation. You can log in and view the conversation here.

neubig mentioned this pull request Aug 7, 2026

feat: add LLM pre-flight validation to prevent saving misconfigured profiles OpenHands/OpenHands#16417

Merged

1 of 5 tasks

This was referenced Aug 8, 2026

feat: add pre-flight LLM validation endpoint (POST /api/profiles/{name}/validate) #4429

Closed

Add pre-flight LLM validation endpoint to prevent saving misconfigured LLM profiles OpenHands/OpenHands#16416

Closed

all-hands-bot added the review-this label Aug 13, 2026

all-hands-bot reviewed Aug 13, 2026

View reviewed changes

Comment thread openhands-agent-server/openhands/agent_server/profiles_router.py Outdated

Comment thread tests/agent_server/test_profiles_router.py Outdated

github-actions Bot mentioned this pull request Aug 14, 2026

chore(main): release 1.38.1 OpenHands/typescript-client#343

Merged

fix: preflight validation uses async runtime dispatch (aresponses/aco…

f06f879

…mpletion) Co-authored-by: openhands <openhands@all-hands.dev>

neubig requested a review from all-hands-bot August 14, 2026 23:40

style: ruff-format test_profiles_router responses test

b6f03af

all-hands-bot approved these changes Aug 14, 2026

View reviewed changes

Comment thread openhands-agent-server/openhands/agent_server/profiles_router.py Outdated

Comment thread openhands-agent-server/openhands/agent_server/profiles_router.py

refactor: log transient error type; document LLMServiceUnavailableErr…

281490e

…or trade-off

neubig requested a review from all-hands-bot August 14, 2026 23:50

This was referenced Oct 15, 2026

Qualify Claude ACP outcome schema alternatives for ScopeJudge (#117) geeen-jp/software-agent-sdk#118

Merged

fix(condenser): show the summarizer whole events, not their 500-char preview Ergartis/software-agent-sdk#2

Merged

Navigation Menu

Repository navigation

feat: add pre-flight LLM validation endpoint (POST /api/profiles/{name}/validate) - #4422

feat: add pre-flight LLM validation endpoint (POST /api/profiles/{name}/validate)#4422
neubig merged 5 commits into
mainfrom
feat/llm-preflight-validation

neubig commented Aug 7, 2026 •

edited by github-actions Bot

Loading

Uh oh!

github-actions Bot commented Aug 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 7, 2026 •

edited

Loading

Uh oh!

all-hands-bot commented Aug 7, 2026

Uh oh!

all-hands-bot commented Aug 10, 2026

Uh oh!

all-hands-bot commented Aug 13, 2026 •

edited

Loading

Uh oh!

all-hands-bot left a comment

Uh oh!

Uh oh!

Uh oh!

neubig commented Aug 14, 2026

Uh oh!

all-hands-bot commented Aug 14, 2026 •

edited

Loading

Uh oh!

all-hands-bot left a comment

Uh oh!

Uh oh!

Uh oh!

all-hands-bot commented Aug 14, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

neubig commented Aug 7, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Summary

REST API contract changes

How to Test

Uh oh!

github-actions Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Python API breakage checks — ✅ PASSED

Uh oh!

github-actions Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

REST API breakage checks (OpenAPI) — ✅ PASSED

Uh oh!

github-actions Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

all-hands-bot commented Aug 7, 2026

Uh oh!

all-hands-bot commented Aug 10, 2026

Uh oh!

all-hands-bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

all-hands-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

neubig commented Aug 14, 2026

Uh oh!

all-hands-bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

all-hands-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

all-hands-bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

neubig commented Aug 7, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Aug 7, 2026 •

edited

Loading

github-actions Bot commented Aug 7, 2026 •

edited

Loading

github-actions Bot commented Aug 7, 2026 •

edited

Loading

all-hands-bot commented Aug 13, 2026 •

edited

Loading

all-hands-bot commented Aug 14, 2026 •

edited

Loading

all-hands-bot commented Aug 14, 2026 •

edited

Loading