test(a2a): assert the property agent returns listings, not just non-empty text

[devin-ai-integration]

TLDR

Problem this solves:

• Property-agent e2e test passed even when the agent returned zero listings
• It only asserted the reply text was non-empty, never the tool output
• Its query (USSFO, sale) can never match the live catalogue

How it solves it:

• Query a catalogue slice that has data (GBLON, long_term)
• Model the search_results artifact and assert real listings came back
• Test now fails if search_properties returns nothing

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Captured against a live proxy at commit 6fd8a68a66, driving the real movehome.org agent registered through the proxy (no mocks). The agent runs its search_properties skill and returns a search_results artifact

New query the test now sends (has data), and the old query it used to send (structurally empty):

$ AID=$(curl -s -X POST http://localhost:4000/v1/agents -H 'Authorization: Bearer sk-1234' \
    -H 'content-type: application/json' -d "$BODY" | jq -r .agent_id)   # movehome card, url -> origin

# AFTER (new query): un_locode=GBLON, service_type=long_term
$ curl -s -X POST http://localhost:4000/a2a/$AID -H 'Authorization: Bearer sk-1234' \
    -d '{"jsonrpc":"2.0","id":"1","method":"message/send","params":{"message":{"role":"user",
        "parts":[{"kind":"data","data":{"skill":"search_properties",
        "params":{"un_locode":"GBLON","service_type":"long_term","limit":3}}}],"messageId":"m1"}}}'
GBLON/long_term -> status: Found 21 listings; returning 3. | artifact total: 21 first listing: prop-gb-rlf-001141

# BEFORE (old query): un_locode=USSFO, service_type=sale, asking_price_max=2_000_000
$ curl -s -X POST http://localhost:4000/a2a/$AID ... "params":{"un_locode":"USSFO","service_type":"sale","asking_price_max":2000000,"limit":3}
USSFO/sale -> status: No listings matched the search criteria. | artifact total: 0 first listing: None

The whole live catalogue is 45 listings, all long_term, in GB/TH locodes; there are no sale listings and no US locodes, so the previous USSFO/sale query could never return a property

Mutation check against the live proxy: pointing the strengthened test back at the old USSFO/sale query fails on assert results.total > 0 (assert 0 > 0), where before the change it passed. With the GBLON/long_term query the full a2a suite is 9 passed

Type

✅ Test

Changes

test_real_world_agent_replies_to_property_query used to register the real movehome.org agent, send a search_properties invocation for San Francisco sale listings under $2M, and assert only that response.result.text.strip() != "". The agent completes the task with the text "No listings matched the search criteria." for that query, which is non-empty, so the test passed without the agent ever returning a property. It was checking that a task completed with some text, not that the tool actually ran and matched

This changes the query to a slice of the catalogue that has data (un_locode="GBLON", service_type="long_term") and asserts on the tool output itself. a2a_client.py gains typed models for the search_results artifact (A2AArtifact, A2ASearchResults, A2AListing, A2AListingLocation) plus a search_results accessor on A2AResult, and the test now asserts the artifact is present, total > 0, the listings are non-empty, each has a raia_id, and each location.un_locode matches the requested location. If search_properties returns nothing the test fails

QA runbook

• tests/e2e/a2a/test_a2a_agent_e2e.py::TestA2AAgentLifecycle::test_real_world_agent_replies_to_property_query - the real movehome.org agent, registered through the proxy, actually runs its search_properties skill and returns property cards
  • Register the agent: fetch https://movehome.org/.well-known/agent.json, set its url to https://movehome.org, then POST it to /v1/agents with the master key as {"agent_name": "...", "agent_card_params": <card>}
  • POST /a2a/{agent_id} with a bearer virtual key and a message/send whose message carries a DataPart {"skill":"search_properties","params":{"un_locode":"GBLON","service_type":"long_term","limit":3}}
  • Expect a completed task whose artifacts[0] is search_results with total > 0 and a non-empty listings array, each listing having a raia_id and location.un_locode == "GBLON"
  • Sanity check: this test makes sense to add and is not hand-wavey (it asserts real listings came back, not just spend > 0 style plumbing), and it is not more flaky than any live real-world-agent test; it depends on the public catalogue having at least one GBLON long_term listing, which is the exact condition the test exists to prove

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/7d46ce10c3ba4fb2847da198d48c00ef

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Greptile Summary

Strengthens the live property-agent end-to-end test.

• Queries a catalogue segment expected to contain long-term London listings.
• Models structured property-search artifacts in the A2A test client.
• Verifies that returned artifacts contain listings with IDs and the requested location.

Confidence Score: 5/5

The PR appears safe to merge, with the test now validating actual property listings rather than merely a non-empty response.

The changes are confined to the end-to-end test harness and strengthen its assertions without introducing an established behavioral, security, or build failure.

Important Files Changed

Filename	Overview
tests/e2e/a2a/a2a_client.py	Adds typed models and an accessor for reading property search results from A2A task artifacts; no actionable defect was established.
tests/e2e/a2a/test_a2a_agent_e2e.py	Replaces a non-empty-text check with assertions that the live agent returned matching property listings.

_{Reviews (1): Last reviewed commit: "test(a2a): assert the property agent ret..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!