Skip to content

ci: relax test_function_call_required #6786

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 31, 2025
Merged

ci: relax test_function_call_required #6786

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 17 additions & 4 deletions test/srt/test_openai_function_calling.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -357,12 +357,25 @@ def test_function_call_required(self):
args_obj = json.loads(arguments)

self.assertEqual(
function_name, "get_weather", "Function name should be 'get_weather'"
function_name,
"get_weather",
f"Function name should be 'get_weather', got: {function_name}",
)
self.assertIn("city", args_obj, "Function arguments should have 'city'")
self.assertIn(
"Paris", args_obj["city"], "Parameter city should contain 'Paris'"
) # might be flaky
"city", args_obj, f"Function arguments should have 'city', got: {args_obj}"
)

# Make the test more robust by checking type and accepting valid responses
city_value = args_obj["city"]
self.assertIsInstance(
city_value,
str,
f"Parameter city should be a string, got: {type(city_value)}",
)
self.assertTrue(
"Paris" in city_value or "France" in city_value,
f"Parameter city should contain either 'Paris' or 'France', got: {city_value}",
Comment on lines +376 to +377
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot May 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current check for city_value is case-sensitive (e.g., it looks for "Paris" and "France"). LLM outputs can sometimes vary in casing (e.g., "paris" or "france"). To make this test even more robust and prevent potential flakiness due to case variations, have you considered making this check case-insensitive?

For example, you could convert city_value to lowercase before checking for the substrings. This would ensure the test passes regardless of the casing returned by the model for these specific keywords.

Suggested change
"Paris" in city_value or "France" in city_value,
f"Parameter city should contain either 'Paris' or 'France', got: {city_value}",
"paris" in city_value.lower() or "france" in city_value.lower(),
f"Parameter city should contain either 'Paris' or 'France' (case-insensitive), got: {city_value}",

)

def test_function_call_specific(self):
"""
Expand Down
Loading