Fixes Openai realtime implementation, fixing tool usage in particular.

[josancamon19]

Fixes for using openai realtime api. Tool calling is not working in current library version, this imp fixes it, additionally, receiving user role messages was triggering an exception.

[davidmigloz]

Hey @josancamon19 ,

Thanks for the report and the PR draft 🙂

I've pushed a new commit that fixes the issue and adds a test.
I'll release a hotfix, let me know if it fixes the issue on your side as well!

[davidmigloz]

Released in https://pub.dev/packages/openai_realtime_dart/changelog#0012

[caleb654]

I'm on the latest version and tool usage doesn't seem to be working. I tried adding a basic tool in both ways shown in the docs but the model does not know that the tool exists.

[davidmigloz]

Hey @caleb654,

Can you provide a test that reproduces the issue you're experiencing?

I've just rerun this test that tests tool use, and it's still passing:

langchain_dart/packages/openai_realtime_dart/test/client_test.dart

Lines 88 to 197 in 194534c

	test('Tool calling test', timeout: const Timeout(Duration(minutes: 5)),
	() async {
	final realtimeEvents = <RealtimeEvent>[];
	bool toolCalled = false;
	final client = RealtimeClient(
	apiKey: Platform.environment['OPENAI_API_KEY'],
	debug: true,
	);
	await client.addTool(
	const ToolDefinition(
	name: 'get_weather',
	description: 'Retrieves the weather for a location '
	'given its latitude and longitude coordinate pair.',
	parameters: {
	'type': 'object',
	'properties': {
	'lat': {
	'type': 'number',
	'description': 'Latitude of the location',
	},
	'lng': {
	'type': 'number',
	'description': 'Longitude of the location',
	},
	},
	'required': ['lat', 'lng'],
	},
	),
	(Map<String, dynamic> params) async {
	toolCalled = true;
	final result = await HttpClient()
	.getUrl(
	Uri.parse(
	'https://api.open-meteo.com/v1/forecast?'
	'latitude=${params['lat']}&'
	'longitude=${params['lng']}&'
	'current=temperature_2m,wind_speed_10m',
	),
	)
	.then((request) => request.close())
	.then((res) => res.transform(const Utf8Decoder()).join())
	.then(jsonDecode);
	return result;
	},
	);
	client.on(RealtimeEventType.all, realtimeEvents.add);
	final isConnected = await client.connect();
	expect(isConnected, isTrue);
	// Should receive "session.created" and send "session.update"
	await client.waitForSessionCreated();
	expect(realtimeEvents.length, equals(2));
	// Send user message
	const content1 = [
	ContentPart.inputText(
	text: "What's the current weather in San Francisco, CA?",
	),
	];
	await client.sendUserMessageContent(content1);
	expect(realtimeEvents.length, equals(4));
	final itemEvent = realtimeEvents[2];
	expect(itemEvent, isA<RealtimeEventConversationItemCreate>());
	final responseEvent = realtimeEvents[3];
	expect(responseEvent, isA<RealtimeEventResponseCreate>());
	final itemCreated = await client.waitForNextCompletedItem();
	expect(itemCreated?.item, isNotNull);
	// Should receive "function_call" from assistant
	final aItem1 = await client.waitForNextCompletedItem();
	expect(aItem1?.item, isNotNull);
	final aItem1Data = aItem1!.item as ItemFunctionCall;
	expect(aItem1Data.type, equals(ItemType.functionCall));
	expect(aItem1Data.status, equals(ItemStatus.completed));
	expect(aItem1Data.name, equals('get_weather'));
	expect(aItem1Data.arguments, isNotNull);
	final aItem1Args =
	jsonDecode(aItem1Data.arguments) as Map<String, dynamic>;
	expect(aItem1Args['lat'], isNotNull);
	expect(aItem1Args['lng'], isNotNull);
	// Should sent "function_call_output" to assistant
	final fItem1 = await client.waitForNextCompletedItem();
	expect(fItem1?.item, isNotNull);
	final fItem1Data = fItem1!.item as ItemFunctionCallOutput;
	expect(fItem1Data.type, equals(ItemType.functionCallOutput));
	expect(fItem1Data.callId, isNotEmpty);
	expect(fItem1Data.output, isNotEmpty);
	expect(toolCalled, isTrue);
	// Should receive response from assistant
	final aItem2 = await client.waitForNextCompletedItem();
	expect(aItem2?.item, isNotNull);
	final aItem2Data = aItem2!.item as ItemMessage;
	expect(aItem2Data.type, equals(ItemType.message));
	expect(aItem1Data.status, equals(ItemStatus.completed));
	expect(aItem2Data.role, equals(ItemRole.assistant));
	final aItem2Formatted = aItem2.formatted;
	expect(aItem2Formatted?.transcript, isNotEmpty);
	// Should close the RealtimeClient connection
	await client.disconnect();
	expect(client.isConnected(), isFalse);
	});
	});

[caleb654]

Sorry seems to be working now. I think my implementation was just messed up.