[typescript] Save output.data with text content instead of response data

[rossdanlm]

[typescript] Save output.data with text content instead of response data

This comes after Sarmad's schema updates in #589. To keep diffs small and easier to review, this simply converts from model-specific outputs --> pure text. I have a diff in #610 which converts from pure text --> OutputData format.

We only needed to update the hf.py and openai.py, because palm.py already returns output in the form of string | null type.

Ran yarn automated tests, but there aren't any specifically for openai. I also ran the typescript demos to make sure that they still work. Run these commands from aiconfig top-level dir:

npx ts-node typescript/demo/function-call-stream.ts
npx ts-node typescript/demo/demo.ts
npx ts-node typescript/demo/test-hf.ts

For the extensions, we only have typescript for hf.ts (trivial: just changed response to response.generated_text), while llama.ts already outputs it in text format so no changes needed

TODO

I still need to add function call support directly to OutputData format. See

[saqadri]

FYI to run demo.ts, simply run

ts-node demo.ts

in that folder. You may need to install ts-node if you don't already have it, from here: https://www.npmjs.com/package/ts-node#installation

[rholinshead]

Can you make sure to test the function calling example,

npx ts-node demo/function-call-stream.ts

I think it could be broken by the change to move it to metadata here?

Also, just curious why you chose to do this first and then move to the OutputData type in #610 instead of doing it in one go? I agree that using OutputData with 'string' kind is the correct way to go

[saqadri]

Requesting changes on the breaking changes introduced here.

[saqadri]

I agree that using OutputData with 'string' kind is the correct way to go

@rholinshead is that significantly different on the frontend? It would be a much simpler JSON/YAML file for string outputs if the "string" kind was inferred if the typeof output.data was string.

[rossdanlm]

Can you make sure to test the function calling example,

npx ts-node demo/function-call-stream.ts

I think it could be broken by the change to move it to metadata here?

Also, just curious why you chose to do this first and then move to the OutputData type in #610 instead of doing it in one go? I agree that using OutputData with 'string' kind is the correct way to go

cc thanks @rholinshead for flagging, you are totally correct this breaks because of the way we implement it in places like this, but moving forwards this isn't something we should continue to support and we should use the function call directly instead from OutputData or metadata

@saqadri I know this breaks existing code API implementation that may be calling into this, but we can't both choose to support old formats while also wanting to make our output types easy to parse and understand. The getOutputText() methods are still supported, it's just if anyone is engaging with the old data structure their implementation will break

[saqadri]

Can you make sure to test the function calling example,

npx ts-node demo/function-call-stream.ts

I think it could be broken by the change to move it to metadata here?

Also, just curious why you chose to do this first and then move to the OutputData type in #610 instead of doing it in one go? I agree that using OutputData with 'string' kind is the correct way to go

cc thanks @rholinshead for flagging, you are totally correct this breaks because of the way we implement it in places like this, but moving forwards this isn't something we should continue to support and we should use the function call directly instead from OutputData or metadata

@saqadri I know this breaks existing code API implementation that may be calling into this, but we can't both choose to support old formats while also wanting to make our output types easy to parse and understand. The getOutputText() methods are still supported, it's just if anyone is engaging with the old data structure their implementation will break

Can you share how the function call script should be updated to support the new format? One other option can be adding output kind "function" to the schema to it's distinguished from plain string.

We cannot have breaking changes without explicitly mentioning it and having a clear major version bump. The issue is there are several implicit breaking changes introduced in this stack

[rossdanlm]

Can you share how the function call script should be updated to support the new format? One other option can be adding output kind "function" to the schema to it's distinguished from plain string.

Saw your comments, I didn't have time to update the diff summary before you replied but just did. Basically we need to do this:

I updated the change to the function-call-stream.ts file

We cannot have breaking changes without explicitly mentioning it and having a clear major version bump. The issue is there are several implicit breaking changes introduced in this stack

Alright, I'm in favour of doing a clear major version (I haven't posted changelog yet for this reason). Let's also include in this NOT to support arbitrary JSON in OutputData schema.

[rossdanlm]

Updated based on feedback, main changes:

• sending rawResponse as a field in metadata for all non-streaming options
• no longer using delta instead of accumulatedData, created task for it: [ts] HuggingFace extension uses delta instead of accumulatedText in output.data #620
• Made note to support function calling more directly as part of OutputData. I will do this end of diff stack
• Supported all old AIConfig schemas with backwards compatbility inside of the getOutputText() methods

Note that this change will introduce breaking the SDK implementation. Ex: instead of check for data.content you now just check data instead. I will land changelog for today before landing these changes, so this won't be included or mentioned in that

[rholinshead]

I agree that using OutputData with 'string' kind is the correct way to go

@rholinshead is that significantly different on the frontend? It would be a much simpler JSON/YAML file for string outputs if the "string" kind was inferred if the typeof output.data was string.

It isn't significantly different on the front end, it's just checking if the type is a string or an object first instead of just checking the keys of the object. In general, was just thinking it is a much cleaner API to get rid of the redundant union types (i.e. can represent string output in 2 supported ways) that don't have a clear standard or recommendation.

If it's a lot nicer for the JSON/YAML format then maybe it's worth supporting raw string there, we should just consider:

• what is our recommendation / standard for how we think it should be used? All our example configs should be consistent
• do we even want to consider {kind: 'string'} then, or just remove it to have the single way (raw text as the data value for text outputs)

[rholinshead]

Let's also include in this NOT to support arbitrary JSON in OutputData schema.

Based on previous PR discussion with @saqadri I think we should still support arbitrary JSON for now and just provide the new structured types. We can revisit later to make a better informed decision with more data (e.g. once we see more examples of user-created parsers to know how they're using it)

[rholinshead]

nit, can remove

[rossdanlm]

I wanted to keep it because we have other console.logs() earlier and it shows more clearly in the demo how function_call is being used. Ex: https://github.com/lastmile-ai/aiconfig/blob/main/typescript/demo/function-call-stream.ts#L176

[rholinshead]

I think we still need to handle the existing outputMessage-as-ChatCompletionMessageParam case here for backwards compatibility on deserialize. We can reserialize in our updated format, but it's possible the config being loaded has the old format

I think you can do similar check to above

if (typeof output.data === "string") {
   ... handle how you do here
} else if (output.data.hasOwnProperty("role")) {
  // for backwards compatibility
  const outputMessage = output.data as unknown as Chat.ChatCompletionMessageParam;
  if (outputMessage.role === "assistant") {
    messages.push(outputMessage);
  }
}

[rossdanlm]

You are correct, thank you for catching!

[saqadri]

I don't think this is a good model. If the only way for someone to support function calling is by delving into the raw response, then we've missed a critical scenario.

[rossdanlm]

Ok after rebasing onto #628, I will update but do this at the end of the diff stack. I just want to keep this for string-only to make it easier to review, then the rest of diffs will be easier to followup

[saqadri]

Approving to unblock :). We are aligned on the acceptable breaking changes to the SDK while still supporting previous aiconfig schemas.

Also, as you land this stack please document the breaking changes explicitly in our changelog doc for next week so we can communicate it explicitly

[saqadri]

Delete this