[RFC] Output data type #589

saqadri · 2023-12-22T18:30:34Z

[RFC] Output data type

Add some structure to the outputs that allow the frontend to know how to render them.

Still need to update the Python types, and modelparsers to return data in this type.

schema/aiconfig.schema.json

rholinshead · 2023-12-22T18:52:51Z

typescript/types.ts

+    | {
+        kind: "string" | "file_uri" | "base64";
+        value: string;
+      };


Ah, right, I guess we can rely on the mime_type below to know the represented type

rholinshead · 2023-12-22T18:54:17Z

typescript/types.ts

-  data: JSONValue;
-
+  data:
+    | JSONValue


Long-term, do we want to support arbitrary JSONValue here or just let them dump things into metadata so we can always have a kind/value?

We can decide that. My 2c are it's still good to support arbitrary JSONValue here, but we can revisit that. It will require a breaking change to the schema too, so we can consider this for v2 schema

saqadri · 2023-12-23T02:40:41Z

python/src/aiconfig/schema.py

+class OutputData(BaseModel):
+    """
+    OutputData represents the output content in a standard format.
+    """
+
+    kind: Literal["string", "file_uri", "base64"]
+    value: str
+
+
 class ExecuteResult(BaseModel):
+    """
+    ExecuteResult represents the result of executing a prompt.
+    """
+
    # Type of output
    output_type: Literal["execute_result"]
    # nth choice.
    execution_count: Union[int, None] = None
    # The result of the executing prompt.
-    data: Any
+    data: Union[Any, OutputData]
    # The MIME type of the result. If not specified, the MIME type will be assumed to be plain text.
    mime_type: Optional[str] = None
    # Output metadata
    metadata: Dict[str, Any]


 class Error(BaseModel):
+    """
+    Error represents an error that occurred while executing a prompt.
+    """
+


These are the only changes -- the rest are autoformatting changes

Add some structure to the outputs that allow the frontend to know how to render them. Still need to update the Python types, and modelparsers to return data in this type.

saqadri · 2023-12-23T03:51:54Z

@rossdanlm I'm going to ship this -- can you take up the model parser updates to satisfy these types. Can start with a couple of them (maybe Dalle and GPT) to see what it's like. cc @rholinshead

This comes after Sarmad's schema updates in #589 We only needed to update the `hf.py` and `openai.py`, because `palm.py` aalready returns output in the form of `string | null` type. Ran yarn automated tests, but there aren't any specifically for openai. Not sure how to run the `demo.ts` file which would also be a reasonable test to ensure everything there works too

This comes after Sarmad's schema updates in #589 We only needed to update the `hf.py` and `openai.py`, because `palm.py` aalready returns output in the form of `string | null` type. Ran yarn automated tests, but there aren't any specifically for openai. Not sure how to run the `demo.ts` file which would also be a reasonable test to ensure everything there works too For the extensions, we only have typescript for `hf.ts` (trivial: just changed `response` to `response.generated_text`), while `llama.ts` already outputs it in text format so no changes needed

This comes after Sarmad's schema updates in #589. To keep diffs small and easier to review, this simply converts from model-specific outputs --> pure text. I have a diff in #610 which converts from pure text --> `OutputData` format. We only needed to update the `hf.py` and `openai.py`, because `palm.py` aalready returns output in the form of `string | null` type. Ran yarn automated tests, but there aren't any specifically for openai. Not sure how to run the `demo.ts` file which would also be a reasonable test to ensure everything there works too For the extensions, we only have typescript for `hf.ts` (trivial: just changed `response` to `response.generated_text`), while `llama.ts` already outputs it in text format so no changes needed

Add schema explicitly for JSON Adding the $schema property to JSON with the schemastore schema. Currently our schema has just one version but in the future we should respect the schema that the config already has or introduce proper versioning. Test Plan: ``` from aiconfig import AIConfigRuntime # Load the aiconfig (without $schema). config = AIConfigRuntime.load('travel.aiconfig.json') config.save() # Ensure $schema is specified and I can get IntelliSense in the json file now. ``` --- Stack created with [Sapling](https://sapling-scm.com). Best reviewed with [ReviewStack](https://reviewstack.dev/lastmile-ai/aiconfig/pull/598). * __->__ #598 * #589

This comes after Sarmad's schema updates in #589. To keep diffs small and easier to review, this simply converts from model-specific outputs --> pure text. I have a diff in #610 which converts from pure text --> `OutputData` format. We only needed to update the `hf.py` and `openai.py`, because `palm.py` aalready returns output in the form of `string | null` type. Ran yarn automated tests, but there aren't any specifically for openai. Not sure how to run the `demo.ts` file which would also be a reasonable test to ensure everything there works too For the extensions, we only have typescript for `hf.ts` (trivial: just changed `response` to `response.generated_text`), while `llama.ts` already outputs it in text format so no changes needed

This comes after Sarmad's schema updates in #589. To keep diffs small and easier to review, this simply converts from model-specific outputs --> pure text. I have a diff in #610 which converts from pure text --> `OutputData` format. We only needed to update the `hf.py` and `openai.py`, because `palm.py` already returns output in the form of `string | null` type. Ran yarn automated tests, but there aren't any specifically for openai. I also ran the typescript demos to make sure that they still work. Run these commands from `aiconfig` top-level dir: ``` npx ts-node typescript/demo/function-call-stream.ts npx ts-node typescript/demo/demo.ts npx ts-node typescript/demo/test-hf.ts ``` For the extensions, we only have typescript for `hf.ts` (trivial: just changed `response` to `response.generated_text`), while `llama.ts` already outputs it in text format so no changes needed ## TODO I still need to add function call support directly to `OutputData` format. See

…ata (#603) [typescript] Save output.data with text content instead of response data This comes after Sarmad's schema updates in #589. To keep diffs small and easier to review, this simply converts from model-specific outputs --> pure text. I have a diff in #610 which converts from pure text --> `OutputData` format. We only needed to update the `hf.py` and `openai.py`, because `palm.py` already returns output in the form of `string | null` type. Ran yarn automated tests, but there aren't any specifically for openai. I also ran the typescript demos to make sure that they still work. Run these commands from `aiconfig` top-level dir: ``` npx ts-node typescript/demo/function-call-stream.ts npx ts-node typescript/demo/demo.ts npx ts-node typescript/demo/test-hf.ts ``` For the extensions, we only have typescript for `hf.ts` (trivial: just changed `response` to `response.generated_text`), while `llama.ts` already outputs it in text format so no changes needed ## TODO I still need to add function call support directly to `OutputData` format. See

This comes after Sarmad's schema updates in lastmile-ai#589. To keep diffs small and easier to review, this simply converts from model-specific outputs --> pure text. I have a diff in lastmile-ai#610 which converts from pure text --> `OutputData` format. We only needed to update the `hf.py` and `openai.py`, because `palm.py` already returns output in the form of `string | null` type. Ran yarn automated tests, but there aren't any specifically for openai. I also ran the typescript demos to make sure that they still work. Run these commands from `aiconfig` top-level dir: ``` npx ts-node typescript/demo/function-call-stream.ts npx ts-node typescript/demo/demo.ts npx ts-node typescript/demo/test-hf.ts ``` For the extensions, we only have typescript for `hf.ts` (trivial: just changed `response` to `response.generated_text`), while `llama.ts` already outputs it in text format so no changes needed ## TODO I still need to add function call support directly to `OutputData` format. See

saqadri mentioned this pull request Dec 22, 2023

Add YAML serialization to aiconfig #583

Merged

saqadri marked this pull request as ready for review December 22, 2023 18:39

saqadri requested review from rholinshead, suyoglastmileai, Ankush-lastmile, jonathanlastmileai and rossdanlm as code owners December 22, 2023 18:39

rholinshead reviewed Dec 22, 2023

View reviewed changes

schema/aiconfig.schema.json Show resolved Hide resolved

schema/aiconfig.schema.json Outdated Show resolved Hide resolved

rholinshead reviewed Dec 22, 2023

View reviewed changes

saqadri force-pushed the pr589 branch from 5bf34da to 8b18ff2 Compare December 23, 2023 02:39

saqadri commented Dec 23, 2023

View reviewed changes

rholinshead approved these changes Dec 23, 2023

View reviewed changes

[RFC] Output data type

a741de9

Add some structure to the outputs that allow the frontend to know how to render them. Still need to update the Python types, and modelparsers to return data in this type.

saqadri force-pushed the pr589 branch from 8b18ff2 to a741de9 Compare December 23, 2023 03:45

saqadri merged commit 307f791 into main Dec 23, 2023
4 checks passed

saqadri mentioned this pull request Dec 23, 2023

Add schema explicitly for JSON #598

Merged

rossdanlm mentioned this pull request Dec 24, 2023

[typescript] Save output.data with text content instead of response data #603

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC] Output data type #589

[RFC] Output data type #589

saqadri commented Dec 22, 2023 •

edited

Loading

rholinshead Dec 22, 2023 •

edited

Loading

rholinshead Dec 22, 2023

saqadri Dec 23, 2023

saqadri Dec 23, 2023

saqadri commented Dec 23, 2023

[RFC] Output data type #589

[RFC] Output data type #589

Conversation

saqadri commented Dec 22, 2023 • edited Loading

rholinshead Dec 22, 2023 • edited Loading

Choose a reason for hiding this comment

rholinshead Dec 22, 2023

Choose a reason for hiding this comment

saqadri Dec 23, 2023

Choose a reason for hiding this comment

saqadri Dec 23, 2023

Choose a reason for hiding this comment

saqadri commented Dec 23, 2023

saqadri commented Dec 22, 2023 •

edited

Loading

rholinshead Dec 22, 2023 •

edited

Loading