Azure OpenAI: TypeSpec updates for 2023-12-01-preview (#26737)

* baseline: add new label with no changes

* reapplication of changes from PR repository

* incremental: chat inputs, including tools

* incremental, in-progress: chat message rework

* incremental: chat completions updated (undoc'd/unverified)

* incremental: provisional completeness for chat

* Incremental: RAI updates

* RAI, data-plane, tsp compilation fixes

* minor updates after starting .NET integration

* small fix to structured text items for vision

* small but important function fix

* trial and error: OYD apiKey->key

* minor edit

* java emitter config merge

* some PR feedback + tool_calls role fix

* small but important: url is optional in protected_material_code

* add missing layer to image_url

* Add speculative finish_details for gpt-4-vision-preview

* Reorging files for extensions

* Using generics

* Removed generics approach

* Fixed various typos and added docs

* Removed super class parameters

* Removed LRO entries in client.tsp, since they are removed from spec anyway

* WIP

* removed LRO related definitions

* Added @client to hide the LRO op

* Moved visibility modifiers to their own source file

* Annotated Key as removed

* Restored LRO as private methods

* PR feedback

* Hid LROs (#26898)

* format, recompile, backport optional filename for whisper

* PR feedback: naming/docs updates

* Added TypeSpec customization support for Java (#26916)

* Update spell checking instructions (#26874)

Having one custom-words.txt for the entire repo has been causing a number of merging issues so we are trying to move away from it and instead have teams use override in the cspell configuration file directly.

* Update readme.python.md (#26894)

* Update readme.python.md

* Update readme.python.md

* Update readme.python.md

* Update readme.python.md

---------

Co-authored-by: Yuchao Yan <[email protected]>

* [Cognitive Services - Azure AI Content Safety] Update package print name. (#26911)

* Update package print name.

* update.

* update name.

* Added TypeSpec customization support for Java

---------

Co-authored-by: Wes Haggard <[email protected]>
Co-authored-by: ChenxiJiang333 <[email protected]>
Co-authored-by: Yuchao Yan <[email protected]>
Co-authored-by: jiezhou_microsoft <[email protected]>

* Renamed customization class (#26919)

* Update specification/cognitiveservices/OpenAI.Inference/models/completions/chat_completions.tsp

Co-authored-by: Deyaaeldeen Almahallawi <[email protected]>

* recompile after accepted/committed feedback fix

* PR feedback: non-optional content_filter_results on prompt model

* very minor change: one file fix via npx prettier

* missing second half of prettier fix: copied file to data-plane

---------

Co-authored-by: Jose Alvarez <[email protected]>
Co-authored-by: Jose Alvarez <[email protected]>
Co-authored-by: Shawn Fang <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>
Co-authored-by: ChenxiJiang333 <[email protected]>
Co-authored-by: Yuchao Yan <[email protected]>
Co-authored-by: jiezhou_microsoft <[email protected]>
Co-authored-by: Deyaaeldeen Almahallawi <[email protected]>

Author: 8 people

specification/cognitiveservices/OpenAI.Inference/client.tsp

@@ -1,46 +1,43 @@
 import "@azure-tools/typespec-client-generator-core";
+
+import "./client/custom_visibility.tsp";
 import "./main.tsp";
 
 using Azure.ClientGenerator.Core;
 
-// Azure-specific long-running operations should be treated as implementation details that are wrapped into
-// appropriately merged public surface.
-@@access(Azure.OpenAI.beginAzureBatchImageGeneration, Access.internal);
-@@access(Azure.OpenAI.getAzureBatchImageGenerationOperationStatus,
-  Access.internal
-);
-
-// Azure-specific Chat Completions with extensions should be handled by clients as a conditional selection within the
-// shared Chat Completions route, with the selection gated by the presence or non-presence of additional child
-// configuration options on the request payload options model.
-@@access(Azure.OpenAI.getChatCompletionsWithAzureExtensions, Access.internal);
-
-// Some models from routes with suppressed visibility are still desired for custom public surface.
-@@access(Azure.OpenAI.ImageGenerationOptions, Access.public);
-@@usage(Azure.OpenAI.ImageGenerationOptions, Usage.input | Usage.output);
-@@access(Azure.OpenAI.ImageLocation, Access.public);
-@@usage(Azure.OpenAI.ImageLocation, Usage.input | Usage.output);
-@@access(Azure.OpenAI.ImageSize, Access.public);
-@@usage(Azure.OpenAI.ImageGenerations, Usage.input | Usage.output);
-
-// Java will need to have those models expose in public surface.
-@@access(Azure.OpenAI.AudioTranscription, Access.public, "java");
-@@access(Azure.OpenAI.AudioTranscriptionOptions, Access.public, "java");
-@@access(Azure.OpenAI.AudioTranslation, Access.public, "java");
-@@access(Azure.OpenAI.AudioTranslationOptions, Access.public, "java");
-@@access(Azure.OpenAI.AzureOpenAIOperationState, Access.public, "java");
-
-@@access(Azure.OpenAI.AzureCognitiveSearchIndexFieldMappingOptions,
-  Access.public
-);
-@@usage(Azure.OpenAI.AzureCognitiveSearchIndexFieldMappingOptions, Usage.input);
-
-@@access(Azure.OpenAI.AzureCognitiveSearchQueryType, Access.public);
-@@usage(Azure.OpenAI.AzureCognitiveSearchQueryType, Usage.input);
-
-@@access(Azure.OpenAI.AzureCognitiveSearchChatExtensionConfiguration,
-  Access.public
-);
-@@usage(Azure.OpenAI.AzureCognitiveSearchChatExtensionConfiguration,
-  Usage.input
-);
+#suppress "@azure-tools/typespec-azure-core/casing-style" "OpenAI is a case-sensitive name"
+@TypeSpec.Versioning.useDependency(
+  Azure.OpenAI.ServiceApiVersions.v2023_12_01_Preview
+)
+namespace Client;
+
+#suppress "@azure-tools/typespec-azure-core/casing-style" "OpenAI is a case-sensitive name"
+@client({
+  name: "OpenAIClient",
+  service: Azure.OpenAI,
+})
+interface OpenAIClient {
+  #suppress "@azure-tools/typespec-azure-core/byos" "representation of existing multipart/form-data operation"
+  #suppress "@azure-tools/typespec-azure-core/use-standard-operations" "non-model response value needed"
+  getAudioTranscriptionAsPlainText is Azure.OpenAI.getAudioTranscriptionAsPlainText;
+
+  #suppress "@azure-tools/typespec-azure-core/byos" "representation of existing multipart/form-data operation"
+  #suppress "@azure-tools/typespec-azure-core/use-standard-operations" "non-model response value needed"
+  getAudioTranscriptionAsResponseObject is Azure.OpenAI.getAudioTranscriptionAsResponseObject;
+
+  #suppress "@azure-tools/typespec-azure-core/byos" "representation of existing multipart/form-data operation"
+  #suppress "@azure-tools/typespec-azure-core/use-standard-operations" "non-model response value needed"
+  getAudioTranslationAsPlainText is Azure.OpenAI.getAudioTranslationAsPlainText;
+
+  #suppress "@azure-tools/typespec-azure-core/byos" "representation of existing multipart/form-data operation"
+  #suppress "@azure-tools/typespec-azure-core/use-standard-operations" "non-model response value needed"
+  getAudioTranslationAsResponseObject is Azure.OpenAI.getAudioTranslationAsResponseObject;
+
+  getCompletions is Azure.OpenAI.getCompletions;
+  getChatCompletions is Azure.OpenAI.getChatCompletions;
+  getChatCompletionsWithAzureExtensions is Azure.OpenAI.getChatCompletionsWithAzureExtensions;
+
+  getImageGenerations is Azure.OpenAI.getImageGenerations;
+
+  getEmbeddings is Azure.OpenAI.getEmbeddings;
+}

specification/cognitiveservices/OpenAI.Inference/client/custom_visibility.tsp

@@ -0,0 +1,47 @@
+import "@azure-tools/typespec-client-generator-core";
+import "../main.tsp";
+
+using Azure.ClientGenerator.Core;
+
+// Azure-specific Chat Completions with extensions should be handled by clients as a conditional selection within the
+// shared Chat Completions route, with the selection gated by the presence or non-presence of additional child
+// configuration options on the request payload options model.
+@@access(Azure.OpenAI.getChatCompletionsWithAzureExtensions, Access.internal);
+
+// Some models from routes with suppressed visibility are still desired for custom public surface.
+@@access(Azure.OpenAI.ImageGenerationOptions, Access.public);
+@@usage(Azure.OpenAI.ImageGenerationOptions, Usage.input | Usage.output);
+@@access(Azure.OpenAI.ImageSize, Access.public);
+@@usage(Azure.OpenAI.ImageGenerations, Usage.input | Usage.output);
+
+// Java will need to have those models expose in public surface.
+@@access(Azure.OpenAI.AudioTranscription, Access.public, "java");
+@@access(Azure.OpenAI.AudioTranscriptionOptions, Access.public, "java");
+@@access(Azure.OpenAI.AudioTranslation, Access.public, "java");
+@@access(Azure.OpenAI.AudioTranslationOptions, Access.public, "java");
+
+@@access(Azure.OpenAI.AzureCognitiveSearchIndexFieldMappingOptions,
+  Access.public
+);
+@@usage(Azure.OpenAI.AzureCognitiveSearchIndexFieldMappingOptions, Usage.input);
+
+@@access(Azure.OpenAI.AzureCognitiveSearchQueryType, Access.public);
+@@usage(Azure.OpenAI.AzureCognitiveSearchQueryType, Usage.input);
+
+@@access(Azure.OpenAI.AzureCognitiveSearchChatExtensionConfiguration,
+  Access.public
+);
+@@usage(Azure.OpenAI.AzureCognitiveSearchChatExtensionConfiguration,
+  Usage.input
+);
+
+// Remap "model" nomenclature to "DeploymentName" in appropriate containers; comments should clarify the dual use
+@@projectedName(Azure.OpenAI.ImageGenerationOptions.`model`,
+  "csharp",
+  "DeploymentName"
+);
+
+@@projectedName(Azure.OpenAI.ChatResponseMessage.context,
+  "csharp",
+  "AzureExtensionsContext"
+);

specification/cognitiveservices/OpenAI.Inference/examples/2023-12-01-preview/audio_transcription_object.json

@@ -0,0 +1,16 @@
+{
+  "operationId": "GetAudioTranscriptionAsResponseObject",
+  "title": "Gets transcribed text and associated metadata from provided spoken audio data.",
+  "parameters": {
+    "endpoint": "{endpoint}",
+    "api-version": "2023-12-01-preview",
+    "deploymentId": "<deployment-id>"
+  },
+  "responses": {
+    "200": {
+      "body": {
+        "text": "A structured object when requesting json or verbose_json"
+      }
+    }
+  }
+}

specification/cognitiveservices/OpenAI.Inference/examples/2023-12-01-preview/audio_transcription_text.json

@@ -0,0 +1,16 @@
+{
+  "operationId": "GetAudioTranscriptionAsPlainText",
+  "title": "Gets transcribed text and associated metadata from provided spoken audio data.",
+  "parameters": {
+    "endpoint": "{endpoint}",
+    "api-version": "2023-12-01-preview",
+    "deploymentId": "<deployment-id>",
+    "body": "---multipart-boundary\nContent-Disposition: form-data; name=\"file\"; filename=\"file.wav\"\nContent-Type: application/octet-stream\n\nRIFF..audio.data.omitted\n---multipart-boundary--"
+  },
+  "responses": {
+    "200": {
+      "type": "string",
+      "example": "plain text when requesting text, srt, or vtt"
+    }
+  }
+}

specification/cognitiveservices/OpenAI.Inference/examples/2023-12-01-preview/audio_translation_object.json

@@ -0,0 +1,17 @@
+{
+  "operationId": "GetAudioTranslationAsResponseObject",
+  "title": "Gets English language transcribed text and associated metadata from provided spoken audio data.",
+  "parameters": {
+    "endpoint": "{endpoint}",
+    "api-version": "2023-12-01-preview",
+    "deploymentId": "<deployment-id>",
+    "body": "---multipart-boundary\nContent-Disposition: form-data; name=\"file\"; filename=\"file.wav\"\nContent-Type: application/octet-stream\n\nRIFF..audio.data.omitted\n---multipart-boundary--"
+  },
+  "responses": {
+    "200": {
+      "body": {
+        "text": "A structured object when requesting json or verbose_json"
+      }
+    }
+  }
+}

specification/cognitiveservices/OpenAI.Inference/examples/2023-12-01-preview/audio_translation_text.json

@@ -0,0 +1,16 @@
+{
+  "operationId": "GetAudioTranslationAsPlainText",
+  "title": "Gets English language transcribed text and associated metadata from provided spoken audio data.",
+  "parameters": {
+    "endpoint": "{endpoint}",
+    "api-version": "2023-12-01-preview",
+    "deploymentId": "<deployment-id>",
+    "body": "---multipart-boundary\nContent-Disposition: form-data; name=\"file\"; filename=\"file.wav\"\nContent-Type: application/octet-stream\n\nRIFF..audio.data.omitted\n---multipart-boundary--"
+  },
+  "responses": {
+    "200": {
+      "type": "string",
+      "example": "plain text when requesting text, srt, or vtt"
+    }
+  }
+}

specification/cognitiveservices/OpenAI.Inference/examples/2023-12-01-preview/chat_completions.json

@@ -0,0 +1,44 @@
+{
+  "operationId": "GetChatCompletions",
+  "title": "Creates a completion for the provided prompt, parameters and chosen model.",
+  "parameters": {
+    "endpoint": "{endpoint}",
+    "api-version": "2023-12-01-preview",
+    "deploymentId": "<deployment-id>",
+    "body": {
+      "messages": [
+        {
+          "role": "system",
+          "content": "you are a helpful assistant that talks like a pirate"
+        },
+        {
+          "role": "user",
+          "content": "can you tell me how to care for a parrot?"
+        }
+      ]
+    }
+  },
+  "responses": {
+    "200": {
+      "body": {
+        "id": "chatcmpl-7R1nGnsXO8n4oi9UPz2f3UHdgAYMn",
+        "created": 1686676106,
+        "choices": [
+          {
+            "index": 0,
+            "finish_reason": "stop",
+            "message": {
+              "role": "assistant",
+              "content": "Ahoy matey! So ye be wantin' to care for a fine squawkin' parrot, eh? Well, shiver me timbers, let ol' Cap'n Assistant share some wisdom with ye! Here be the steps to keepin' yer parrot happy 'n healthy:\n\n1. Secure a sturdy cage: Yer parrot be needin' a comfortable place to lay anchor! Be sure ye get a sturdy cage, at least double the size of the bird's wingspan, with enough space to spread their wings, yarrrr!\n\n2. Perches 'n toys: Aye, parrots need perches of different sizes, shapes, 'n textures to keep their feet healthy. Also, a few toys be helpin' to keep them entertained 'n their minds stimulated, arrrh!\n\n3. Proper grub: Feed yer feathered friend a balanced diet of high-quality pellets, fruits, 'n veggies to keep 'em strong 'n healthy. Give 'em fresh water every day, or ye’ll have a scurvy bird on yer hands!\n\n4. Cleanliness: Swab their cage deck! Clean their cage on a regular basis: fresh water 'n food daily, the floor every couple of days, 'n a thorough scrubbing ev'ry few weeks, so the bird be livin' in a tidy haven, arrhh!\n\n5. Socialize 'n train: Parrots be a sociable lot, arrr! Exercise 'n interact with 'em daily to create a bond 'n maintain their mental 'n physical health. Train 'em with positive reinforcement, treat 'em kindly, yarrr!\n\n6. Proper rest: Yer parrot be needin' ’bout 10-12 hours o' sleep each night. Cover their cage 'n let them slumber in a dim, quiet quarter for a proper night's rest, ye scallywag!\n\n7. Keep a weather eye open for illness: Birds be hidin' their ailments, arrr! Be watchful for signs of sickness, such as lethargy, loss of appetite, puffin' up, or change in droppings, and make haste to a vet if need be.\n\n8. Provide fresh air 'n avoid toxins: Parrots be sensitive to draft and pollutants. Keep yer quarters well ventilated, but no drafts, arrr! Be mindful of toxins like Teflon fumes, candles, or air fresheners.\n\nSo there ye have it, me hearty! With proper care 'n commitment, yer parrot will be squawkin' \"Yo-ho-ho\" for many years to come! Good luck, sailor, and may the wind be at yer back!"
+            }
+          }
+        ],
+        "usage": {
+          "completion_tokens": 557,
+          "prompt_tokens": 33,
+          "total_tokens": 590
+        }
+      }
+    }
+  }
+}

specification/cognitiveservices/OpenAI.Inference/examples/2023-12-01-preview/completions.json

@@ -0,0 +1,38 @@
+{
+  "operationId": "GetCompletions",
+  "title": "Creates a completion for the provided prompt, parameters and chosen model.",
+  "parameters": {
+    "endpoint": "{endpoint}",
+    "api-version": "2023-12-01-preview",
+    "deploymentId": "<deployment-id>",
+    "body": {
+      "prompt": [
+        "tell me a joke about mango"
+      ],
+      "max_tokens": 32,
+      "temperature": 1.0,
+      "n": 1
+    }
+  },
+  "responses": {
+    "200": {
+      "body": {
+        "id": "cmpl-7QmVI15qgYVllxK0FtxVGG6ywfzaq",
+        "created": 1686617332,
+        "choices": [
+          {
+            "text": "es\n\nWhat do you call a mango who's in charge?\n\nThe head mango.",
+            "index": 0,
+            "finish_reason": "stop",
+            "logprobs": null
+          }
+        ],
+        "usage": {
+          "completion_tokens": 20,
+          "prompt_tokens": 6,
+          "total_tokens": 26
+        }
+      }
+    }
+  }
+}