diff --git a/.gitbook/assets/1.png b/.gitbook/assets/1.png new file mode 100644 index 0000000..2dadf07 Binary files /dev/null and b/.gitbook/assets/1.png differ diff --git a/.gitbook/assets/2.png b/.gitbook/assets/2.png new file mode 100644 index 0000000..01ae5e3 Binary files /dev/null and b/.gitbook/assets/2.png differ diff --git a/.gitbook/assets/3.png b/.gitbook/assets/3.png new file mode 100644 index 0000000..a433dfc Binary files /dev/null and b/.gitbook/assets/3.png differ diff --git a/.gitbook/assets/4.png b/.gitbook/assets/4.png new file mode 100644 index 0000000..c1f1b5d Binary files /dev/null and b/.gitbook/assets/4.png differ diff --git a/.gitbook/assets/astra_voice_agent.gif b/.gitbook/assets/astra_voice_agent.gif new file mode 100644 index 0000000..e9e182b Binary files /dev/null and b/.gitbook/assets/astra_voice_agent.gif differ diff --git a/.gitbook/assets/docker_restart_server.gif b/.gitbook/assets/docker_restart_server.gif new file mode 100644 index 0000000..01291c2 Binary files /dev/null and b/.gitbook/assets/docker_restart_server.gif differ diff --git a/.gitbook/assets/docker_setting.gif b/.gitbook/assets/docker_setting.gif new file mode 100644 index 0000000..edc0611 Binary files /dev/null and b/.gitbook/assets/docker_setting.gif differ diff --git a/.gitbook/assets/file.excalidraw.svg b/.gitbook/assets/file.excalidraw.svg new file mode 100644 index 0000000..63d4a7c --- /dev/null +++ b/.gitbook/assets/file.excalidraw.svg @@ -0,0 +1,21 @@ + + + eyJ2ZXJzaW9uIjoiMSIsImVuY29kaW5nIjoiYnN0cmluZyIsImNvbXByZXNzZWQiOnRydWUsImVuY29kZWQiOiJ4nM2WXU/bMFx1MDAxNIbv+yuq7HbyfOKvmjtcdTAwMDZcYlx1MDAxONouYFx1MDAxM6jTNIXEba2aOEpcXEpB/PfZKcRpXHUwMDA1XHUwMDEz00BCkVwi+Xz4Pefk0VHuXHUwMDA2w2HiVpVKdoaJuskzo4s6WyZcdTAwMWaD/VrVjbald6XtubGLOm8jZ85Vzc6nT1lVoal2l9bOUW6v1mnKqCtVusZcdTAwMDf+9Ofh8K5994RqlbusnFx1MDAxYdUmtK6oXHUwMDA1dNv4zZatLIw4XHUwMDA1XHUwMDBlVJAuQjf7Xs6pwrsnmWlU9Fx1MDAwNFPy9WJ6KNj35e5cdTAwMTGMT09cdTAwMGVscT7C46g60cacuZVpq2qsbz76XHUwMDFhV9u5OteFmz1OoGd/Lqu2i+msVE3oXHUwMDFlOqutslxcu1WwYdxZ1yPYXHUwMDE5RsuNP0lcdTAwMDJcYtKRxDxlnSOkMkxcdTAwMTDdsK5L2bPG1qGUXHUwMDBmoMJcdTAwMTOLuczy+dRXVFx1MDAxNl2Mq7OyqbLaf6BcdTAwMTi3fGyScSRcdTAwMTmBnsRM6enMXHUwMDA1XHUwMDFmTtFcYlM+XHUwMDEyPX3VXHUwMDBlXHUwMDFlOJNcdTAwMTQzmcaGg2p1XFy0XGb8iuOusyt1XHUwMDFjUsqFMf2ZlcXDzFx1MDAxZVmJtJBcdTAwMDfLfWwrxFx1MDAxZvQoi1xui6rI1jSASFx0YOBU0lEkyuhyvi1vbD6PXHUwMDAwXHJ6Wq9cdTAwMDauXHUwMDA0mqaMYvlicJ1eXHUwMDFkXexN5OnB4Xjxo0qPv9T1zftcdTAwMDZcdTAwMTdcYsZoi5CQSbBEQlx1MDAxMErE25Hrx4vSbZGOXV9cdTAwMTOS5Fx0cFx1MDAxOXBOMVx1MDAwNfFcdTAwMGXBZdQvgddcdTAwMDJXXHUwMDE5o6vmaWzZ89hygT27/OXYXHUwMDFl2rFOi5P939dyLCp2ezIpj3bfN7aMXHUwMDBiv29cdTAwMDFvXHUwMDAwsuaWIcGIfNOVXHUwMDBiglwiv1x1MDAxZDY0XCK3RFwi8OvrKXRcdTAwMDFTITFcdTAwMDB5XHUwMDBidNP/Q5eDJOSf0fXv9tLE/1WcOX9lV5pvTVx1MDAxN2f6Vm1cXJNcXGu1/Pz3eVx1MDAwZlx1MDAxZXpcYuSpttf7wf1cdTAwMWaINT9DIn0= + + + + + \ No newline at end of file diff --git a/.gitbook/assets/graph_designer.gif b/.gitbook/assets/graph_designer.gif new file mode 100644 index 0000000..955fafa Binary files /dev/null and b/.gitbook/assets/graph_designer.gif differ diff --git a/.gitbook/assets/hello_world_python.gif b/.gitbook/assets/hello_world_python.gif new file mode 100644 index 0000000..0a52f31 Binary files /dev/null and b/.gitbook/assets/hello_world_python.gif differ diff --git a/.gitbook/assets/image.png b/.gitbook/assets/image.png new file mode 100644 index 0000000..28dcfa1 Binary files /dev/null and b/.gitbook/assets/image.png differ diff --git a/.gitbook/assets/star_the_repo_confetti.gif b/.gitbook/assets/star_the_repo_confetti.gif new file mode 100644 index 0000000..d41e7d4 Binary files /dev/null and b/.gitbook/assets/star_the_repo_confetti.gif differ diff --git a/.gitbook/assets/star_the_repo_confetti_higher_quality.gif b/.gitbook/assets/star_the_repo_confetti_higher_quality.gif new file mode 100644 index 0000000..1055515 Binary files /dev/null and b/.gitbook/assets/star_the_repo_confetti_higher_quality.gif differ diff --git a/.gitbook/assets/voice_agent_archietcuture.png b/.gitbook/assets/voice_agent_archietcuture.png new file mode 100644 index 0000000..d7b5543 Binary files /dev/null and b/.gitbook/assets/voice_agent_archietcuture.png differ diff --git a/README.md b/README.md new file mode 100644 index 0000000..e376fbd --- /dev/null +++ b/README.md @@ -0,0 +1,27 @@ +--- +layout: + title: + visible: true + description: + visible: false + tableOfContents: + visible: true + outline: + visible: true + pagination: + visible: true +--- + +# 🌟 Welcome to TEN + +Welcome to TEN! TEN stands for Transformative Extensions Network, an open-source framework for quickly creating multimodal AI agents. It features the all-in-one TEN Manager, written in Rust, supports multimodal protocols, includes the TEN Runtime, offers the Graph Designer, and has an Extension Store to share in community. + +Here, you’ll get an overview of all the incredible features TEN offers to help you build engaging and interactive experiences. + +Before we get started, please click the [**star button on our GitHub repo**](https://github.com/rte-design/astra.ai) to stay updated with one of the AI agents powered by TEN. + +

Star us to stay updated

+ +### Jump right in + +
Getting startedQuickstart to build locally1.pngquickstart.md
Customize agentTwo ways to customize4.pngpublish_your_docs.md
Create hello world extensionBuild extensions right away3.pngcreate_a_hello_world_extension.md
diff --git a/SUMMARY.md b/SUMMARY.md new file mode 100644 index 0000000..b0bee75 --- /dev/null +++ b/SUMMARY.md @@ -0,0 +1,27 @@ +# Table of contents + +* [🌟 Welcome to TEN](README.md) + +## πŸŽ‰ Getting Started + +* [Quickstart](getting_started/quickstart.md) +* [Customize your agent](getting_started/publish_your_docs.md) +* [Create a hello world extension](getting_started/create_a_hello_world_extension.md) + +## TEN framework + +* [Testing](ten_framework/testing.md) + +## TEN Service + +* [🚧 TEN architecture(beta)](ten_service/ten_architecture_beta.md) +* [🚧 TEN schema(beta)](ten_service/ten_schema_beta.md) +* [TEN API(beta)](ten_service/ten_api_beta.md) +* [🚧 TEN Message type and name(beta)](ten_service/ten_message_type_and_name_beta.md) +* [🚧 Astra AI agent architecture(beta)](ten_service/astra_ai_agent_architecture_beta.md) + +## Tutorials + +* [How to debug with logs](tutorials/how_to_debug_with_logs.md) +* [🚧 How to build extension with Go(beta)](tutorials/how_to_build_extension_with_go_beta.md) +* [🚧 How to build extension with C++(beta)](tutorials/how_to_build_extension_with_c++_beta.md) diff --git a/assets/gifs/astra_voice_agent.gif b/assets/gifs/astra_voice_agent.gif new file mode 100644 index 0000000..e9e182b Binary files /dev/null and b/assets/gifs/astra_voice_agent.gif differ diff --git a/assets/gifs/docker_setting.gif b/assets/gifs/docker_setting.gif new file mode 100644 index 0000000..edc0611 Binary files /dev/null and b/assets/gifs/docker_setting.gif differ diff --git a/assets/gifs/graph_designer.gif b/assets/gifs/graph_designer.gif new file mode 100644 index 0000000..955fafa Binary files /dev/null and b/assets/gifs/graph_designer.gif differ diff --git a/assets/gifs/star_the_repo_confetti.gif b/assets/gifs/star_the_repo_confetti.gif new file mode 100644 index 0000000..d41e7d4 Binary files /dev/null and b/assets/gifs/star_the_repo_confetti.gif differ diff --git a/assets/gifs/star_the_repo_confetti_higher_quality.gif b/assets/gifs/star_the_repo_confetti_higher_quality.gif new file mode 100644 index 0000000..1055515 Binary files /dev/null and b/assets/gifs/star_the_repo_confetti_higher_quality.gif differ diff --git a/assets/imgs/astra_banner.jpg b/assets/imgs/astra_banner.jpg new file mode 100644 index 0000000..bcf20d1 Binary files /dev/null and b/assets/imgs/astra_banner.jpg differ diff --git a/assets/imgs/banner_image_without_tagline.png b/assets/imgs/banner_image_without_tagline.png new file mode 100644 index 0000000..e571a29 Binary files /dev/null and b/assets/imgs/banner_image_without_tagline.png differ diff --git a/assets/imgs/docker_setting.png b/assets/imgs/docker_setting.png new file mode 100644 index 0000000..415b412 Binary files /dev/null and b/assets/imgs/docker_setting.png differ diff --git a/assets/imgs/image.png b/assets/imgs/image.png new file mode 100644 index 0000000..28dcfa1 Binary files /dev/null and b/assets/imgs/image.png differ diff --git a/assets/imgs/voice_agent_archietcuture.png b/assets/imgs/voice_agent_archietcuture.png new file mode 100644 index 0000000..d7b5543 Binary files /dev/null and b/assets/imgs/voice_agent_archietcuture.png differ diff --git a/getting_started/create_a_hello_world_extension.md b/getting_started/create_a_hello_world_extension.md new file mode 100644 index 0000000..a1cc483 --- /dev/null +++ b/getting_started/create_a_hello_world_extension.md @@ -0,0 +1,195 @@ +--- +layout: + title: + visible: true + description: + visible: false + tableOfContents: + visible: true + outline: + visible: true + pagination: + visible: true +--- + +# Create a hello world extension + +In this chapter, we are going to create a `Hello World` extension step by step, available in Python, Go, and C++. Feel free to choose whichever language you prefer. So buckle up. + +## Prerequisites + +Before diving into this chapter, you’ll need to be familiar with the [basics covered earlier](quickstart.md). Specifically, ensure you understand how to use `docker compose up` and are aware of the services running in the background. + +## 1. Compose up the servers + +First things first, let’s start by composing the servers. Run the following command: + +{% hint style="info" %} +If the caption says `Terminal`, it means you are running the command locally. If the caption says `Bash`, it means you are running the command in the Docker container. +{% endhint %} + +{% code title=">_ Terminal" %} + +```bash +docker compose up +``` + +{% endcode %} + +Once the command is entered, you should see output similar to this: + +
....
+Attaching to astra_agents_dev, astra_graph_designer, astra_playground
+astra_agents_dev      | >> run graph designer server
+astra_agents_dev      | cd agents && tman dev-server
+astra_agents_dev      | :-)  Starting server at http://0.0.0.0:49483
+astra_graph_designer  |   β–² Next.js 14.2.4
+astra_graph_designer  |   - Local:        http://localhost:3000
+astra_graph_designer  |   - Network:      http://0.0.0.0:3000
+astra_graph_designer  |
+astra_graph_designer  |  βœ“ Starting...
+astra_playground      |   β–² Next.js 14.2.4
+astra_playground      |   - Local:        http://localhost:3000
+astra_playground      |   - Network:      http://0.0.0.0:3000
+astra_playground      |
+astra_playground      |  βœ“ Starting...
+astra_graph_designer  |  βœ“ Ready in 293ms
+astra_playground      |  βœ“ Ready in 293ms
+...
+
+ +Now, we’ve got the following services running: + +β€’ `astra_agents_dev` at `http://0.0.0.0:49483` (the backend server for the Graph Designer) + +β€’ `astra_graph_designer` at `http://localhost:3000` (the UI of the Graph Designer) + +β€’ `astra_playground` at `http://localhost:3001` (the UI of the Astra agent) + +## 2. Enter the docker container + +To work within the isolated environment, run the following command: + +{% code title=">_ Terminal" %} + +```bash +docker exec -it astra_agents_dev bash +``` + +{% endcode %} + +## 3. Create the hello world extension + +By running the following commands, an extension called `hello_world` will be created in Python, Go, or C++. + +{% tabs %} +{% tab title="Python" %} +
cd agents/ten_packages/extension
+
+tman install extension default_extension_python --template-mode --template-data package_name=hello_world --template-data class_name_prefix=HelloWorld
+
+cd /app
+
+{% endtab %} + +{% tab title="Go" %} +
cd agents/ten_packages/extension
+
+tman install extension default_extension_go --template-mode --template-data package_name=hello_world --template-data class_name_prefix=HelloWorld
+
+cd /app
+
+{% endtab %} + +{% tab title="C++" %} +
cd agents/ten_packages/extension
+
+tman install extension default_extension_cpp --template-mode --template-data package_name=hello_world --template-data class_name_prefix=HelloWorld
+
+cd /app
+
+{% endtab %} +{% endtabs %} + +After running the command, the log will display something like this: + +
...
+Resolving packages...
+:-)  Install successfully in xxx seconds
+...
+
+ +## 4. Adding API to the extension + +Navigate into the `hello_world` directory and open manifest.json. Add the API objects with `data_in` and `cmd_out`, which we will use shortly within the Graph Designer: + +
{
+  "type": "extension",
+  "name": "hello_world",
+  "version": "0.4.1",
+  "language": "python",
+  "dependencies": [
+    {
+      "type": "system",
+      "name": "rte_runtime_python",
+      "version": "0.4.1"
+    }
+  ],
+  "api": {
+    "data_in": [
+      {
+        "name": "text_data",
+        "property": {
+          "text": {
+            "type": "string"
+          },
+          "is_final": {
+            "type": "bool"
+          }
+        }
+      }
+    ],
+    "cmd_out": [
+      {
+        "name": "flush"
+      }
+    ]
+  }
+}
+
+ +For detailed information on the API and schema, please refer to [ten_api_beta.md](../ten_service/ten_api_beta.md "mention") and [ten_schema_beta.md](../ten_service/ten_schema_beta.md "mention"). + +## 5. Build the extension + +Let's use `cd /app` command to go back to the root of the project, and run `make build` to build the extension. + +{% code title=">_ Bash" %} + +```bash +cd /app + +make build +``` + +{% endcode %} + +## 6. Restart the server + +You don’t need to restart the server when you first build the agent. However, after making minor updates, if refreshing the page doesn’t apply the changes, you’ll need to restart the server in Docker to ensure the updates take effect. + +

Restart the server for astra_agents_dev

+ +## 7. Verify the extension + +Open `http://localhost:3001` in your browser. You should see `hello_world` in the left menu. Drag it to the canvas, and connect it to the `text_data` input and `flash` output. + +You see the green and red color indicting the possible routes of node connecting. + +

hello_world extension

+ +Congratulations! You’ve successfully created your first `hello_world` extension, and it’s working seamlessly within the Graph Designer canvas. + +## 8. Check the network requests + +Open Chrome DevTools, navigate to the Network panel, and monitor the requests. You should see the status codes returning as 200, indicating that the changes have been successfully processed. diff --git a/getting_started/publish_your_docs.md b/getting_started/publish_your_docs.md new file mode 100644 index 0000000..4afadaf --- /dev/null +++ b/getting_started/publish_your_docs.md @@ -0,0 +1,69 @@ +--- +layout: + title: + visible: true + description: + visible: false + tableOfContents: + visible: true + outline: + visible: true + pagination: + visible: true +--- + +# Customize your agent + +There are two primary methods to customize the Astra agent locally: + +## Using the power of graph designer (recommended) + +
+ +The Graph Designer is a user-friendly, visual tool that allows you to create and modify the behavior and responses of the Astra agent without needing to write code. This approach is highly recommended for its ease of use and efficiency. By leveraging the Graph Designer, you can quickly design complex interactions and workflows through a graphical interface, making it accessible even for those with limited programming experience. + +## Editing the code yourself + +For those who prefer a more hands-on approach or need advanced customization, you can directly edit the source code of the Astra agent. This method requires a good understanding of the codebase and programming languages used in the project. By manually editing the code, you have full control over every aspect of the agent’s functionality, allowing for highly tailored and specific customizations. This approach is ideal for developers who need to implement unique features or integrate the agent with other systems. + +{% tabs %} +{% tab title="property.json" %} +
// ...
+"nodes": [
+    {
+        "type": "extension",
+        "extension_group": "default",
+        "addon": "agora_rtc",
+        "name": "agora_rtc",
+        "property": {
+            "app_id": "app-id",
+            "token": "<agora_token>",
+            "channel": "astra_agents_test",
+            "stream_id": 1234,
+            "remote_stream_id": 123,
+            "subscribe_audio": true,
+            "publish_audio": true,
+            "publish_data": true,
+            "enable_agora_asr": true,
+            "agora_asr_vendor_name": "microsoft",
+            "agora_asr_language": "en-US",
+            "agora_asr_vendor_key": "stt-key",
+            "agora_asr_vendor_region": "stt-region",
+            "agora_asr_session_control_file_path": "session_control.conf"
+        }
+    },
+// ...
+
+ +{% endtab %} + +{% tab title="config.go" %} +
// ...
+// Default graph name
+graphNameDefault = "va.openai.azure"
+// ...
+
+{% endtab %} +{% endtabs %} + +Save both files, then run `make build`. Your agent, complete with the intended extensions, should now be up and running. diff --git a/getting_started/quickstart.md b/getting_started/quickstart.md new file mode 100644 index 0000000..bba0563 --- /dev/null +++ b/getting_started/quickstart.md @@ -0,0 +1,151 @@ +--- +layout: + title: + visible: true + description: + visible: false + tableOfContents: + visible: true + outline: + visible: true + pagination: + visible: true +--- + +# Quickstart + +In this chapter, we’ll build the Astra agent together. For additional help, check out the YouTube video tutorial at the end. + +
+ +## Prerequisites + +{% tabs %} +{% tab title="API Keys" %} + +* Agora App ID and App Certificate([read here on how](https://docs.agora.io/en/video-calling/get-started/manage-agora-account?platform=web)) +* Azure [speech-to-text](https://azure.microsoft.com/en-us/products/ai-services/speech-to-text) and [text-to-speech](https://azure.microsoft.com/en-us/products/ai-services/text-to-speech) API keys +* [OpenAI](https://openai.com/index/openai-api/) API key +{% endtab %} + +{% tab title="Installations" %} + +* [Docker](https://www.docker.com/) / [Docker Compose](https://docs.docker.com/compose/) +* [Node.js(LTS) v18](https://nodejs.org/en) +{% endtab %} + +{% tab title="Minimum system requirements" %} +:tada: CPU >= 2 Core + +:smile: RAM >= 4 GB +{% endtab %} +{% endtabs %} + +**Docker setting on Apple Silicon** + +{% hint style="info" %} +You will need to uncheck "Use Rosetta for x86\_64/amd64 emulation on Apple Silicon" option for Docker if you are on Apple Silicon, otherwise the server is not going to work. +{% endhint %} + +

Make sure the box is unchecked

+ +## Next step + +**1. Prepare config files** + +In the root of the project, use `cd` command to create \`.env\` file from example . It will be used to store information for \`docker compose\` later. + +{% code title=">_ Terminal" %} + +```sh +cp ./.env.example ./.env +``` + +{% endcode %} + +**2. Setup API keys & Environment variables in .env file** + +Open the `.env` file and fill in the keys and regions. This is also where you can choose to use any different extensions: + +{% code title=".env" %} + +```bash +# Agora App ID and App Certificate +AGORA_APP_ID= +# Certificate is only required when enabled within Agora.io account +AGORA_APP_CERTIFICATE= + +# Azure STT key and region +AZURE_STT_KEY= +AZURE_STT_REGION= + +# Azure TTS key and region +AZURE_TTS_KEY= +AZURE_TTS_REGION= + +# OpenAI API key +OPENAI_API_KEY= +``` + +{% endcode %} + +**3. Start agent builder toolkit containers** + +In the same directory, run the `docker` command to compose containers: + +{% code title=">_ Terminal" %} + +```bash +docker compose up +``` + +{% endcode %} + +**4. Build Astra agent** + +Open up a separate terminal window, build the agent and start the server: + +{% code title=">_ Bash" %} + +```bash +docker exec -it astra_agents_dev bash +make build +``` + +{% endcode %} + +**5. Start the server** + +Now the server is running at port: 8080. + +{% code title=">_ Bash" %} + +```bash +make run-server +``` + +{% endcode %} + +## **Finish and verify your agent** + +You can open [https://localhost:3001](https://localhost:3001/) in browser to use your graph designer. Simultaneously, open another tab at [https://localhost:3000](https://localhost:3000/) to see the customized voice agent up and running. + +Now you have the power of the Graph Designer at your fingertips to perform the magic of agent customization yourself. πŸŽ‰ + +**Graph designer** + +TEN Graph Designer (beta), a tool that requires zero coding knowledge and makes the experience of creating agentic applications smoother. + +

Graph Designer

+ +## Video tutorials + +{% tabs %} +{% tab title="English" %} +{% embed url="" %} +{% endtab %} + +{% tab title="δΈ­ζ–‡" %} +{% embed url="" %} +{% endtab %} +{% endtabs %} diff --git a/ten_framework/testing.md b/ten_framework/testing.md new file mode 100644 index 0000000..165108a --- /dev/null +++ b/ten_framework/testing.md @@ -0,0 +1,81 @@ +# Testing + +## For Users of the TEN Framework + +(To be added.) + +## For Developers of the TEN Framework + +The TEN framework includes three types of test suites: + +* Unit Tests +* Smoke Tests +* Integration Tests + +The TEN framework uses `gtest` as the testing framework for unit tests and smoke tests, and `pytest` as the testing framework for integration tests. + +### Unit Tests + +The source directory for unit tests is `test/ten_runtime/unit`. If you need to add new unit test cases, please place them in the `test/ten_runtime/unit directory`. + +For C++ projects, after successful compilation, you can navigate to the output directory (e.g., `out/linux/x64/test/standalone/`, depending on the target OS and CPU) and execute the following command: + +``` shell +./ten_runtime_unit_test +``` + +### Smoke Tests + +The source directory for smoke tests is `test/ten_runtime/smoke`. If you need to add new smoke test cases, please place them in the `test/ten_runtime/smoke` directory. + +Navigate to the output directory (e.g., `out/linux/x64/test/standalone`, depending on the target OS and CPU) and execute the following command: + +``` shell +./ten_runtime_smoke_test +``` + +### Loop Testing + +You can perform multiple rounds of testing using the following commands. + +To run only unit tests: + +``` shell +failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_unit_test; failed=$?; done; +``` + +To run only smoke tests: + +``` shell +failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_smoke_test; failed=$?; done; +``` + +To run both unit tests and smoke tests: + +``` shell +failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_unit_test; failed=$?; if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_smoke_test; failed=$?; done; +``` + +### Integration Tests + +Integration tests are black-box tests for the TEN framework. The source directory for integration tests is `test/ten_runtime/integration`. These tests are used to validate real-world execution scenarios. + +The file structure for integration tests is as follows: + +``` text +test/ten_runtime/integration/ + β”œβ”€β”€ test_1/ + β”‚ β”œβ”€β”€ test_case.py + β”‚ └── ... + β”œβ”€β”€ test_2/ + β”‚ β”œβ”€β”€ test_case.py + β”‚ └── ... + └── ... +``` + +To execute the integration tests, use the following command: + +``` shell +cd path/to/out/ +pytest test/ten_runtime/integration/ +``` diff --git a/ten_service/astra_ai_agent_architecture_beta.md b/ten_service/astra_ai_agent_architecture_beta.md new file mode 100644 index 0000000..ac0db78 --- /dev/null +++ b/ten_service/astra_ai_agent_architecture_beta.md @@ -0,0 +1,20 @@ +--- +hidden: true +layout: + title: + visible: true + description: + visible: false + tableOfContents: + visible: true + outline: + visible: true + pagination: + visible: true +--- + +# 🚧 Astra AI agent architecture(beta) + +## Astra diagram + +

Astra voice agent diagram

diff --git a/ten_service/ten_api_beta.md b/ten_service/ten_api_beta.md new file mode 100644 index 0000000..578ae8e --- /dev/null +++ b/ten_service/ten_api_beta.md @@ -0,0 +1,1015 @@ +--- +hidden: true +layout: + title: + visible: true + description: + visible: false + tableOfContents: + visible: true + outline: + visible: true + pagination: + visible: true +--- + +# TEN API(beta) + +## TEN Platform API Specification + +### C Core + +#### Error + +> > Default errno, for those users only care error msgs. +> +> > Invalid json. +> +> > Invalid argument. +> +> > Invalid graph. +> +> > The TEN world is closed. + +### C++ + +#### Message + +| Name | Link | +| --------------------------------- | -------------------------------------- | +| msg\_t::get\_type | `Link ` | +| cmd\_t::get\_name | `Link ` | +| msg\_t::set\_dest | `Link ` | +| msg\_t::from\_json | `Link ` | +| msg\_t::to\_json | `Link ` | +| msg\_t::is\_property\_exist | `Link ` | +| msg\_t::get\_property\_uint8 | `Link ` | +| msg\_t::get\_property\_uint16 | `Link ` | +| msg\_t::get\_property\_uint32 | `Link ` | +| msg\_t::get\_property\_uint64 | `Link ` | +| msg\_t::get\_property\_int8 | `Link ` | +| msg\_t::get\_property\_int16 | `Link ` | +| msg\_t::get\_property\_int32 | `Link ` | +| msg\_t::get\_property\_int64 | `Link ` | +| msg\_t::get\_property\_float32 | `Link ` | +| msg\_t::get\_property\_float64 | `Link ` | +| msg\_t::get\_property\_string | `Link ` | +| msg\_t::get\_property\_bool | `Link ` | +| msg\_t::get\_property\_ptr | `Link ` | +| msg\_t::get\_property\_buf | `Link ` | +| msg\_t::get\_property\_to\_json | `Link ` | +| msg\_t::set\_property\_uint8 | `Link ` | +| msg\_t::set\_property\_uint16 | `Link ` | +| msg\_t::set\_property\_uint32 | `Link ` | +| msg\_t::set\_property\_uint64 | `Link ` | +| msg\_t::set\_property\_int8 | `Link ` | +| msg\_t::set\_property\_int16 | `Link ` | +| msg\_t::set\_property\_int32 | `Link ` | +| msg\_t::set\_property\_int64 | `Link ` | +| msg\_t::set\_property\_float32 | `Link ` | +| msg\_t::set\_property\_float64 | `Link ` | +| msg\_t::set\_property\_string | `Link ` | +| msg\_t::set\_property\_bool | `Link ` | +| msg\_t::set\_property\_ptr | `Link ` | +| msg\_t::set\_property\_buf | `Link ` | +| msg\_t::set\_property\_from\_json | `Link ` | + +C++ message API + +> Get the message type. + +> Get the message name. + +> Set the destination of the message. + +> Convert the message to JSON string. + +> Convert the message from JSON string. + +> Check if the property exists. path can not be empty. + +> Get the property from the message in the specified type. path can not be empty. + +> Get the property from the message in JSON format. path can not be empty. + +> Set the property in the message from JSON string. path can not be empty. + +#### Command + +| Name | Link | +| ----------------------------- | -------------------------------- | +| cmd\_t::create(const char \*) | `Link ` | +| cmd\_t::create\_from\_json | `Link ` | + +C++ command API + +> Create a new command with the specified command name. + +> Create a new command from a JSON string. + +#### Connect Command + +| Name | Link | +| ------------------------- | ------------------------------ | +| cmd\_connect\_t::create() | `Link ` | + +C++ connect command API + +> Create a new connect command. + +#### Status Command + +| Name | Link | +| --------------------------------------- | ------------------------------------------- | +| cmd\_result\_t::create(STATUS\_CODE) | `Link ` | +| cmd\_result\_t::get\_detail\ | `Link ` | +| cmd\_result\_t::get\_detail\_to\_json | `Link ` | +| cmd\_result\_t::set\_detail\ | `Link ` | +| cmd\_result\_t::set\_detail\_from\_json | `Link ` | +| cmd\_result\_t::get\_status\_code | `Link ` | + +C++ status command API + +> Create a new status command with the specified status code. + +> Get the detail of the status command in JSON format. + +> Set the detail of the status command from JSON string. + +> Get the status code from the status command. + +#### Timeout Command + +| Name | Link | +| --------------------------------- | ------------------------------------ | +| cmd\_timeout\_t::create() | `Link ` | +| cmd\_timeout\_t::get\_timer\_id() | `Link ` | + +C++ timeout command API + +> Create a new timeout command. + +> Get the corresponding timer ID from the timeout command. + +#### Timer Command + +| Name | Link | +| ----------------------- | ---------------------------- | +| cmd\_timer\_t::create() | `Link ` | + +C++ timer command API + +> Create a new timer command. + +#### Close App Command + +| Name | Link | +| ---------------------------- | -------------------------------- | +| cmd\_close\_app\_t::create() | `Link ` | + +C++ close app command API + +> Create a new close app command. + +#### Close Engine Command + +| Name | Link | +| ------------------------------- | ----------------------------------- | +| cmd\_close\_engine\_t::create() | `Link ` | + +C++ close engine command API + +> Create a new close engine command. + +#### Data Message + +| Name | Link | +| -------------------- | --------------------------- | +| data\_t::create() | `Link ` | +| data\_t::get\_buf | `Link ` | +| data\_t::lock\_buf | `Link ` | +| data\_t::unlock\_buf | `Link ` | + +C++ data message API + +> Create a data message. + +> Get the buffer of the data message. The operation is deep-copy. + +> Borrow the ownership of the buffer from the data message. + +> Give the ownership of the buffer back to the data message. + +#### Image frame Message + +| Name | Link | +| -------------------------------- | ------------------------------------- | +| image\_frame\_t::create() | `Link ` | +| image\_frame\_t::get\_width | `Link ` | +| image\_frame\_t::set\_width | `Link ` | +| image\_frame\_t::get\_height | `Link ` | +| image\_frame\_t::set\_height | `Link ` | +| image\_frame\_t::get\_timestamp | `Link ` | +| image\_frame\_t::set\_timestamp | `Link ` | +| image\_frame\_t::get\_pixel\_fmt | `Link ` | +| image\_frame\_t::set\_pixel\_fmt | `Link ` | +| image\_frame\_t::is\_eof | `Link ` | +| image\_frame\_t::set\_is\_eof | `Link ` | +| image\_frame\_t::alloc\_buf | `Link ` | +| image\_frame\_t::lock\_buf | `Link ` | +| image\_frame\_t::unlock\_buf | `Link ` | + +C++ image frame message API + +> Create a image frame message. + +> Get/set the width of the image frame. + +> Get/set the height of the image frame. + +> Get/set the timestamp of the image frame. + +> Get/set the pixel format type of the image frame. + +> Get/set the end of file flag of the image frame. + +> Allocate a buffer for the image frame. + +> Borrow the ownership of the buffer from the image frame message. + +> Give the ownership of the buffer back to the image frame message. + +#### Pcm frame Message + +| Name | Link | +| ----------------------------------------- | --------------------------------------------- | +| pcm\_frame\_t::create() | `Link ` | +| pcm\_frame\_t::get\_timestamp | `Link ` | +| pcm\_frame\_t::set\_timestamp | `Link ` | +| pcm\_frame\_t::get\_sample\_rate | `Link ` | +| pcm\_frame\_t::set\_sample\_rate | `Link ` | +| pcm\_frame\_t::get\_channel\_layout | `Link ` | +| pcm\_frame\_t::set\_channel\_layout | `Link ` | +| pcm\_frame\_t::get\_samples\_per\_channel | `Link ` | +| pcm\_frame\_t::set\_samples\_per\_channel | `Link ` | +| pcm\_frame\_t::get\_bytes\_per\_sample | `Link ` | +| pcm\_frame\_t::set\_bytes\_per\_sample | `Link ` | +| pcm\_frame\_t::get\_number\_of\_channels | `Link ` | +| pcm\_frame\_t::set\_number\_of\_channels | `Link ` | +| pcm\_frame\_t::get\_data\_fmt | `Link ` | +| pcm\_frame\_t::set\_data\_fmt | `Link ` | +| pcm\_frame\_t::get\_line\_size | `Link ` | +| pcm\_frame\_t::set\_line\_size | `Link ` | +| pcm\_frame\_t::is\_eof | `Link ` | +| pcm\_frame\_t::set\_is\_eof | `Link ` | +| pcm\_frame\_t::alloc\_buf | `Link ` | +| pcm\_frame\_t::lock\_buf | `Link ` | +| pcm\_frame\_t::unlock\_buf | `Link ` | + +C++ pcm frame message API + +> Create a pcm frame message. + +> Get/set the timestamp of the pcm frame. + +> Get/set the sample rate of the pcm frame. + +> Get/set the channel layout of the pcm frame. + +> Get/set the samples per channel of the pcm frame. + +> Get/set the bytes per sample of the pcm frame. + +> Get/set the number of channels of the pcm frame. + +> Get/set the data format of the pcm frame. + +> Get/set line size of the pcm frame. + +> Get/set the end of file flag of the pcm frame. + +> Allocate a buffer for the pcm frame. + +> Borrow the ownership of the buffer from the pcm frame message. + +> Give the ownership of the buffer back to the pcm frame message. + +#### Addon + +| Name | Link | +| ----------------------------------------------- | -------------------------------------------------- | +| RTE\_CPP\_REGISTER\_ADDON\_AS\_EXTENSION | `Link ` | +| RTE\_CPP\_REGISTER\_ADDON\_AS\_EXTENSION\_GROUP | `Link ` | + +C++ addon API + +> Register a C++ class as an RTE extension addon. + +> Register a C++ class as an RTE extension group addon. + +#### App + +| Name | Link | +| ---------------------------- | ------------------------- | +| app\_t::run | `Link ` | +| app\_t::close | `Link ` | +| app\_t::wait | `Link ` | +| Callback: app\_t::on\_init | `Link ` | +| Callback: app\_t::on\_deinit | `Link ` | + +C++ app API + +> Run the app. + +> Close the app. + +> Wait for the app to close. + +#### Extension + +| Name | Link | +| ------------------------------ | ------------------------------------ | +| extension\_t::on\_init | `Link ` | +| extension\_t::on\_deinit | `Link ` | +| extension\_t::on\_start | `Link ` | +| extension\_t::on\_stop | `Link ` | +| extension\_t::on\_cmd | `Link ` | +| extension\_t::on\_data | `Link ` | +| extension\_t::on\_pcm\_frame | `Link ` | +| extension\_t::on\_image\_frame | `Link ` | + +C++ extension API + +#### Extension Group + +| Name | Link | +| -------------------------------------------- | ------------------------------------------------- | +| extension\_group\_t::on\_init | `Link ` | +| extension\_group\_t::on\_deinit | `Link ` | +| extension\_group\_t::on\_create\_extensions | `Link ` | +| extension\_group\_t::on\_destroy\_extensions | `Link ` | + +C++ extension group API + +#### Metadata Info + +| Name | Link | +| ---------------------- | ----------------------------- | +| metadata\_info\_t::set | `Link ` | + +C+ + metadata info API + +> Set the metadata info. + +#### TEN Proxy + +| Name | Link | +| ---------------------------------- | --------------------------------------- | +| rte\_proxy\_t::rte\_proxy\_t | `Link ` | +| rte\_proxy\_t::acquire\_lock\_mode | `Link ` | +| rte\_proxy\_t::release\_lock\_mode | `Link ` | +| rte\_proxy\_t::notify | `Link ` | + +C++ rte proxy API + +> Create an RTE proxy instance from a RTE instance. + +> Acquire the lock mode. + +> Release the lock mode. + +> Enable the `notify_func` to be called in the RTE extension thread. + +TEN + +| Name | Link | +| ---------------------------------------- | --------------------------------------------- | +| rte\_t::send\_cmd | `Link ` | +| rte\_t::send\_json | `Link ` | +| rte\_t::send\_data | `Link ` | +| rte\_t::send\_image\_frame | `Link ` | +| rte\_t::send\_pcm\_frame | `Link ` | +| rte\_t::return\_result\_directly | `Link ` | +| rte\_t::return\_result | `Link ` | +| rte\_t::is\_property\_exist | `Link ` | +| rte\_t::get\_property\_uint8 | `Link ` | +| rte\_t::get\_property\_uint16 | `Link ` | +| rte\_t::get\_property\_uint32 | `Link ` | +| rte\_t::get\_property\_uint64 | `Link ` | +| rte\_t::get\_property\_int8 | `Link ` | +| rte\_t::get\_property\_int16 | `Link ` | +| rte\_t::get\_property\_int32 | `Link ` | +| rte\_t::get\_property\_int64 | `Link ` | +| rte\_t::get\_property\_float32 | `Link ` | +| rte\_t::get\_property\_float64 | `Link ` | +| rte\_t::get\_property\_string | `Link ` | +| rte\_t::get\_property\_bool | `Link ` | +| rte\_t::get\_property\_ptr | `Link ` | +| rte\_t::get\_property\_buf | `Link ` | +| rte\_t::get\_property\_to\_json | `Link ` | +| rte\_t::set\_property\_from\_json | `Link ` | +| rte\_t::is\_cmd\_connected | `Link ` | +| rte\_t::addon\_create\_extension\_async | `Link ` | +| rte\_t::addon\_destroy\_extension\_async | `Link ` | +| rte\_t::on\_init\_done | `Link ` | +| rte\_t::on\_deinit\_done | `Link ` | +| rte\_t::on\_start\_done | `Link ` | +| rte\_t::on\_stop\_done | `Link ` | +| rte\_t::on\_create\_extensions\_done | `Link ` | +| rte\_t::on\_destroy\_extensions\_done | `Link ` | +| rte\_t::get\_attached\_target | `Link ` | + +C++ TEN API + +> Send the cmd with a response handler. +> +> When the sending action is successful, the unique\_ptr will be released to represent that the ownership of the cmd has been transferred to the TEN runtime. Conversely, if the sending action fails, the unique\_ptr will not perform any action, indicating that the ownership of the cmd remains with the user. +> +> The type of response\_handler\_func\_t is void(rte\_t &, std::unique\_ptr\) + +> Send the command created from the json string without a response handler. + +> Send the command created from the json string with a response handler. +> +> The type of response\_handler\_func\_t is void(rte\_t &, std::unique\_ptr\) + +> Send the data message to the TEN. +> +> When the sending action is successful, the unique\_ptr will be released to represent that the ownership of the data has been transferred to the TEN runtime. Conversely, if the sending action fails, the unique\_ptr will not perform any action, indicating that the ownership of the data remains with the user. + +> Send the image frame to the TEN. +> +> When the sending action is successful, the unique\_ptr will be released to represent that the ownership of the frame has been transferred to the TEN runtime. Conversely, if the sending action fails, the unique\_ptr will not perform any action, indicating that the ownership of the frame remains with the user. + +> Send the PCM frame to the TEN. +> +> When the sending action is successful, the unique\_ptr will be released to represent that the ownership of the frame has been transferred to the TEN runtime. Conversely, if the sending action fails, the unique\_ptr will not perform any action, indicating that the ownership of the frame remains with the user. + +> Return the status command directly. +> +> When the returning action is successful, the unique\_ptr will be released to represent that the ownership of the cmd has been transferred to the RTE runtime. Conversely, if the sending action fails, the unique\_ptr will not perform any action, indicating that the ownership of the cmd remains with the user. + +> Return the status command corresponding to the target command. +> +> When the returning action is successful, the unique\_ptr will be released to represent that the ownership of the cmd has been transferred to the TEN runtime. Conversely, if the sending action fails, the unique\_ptr will not perform any action, indicating that the ownership of the cmd remains with the user. + +> Check if the property exists. path can not be empty. + +> Get the property from the TEN in JSON format. path can not be empty. + +> Set the property in the TEN from JSON string. path can not be empty. + +> Check if the command is connected in the graph. + +> Create an TEN extension instance with the specified `instance_name` from the specified addon specified with the `addon_name` asynchronously. + +> Destroy an TEN extension instance. + +> Notify the TEN that the `on_init` callback is done. + +> Notify the TEN that the `on_deinit` callback is done. + +> Notify the TEN that the `on_start` callback is done. + +> Notify the TEN that the `on_stop` callback is done. + +> Notify the TEN that the `on_create_extensions` callback is done. + +> Notify the TEN that the `on_destroy_extensions` callback is done. + +> Get the attached target. + +### Golang + +#### Message + +| Name | Link | +| ----------------------------- | ---------------------------------------- | +| msg::GetType | `Link ` | +| msg::GetName | `Link ` | +| msg::ToJSON | `Link ` | +| msg::GetPropertyInt8 | `Link ` | +| msg::GetPropertyInt16 | `Link ` | +| msg::GetPropertyInt32 | `Link ` | +| msg::GetPropertyInt64 | `Link ` | +| msg::GetPropertyUint8 | `Link ` | +| msg::GetPropertyUint16 | `Link ` | +| msg::GetPropertyUint32 | `Link ` | +| msg::GetPropertyUint64 | `Link ` | +| msg::GetPropertyBool | `Link ` | +| msg::GetPropertyPtr | `Link ` | +| msg::GetPropertyString | `Link ` | +| msg::GetPropertyBytes | `Link ` | +| msg::GetPropertyToJSONBytes | `Link ` | +| msg::SetPropertyString | `Link ` | +| msg::SetPropertyBytes | `Link ` | +| msg::SetProperty | `Link ` | +| msg::SetPropertyFromJSONBytes | `Link ` | + +Golang message API + +**GetType() MsgType** + +Get the message type. + +**GetName() (string, error)** + +Get the name of the message. + +**ToJSON() string** + +Get the JSON string of the message. + +**GetPropertyInt8(path string) (int8, error)** + +Get the property from the message in int8 type. + +**GetPropertyInt16(path string) (int16, error)** + +Get the property from the message in int16 type. + +**GetPropertyInt32(path string) (int32, error)** + +Get the property from the message in int32 type. + +**GetPropertyInt64(path string) (int64, error)** + +Get the property from the message in int64 type. + +**GetPropertyUint8(path string) (uint8, error)** + +Get the property from the message in uint8 type. + +**GetPropertyUint16(path string) (uint16, error)** + +Get the property from the message in uint16 type. + +**GetPropertyUint32(path string) (uint32, error)** + +Get the property from the message in uint32 type. + +**GetPropertyUint64(path string) (uint64, error)** + +Get the property from the message in uint64 type. + +**GetPropertyBool(path string) (bool, error)** + +Get the property from the message in bool type. + +**GetPropertyPtr(path string) (any, error)** + +Get the property from the message in ptr type. + +**GetPropertyString(path string) (string, error)** + +Get the property from the message in string type. + +**GetPropertyBytes(path string) (\[]byte, error)** + +Get the property from the message in bytes type. + +**GetPropertyToJSONBytes(path string) (\[]byte, error)** + +Get the property from the message in JSON bytes type. + +**SetPropertyString(path string, value string) error** + +Set the property in string type. + +**SetPropertyBytes(path string, value \[]byte) error** + +Set the property in bytes type. + +**SetProperty(path string, value any) error** + +Set the property. + +**SetPropertyFromJSONBytes(path string, value \[]byte) error** + +SEt the property from the JSON bytes. + +#### Command + +| Name | Link | +| ------------------- | ------------------------------------------------ | +| NewCmd | `Link ` | +| NewCmdFromJSONBytes | `Link ` | + +Golang custom command API + +**NewCmd(cmdName string) (Cmd, error)** + +Create a new custom command width the specified command name. + +**NewCmdFromJSONBytes(data \[]byte) (Cmd, error)** + +Create a new custom command from the specified JSON bytes. + +#### Status Command + +| Name | Link | +| ------------------------ | ----------------------------------- | +| NewCmdResult | `Link ` | +| CmdResult::GetStatusCode | `Link ` | + +Golang status command API + +**NewCmdResult(statusCode StatusCode, detail any) (CmdResult, error)** + +Create a new status command. + +**GetStatusCode() (StatusCode, error)** + +Get the status code from the status command. + +#### Data Message + +| Name | Link | +| ------------------------------ | ---------------------------- | +| [Data::NewData](Data::NewData) | `Link ` | +| [Data::GetBuf](Data::GetBuf) | `Link ` | + +Golang data message API + +**NewData(bytes \[]byte) (Data, error)** + +Create a new data message. + +**GetBuf() (\[]byte, error)** + +Get the data buffer from the data message. Note that this function performs a deep copy of the data buffer. + +#### Image Frame Message + +#### Pcm Frame Message + +#### Error + +> Default errno, for those users only care error msgs. + +> Invalid json. + +> Invalid argument. + +> Invalid type. + +| Name | Link | +| ---------------- | ------------------------ | +| TENError::Error | `Link ` | +| TENError::ErrNo | `Link ` | +| TENError::ErrMsg | `Link ` | + +Golang error API + +**Error() string** + +**ErrNo() uint32** + +Get the error number. + +**ErrMsg() string** + +Get the error message. + +#### Addon + +| Name | Link | +| ----------------------------- | ----------------------------------------------- | +| Addon::OnInit | `Link ` | +| Addon::OnDeinit | `Link ` | +| Addon::OnCreateInstance | `Link ` | +| RegisterAddonAsExtension | `Link ` | +| RegisterAddonAsExtensionGroup | `Link ` | +| UnloadAllAddons | `Link ` | +| NewDefaultExtensionAddon | `Link ` | +| NewDefaultExtensionGroupAddon | `Link ` | + +Golang addon API + +**OnInit(rte Rte, manifest MetadataInfo, property MetadataInfo)** + +Initialize the addon. + +**OnDeinit(rte Rte)** + +De-initialize the addon. + +**OnCreateInstance(rte Rte, name string) any** + +Create an instance of the addon. + +**RegisterAddonAsExtension(name string, addon \*Addon) error** + +Register the addon as an extension addon to the TEN runtime environment. + +**RegisterAddonAsExtensionGroup(name string, addon \*Addon) error** + +Register the addon as an extension group addon to the TEN runtime environment. + +**UnloadAllAddons() error** + +Un-register all the addons from the TEN runtime environment. + +**NewDefaultExtensionAddon(constructor func(name string) Extension) \*Addon** + +Create a new default extension addon. + +**NewDefaultExtensionGroupAddon(constructor func(name string) ExtensionGroup) \*Addon** + +Create a new default extension group addon. + +#### App + +| Name | Link | +| ------------- | ------------------------ | +| NewApp | `Link ` | +| App::OnInit | `Link ` | +| App::OnDeinit | `Link ` | +| App::Run | `Link ` | +| App::Close | `Link ` | +| App::Wait | `Link ` | + +Golang app API + +**NewApp(iApp IApp) (App, error)** + +Create a new app. + +**OnInit(rte Rte, manifest MetadataInfo, property MetadataInfo)** + +Initialize the app. + +**OnDeinit(rte Rte)** + +De-initialize the app. + +**Run(runInBackground bool)** + +Run the app. + +**Close()** + +Close the app. + +**Wait()** + +Wait the app to be closed. + +#### Extension + +| Name | Link | +| ------------------------ | ---------------------------------- | +| Extension::OnInit | `Link ` | +| Extension::OnStart | `Link ` | +| Extension::OnStop | `Link ` | +| Extension::OnDeinit | `Link ` | +| Extension::OnCmd | `Link ` | +| Extension::OnData | `Link ` | +| Extension::OnImageFrame | `Link ` | +| Extension::OnPcmFrame | `Link ` | +| Extension::WrapExtension | `Link ` | + +Golang extension API + +**OnInit(rte Rte, manifest MetadataInfo, property MetadataInfo)** + +**OnStart(rte Rte)** + +**OnStop(rte Rte)** + +**OnDeinit(rte Rte)** + +**OnCmd(rte Rte, cmd Cmd)** + +**OnData(rte Rte, data Data)** + +**OnImageFrame(rte Rte, imageFrame ImageFrame)** + +**OnPcmFrame(rte Rte, pcmFrame PcmFrame)** + +**WrapExtension(ext Extension, name string) \*extension** + +Create a new extension instance with the specified name. + +#### Extension Group + +| Name | Link | +| ----------------------------------- | ----------------------------------------------- | +| ExtensionGroup::OnInit | `Link ` | +| ExtensionGroup::OnDeinit | `Link ` | +| ExtensionGroup::OnCreateExtensions | `Link ` | +| ExtensionGroup::OnDestroyExtensions | `Link ` | +| WrapExtensionGroup | `Link ` | + +Golang extension group API + +**OnInit(rte Rte, manifest MetadataInfo, property MetadataInfo)** + +**OnDeinit(rte Rte)** + +**OnCreateExtensions(rte Rte)** + +**OnDestroyExtensions(rte Rte, extensions \[]Extension)** + +**WrapExtensionGroup(extGroup ExtensionGroup, name string) \*extensionGroup** + +Create a new extension group instance with the specified name. + +#### Metadata Info + +| Name | Link | +| ----------------- | ----------------------------- | +| MetadataInfo::Set | `Link ` | + +Golang metadata info API + +**Set(metadataType MetadataType, value string)** + +Set the metadata info. + +TEN + +| Name | Link | +| ------------------------------- | ------------------------------------------ | +| rte::GetPropertyInt8 | `Link ` | +| rte::GetPropertyInt16 | `Link ` | +| rte::GetPropertyInt32 | `Link ` | +| rte::GetPropertyInt64 | `Link ` | +| rte::GetPropertyUint8 | `Link ` | +| rte::GetPropertyUint16 | `Link ` | +| rte::GetPropertyUint32 | `Link ` | +| rte::GetPropertyUint64 | `Link ` | +| rte::GetPropertyBool | `Link ` | +| rte::GetPropertyPtr | `Link ` | +| rte::GetPropertyString | `Link ` | +| rte::GetPropertyBytes | `Link ` | +| rte::GetPropertyToJSONBytes | `Link ` | +| rte::SetPropertyString | `Link ` | +| rte::SetPropertyBytes | `Link ` | +| rte::SetProperty | `Link ` | +| rte::SetPropertyAsync | `Link ` | +| rte::SetPropertyFromJSONBytes | `Link ` | +| rte::ReturnResult | `Link ` | +| rte::ReturnResultDirectly | `Link ` | +| rte::SendJSON | `Link ` | +| rte::SendJSONBytes | `Link ` | +| rte::SendCmd | `Link ` | +| rte::SendData | `Link ` | +| rte::SendImageFrame | `Link ` | +| rte::SendPcmFrame | `Link ` | +| rte::OnStartDone | `Link ` | +| rte::OnStopDone | `Link ` | +| rte::OnInitDone | `Link ` | +| rte::OnDeinitDone | `Link ` | +| rte::OnCreateExtensionsDone | `Link ` | +| rte::OnDestroyExtensionsDone | `Link ` | +| rte::OnCreateInstanceDone | `Link ` | +| rte::IsCmdConnected | `Link ` | +| rte::AddonCreateExtensionAsync | `Link ` | +| rte::AddonDestroyExtensionAsync | `Link ` | + +Golang rte API + +**GetPropertyInt8(path string) (int8, error)** + +Get the property from the RTE in int8 type. + +**GetPropertyInt16(path string) (int16, error)** + +Get the property from the RTE in int16 type. + +**GetPropertyInt32(path string) (int32, error)** + +Get the property from the RTE in int32 type. + +**GetPropertyInt64(path string) (int64, error)** + +Get the property from the RTE in int64 type. + +**GetPropertyUint8(path string) (uint8, error)** + +Get the property from the RTE in uint8 type. + +**GetPropertyUint16(path string) (uint16, error)** + +Get the property from the RTE in uint16 type. + +**GetPropertyUint32(path string) (uint32, error)** + +Get the property from the RTE in uint32 type. + +**GetPropertyUint64(path string) (uint64, error)** + +Get the property from the RTE in uint64 type. + +**GetPropertyBool(path string) (bool, error)** + +Get the property from the RTE in bool type. + +**GetPropertyPtr(path string) (any, error)** + +Get the property from the RTE in ptr type. + +**GetPropertyString(path string) (string, error)** + +Get the property from the RTE in string type. + +**GetPropertyBytes(path string) (\[]byte, error)** + +Get the property from the RTE in bytes type. + +**GetPropertyToJSONBytes(path string) (\[]byte, error)** + +Get the property from the RTE in JSON bytes type. + +**SetProperty(path string, value any) error** + +Set the property to the RTE. + +**SetPropertyString(path string, value string) error** + +Set the property to the RTE in string type. + +**SetPropertyBytes(path string, value \[]byte) error** + +Set the property to the RTE in bytes type. + +**SetPropertyFromJSONBytes(path string, value \[]byte) error** + +Set the property to the RTE from the JSON bytes. + +**SetPropertyAsync(path string, v any, callback func(Rte, error)) error** + +Set the property to the RTE asynchronously. + +**ReturnResult(statusCmd CmdResult, cmd Cmd) error** + +Return the result. + +**ReturnResultDirectly(statusCmd CmdResult) error** + +Return the result directly. + +**SendJSON(json string, handler ResponseHandler) error** + +Send the JSON. + +**SendJSONBytes(json \[]byte, handler ResponseHandler) error** + +Send the JSON bytes. + +**SendCmd(cmd Cmd, handler ResponseHandler) error** + +Send the command. + +**SendData(data Data) error** + +Send the data. + +**SendImageFrame(imageFrame ImageFrame) error** + +Send the image frame. + +**SendPcmFrame(pcmFrame PcmFrame) error** + +Send the PCM frame. + +**OnInitDone(manifest MetadataInfo, property MetadataInfo) error** + +OnInit done. + +**OnStartDone() error** + +OnStart done. + +**OnStopDone() error** + +OnStop done. + +**OnDeinitDone() error** + +OnDeinit done. + +**OnCreateExtensionsDone(extensions ...Extension) error** + +OnCreateExtensions done. + +**OnDestroyExtensionsDone() error** + +OnDestroyExtensions done. + +**OnCreateInstanceDone(instance any) error** + +OnCreateInstance done. + +**IsCmdConnected(cmdName string) (bool, error)** + +Is the command connected. + +**AddonCreateExtensionAsync(addonName string, instanceName string, callback func(rte Rte, p Extension)) error** + +Create an extension from the addon specified by the addon name and instance name. + +**AddonDestroyExtensionAsync(ext Extension, callback func(rte Rte)) error** + +Destroy a specified extension. diff --git a/ten_service/ten_architecture_beta.md b/ten_service/ten_architecture_beta.md new file mode 100644 index 0000000..580e8d4 --- /dev/null +++ b/ten_service/ten_architecture_beta.md @@ -0,0 +1,60 @@ +--- +layout: + title: + visible: true + description: + visible: false + tableOfContents: + visible: true + outline: + visible: true + pagination: + visible: true +--- + +# 🚧 TEN architecture(beta) + +## TEN architecture + +Now let's discuss what's under the hood. The TEN architecture is composed of various TEN extensions, developed in different programming languages. These extensions are interconnected using Graph, which describes their relationships and illustrates the flow of data. Furthermore, sharing and downloading extensions are simplified through the TEN Extension Store and the TEN Package Manager. + +## TEN extension + +An extension is the fundamental unit of composition within the TEN framework. Developers can create extensions in various programming languages and combine them to build diverse scenarios and applications. TEN emphasizes cross-language collaboration, allowing extensions written in different languages to work together seamlessly within the same application or service. + +For example, if an application requires real-time communication (RTC) features and advanced AI capabilities, a developer might choose to write RTC-related extensions in C++ for its performance advantages in processing audio and video data. Meanwhile, they could develop AI extensions in Python to leverage its extensive libraries and frameworks for data analysis and machine learning tasks. + + + +## TEN graph + +A Graph in TEN describes the data flow between extensions, orchestrating their interactions. For example, the text output from a speech-to-text (STT) extension might be directed to a large language model (LLM) extension. Essentially, a Graph defines which extensions are involved and the direction of data flow between them. Developers can customize this flow, directing outputs from one extension, such as an STT, into another, like an LLM. + +In TEN, there are four main types of data flow between extensions, they are **Command**, **Data**, **Image Frame** and **PCM Frame**. + +By specifying the direction of these data types in the Graph, developers can enable mutual invocation and unidirectional data flow between plugins. This is especially useful for PCM and image data types, simplifying audio and video processing. + + + +## TEN agent app + +A TEN Agent App is a runnable server-side application that combines multiple Extensions following Graph rules to accomplish more sophisticated operations. A TEN Agent App is a robust, server-side application that executes complex operations by integrating multiple Extensions within a flexible framework defined by Graph rules. These Graph rules orchestrate the interplay between various Extensions, enabling the app to perform sophisticated tasks that go beyond the capabilities of individual components. + +By leveraging this architecture, a TEN Agent App can seamlessly manage and coordinate different functionalities, ensuring that each Extension interacts harmoniously with others. This design allows developers to create powerful and scalable applications capable of handling intricate workflows and data processing requirements. + + + +## TEN extension store + +The TEN Store is a centralized platform designed to foster collaboration and innovation among developers by providing a space where they can share their extensions. This allows developers to contribute to the community, showcase their work, and receive feedback from peers, enhancing the overall quality and functionality of the TEN ecosystem. + +In addition to sharing their own extensions, developers can also access a wide array of extensions created by others. This extensive library of extensions makes it easier to find tools and functionalities that can be integrated into their own projects, accelerating development and promoting best practices within the community. The TEN Store thus serves as a valuable resource for both novice and experienced developers looking to expand their capabilities and leverage the collective expertise of the community. + + + +## TEN package manager + +The TEN Package Manager streamlines the entire process of handling TEN extensions, making it easy to upload, share, download, and install them. It significantly simplifies the workflow by allowing extensions to specify their dependencies on other extensions and the environment. This ensures that all necessary components are automatically managed and installed, reducing the potential for errors and conflicts. + +By automatically managing these dependencies, the TEN Package Manager makes the installation and release of extensions extremely convenient and intuitive. This tool not only saves time but also enhances the user experience by ensuring that every extension works seamlessly within the larger ecosystem. This level of automation and ease of use encourages the development and distribution of more robust and complex extensions, further enriching the TEN framework. + diff --git a/ten_service/ten_message_type_and_name_beta.md b/ten_service/ten_message_type_and_name_beta.md new file mode 100644 index 0000000..7cd2534 --- /dev/null +++ b/ten_service/ten_message_type_and_name_beta.md @@ -0,0 +1,252 @@ +--- +layout: + title: + visible: true + description: + visible: false + tableOfContents: + visible: true + outline: + visible: true + pagination: + visible: true +--- + +# 🚧 TEN Message type and name(beta) + +## Message Type and Name + +Below is an overview diagram of an TEN platform message. + +``` +β”Œβ”€β”€ has response +β”‚ └── command +β”‚ β”œβ”€β”€ TEN platform built-in command +β”‚ β”‚ => message names start with `ten::` +β”‚ └── Non-TEN platform built-in command +β”‚ => message names do not start with `ten::` +└── no response + β”œβ”€β”€ data + β”‚ β”œβ”€β”€ TEN platform built-in data + β”‚ β”‚ => message names start with `ten::` + β”‚ └── Non-TEN platform built-in data + β”‚ => message names do not start with `ten::` + β”œβ”€β”€ image_frame + β”‚ β”œβ”€β”€ TEN platform built-in image_frame + β”‚ β”‚ => message names start with `ten::` + β”‚ └── Non-TEN platform built-in image_frame + β”‚ => message names do not start with `ten::` + └── pcm_frame + β”œβ”€β”€ TEN platform built-in pcm_frame + β”‚ => message names start with `ten::` + └── Non-TEN platform built-in pcm_frame + => message names do not start with `ten::` +``` + +### Differentiating Message Type and Message Name + +When different messages have different functionalities, message types are used to differentiate them. Functionalities refer to what the TEN Platform provides for a message, and one of the functionalities is an API. For example, the TEN Platform provides an API for getting/setting image frame data. Other functionalities include whether the message has a response, etc. For instance: + +* If a message has a response (referred to as a status command in the TEN Platform), there will be a message type representing this type of message. +* When a message has an image buffer (YUV422, RGB, etc.) that can be accessed, there will be a message type representing this message with the field. This message type provides the API for getting/setting image frame data. +* When a message has an audio buffer that can be accessed, a message type called pcm frame is created. This message type provides the API for getting/setting pcm frame data. + +Message names are used to differentiate the different purposes of messages within the same message type. + +### Message Type + +The TEN Platform messages have four types: + +1. command +2. data +3. image frame +4. pcm frame + +The difference between commands and non-commands is that commands have a response (referred to as a status command in the TEN Platform), while non-commands do not have a response. + +The corresponding extension message callbacks are: + +1. OnCmd +2. OnData +3. OnImageFrame +4. OnPcmFrame + +Although there are currently four message types, it is not certain if there will only be these four types in the future. There may be new types added. If we consider this, merging the four extension message callbacks into one OnMsg can avoid the issue of adding a new extension message callback for each new message type. However, this may inconvenience users, as it means they would have to handle different message types within the OnMsg function (pseudo code). + +> ```c++ +> OnMsg(msg) { +> switch (msg->type) { +> case command || connect || timer || ...: +> OnCmd(msg); +> break; +> case data: +> OnData(msg); +> break; +> case image_frame: +> OnImageFrame(msg); +> break; +> case pcm_frame: +> OnPcmFrame(msg); +> break; +> } +> } +> ``` + +Additionally, in the TEN graph, there is a distinction between cmd\_in, cmd\_out, data\_in, data\_out, etc. So, following a unified approach, the interface of the extension should still maintain the differentiation of different message types. + +Note + +If no message type is specified, the default type is cmd. + +### Message Name + +Message Name is used within the TEN runtime to differentiate messages with different purposes under the same message type. The extension determines what actions to take based on the differentiation of message names. + +The naming convention for message names is as follows: + +1. The first character can only be a-z, A-Z, or \_. +2. Other characters can be a-z, A-Z, 0-9, or \_. + +### Creating Messages + +Non-TEN platform built-in command: + +```json +{ + "ten": { + "type": "cmd", // mandatory + "name": "hello_world" // mandatory + } +} +``` + +TEN platform built-in command: + +```json +{ + "ten": { + "type": "cmd", // mandatory + "name": "ten::connect" // mandatory + } +} +``` + +Data: + +```json +{ + "ten": { + "type": "data", // mandatory + "name": "foo" // optional + } +} +``` + +Image Frame: + +```json +{ + "ten": { + "type": "image_frame", // mandatory + "name": "foo" // optional + } +} +``` + +PCM Frame: + +```json +{ + "ten": { + "type": "pcm_frame", // mandatory + "name": "foo" // optional + } +} +``` + +### An Optimization in TEN Runtime for Message Names + +Within the TEN runtime, each message is recorded with two fields: + +1. Message type + + Enum. +2. Message name + + String. + +Since message names are in string format, there is a certain performance overhead when comparing strings. Therefore, when the TEN runtime encounters certain specific message names, it can use a message type to represent that message. For example, when it sees a message name like ten::connect, the TEN runtime can optimize it from: + +* Message type: cmd // mandatory +* Message name: ten::connect // mandatory + +to: + +* Message type: connect // mandatory +* Message name: ten::connect // optional + +This optimization can only be applied to message names recognized by the TEN platform, so it is only possible for TEN platform built-in message names. Users cannot perform this optimization themselves, and new message types cannot be added by users. Since built-in messages are limited, the newly added message types are also limited, aligning with the enum type of message types. + +In summary, message types other than cmd, data, image\_frame, and pcm\_frame are generated through this optimization, and these additional message types correspond to a special message name, which is always an TEN built-in message. + +This optimization is not only applicable to commands but also to other message types. For example: + +* Message type: image\_frame // mandatory +* Message name: ten::empty\_image\_frame // mandatory + +can be optimized to: + +* Message type: image\_frame\_empty // mandatory +* Message name: ten::empty\_image\_frame // optional + +Since the TEN platform modifies the type field of the message, users can also see and use this optimization. This is not a bad thing as users can leverage it to speed up message analysis. Therefore, users can use two methods to determine the message: + +1. Using the message name + + ```c++ + if (message_name == "ten::timer") + ``` +2. Using the message type + + ```c++ + if (message_type == MSG_TYPE_TIMER) + ``` + +### Creating Messages (with the mentioned optimization) + +TEN platform ηš„ built-in command: + +```json +{ + "ten": { + "type": "cmd", // mandatory + "name": "ten::connect" // mandatory + } +} +``` + +Or + +```json +{ + "ten": { + "type": "connect" // mandatory + } +} +``` + +### When to Specify Message Type or Message Name + +When it is not possible to determine the message type or message name in the context, it is necessary to specify the message type or message name. + +For example: + +* Message from JSON + + When the message type or message name is unknown, it is necessary to specify the message type and message name in the JSON. +* Command from JSON + + When it is known that the message is a command but the command name is unknown, it is necessary to specify the message name in the JSON. +* Connect command from JSON + + When it is known that the message is a connect command, the message name can only be ten::connect. Therefore, there is no need to specify the message type or message name in the JSON. diff --git a/ten_service/ten_schema_beta.md b/ten_service/ten_schema_beta.md new file mode 100644 index 0000000..a90d9b3 --- /dev/null +++ b/ten_service/ten_schema_beta.md @@ -0,0 +1,512 @@ +--- +layout: + title: + visible: true + description: + visible: false + tableOfContents: + visible: true + outline: + visible: true + pagination: + visible: true +--- + +# 🚧 TEN schema(beta) + +## Overview + +TEN Schema provides a way to describe the data structure of TEN Value. TEN Value is a structured data type used in TEN Runtime to store information such as properties. TEN Schema can be used to define the data structure of configuration files for Extensions, as well as the data structure of messages sent and received by Extensions. + +The most common use case is when there are two Extensions - A and B, and A sends a command to B. In this scenario, TEN Schema is involved in the following places: + +* Configuration file of Extension A. +* Configuration file of Extension B. +* Data structure of the message carried by the command sent by A as a producer. +* Data structure of the message carried by the command received by B as a consumer. + +### Configuration Files + +RTE Extensions typically have two configuration files: + +`property.json`: This file contains the business-specific configuration of the Extension, such as: + +{% code title="property.josn" %} +```json +{ + "app_id": "123456", + "channel": "test", + "log": { + "level": 1, + "redirect_stdout": true, + "file": "api.log" + } +} +``` +{% endcode %} + +`manifest.json`: This file contains the immutable information of the Extension, including metadata (type, name, version, etc.) and schema definitions. For example: + +{% code title="manifest.json" %} +```json +{ + "type": "extension", + "name": "A", + "version": "1.0.0", + "language": "cpp", + "dependencies": [], + "api": {} +} +``` +{% endcode %} + +Developers can customize the logic for loading configuration files in the Extension `on_init()` function. By default, TEN Extension automatically loads the `property.json` and `manifest.json` files in the directory. After the Extension`:on_start()` callback is triggered, developers can use rte:`get_property()` to retrieve the contents of `property.json`. For example: + +````cpp +``` +void on_init(rte::rte_t &rte, rte::metadata_info_t &manifest, + rte::metadata_info_t &property) override { + // You can load custom configuration files using the following method. In this case, the default property.json will not be loaded. + // property.set(RTE_METADATA_JSON_FILENAME, "a.json"); + + rte.on_init_done(manifest, property); +} +```` + +After the `rte.on_init_done()` is triggered, the content of property.json will be stored in TEN Runtime as an TEN Value, with the type of Object. Here are some details: + +* `app_id` is a Value of type String, and developers can retrieve it using `rte.get_property_string("app_id")`, specifying the type as string. +* `log` is a Value of type Object. In the TEN Value system, Object is a composite structured data that contains other TEN Values in key-value pairs. For example, `level` is a field in `log` and its type is int64. + + + +{% hint style="info" %} +Note + +This involves one of the purposes of TEN Schema: to explicitly declare the precision of Values. + +For JSON, an integer value like `1` is by default parsed as `int64` in x64. When TEN Runtime reads JSON, it can only handle it as `int64` by default. Even if the developer expects the type to be `uint8`, this information cannot be inferred from JSON alone. This is where TEN Schema comes in, to explicitly declare the type of `level` as `uint8`. After parsing property.json with TEN Runtime, the value of `level` will be evaluated based on the definition in the TEN Schema. If it meets the storage requirements of `uint8`, it will be stored as `uint8` in the TEN Value. + +Note + +One of the rules followed by TEN Schema is to ensure data integrity and no loss of precision during type conversion. It does not support cross-type re-parsing, such as converting int to double or string. +{% endhint %} + + + +### TEN Schema + +The type in TEN Schema is exactly the same as the type in TEN Value, including: + +
integerunsigned integerfloat pointother
int8uint8float32string
int16uint16float64bool
int32uint32buf
int64uint64ptr
array
object
+ + + +Except for the simple data types (array and object), the declaration is as follows: + +```json +{ + "type": "int8" +} +``` + +For the array type, the declaration is as follows: + +```json +{ + "type": "array", + "items": { + "type": "int8" + } +} +``` + +For the object type, the declaration is as follows: + +```json +{ + "type": "object", + "properties": { + "field_1": { + "type": "int8" + }, + "field_2": { + "type": "bool" + } + } +} +``` + +Using the example of property.json mentioned above, its corresponding TEN Schema is as follows: + +```json +{ + "type": "object", + "properties": { + "app_id": { + "type": "string" + }, + "channel": { + "type": "string" + }, + "log": { + "type": "object", + "properties": { + "level": { + "type": "uint8" + }, + "redirect_stdout": { + "type": "bool" + }, + "file": { + "type": "string" + } + } + } + } +} +``` + +Note + +* Currently, TEN Schema only supports the type keyword to meet the basic needs. It will be gradually improved based on future requirements. + +### manifest.json + +The definition of the TEN Schema for the Extension needs to be saved in the manifest.json file. Similar to property.json, after the rte.on\_init\_done() is executed, manifest.json will be parsed by the TEN Runtime. This allows the TEN Runtime to validate the data integrity based on the schema definitions when loading property.json. + +The TEN Schema is placed under the api section in manifest.json. The api section is a JSON object that contains the definitions of all the schemas involved in the Extension, including the configurations mentioned above and the messages (TEN cmd/data/image\_frame/pcm\_frame) that will be introduced next. The structure of the api section is as follows: + +{% code title="manifest.json" %} +```json +{ + "property": {}, + "cmd_in": [], + "cmd_out": [], + "data_in": [], + "data_out": [], + "image_frame_in": [], + "image_frame_out": [], + "pcm_frame_in": [], + "pcm_frame_out": [], + "interface_in": [], + "interface_out": [] +} +``` +{% endcode %} + +The schema for property.json is placed under the property section. The content of the manifest.json corresponding to the example mentioned above should be as follows: + +{% code title="property.json" %} +```json +{ + "type": "extension", + "name": "A", + "version": "1.0.0", + "language": "cpp", + "dependencies": [], + "api": { + "property": { + "app_id": { + "type": "string" + }, + "channel": { + "type": "string" + }, + "log": { + "type": "object", + "properties": { + "level": { + "type": "uint8" + }, + "redirect_stdout": { + "type": "bool" + }, + "file": { + "type": "string" + } + } + } + } + } +} +``` +{% endcode %} + +### Messages + +Extensions can exchange messages, including cmd/data/image\_frame/pcm\_frame. If it is used as a producer (e.g., calling rte.send\_cmd()), the corresponding schema is defined in the xxx\_in section of manifest.json, such as cmd\_in. Similarly, if it is used as a consumer (e.g., receiving messages in the on\_cmd() callback), the corresponding schema is defined in the xxx\_out section of manifest.json, such as cmd\_out. + +The schema definition for messages is indexed by name. + +cmd must specify the name. For example: + +```json +{ + "api": { + "cmd_in": [ + { + "name": "start", + "property": { + "app_id": { + "type": "string" + } + } + } + ] + } +} +``` + +For data/image\_frame/pcm\_frame, the name is optional. This means that the name in the schema definition is optional; schemas without a specified name will be considered as the default schema. In other words, for any message, if a name exists, the schema will be indexed by that name; if not found, the default schema will be used. For example: + +```json +{ + "api": { + "data_in": [ + { + "property": { + "width": { + "type": "uint32" + } + } + }, + { + "name": "speech", + "property": { + "width": { + "type": "uint16" + } + } + } + ] + } +} +``` + +Similarly, when using JSON as the property of a message, if a schema exists, the types and precision will be converted according to the definitions in the schema. + +For the example mentioned above, Extension A's cmd\_out (i.e., A calling rte.send\_cmd()) corresponds to Extension B's cmd\_in (i.e., B receiving the message in the on\_cmd() callback). + +Note + +* Currently, if the cmd sent by A does not comply with the schema definition, B will not receive the cmd, and the TEN Runtime will return an error to A. + +Compared to data/image\_frame/pcm\_frame, cmd has an ack (represented as status cmd in TEN). This means that when defining the cmd schema, you can also define the schema for the corresponding status cmd. For example: + +```json +{ + "api": { + "cmd_in": [ + { + "name": "start", + "property": { + "app_id": { + "type": "string" + } + }, + "result": { + "property": { + "detail": { + "type": "string" + }, + "code": { + "type": "uint8" + } + } + } + } + ] + } +} +``` + +Note + +* "status" is the annotation key used to define the status schema. +* "rte/detail" is the annotation key used to define the type of detail in the status. It is used in APIs like rte.return\_string(), cmd.get\_detail(), etc. +* "status/property" is the annotation key that works the same as the property in cmd. +* The "status" in cmd\_in refers to the response received from the downstream Extension as a producer. Similarly, the "status" in cmd\_out refers to the response sent back to the upstream as a consumer. + +### Example + +Taking the rte\_stt\_asr\_filter extension as an example, let's see how to define the schema. + +First, the property.json file under the rte\_stt\_asr\_filter extension is not used. Its configuration is specified in the Graph as follows: + +```json +{ + "type": "extension", + "name": "rte_stt_asr_filter", + "addon": "rte_stt_asr_filter", + "extension_group": "rtc_group", + "property": { + "app_id": "1a4dcbc3", + "api_key": "b6e21445580c80a2a62a9c0394bc5e83", + "api_secret": "ZTliNzFhODU2MDMzYzQzYzUxODNmY2Ix", + "plugin_path": "addon/extension/rte_stt_asr_filter/lib/liblinux_audio_hy_extension.so", + "data_encoding": "raw" + } +} +``` + +The corresponding schema should be: + +```json +{ + "api": { + "property": { + "app_id": { + "type": "string" + }, + "api_key": { + "type": "string" + }, + "api_secret": { + "type": "string" + }, + "plugin_path": { + "type": "string" + }, + "data_encoding": { + "type": "string" + } + } + } +} +``` + +Next, it will receive the following five cmds: + +```cpp +if (command == COMMAND_START) { + handleStart(rte, std::move(cmd)); +} else if (command == COMMAND_STOP) { + handleStop(rte, std::move(cmd)); +} else if (command == COMMAND_ON_USER_AUDIO_TRACK_SUBSCRIBED) { + handleUserAudioTrackSubscribed(rte, std::move(cmd)); +} else if (command == COMMAND_ON_USER_AUDIO_TRACK_STATE_CHANGED) { + handleUserAudioTrackStateChanged(rte, std::move(cmd)); +} else if (command == COMMAND_QUERY) { + handleQuery(rte, std::move(cmd)); +} else { + RTE_LOGE("FATAL: unknown command: %s", command.c_str()); + rte.return_string(RTE_STATUS_CODE_ERROR, "not implemented", std::move(cmd)); +} +``` + +Let's take start and onUserAudioTrackSubscribed cmds as examples. + +start + +```cpp +class Config +{ + // ... + + private: + std::vector languages_; + std::string licenseFilePath_; +}; + +void from_json(const nlohmann::json& j, Config& p) +{ + try { + j.at("languages").get_to(p.languages_); + } catch (std::exception& e) { + RTE_LOGW("Failed to parse 'languages' property: %s", e.what()); + } + + try { + j.at("licenseFilePath").get_to(p.licenseFilePath_); + } catch (std::exception& e) { + RTE_LOGW("Failed to parse 'licenseFilePath' property: %s", e.what()); + } +} +``` + +For the start cmd, it includes two properties: + +* `languages`: an array type with elements of string type. +* `licenseFilePath`: a string type. + +So, the corresponding schema should be: + +```json +{ + "api": { + "cmd_in": [ + { + "name": "start", + "property": { + "languages": { + "type": "array", + "items": { + "type": "string" + } + }, + "licenseFilePath": { + "type": "string" + } + }, + "result": { + "property": { + "detail": { + "type": "string" + } + } + } + } + ] + } +} +``` + +onUserAudioTrackSubscribed + +```cpp +void rte_stt_asr_filter_extension_t::handleUserAudioTrackSubscribed( + rte::rte_t &rte, std::unique_ptr cmd) { + auto user_id = cmd->get_property_string("userId"); + auto *track = + cmd->get_property("audioTrack"); + + // ... + + rte.return_string(RTE_STATUS_CODE_OK, "done", std::move(cmd)); +} +``` + +For the onUserAudioTrackSubscribed cmd, it includes two properties: + +* `userId`: a string type. +* `audioTrack`: a ptr type pointing to agora::rtc::IRemoteAudioTrack. + +So, the corresponding schema should be: + +```json +{ + "api": { + "cmd_in": [ + { + "name": "onUserAudioTrackSubscribed", + "property": { + "userId": { + "type": "string" + }, + "audioTrack": { + "type": "ptr" + } + }, + "result": { + "property": { + "detail": { + "type": "string" + } + } + } + } + ] + } +} +``` diff --git a/tutorials/how_to_build_extension_with_c++_beta.md b/tutorials/how_to_build_extension_with_c++_beta.md new file mode 100644 index 0000000..dc85eb9 --- /dev/null +++ b/tutorials/how_to_build_extension_with_c++_beta.md @@ -0,0 +1,1118 @@ +--- +hidden: true +layout: + title: + visible: true + description: + visible: false + tableOfContents: + visible: true + outline: + visible: true + pagination: + visible: true +--- + +# 🚧 How to build extension with C++(beta) + +## Overview + +This tutorial introduces how to develop an TEN extension using C++, as well as how to debug and deploy it to run in an TEN app. This tutorial covers the following topics: + +* How to create a C++ extension development project using arpm. +* How to use TEN API to implement the functionality of the extension, such as sending and receiving messages. +* How to write unit test cases and debug the code. +* How to deploy the extension locally to an app and perform integration testing within the app. +* How to debug the extension code within the app. + +Note + +Unless otherwise specified, the commands and code in this tutorial are executed in a Linux environment. Since TEN has a consistent development approach and logic across all platforms (e.g., Windows, Mac), this tutorial is also suitable for other platforms. + +### Preparation + +* Download the latest arpm and configure the PATH. You can check if it is configured correctly with the following command: + + ``` + $ arpm -h + ``` + + If the configuration is successful, it will display the help information for arpm as follows: + + ``` + Usage: arpm [OPTIONS] + + Commands: + install Install a package. For more detailed usage, run 'install -h' + publish Publish a package. For more detailed usage, run 'publish -h' + dev-server Install a package. For more detailed usage, run 'dev-server -h' + help Print this message or the help of the given subcommand(s) + + Options: + --config-file The location of config.json + -h, --help Print help + -V, --version Print version + ``` +* Download the latest standalone\_gn and configure the PATH. For example: + + Note + + standalone\_gn is the C++ build system for the TEN platform. To facilitate developers, TEN provides a standalone\_gn toolchain for building C++ extension projects. + + ``` + $ export PATH=/path/to/standalone_gn:$PATH + ``` + + You can check if the configuration is successful with the following command: + + ``` + $ ag -h + ``` + + If the configuration is successful, it will display the help information for standalone\_gn as follows: + + ``` + usage: ag [-h] [-v] [--verbose | --no-verbose] [--out_file OUT_FILE] [--out_dir OUT_DIR] command target_OS target_CPU build_type + + An easy-to-use Google gn wrapper + + positional arguments: + command possible commands are: + gen build rebuild refs clean + graph uninstall explain_build desc check + show_deps show_input show_input_output path args + target_OS possible OS values are: + win mac ios android linux + target_CPU possible values are: + x64 x64 arm arm64 + build_type possible values are: + debug release + + options: + -h, --help show this help message and exit + -v, --version show program's version number and exit + --verbose, --no-verbose + dump verbose outputs + --out_file OUT_FILE dump command output to a file + --out_dir OUT_DIR build output dir, default is 'out/' + ``` + + Note + + * gn depends on python3, please make sure that Python 3.10 or above is installed. +* Install a C/C++ compiler, either clang/clang++ or gcc/g++. + +In addition, we provide a base compilation image where all of the above dependencies are already installed and configured. You can refer to the [ASTRA.ai](https://github.com/rte-design/ASTRA.ai) project on GitHub. + +### Creating C++ extension project + + + +#### Creating Based on Templates + + + +Assuming we want to create a project named first\_cxx\_extension, we can use the following command to create it: + +``` +$ arpm install extension default_extension_cpp --template-mode --template-data package_name=first_cxx_extension +``` + +Note + +The above command indicates that we are installing an TEN package using the default\_extension\_cpp template to create an extension project named first\_cxx\_extension. + +* \--template-mode indicates installing the TEN package as a template. The template rendering parameters can be specified using --template-data. +* extension is the type of TEN package to install. Currently, TEN provides app/extension\_group/extension/system packages. In the following sections on testing extensions in an app, we will use several other types of packages. +* default\_extension\_cpp is the default C++ extension provided by TEN. Developers can also specify other C++ extensions available in the store as templates. + +After the command is executed, a directory named first\_cxx\_extension will be generated in the current directory, which is our C++ extension project. The directory structure is as follows: + +``` +. +β”œβ”€β”€ BUILD.gn +β”œβ”€β”€ manifest.json +β”œβ”€β”€ property.json +└── src + └── main.cc +``` + +Where: + +* src/main.cc contains a simple implementation of the extension, including calls to the C++ API provided by TEN. We will discuss how to use the TEN API in the next section. +* manifest.json and property.json are the standard configuration files for TEN extensions. In manifest.json, metadata information such as the version, dependencies, and schema definition of the extension are typically declared. property.json is used to declare the business configuration of the extension. +* BUILD.gn is the configuration file for standalone\_gn, used to compile the C++ extension project. + +The property.json file is initially an empty JSON file, like this: + +``` +{} +``` + +The manifest.json file will include the rte\_runtime dependency by default, like this: + +``` +{ + "type": "extension", + "name": "first_cxx_extension", + "version": "0.2.0", + "language": "cpp", + "dependencies": [ + { + "type": "system", + "name": "rte_runtime", + "version": "0.2.0" + } + ], + "api": {} +} +``` + +Note + +* Please note that according to TEN's naming convention, the name should be alphanumeric. This is because when integrating the extension into an app, a directory will be created based on the extension name. TEN also provides the functionality to automatically load the manifest.json and property.json files from the extension directory. +* Dependencies are used to declare the dependencies of the extension. When installing TEN packages, arpm will automatically download the dependencies based on the declarations in the dependencies section. +* The api section is used to declare the schema of the extension. Refer to `usage of rte schema `. + +#### Manual Creation + + + +Developers can also manually create a C++ extension project or transform an existing project into an TEN extension project. + +First, ensure that the project's output target is a shared library. Then, refer to the example above to create `property.json` and `manifest.json` in the project's root directory. The `manifest.json` should include information such as `type`, `name`, `version`, `language`, and `dependencies`. Specifically: + +* `type` must be `extension`. +* `language` must be `cpp`. +* `dependencies` should include `rte_runtime`. + +Finally, configure the build settings. The `default_extension_cpp` provided by TEN uses `standalone_gn` as the build toolchain. If developers are using a different build toolchain, they can refer to the configuration in `BUILD.gn` to set the compilation parameters. Since `BUILD.gn` contains the directory structure of the TEN package, we will discuss it in the next section (Downloading Dependencies). + +### Download Dependencies + + + +To download dependencies, execute the following command in the extension project directory: + +``` +$ arpm install +``` + +After the command is executed successfully, a `.rte` directory will be generated in the current directory, which contains all the dependencies of the current extension. + +Note + +* There are two modes for extensions: development mode and runtime mode. In development mode, the root directory is the source code directory of the extension. In runtime mode, the root directory is the app directory. Therefore, the placement path of dependencies is different in these two modes. The `.rte` directory mentioned here is the root directory of dependencies in development mode. + +The directory structure is as follows: + +``` +. +β”œβ”€β”€ BUILD.gn +β”œβ”€β”€ manifest.json +β”œβ”€β”€ property.json +β”œβ”€β”€ .rte +β”‚ └── app +β”‚ β”œβ”€β”€ addon +β”‚ β”œβ”€β”€ include +β”‚ └── lib +└── src + └── main.cc +``` + +Where: + +* `.rte/app/include` is the root directory for header files. +* `.rte/app/lib` is the root directory for precompiled dynamic libraries of TEN runtime. + +If it is in runtime mode, the extension will be placed in the `addon/extension` directory of the app, and the dynamic libraries will be placed in the `lib` directory of the app. The structure is as follows: + +``` +. +β”œβ”€β”€ BUILD.gn +β”œβ”€β”€ manifest.json +β”œβ”€β”€ property.json +β”œβ”€β”€ addon +β”‚ └── extension +β”‚ └── first_cxx_extension +β”œβ”€β”€ include +└── lib +``` + +So far, an TEN C++ extension project has been created. + +### BUILD.gn + + + +The content of `BUILD.gn` for `default_extension_cpp` is as follows: + +``` +import("//exts/rte/base_options.gni") +import("//exts/rte/rte_package.gni") + +config("common_config") { + defines = common_defines + include_dirs = common_includes + cflags = common_cflags + cflags_c = common_cflags_c + cflags_cc = common_cflags_cc + cflags_objc = common_cflags_objc + cflags_objcc = common_cflags_objcc + libs = common_libs + lib_dirs = common_lib_dirs + ldflags = common_ldflags +} + +config("build_config") { + configs = [ ":common_config" ] + + # 1. The `include` refers to the `include` directory in current extension. + # 2. The `//include` refers to the `include` directory in the base directory + # of running `ag gen`. + # 3. The `.rte/app/include` is used in extension standalone building. + include_dirs = [ + "include", + "//include", + "//include/nlohmann_json", + ".rte/app/include", + ".rte/app/include/nlohmann_json", + ] + + lib_dirs = [ + "lib", + "//lib", + ".rte/app/lib", + ] + + if (is_win) { + libs = [ + "rte_runtime.dll.lib", + "utils.dll.lib", + ] + } else { + libs = [ + "rte_runtime", + "utils", + ] + } +} + +rte_package("first_cxx_extension") { + package_type = "develop" # develop | release + package_kind = "extension" + + manifest = "manifest.json" + property = "property.json" + + if (package_type == "develop") { + # It's 'develop' package, therefore, need to build the result. + build_type = "shared_library" + + sources = [ "src/main.cc" ] + + configs = [ ":build_config" ] + } +} +``` + +Let's first take a look at the `rte_package` target, which declares a build target for an TEN package. + +* The `package_kind` is set to `extension`, and the `build_type` is set to `shared_library`. This means that the expected output of the compilation is a shared library. +* The `sources` field specifies the source file(s) to be compiled. If there are multiple source files, they need to be added to the `sources` field. +* The `configs` field specifies the build configurations. It references the `build_config` defined in this file. + +Next, let's look at the content of `build_config`. + +* The `include_dirs` field defines the search paths for header files. + * The difference between `include` and `//include` is that `include` refers to the `include` directory in the current extension directory, while `//include` is based on the working directory of the `ag gen` command. So, if the compilation is executed in the extension directory, it will be the same as `include`. But if it is executed in the app directory, it will be the `include` directory in the app. + * `.rte/app/include` is used for standalone development and compilation of the extension, which is the scenario being discussed in this tutorial. In other words, the default `build_config` is compatible with both development mode and runtime mode compilation. +* The `lib_dirs` field defines the search paths for dependency libraries. The difference between `lib` and `//lib` is similar to `include`. +* The `libs` field defines the dependent libraries. `rte_runtime` and `utils` are libraries provided by TEN. + +Therefore, if developers are using a different build toolchain, they can refer to the above configuration and set the compilation parameters in their own build toolchain. For example, if using g++ to compile: + +``` +$ g++ -shared -fPIC -I.rte/app/include/ -L.rte/app/lib -lrte_runtime -lutils -Wl,-rpath=\$ORIGIN -Wl,-rpath=\$ORIGIN/../../../lib src/main.cc +``` + +The setting of `rpath` is also considered for the runtime mode, where the rte\_runtime dependency of the extension is placed in the `app/lib` directory. + +### Implementation of Extension Functionality + + + +For developers, there are two things to do: + +* Create an extension as a channel for interacting with TEN runtime. +* Register the extension as an addon in TEN, allowing it to be used in the graph through a declarative approach. + +#### Creating the Extension Class + + + +The extension created by developers needs to inherit the `rte::extension_t` class. The main definition of this class is as follows: + +``` +class extension_t { +protected: + explicit extension_t(const std::string &name) {...} + + virtual void on_init(rte_t &rte, metadata_info_t &manifest, + metadata_info_t &property) { + rte.on_init_done(manifest, property); + } + + virtual void on_start(rte_t &rte) { rte.on_start_done(); } + + virtual void on_stop(rte_t &rte) { rte.on_stop_done(); } + + virtual void on_deinit(rte_t &rte) { rte.on_deinit_done(); } + + virtual void on_cmd(rte_t &rte, std::unique_ptr cmd) { + auto cmd_result = rte::cmd_result_t::create(RTE_STATUS_CODE_OK); + cmd_result->set_property("detail", "default"); + rte.return_result(std::move(cmd_result), std::move(cmd)); + } + + virtual void on_data(rte_t &rte, std::unique_ptr data) {} + + virtual void on_pcm_frame(rte_t &rte, std::unique_ptr frame) {} + + virtual void on_image_frame(rte_t &rte, + std::unique_ptr frame) {} +} +``` + +In the markdown content you provided, there are descriptions of the lifecycle functions and message handling functions in Chinese. Here is the translation: + +Lifecycle Functions: + +* on\_init: Used to initialize the extension instance, such as setting the extension's configuration. +* on\_start: Used to start the extension instance, such as establishing connections to external services. The extension will not receive messages until on\_start is completed. In on\_start, you can use the rte.get\_property API to retrieve the extension's configuration. +* on\_stop: Used to stop the extension instance, such as closing connections to external services. +* on\_deinit: Used to destroy the extension instance, such as releasing memory resources. + +Message Handling Functions: + +* on\_cmd/on\_data/on\_pcm\_frame/on\_image\_frame: These are callback methods used to receive messages of four different types. For more information on TEN message types, you can refer to the [message-type-and-name](https://github.com/rte-design/ASTRA.ai/blob/main/docs/message-type-and-name.md) + +The rte::extension\_t class provides default implementations for these functions, and developers can override them according to their needs. + +#### Registering the Extension + + + +After defining the extension, it needs to be registered as an addon in the TEN runtime. For example, in the `first_cxx_extension/src/main.cc` file, the registration code is as follows: + +``` +RTE_CPP_REGISTER_ADDON_AS_EXTENSION(first_cxx_extension, first_cxx_extension_extension_t); +``` + +* RTE\_CPP\_REGISTER\_ADDON\_AS\_EXTENSION is a macro provided by the TEN runtime for registering extension addons. + * The first parameter is the name of the addon, which serves as a unique identifier for the addon. It will be used to define the extension in the graph using a declarative approach. + * The second parameter is the implementation class of the extension, which is the class that inherits from rte::extension\_t. + +Please note that the addon name must be unique because it is used as a unique index to find the implementation in the graph. + +#### on\_init + + + +Developers can set the extension's configuration in the on\_init() function, as shown in the example: + +``` +void on_init(rte::rte_t& rte, rte::metadata_info_t& manifest, + rte::metadata_info_t& property) override { + property.set(RTE_METADATA_JSON_FILENAME, "customized_property.json"); + rte.on_init_done(manifest, property); +} +``` + +Both the property and manifest can be customized using the set() method. In the example, the first parameter RTE\_METADATA\_JSON\_FILENAME indicates that the custom property is stored as a local file, and the second parameter is the file path relative to the extension directory. So in this example, when the app loads the extension, it will load `/addon/extension/first_cxx_extension/customized_property.json`. + +TEN's on\_init provides default logic for loading default configurations. If developers do not call property.set(), the property.json file in the extension directory will be loaded by default. Similarly, if manifest.set() is not called, the manifest.json file in the extension directory will be loaded by default. In the example, since property.set() is called, the property.json file will not be loaded by default. + +Please note that on\_init is an asynchronous method, and developers need to call rte.on\_init\_done() to inform the TEN runtime that on\_init has completed as expected. + +#### on\_start + + + +When on\_start is called, it means that on\_init\_done() has been executed and the extension's property has been loaded. From this point on, the extension can access the configuration. For example: + +``` +void on_start(rte::rte_t& rte) override { + auto prop = rte.get_property_string("some_string"); + // do something + + rte.on_start_done(); +} +``` + +rte.get\_property\_string() is used to retrieve a property of type string with the name "some\_string". If the property does not exist or the type does not match, an error will be returned. If the extension's configuration contains the following content: + +``` +{ + "some_string": "hello world" +} +``` + +Then the value of prop will be "hello world". + +Similar to on\_init, on\_start is also an asynchronous method, and developers need to call rte.on\_start\_done() to inform the TEN runtime that on\_start has completed as expected. + +For more information, you can refer to the API documentation: rte api doc. + +#### Error Handling + + + +As shown in the previous example, if "some\_string" does not exist or is not of type string, rte.get\_property\_string() will return an error. You can handle the error as follows: + +``` +void on_start(rte::rte_t& rte) override { + rte::error_t err; + auto prop = rte.get_property_string("some_string", &err); + + // error handling + if (!err.is_success()) { + RTE_LOGE("Failed to get property: %s", err.errmsg()); + } + + rte.on_start_done(); +} +``` + +#### Message Handling + + + +TEN provides four types of messages: `cmd`, `data`, `image_frame`, and `pcm_frame`. Developers can handle these four types of messages by implementing the `on_cmd`, `on_data`, `on_image_frame`, and `on_pcm_frame` callback methods. + +Taking `cmd` as an example, let's see how to receive and send messages. + +Assume that `first_cxx_extension` receives a `cmd` with the name `hello`, which includes the following properties: + +| name | type | +| ---------------- | ------ | +| app\_id | string | +| client\_type | int8 | +| payload | object | +| payload.err\_no | uint8 | +| payload.err\_msg | string | + +The processing logic of `first_cxx_extension` for the `hello` cmd is as follows: + +* If the `app_id` or `client_type` parameters are invalid, return an error: + + ``` + { + "err_no": 1001, + "err_msg": "Invalid argument." + } + ``` +* If `payload.err_no` is greater than 0, return an error with the content from the `payload`. +* If `payload.err_no` is equal to 0, forward the `hello` cmd downstream for further processing. After receiving the processing result from the downstream extension, return the result. + +**Describing the Extension's Behavior in manifest.json** + + + +Based on the above description, the behavior of `first_cxx_extension` is as follows: + +* It receives a `cmd` named `hello` with properties. +* It may send a `cmd` named `hello` with properties. +* It receives a response from a downstream extension, which includes error information. +* It returns a response to an upstream extension, which includes error information. + +For a TEN extension, you can describe the above behavior in the `manifest.json` file of the extension, including: + +* What messages the extension receives, their names, and the structure definition of their properties (schema definition). +* What messages the extension generates/sends, their names, and the structure definition of their properties. +* Additionally, for `cmd` type messages, a response definition is required (referred to as a result in TEN). + +With these definitions, TEN runtime will perform validity checks based on the schema definition before delivering messages to the extension or when the extension sends messages through TEN runtime. It also helps the users of the extension to see the protocol definition. + +The schema is defined in the `api` field of the `manifest.json` file. `cmd_in` defines the cmds that the extension will receive, and `cmd_out` defines the cmds that the extension will send. + +Note + +For the usage of schema, refer to: [rte-schema](https://github.com/rte-design/ASTRA.ai/blob/main/docs/rte-schema.md) + +Based on the above description, the content of `manifest.json` for `first_cxx_extension` is as follows: + +``` +{ + "type": "extension", + "name": "first_cxx_extension", + "version": "0.2.0", + "language": "cpp", + "dependencies": [ + { + "type": "system", + "name": "rte_runtime", + "version": "0.2.0" + } + ], + "api": { + "cmd_in": [ + { + "name": "hello", + "property": { + "app_id": { + "type": "string" + }, + "client_type": { + "type": "int8" + }, + "payload": { + "type": "object", + "properties": { + "err_no": { + "type": "uint8" + }, + "err_msg": { + "type": "string" + } + } + } + }, + "required": ["app_id", "client_type"], + "result": { + "property": { + "err_no": { + "type": "uint8" + }, + "err_msg": { + "type": "string" + } + }, + "required": ["err_no"] + } + } + ], + "cmd_out": [ + { + "name": "hello", + "property": { + "app_id": { + "type": "string" + }, + "client_type": { + "type": "string" + }, + "payload": { + "type": "object", + "properties": { + "err_no": { + "type": "uint8" + }, + "err_msg": { + "type": "string" + } + } + } + }, + "required": ["app_id", "client_type"], + "result": { + "property": { + "err_no": { + "type": "uint8" + }, + "err_msg": { + "type": "string" + } + }, + "required": ["err_no"] + } + } + ] + } +} +``` + +**Getting Request Data** + + + +In the `on_cmd` method, the first step is to retrieve the request data, which is the property in the cmd. We define a `request_t` class to represent the request data. + +Create a file called `model.h` in the `include` directory of your extension project with the following content: + +``` +#pragma once + +#include "nlohmann/json.hpp" +#include +#include + +namespace first_cxx_extension_extension { + +class request_payload_t { +public: + friend void from_json(const nlohmann::json &j, request_payload_t &payload); + + friend class request_t; + +private: + uint8_t err_no; + std::string err_msg; +}; + +class request_t { +public: + friend void from_json(const nlohmann::json &j, request_t &request); + +private: + std::string app_id; + int8_t client_type; + request_payload_t payload; +}; + +} // namespace first_cxx_extension_extension +``` + +In the `src` directory, create a file called `model.cc` with the following content: + +``` +#include "model.h" + +namespace first_cxx_extension_extension { +void from_json(const nlohmann::json &j, request_payload_t &payload) { + if (j.contains("err_no")) { + j.at("err_no").get_to(payload.err_no); + } + + if (j.contains("err_msg")) { + j.at("err_msg").get_to(payload.err_msg); + } +} + +void from_json(const nlohmann::json &j, request_t &request) { + if (j.contains("app_id")) { + j.at("app_id").get_to(request.app_id); + } + + if (j.contains("client_type")) { + j.at("client_type").get_to(request.client_type); + } + + if (j.contains("payload")) { + j.at("payload").get_to(request.payload); + } +} +} // namespace first_cxx_extension_extension +``` + +To parse the request data, you can use the `get_property` API provided by TEN. Here is an example of how to implement it: + +``` +// model.h + +class request_t { +public: + void from_cmd(rte::cmd_t &cmd); + + // ... +} + +// model.cc + +void request_t::from_cmd(rte::cmd_t &cmd) { + app_id = cmd.get_property_string("app_id"); + client_type = cmd.get_property_int8("client_type"); + + auto payload_str = cmd.get_property_to_json("payload"); + if (!payload_str.empty()) { + auto payload_json = nlohmann::json::parse(payload_str); + from_json(payload_json, payload); + } +} +``` + +To return a response, you need to create a `cmd_result_t` object and set the properties accordingly. Then, pass the `cmd_result_t` object to TEN runtime to return it to the requester. Here is an example: + +``` +// model.h + +class request_t { +public: + bool validate(std::string *err_msg) { + if (app_id.length() < 64) { + *err_msg = "invalid app_id"; + return false; + } + + return true; + } +} + +// main.cc + +void on_cmd(rte::rte_t &rte, std::unique_ptr cmd) override { + request_t request; + request.from_cmd(*cmd); + + std::string err_msg; + if (!request.validate(&err_msg)) { + auto result = rte::cmd_result_t::create(RTE_STATUS_CODE_ERROR); + result->set_property("err_no", 1); + result->set_property("err_msg", err_msg.c_str()); + + rte.return_result(std::move(result), std::move(cmd)); + } +} +``` + +In the example above, `rte::cmd_result_t::create` is used to create a `cmd_result_t` object with an error code. `result.set_property` is used to set the properties of the `cmd_result_t` object. Finally, `rte.return_result` is called to return the `cmd_result_t` object to the requester. + +**Passing Requests to Downstream Extensions** + + + +If an extension needs to send a message to another extension, it can call the `send_cmd()` API. Here is an example: + +``` +void on_cmd(rte::rte_t &rte, std::unique_ptr cmd) override { + request_t request; + request.from_cmd(*cmd); + + std::string err_msg; + if (!request.validate(&err_msg)) { + // ... + } else { + rte.send_cmd(std::move(cmd)); + } +} +``` + +The first parameter in `send_cmd()` is the command of the request, and the second parameter is the handler for the returned `cmd_result_t`. The second parameter can also be omitted, indicating that no special handling is required for the returned result. If the command was originally sent from a higher-level extension, the runtime will automatically return it to the upper-level extension. + +Developers can also pass a response handler, like this: + +``` +rte.send_cmd( + std::move(cmd), + [](rte::rte_t &rte, std::unique_ptr result) { + rte.return_result_directly(std::move(result)); + }); +``` + +In the example above, the `return_result_directly()` method is used in the response handler. You can see that this method differs from `return_result()` in that it does not pass the original command object. This is mainly because: + +* For TEN message objects (cmd/data/pcm\_frame/image\_frame), ownership is transferred to the extension in the message callback method, such as `on_cmd()`. This means that once the extension receives the command, the TEN runtime will not perform any read/write operations on it. When the extension calls the `send_cmd()` or `return_result()` API, it means that the extension is returning the ownership of the command back to the TEN runtime for further processing, such as message delivery. After that, the extension should not perform any read/write operations on the command. +* The `result` in the response handler (i.e., the second parameter of `send_cmd()`) is returned by the downstream extension, and at this point, the result is already bound to the command, meaning that the runtime has the return path information for the result. Therefore, there is no need to pass the command object again. + +Of course, developers can also process the result in the response handler. + +So far, an example of a simple command processing logic is complete. For other message types such as data, you can refer to the TEN API documentation. + +### Deploying Locally to an App for Integration Testing + + + +arpm provides the ability to publish to a local registry, allowing you to perform integration testing locally without uploading the extension to the central repository. Unlike GO extensions, for C++ extensions, there are no strict requirements on the app's programming language. It can be GO, C++, or Python. + +The deployment process may vary for different apps. The specific steps are as follows: + +* Set up the arpm local registry. +* Upload the extension to the local registry. +* Download the app from the central repository (default\_app\_cpp/default\_app\_go) for integration testing. +* For C++ apps: + * Install the first\_cxx\_extension in the app directory. + * Compile in the app directory. At this point, both the app and the extension will be compiled into the out/linux/x64/app/default\_app\_cpp directory. + * Install the required dependencies in out/linux/x64/app/default\_app\_cpp. The working directory for testing is the current directory. +* For GO apps: + * Install the first\_cxx\_extension in the app directory. + * Compile in the addon/extension/first\_cxx\_extension directory, as the GO and C++ compilation toolchains are different. + * Install the dependencies in the app directory. The working directory for testing is the app directory. +* Configure the graph in the app's manifest.json, specifying the recipient of the message as first\_cxx\_extension, and send test messages. + +#### Uploading the Extension to the Local Registry + + + +First, create a temporary config.json file to set up the arpm local registry. For example, the contents of /tmp/code/config.json are as follows: + +``` +{ + "registry": [ + "file:///tmp/code/repository" + ] +} +``` + +This sets the local directory /tmp/code/repository as the arpm local registry. + +Note + +* Be careful not to place it in \~/.arpm/config.json, as it will affect the subsequent download of dependencies from the central repository. + +Then, in the first\_cxx\_extension directory, execute the following command to upload the extension to the local registry: + +``` +$ arpm --config-file /tmp/code/config.json publish +``` + +After the command completes, the uploaded extension can be found in the /tmp/code/repository/extension/first\_cxx\_extension/0.1.0 directory. + +#### Prepare app for testing (C++) + + + +1. Install default\_app\_cpp as the test app in an empty directory. + +> ``` +> $ arpm install app default_app_cpp +> ``` +> +> After the command is successfully executed, there will be a directory named default\_app\_cpp in the current directory. +> +> Note +> +> * When installing an app, its dependencies will be automatically installed. + +2. Install first\_cxx\_extension that we want to test in the app directory. + +> Execute the following command: +> +> ``` +> $ arpm --config-file /tmp/code/config.json install extension first_cxx_extension +> ``` +> +> After the command is completed, there will be a first\_cxx\_extension directory in the addon/extension directory. +> +> Note +> +> * It is important to note that since first\_cxx\_extension is in the local registry, the configuration file path with the local registry specified by --config-file needs to be the same as when publishing. + +3. Add an extension as a message producer. + +> first\_cxx\_extension is expected to receive a hello cmd, so we need a message producer. One way is to add an extension as a message producer. To conveniently generate test messages, an http server can be integrated into the producer's extension. +> +> First, create an http server extension based on default\_extension\_cpp. Execute the following command in the app directory: +> +> ``` +> $ arpm install extension default_extension_cpp --template-mode --template-data package_name=http_server +> ``` +> +> The main functionality of the http server is: +> +> * Start a thread running the http server in the extension's on\_start(). +> * Convert incoming requests into TEN cmds named hello and send them using send\_cmd(). +> * Expect to receive a cmd\_result\_t response and write its content to the http response. +> +> Here, we use cpp-httplib ([https://github.com/yhirose/cpp-httplib](https://github.com/yhirose/cpp-httplib)) as the implementation of the http server. +> +> First, download httplib.h and place it in the include directory of the extension. Then, add the implementation of the http server in src/main.cc. Here is an example code: +> +> ``` +> #include "httplib.h" +> #include "nlohmann/json.hpp" +> #include "rte_runtime/binding/cpp/rte.h" +> +> namespace http_server_extension { +> +> class http_server_extension_t : public rte::extension_t { +> public: +> explicit http_server_extension_t(const std::string &name) +> : extension_t(name) {} +> +> void on_start(rte::rte_t &rte) override { +> rte_proxy = rte::rte_proxy_t::create(rte); +> srv_thread = std::thread([this] { +> server.Get("/health", +> [](const httplib::Request &req, httplib::Response &res) { +> res.set_content("OK", "text/plain"); +> }); +> +> // Post handler, receive json body. +> server.Post("/hello", [this](const httplib::Request &req, +> httplib::Response &res) { +> // Receive json body. +> auto body = nlohmann::json::parse(req.body); +> body["rte"]["name"] = "hello"; +> +> auto cmd = rte::cmd_t::create_from_json(body.dump().c_str()); +> auto cmd_shared = +> std::make_shared>(std::move(cmd)); +> +> std::condition_variable *cv = new std::condition_variable(); +> +> auto response_body = std::make_shared(); +> +> rte_proxy->notify([cmd_shared, response_body, cv](rte::rte_t &rte) { +> rte.send_cmd( +> std::move(*cmd_shared), +> [response_body, cv](rte::rte_t &rte, +> std::unique_ptr result) { +> auto err_no = result->get_property_uint8("err_no"); +> if (err_no > 0) { +> auto err_msg = result->get_property_string("err_msg"); +> response_body->append(err_msg); +> } else { +> response_body->append("OK"); +> } +> +> cv->notify_one(); +> }); +> }); +> +> std::unique_lock lk(mtx); +> cv->wait(lk); +> delete cv; +> +> res.set_content(response_body->c_str(), "text/plain"); +> }); +> +> server.listen("0.0.0.0", 8001); +> }); +> +> rte.on_start_done(); +> } +> +> void on_stop(rte::rte_t &rte) override { +> // Extension stop. +> +> server.stop(); +> srv_thread.join(); +> delete rte_proxy; +> +> rte.on_stop_done(); +> } +> +> private: +> httplib::Server server; +> std::thread srv_thread; +> rte::rte_proxy_t *rte_proxy{nullptr}; +> std::mutex mtx; +> }; +> +> RTE_CPP_REGISTER_ADDON_AS_EXTENSION(http_server, http_server_extension_t); +> +> } // namespace http_server_extension +> ``` + +Here, a new thread is created in `on_start()` to run the http server because we don't want to block the extension thread. This way, the converted cmd requests are generated and sent from `srv_thread`. In the TEN runtime, to ensure thread safety, we use `rte_proxy_t` to pass calls like `send_cmd()` from threads outside the extension thread. + +This code also demonstrates how to clean up external resources in `on_stop()`. For an extension, you should release the `rte_proxy_t` before `on_stop_done()`, which stops the external thread. + +1. Configure the graph. + +In the app's `manifest.json`, configure `predefined_graph` to specify that the `hello` cmd generated by `http_server` should be sent to `first_cxx_extension`. For example: + +> ``` +> "predefined_graphs": [ +> { +> "name": "testing", +> "auto_start": true, +> "nodes": [ +> { +> "type": "extension_group", +> "name": "http_thread", +> "addon": "default_extension_group" +> }, +> { +> "type": "extension", +> "name": "http_server", +> "addon": "http_server", +> "extension_group": "http_thread" +> }, +> { +> "type": "extension", +> "name": "first_cxx_extension", +> "addon": "first_cxx_extension", +> "extension_group": "http_thread" +> } +> ], +> "connections": [ +> { +> "extension_group": "http_thread", +> "extension": "http_server", +> "cmd": [ +> { +> "name": "hello", +> "dest": [ +> { +> "extension_group": "http_thread", +> "extension": "first_cxx_extension" +> } +> ] +> } +> ] +> } +> ] +> } +> ] +> ``` + +5. Compile the app. + +> Execute the following commands in the app directory: +> +> ``` +> $ ag gen linux x64 debug +> $ ag build linux x64 debug +> ``` +> +> After the compilation is complete, the compilation output for the app and extension will be generated in the directory out/linux/x64/app/default\_app\_cpp. +> +> However, it cannot be run directly at this point as it is missing the dependencies of the extension group. + +6. Install the extension group. + +> Switch to the compilation output directory. +> +> ``` +> $ cd out/linux/x64/app/default_app_cpp +> ``` +> +> Install the extension group. +> +> ``` +> $ arpm install extension_group default_extension_group +> ``` + +7. Start the app. + +> In the compilation output directory, execute the following command: +> +> ``` +> $ ./bin/default_app_cpp +> ``` +> +> After the app starts, you can now test it by sending messages to the http server. For example, use curl to send a request with an invalid app\_id: +> +> ``` +> $ curl --location 'http://127.0.0.1:8001/hello' \ +> --header 'Content-Type: application/json' \ +> --data '{ +> "app_id": "123", +> "client_type": 1, +> "payload": { +> "err_no": 0 +> } +> }' +> ``` +> +> The expected response should be "invalid app\_id". + +### Debugging extension in an app + +#### App (C++) + +A C++ app is compiled into an executable file with the correct `rpath` set. Therefore, debugging a C++ app only requires adding the following configuration to `.vscode/launch.json`: + +``` +"configurations": [ + { + "name": "App (C/C++) (lldb, launch)", + "type": "lldb", + "request": "launch", + "program": "${workspaceFolder}/out/linux/x64/app/default_app_cpp/bin/default_app_cpp", + "args": [], + "cwd": "${workspaceFolder}/out/linux/x64/app/default_app_cpp" + } + ] +``` diff --git a/tutorials/how_to_build_extension_with_go_beta.md b/tutorials/how_to_build_extension_with_go_beta.md new file mode 100644 index 0000000..40bd11d --- /dev/null +++ b/tutorials/how_to_build_extension_with_go_beta.md @@ -0,0 +1,1120 @@ +--- +hidden: true +layout: + title: + visible: true + description: + visible: false + tableOfContents: + visible: true + outline: + visible: true + pagination: + visible: true +--- + +# 🚧 How to build extension with Go(beta) + +## Overview + +Introduction to developing a TEN extension using the GO language, as well as debugging and deploying it to run in an app. + +This tutorial includes the following: + +* How to create a GO extension development project using `arpm`. +* How to use TEN API to implement the functionality of the extension, such as sending and receiving messages. +* How to configure `cgo` if needed. +* How to write unit tests and debug code. +* How to deploy the extension locally in an app for integration testing. +* How to debug extension code in an app. + +{% hint style="info" %} +Unless otherwise specified, the commands and code in this tutorial are executed in a Linux environment. Similar steps can be followed for other platforms. +{% endhint %} + + + +## Prerequisites + +Download the latest version of arpm and configure the PATH. You can check if it is configured correctly by running the following command: + +```bash +$ arpm -h +``` + +If the configuration is correct, it will display the help information for arpm. + +* Install GO 1.20 or above, preferably the latest version. + +Note: + +* TEN GO API uses cgo, so make sure cgo is enabled by default. You can check by running the following command: + +``` +$ go env CGO_ENABLED +``` + +If it returns 1, it means cgo is enabled by default. Otherwise, you can enable cgo by running the following command: + +``` +$ go env -w CGO_ENABLED=1 +``` + +### Creating a GO Extension Project + + + +A GO extension is essentially a go module project that includes the necessary dependencies and configuration files to meet the requirements of a TEN extension. TEN provides a default GO extension template project that developers can use to quickly create a GO extension project. + +#### Creating from Template + + + +To create a project named "first\_go\_extension" based on the default\_extension\_go template, use the following command: + +``` +$ arpm install extension default_extension_go --template-mode --template-data package_name=first_go_extension +``` + +After executing the command, a directory named "first\_go\_extension" will be created in the current directory. This directory will contain the GO extension project with the following structure: + +``` +. +β”œβ”€β”€ default_extension.go +β”œβ”€β”€ go.mod +β”œβ”€β”€ manifest.json +└── property.json +``` + +In this structure: + +* "default\_extension.go" contains a simple extension implementation that includes calls to the TEN GO API. The usage of the TEN API will be explained in the next section. +* "manifest.json" and "property.json" are the standard configuration files for TEN extensions. "manifest.json" is used to declare metadata information such as the version, dependencies, and schema definition of the extension. "property.json" is used to declare the business configuration of the extension. + +The `property.json` file is initially empty and The `manifest.json` file includes a dependency on "rte\_runtime\_go" by default: + +``` +{ + "type": "extension", + "name": "first_go_extension", + "version": "0.1.0", + "language": "go", + "dependencies": [ + { + "type": "system", + "name": "rte_runtime_go", + "version": "0.2.0" + } + ], + "api": {} +} +``` + +Note + +* Please note that according to TEN's naming convention, the name should be alphanumeric because when integrating the extension into an app, a directory will be created based on the extension name. Additionally, TEN will provide functionality to load the manifest.json and property.json files from the extension directory. +* The dependencies section declares the dependencies of the current extension. When installing the TEN package, arpm will automatically download the declared dependencies. +* The api section is used to declare the schema of the extension. For the usage of schema, refer to: [rte-schema](https://github.com/rte-design/ASTRA.ai/blob/main/docs/rte-schema.md) + +TEN GO API is not publicly available and needs to be installed locally using arpm. Therefore, the go.mod file uses the replace directive to reference the TEN GO API. For example: + +``` +replace agora.io/rte => ../../../interface +``` + +When the extension is installed in an app, it will be placed in the addon/extension/ directory. At the same time, the TEN GO API will be installed in the root directory of the app. The expected directory structure is as follows: + +``` +. +β”œβ”€β”€ addon +β”‚ └── extension +β”‚ └── first_go_extension +β”‚ β”œβ”€β”€ default_extension.go +β”‚ β”œβ”€β”€ go.mod +β”‚ β”œβ”€β”€ manifest.json +β”‚ └── property.json +β”œβ”€β”€ go.mod +β”œβ”€β”€ go.sum +β”œβ”€β”€ interface +β”‚ └── rtego +└── main.go +``` + +Therefore, the replace directive in the extension's go.mod file points to the interface directory in the app. + +#### Manual Creation + + + +Alternatively, developers can create a go module project using the go init command and then create the manifest.json and property.json files based on the examples provided above. + +Do not add the dependency on the TEN GO API yet because the required interface directory is not available locally. It needs to be installed using arpm before adding the dependency. + +To convert a newly created go module project or an existing one into an extension project, follow these steps: + +* Create the property.json file in the project directory and add the necessary configuration for the extension. +* Create the manifest.json file in the project directory and specify the type, name, version, language, and dependencies information. Note that these fields are required. + * The type should be extension. + * The language should be go. + * The dependencies should include the dependency on rte\_runtime\_go and any other dependencies as needed. + +### Download Dependencies + + + +Execute the following command in the extension project directory to download dependencies: + +``` +$ arpm install +``` + +After the command is successfully executed, a .rte directory will be generated in the current directory, which contains all the dependencies of the current extension. + +Note + +* There are two modes for an extension: development mode and runtime mode. In development mode, the root directory is the source code directory of the extension. In runtime mode, the root directory is the app directory. Therefore, the placement path of dependencies is different in these two modes. The .rte directory mentioned here is the root directory of dependencies in development mode. + +The directory structure is as follows: + +``` +β”œβ”€β”€ default_extension.go +β”œβ”€β”€ go.mod +β”œβ”€β”€ manifest.json +β”œβ”€β”€ property.json +└── .rte + └── app + +``` + +In this structure, .rte/app/interface is the module for the TEN GO API. + +Therefore, in development mode, the go.mod file of the extension should be modified as follows: + +``` +replace agora.io/rte => ./.rte/app/interface +``` + +If you manually created the extension as mentioned in the previous section, you also need to execute the following command in the extension directory: + +``` +$ go get agora.io/rte +``` + +The expected output should be: + +``` +go: added agora.io/rte v0.0.0-00010101000000-000000000000 +``` + +At this point, a TEN GO extension project has been created. + +### Implementing Extension Functionality + + + +The "go.mod" file uses the "replace" directive to reference the TEN GO API: + +``` +replace agora.io/rte => ../../../interface +``` + +When the extension is installed in an app, it will be placed in the "addon/extension/" directory. The TEN GO API will be installed in the root directory of the app. The expected directory structure is as follows: + +``` +. +β”œβ”€β”€ addon +β”‚ └── extension +β”‚ └── first_go_extension +β”‚ β”œβ”€β”€ default_extension.go +β”‚ β”œβ”€β”€ go.mod +β”‚ β”œβ”€β”€ manifest.json +β”‚ └── property.json +β”œβ”€β”€ go.mod +β”œβ”€β”€ go.sum +β”œβ”€β”€ interface +β”‚ └── rtego +└── main.go +``` + +Therefore, the "replace" directive in the extension's go.mod file points to the "interface" directory in the app. + +#### Manual Creation + + + +Alternatively, developers can manually create a go module project and then add the "manifest.json" and "property.json" files based on the examples provided above. + +For a newly created go module project or an existing one, to convert it into an extension project, follow these steps: + +* Create the "property.json" file in the project directory and add the necessary configuration for the extension. +* Create the "manifest.json" file in the project directory and specify the "type", "name", "version", "language", and "dependencies" information. Note that these fields are required. + * The "type" should be "extension". + * The "language" should be "go". + * The "dependencies" should include the dependency on "rte\_runtime\_go" and any other dependencies as needed. + +### Download Dependencies + + + +Execute the following command in the extension project directory to download dependencies: + +``` +$ arpm install +``` + +After the command is successfully executed, a `.rte` directory will be generated in the current directory, which contains all the dependencies of the current extension. + +Note: + +* There are two modes for an extension: development mode and runtime mode. In development mode, the root directory is the source code directory of the extension. In runtime mode, the root directory is the app directory. Therefore, the placement path of dependencies is different in these two modes. The `.rte` directory mentioned here is the root directory of dependencies in development mode. + +The directory structure is as follows: + +``` +β”œβ”€β”€ default_extension.go +β”œβ”€β”€ go.mod +β”œβ”€β”€ manifest.json +β”œβ”€β”€ property.json +└── .rte + └── app + β”œβ”€β”€ addon + β”œβ”€β”€ include + β”œβ”€β”€ interface + └── lib +``` + +In this structure, `.rte/app/interface` is the module for the TEN GO API. + +Therefore, in development mode, the `go.mod` file of the extension should be modified as follows: + +``` +replace agora.io/rte => ./.rte/app/interface +``` + +If you manually created the extension as mentioned in the previous section, you also need to execute the following command in the extension directory: + +``` +$ go get agora.io/rte +``` + +The expected output should be: + +``` +go: added agora.io/rte v0.0.0-00010101000000-000000000000 +``` + +At this point, a TEN GO extension project has been created. + +### Implementing Extension Functionality + + + +For developers, there are two things to do: + +* Create an extension as a channel for interaction with TEN runtime. +* Register the extension as an addon (referred to as addon in TEN) to use the extension in the graph declaratively. + +#### Create extension struct + + + +The extension created by developers needs to implement the `rtego.Extension` interface, which is defined as follows: + +``` +type Extension interface { + OnInit( + rte Rte, + manifest MetadataInfo, + property MetadataInfo, + ) + OnStart(rte Rte) + OnStop(rte Rte) + OnDeinit(rte Rte) + OnCmd(rte Rte, cmd Cmd) + OnData(rte Rte, data Data) + OnImageFrame(rte Rte, imageFrame ImageFrame) + OnPcmFrame(rte Rte, pcmFrame PcmFrame) +} +``` + +It includes four lifecycle functions and four message handling functions: + +Lifecycle functions: + +* OnInit: Used to initialize the extension instance, such as setting the extension's configuration. +* OnStart: Used to start the extension instance, such as creating connections to external services. The extension will not receive messages until it is started. In OnStart, you can use the `rte.GetProperty` related APIs to get the extension's configuration. +* OnStop: Used to stop the extension instance, such as closing connections to external services. +* OnDeinit: Used to destroy the extension instance, such as releasing memory resources. + +Message handling functions: + +* OnCmd/OnData/OnImageFrame/OnPcmFrame: Callback functions for receiving four types of messages. The message types in TEN can be referred to as [message-type-and-name](https://github.com/rte-design/ASTRA.ai/blob/main/docs/message-type-and-name.md) + +For the implementation of the extension, you may only need to focus on a subset of message types. To facilitate implementation, TEN provides a default `DefaultExtension`. Developers have two options: either directly implement the `rtego.Extension` interface or embed `DefaultExtension` and override the necessary methods. + +For example, the `default_extension_go` template uses the approach of embedding `DefaultExtension`. Here is an example: + +``` +type defaultExtension struct { + rtego.DefaultExtension +} +``` + +#### Register extension + + + +After defining the extension, it needs to be registered as an addon in the TEN runtime. For example, the registration code in `default_extension.go` is as follows: + +``` +func init() { + // Register addon + rtego.RegisterAddonAsExtension( + "default_extension_go", + rtego.NewDefaultExtensionAddon(newDefaultExtension), + ) +} +``` + +* `RegisterAddonAsExtension` is the method to register an addon object as an TEN extension addon. The addon types in TEN also include extension\_group and protocol, which will be covered in the subsequent integration testing section. + * The first parameter is the addon name, which is a unique identifier for the addon. It will be used to define the extension in the graph declaratively. For example: + + ``` + { + "nodes": [ + { + "type": "extension", + "name": "extension_go", + "addon": "default_extension_go", + "extension_group": "default" + } + ] + } + ``` + + In this example, it means using an addon named `default_extension_go` to create an extension instance named `extension_go`. + * The second parameter is an addon object. TEN provides a simple way to create it - `NewDefaultExtensionAddon`, which takes the constructor of the business extension as a parameter. For example: + + ``` + func newDefaultExtension(name string) rtego.Extension { + return &defaultExtension{} + } + ``` + +Note + +* It is important to note that the addon name must be unique because in the graph, the addon name is used as a unique index to find the implementation. Here, change the first parameter to `first_go_extension`. + +#### OnInit + + + +Developers can set the extension's configuration in OnInit(), as shown below: + +``` +func (p *defaultExtension) OnInit(rte rtego.Rte, property rtego.MetadataInfo, manifest rtego.MetadataInfo) { + property.Set(rtego.MetadataTypeJSONFileName, "customized_property.json") + rte.OnInitDone() +} +``` + +* Both `property` and `manifest` can customize the configuration content using the `Set()` method. In the example, the first parameter `rtego.MetadataTypeJSONFileName` indicates that the custom property exists as a local file, and the second parameter is the file path relative to the extension directory. So in the example, when the app loads the extension, it will load `\/addon/extension/first_go_extension/customized_property.json`. +* TEN OnInit provides default configuration loading logic - if developers do not call `property.Set()`, the `property.json` in the extension directory will be loaded by default. Similarly, if `manifest.Set()` is not called, the `manifest.json` in the extension directory will be loaded by default. Also, if developers call `property.Set()`, the `property.json` will not be loaded by default. +* OnInit is an asynchronous method, and developers need to call `rte.OnInitDone()` to inform the TEN runtime that the initialization is expected to be completed. + +Note + +* Please note that `OnInitDone()` is also an asynchronous method, which means that even after `OnInitDone()` returns, developers still cannot use `rte.GetProperty()` to get the configuration. For the extension, you need to wait until the `OnStart()` callback method. + +#### OnStart + + + +When OnStart is called, it indicates that OnInitDone() has already been executed and the extension's property has been loaded. From this point on, the extension can access the configuration. Here is an example: + +``` +func (p *defaultExtension) OnStart(rte rtego.Rte) { + prop, err := rte.GetPropertyString("some_string") + + if err != nil { + // handle error. + } else { + // do something. + } + + rte.OnStartDone() +} +``` + +`rte.GetPropertyString()` is used to retrieve a property of type string with the name "some\_string". If the property does not exist or the type does not match, an error will be returned. If the extension's configuration is as follows: + +``` +{ + "some_string": "hello world" +} +``` + +Then the value of `prop` will be "hello world". + +Similar to OnInit, OnStart is also an asynchronous method, and developers need to call `rte.OnStartDone()` to inform the TEN runtime that the start process is expected to be completed. + +For error handling, as shown in the previous example, `rte.GetPropertyString()` returns an error. For TEN API, errors are generally returned as `rtego.RteError` type. Therefore, you can handle the error as follows: + +``` +func (p *defaultExtension) OnStart(rte rtego.Rte) { + prop, err := rte.GetPropertyString("some_string") + + if err != nil { + // handle error. + var rteErr *rtego.RteError + if errors.As(err, &rteErr) { + log.Printf("Failed to get property, cause: %s.\n", rteErr.ErrMsg()) + } + } else { + // do something. + } + + rte.OnStartDone() +} +``` + +`rtego.RteError` provides the `ErrMsg()` method to retrieve the error message and the `ErrNo()` method to retrieve the error code. + +TEN provides four types of messages: Cmd, Data, ImageFrame, and PcmFrame. Developers can handle these four types of messages by implementing the OnCmd, OnData, OnImageFrame, and OnPcmFrame callback methods. + +Taking Cmd as an example, let's see how to receive and send messages. + +Assume that the first\_go\_extension will receive a Cmd named "hello" with the following properties: + +| name | type | +| ---------------- | ------ | +| app\_id | string | +| client\_type | int8 | +| payload | object | +| payload.err\_no | uint8 | +| payload.err\_msg | string | + +The processing logic of the first\_go\_extension for the "hello" Cmd is as follows: + +* If the app\_id or client\_type parameters are invalid, return an error: + +``` +{ + "err_no": 1001, + "err_msg": "Invalid argument." +} +``` + +* If payload.err\_no is greater than 0, return an error with the content from the payload. +* If payload.err\_no is equal to 0, forward the "hello" Cmd downstream for further processing. After receiving the processing result from the downstream extension, return the result. + +To describe the behavior of the extension in the manifest.json file, you can specify: + +* What messages the extension will receive, including the message name and the structure definition of the properties (schema). +* What messages the extension will send, including the message name and the structure definition of the properties. +* For Cmd messages, a response definition is required (referred to as Result in TEN). + +With these definitions, the TEN runtime will perform validation based on the schema before delivering messages to the extension and before the extension sends messages through the TEN runtime. This ensures the validity of the messages and facilitates the understanding of the extension's protocol by its users. + +The schema is defined in the api field of the manifest.json file. cmd\_in defines the Cmd messages that the extension will receive, and cmd\_out defines the Cmd messages that the extension will send. + +Here is an example of the manifest.json for the first\_go\_extension: + +``` +{ + "type": "extension", + "name": "first_go_extension", + "version": "0.1.0", + "language": "go", + "dependencies": [ + { + "type": "system", + "name": "rte_runtime_go", + "version": "0.2.0" + } + ], + "api": { + "cmd_in": [ + { + "name": "hello", + "property": { + "app_id": { + "type": "string" + }, + "client_type": { + "type": "int8" + }, + "payload": { + "type": "object", + "properties": { + "err_no": { + "type": "uint8" + }, + "err_msg": { + "type": "string" + } + } + } + }, + "required": ["app_id", "client_type"], + "result": { + "property": { + "err_no": { + "type": "uint8" + }, + "err_msg": { + "type": "string" + } + }, + "required": ["err_no"] + } + } + ], + "cmd_out": [ + { + "name": "hello", + "property": { + "app_id": { + "type": "string" + }, + "client_type": { + "type": "string" + }, + "payload": { + "type": "object", + "properties": { + "err_no": { + "type": "uint8" + }, + "err_msg": { + "type": "string" + } + } + } + }, + "required": ["app_id", "client_type"], + "result": { + "property": { + "err_no": { + "type": "uint8" + }, + "err_msg": { + "type": "string" + } + }, + "required": ["err_no"] + } + } + ] + } +} +``` + +**Getting Request Data** + + + +In the `OnCmd` method, the first step is to retrieve the request data, which is the property of the Cmd object. We define a `Request` struct to represent the request data as follows: + +``` +type RequestPayload struct { + ErrNo uint8 `json:"err_no"` + ErrMsg string `json:"err_msg"` +} + +type Request struct { + AppID string `json:"app_id"` + ClientType int8 `json:"client_type"` + Payload RequestPayload `json:"payload"` +} +``` + +For TEN message objects like Cmd, Data, PcmFrame, and ImageFrame, properties can be set. TEN provides getter/setter APIs for properties. The logic to retrieve the request data is to use the `GetProperty` API to parse the property of the Cmd object, as shown below: + +``` +func parseRequestFromCmdProperty(cmd rtego.Cmd) (*Request, error) { + request := &Request{} + + if appID, err := cmd.GetPropertyString("app_id"); err != nil { + return nil, err + } else { + request.AppID = appID + } + + if clientType, err := cmd.GetPropertyInt8("client_type"); err != nil { + return nil, err + } else { + request.ClientType = clientType + } + + if payloadBytes, err := cmd.GetPropertyToJSONBytes("payload"); err != nil { + return nil, err + } else { + err := json.Unmarshal(payloadBytes, &request.Payload) + + rtego.ReleaseBytes(payloadBytes) + + if err != nil { + return nil, err + } + } + + return request, nil +} +``` + +* `GetPropertyString()` and `GetPropertyInt8()` are specialized APIs to retrieve properties of specific types. For example, `GetPropertyString()` expects the property to be of type string, and if it's not, an error will be returned. +* `GetPropertyToJSONBytes()` expects the value of the property to be a JSON-serialized data. This API is provided because TEN runtime does not expect to be bound to a specific JSON library. After obtaining the property as a slice, developers can choose a JSON library for deserialization as needed. +* `rtego.ReleaseBytes()` is used to release the `payloadBytes` because TEN GO binding layer provides a memory pool. + +**Returning a Response** + + + +After parsing the request data, we can now implement the first step of the processing flow - returning an error response if the parameters are invalid. In TEN extensions, a response is represented by a `CmdResult`. So, returning a response involves the following two steps: + +* Creating a `CmdResult` object and setting properties as needed. +* Handing over the created `CmdResult` object to the TEN runtime, which will handle returning it based on the path of the requester. + +Here's the implementation: + +``` +const InvalidArgument uint8 = 1 + +func (r *Request) validate() error { + if len(r.AppID) < 64 { + return errors.New("invalid app_id") + } + + if r.ClientType != 1 { + return errors.New("invalid client_type") + } + + return nil +} + +func (p *defaultExtension) OnCmd( + rte rtego.Rte, + cmd rtego.Cmd, +) { + request, err := parseRequestFromCmdProperty(cmd) + if err == nil { + err = request.validate() + } + + if err != nil { + result, fatal := rtego.NewCmdResult(rtego.Error) + if fatal != nil { + log.Fatalf("Failed to create result, %v\n", fatal) + return + } + + result.SetProperty("err_no", InvalidArgument) + result.SetPropertyString("err_msg", err.Error()) + + rte.ReturnResult(result, cmd) + } +} +``` + +* `rtego.NewCmdResult()` is used to create a `CmdResult` object. The first parameter is the error code - either Ok or Error. This error code is built-in to the TEN runtime and indicates whether the processing was successful. Developers can also use `GetStatusCode()` to retrieve this error code. Additionally, developers can define more detailed business-specific error codes, as shown in the example. +* `result.SetProperty()` is used to set properties in the `CmdResult` object. Properties are set as key-value pairs. `SetPropertyString()` is a specialized API of `SetProperty()` and is provided to reduce the performance overhead of passing GO strings. +* `rte.ReturnResult()` is used to return the `CmdResult` to the requester. The first parameter is the response, and the second parameter is the request. The TEN runtime will handle returning the response to the requester based on the request's path. + +**Passing the Request to Downstream Extensions** + + + +If an extension wants to send a message to another extension, it can call the `SendCmd()` API. Here's an example: + +``` +func (p *defaultExtension) OnCmd( + rte rtego.Rte, + cmd rtego.Cmd, +) { + // ... + + if err != nil { + // ... + } else { + // Dispatch the request to the upstream. + rte.SendCmd(cmd, func(r rtego.Rte, result rtego.CmdResult) { + r.ReturnResultDirectly(result) + }) + } +} +``` + +* The first parameter in `SendCmd()` is the command of the request, and the second parameter is the callback function that handles the `CmdResult` returned by the downstream. The second parameter can also be set to `nil`, indicating that no special handling is required for the result returned by the downstream. If the original command was sent from a higher-level extension, the runtime will automatically return it to the upper-level extension. +* In the example's callback function, `ReturnResultDirectly()` is used. You can see that this method differs from `ReturnResult()` in that it does not pass the original command object. This is mainly because: + * For TEN message objects like Cmd/Data/PcmFrame/ImageFrame, there is an ownership concept. In the extension's message callback method, such as `OnCmd()`, the TEN runtime transfers ownership of the Cmd object to the extension. This means that once the extension receives the Cmd, the TEN runtime will not perform any read or write operations on it. When the extension calls the `SendCmd()` or `ReturnResult()` API, it returns ownership of the Cmd back to the TEN runtime, which handles further processing, such as message delivery. After that, the extension should not perform any read or write operations on the Cmd. + * The `result` in the response handler (the second parameter of `SendCmd()`) is returned by the downstream, and at this point, the result is already bound to the Cmd, meaning the runtime has information about the result's return path. Therefore, there is no need to pass the Cmd object again. + +Of course, developers can also handle the result in the response handler. + +So far, an example of a simple Cmd processing logic is complete. For other message types like Data, you can refer to the TEN API documentation. + +### Deploying Locally to the App for Integration Testing + + + +arpm provides the ability to publish and use a local registry, allowing you to perform integration testing locally without uploading the extension to the central repository. Here are the steps: + +* Set up the arpm local registry. +* Upload the extension to the local registry. +* Download the default\_app\_go from the central repository as the integration testing environment, along with any other dependencies needed. +* Download the first\_go\_extension from the local registry. +* Configure the graph in default\_app\_go to specify the receiver of the message as first\_go\_extension and send test messages. + +#### Uploading the Extension to the Local Registry + + + +Before uploading, restore the dependency path for TEN GO binding in first\_go\_extension/go.mod because it needs to be installed under the app. For example: + +``` +replace agora.io/rte => ../../../interface +``` + +First, create a temporary config.json file to set up the arpm local registry. For example, the contents of /tmp/code/config.json are as follows: + +``` +{ + "registry": [ + "file:///tmp/code/repository" + ] +} +``` + +This sets the local directory /tmp/code/repository as the arpm local registry. + +Note + +* Make sure not to place it in \~/.arpm/config.json as it will affect downloading dependencies from the central repository. + +Then, in the first\_go\_extension directory, execute the following command to upload the extension to the local registry: + +``` +$ arpm --config-file /tmp/code/config.json publish +``` + +After the command completes, the uploaded extension can be found in the directory /tmp/code/repository/extension/first\_go\_extension/0.1.0. + +#### Prepare the test app + + + +1. Install `default_app_go` as the test app in an empty directory. + +``` +$ arpm install app default_app_go +``` + +After the command is successful, you will have a directory named `default_app_go` in the current directory. + +> **Note** +> +> * Since the extension being tested is written in Go, the app must also be written in Go. `default_app_go` is a Go app template provided by TEN. +> * When installing the app, its dependencies will be automatically installed. + +2. Install the `extension_group` in the app directory. + +Switch to the app directory: + +``` +$ cd default_app_go +``` + +Install the `default_extension_group`: + +``` +$ arpm install extension_group default_extension_group +``` + +After the command completes, you will have a directory named `addon/extension_group/default_extension_group`. + +> **Note** +> +> * `extension_group` is a capability provided by TEN to declare physical threads and specify which extension instances run in which extension groups. `default_extension_group` is the default extension group provided by TEN. + +3. Install the `first_go_extension` that we want to test in the app directory. + +Execute the following command: + +``` +$ arpm --config-file /tmp/code/config.json install extension first_go_extension +``` + +After the command completes, you will have a directory named `addon/extension/first_go_extension`. + +> **Note** +> +> * It is important to note that since `first_go_extension` is in the local registry, you need to specify the configuration file path that contains the local registry configuration using `--config-file`, just like when publishing. + +4. Add an extension as a message producer. + +> The first\_go\_extension is expected to receive a hello cmd, so we need a message producer. One way to achieve this is by adding an extension as a message producer. To conveniently generate test messages, we can integrate an HTTP server into the producer's extension. +> +> First, we can create an HTTP server extension based on default\_extension\_go. Execute the following command in the app directory: +> +> ``` +> $ arpm install extension default_extension_go --template-mode --template-data package_name=http_server +> ``` +> +> Modify the module name in addon/extension/http\_server/go.mod to http\_server. +> +> The main functionalities of the HTTP server are as follows: +> +> * Start an HTTP server in the extension's OnStart() method, running as a goroutine. +> * Convert incoming requests into TEN Cmd with the name hello, and then call SendCmd() to send the message. +> * Expect to receive a CmdResult response and write its content to the HTTP response. +> +> The code implementation is as follows: +> +> ``` +> type defaultExtension struct { +> rtego.DefaultExtension +> rte rtego.Rte +> +> server *http.Server +> } +> +> type RequestPayload struct { +> ErrNo uint8 `json:"err_no"` +> ErrMsg string `json:"err_msg"` +> } +> +> type Request struct { +> AppID string `json:"app_id"` +> ClientType int8 `json:"client_type"` +> Payload RequestPayload `json:"payload"` +> } +> +> func newDefaultExtension(name string) rtego.Extension { +> return &defaultExtension{} +> } +> +> func (p *defaultExtension) defaultHandler(writer http.ResponseWriter, request *http.Request) { +> switch request.URL.Path { +> case "/health": +> writer.WriteHeader(http.StatusOK) +> default: +> resultChan := make(chan rtego.CmdResult, 1) +> +> var req Request +> if err := json.NewDecoder(request.Body).Decode(&req); err != nil { +> writer.WriteHeader(http.StatusBadRequest) +> writer.Write([]byte("Invalid request body.")) +> return +> } +> +> cmd, _ := rtego.NewCmd("hello") +> cmd.SetPropertyString("app_id", req.AppID) +> cmd.SetProperty("client_type", req.ClientType) +> +> payloadBytes, _ := json.Marshal(req.Payload) +> cmd.SetPropertyFromJSONBytes("payload", payloadBytes) +> +> p.rte.SendCmd(cmd, func(rte rtego.Rte, result rtego.CmdResult) { +> resultChan <- result +> }) +> +> result := <-resultChan +> +> writer.WriteHeader(http.StatusOK) +> errNo, _ := result.GetPropertyUint8("err_no") +> +> if errNo > 0 { +> errMsg, _ := result.GetPropertyString("err_msg") +> writer.Write([]byte(errMsg)) +> } else { +> writer.Write([]byte("OK")) +> } +> } +> } +> +> func (p *defaultExtension) OnStart(rte rtego.Rte) { +> p.rte = rte +> +> mux := http.NewServeMux() +> mux.HandleFunc("/", p.defaultHandler) +> +> p.server = &http.Server{ +> Addr: ":8001", +> Handler: mux, +> } +> +> go func() { +> if err := p.server.ListenAndServe(); err != nil { +> if err != http.ErrServerClosed { +> panic(err) +> } +> } +> }() +> +> go func() { +> // Check if the server is ready. +> for { +> resp, err := http.Get("http://127.0.0.1:8001/health") +> if err != nil { +> continue +> } +> +> defer resp.Body.Close() +> +> if resp.StatusCode == 200 { +> break +> } +> +> time.Sleep(50 * time.Millisecond) +> } +> +> fmt.Println("http server starts.") +> +> p.rte.OnStartDone() +> }() +> } +> +> func (p *defaultExtension) OnStop(rte rtego.Rte) { +> fmt.Println("defaultExtension OnStop") +> +> if p.server != nil { +> p.server.Shutdown(context.Background()) +> } +> +> rte.OnStopDone() +> } +> +> func init() { +> fmt.Println("defaultExtension init") +> +> // Register addon +> rtego.RegisterAddonAsExtension( +> "http_server", +> rtego.NewDefaultExtensionAddon(newDefaultExtension), +> ) +> } +> ``` + +1. Configure the graph. + +> In the app's `manifest.json`, configure `predefined_graph` to specify the `hello` cmd generated by `http_server` and send it to `first_go_extension`. For example: + +``` +"predefined_graphs": [ + { + "name": "testing", + "auto_start": true, + "nodes": [ + { + "type": "extension_group", + "name": "http_thread", + "addon": "default_extension_group" + }, + { + "type": "extension", + "name": "http_server", + "addon": "http_server", + "extension_group": "http_thread" + }, + { + "type": "extension", + "name": "first_go_extension", + "addon": "first_go_extension", + "extension_group": "http_thread" + } + ], + "connections": [ + { + "extension_group": "http_thread", + "extension": "http_server", + "cmd": [ + { + "name": "hello", + "dest": [ + { + "extension_group": "http_thread", + "extension": "first_go_extension" + } + ] + } + ] + } + ] + } +] +``` + +6. Compile the app and start it. + +> Run the following command in the app directory: + +``` +$ go run scripts/build/main.go +``` + +After the compilation is complete, an executable file `./bin/main` will be generated by default. + +Start the app by running the following command: + +``` +$ ./bin/main +``` + +When the console outputs the log message "http server starts", it means that the HTTP listening port has been successfully started. You can now send requests to test it. + +For example, use curl to send a request with an invalid `app_id`: + +``` +$ curl --location 'http://127.0.0.1:8001/hello' \ + --header 'Content-Type: application/json' \ + --data '{ + "app_id": "123", + "client_type": 1, + "payload": { + "err_no": 0 + } + }' +``` + +You should expect to receive a response with the message "invalid app\_id". + +### Debugging an extension in the app + + + +The compilation of an TEN app is performed through a script defined by TEN, which includes the necessary configuration settings for compilation. In other words, you cannot compile an TEN app directly using "go build". Therefore, you cannot debug the app using the default method and instead need to choose the "attach" method. + +For example, if you want to debug the app in Visual Studio Code, you can add the following configuration to ".vscode/launch.json": + +``` +{ + "version": "0.2.0", + "configurations": [ + { + "name": "app (golang) (go, attach)", + "type": "go", + "request": "attach", + "mode": "local", + "processId": 0, + "stopOnEntry": true + } + ] +} +``` + +First, compile and start the app using the above method. + +Then, in the "RUN AND DEBUG" window of Visual Studio Code, select "app (golang) (go, attach)" and click "Start Debugging". + +Next, in the pop-up process selection window, locate the running app process and start debugging. + +Note: + +* If you encounter the "the scope of ptrace system call application is limited" error on a Linux environment, you can resolve it by running the following command: + +``` +$ sudo sysctl -w kernel.yama.ptrace_scope=0 +``` + +\ diff --git a/tutorials/how_to_debug_with_logs.md b/tutorials/how_to_debug_with_logs.md new file mode 100644 index 0000000..8d45841 --- /dev/null +++ b/tutorials/how_to_debug_with_logs.md @@ -0,0 +1,47 @@ +--- +layout: + title: + visible: true + description: + visible: false + tableOfContents: + visible: true + outline: + visible: true + pagination: + visible: true +--- + +# How to debug with logs + +In this chapter, we’ll cover how to view the logs and understand what they mean. + +For instance, if the Astra AI agent is running at localhost:3000, you might see output similar to the following in the logs: + +
...
+2024/08/13 05:18:38 INFO handlerPing start channelName=agora_aa1aou requestId=435851e4-b5ff-437a-9930-14ae94b1dee7 service=HTTP_SERVER
+2024/08/13 05:18:38 INFO handlerPing end worker="&{ChannelName:agora_aa1aou LogFile:/tmp/astra/app-5ea31f2d5d7a5e48a9dbc583959c1b17-1723525636779876885.log PropertyJsonFile:/tmp/property-5ea31f2d5d7a5e48a9dbc583959c1b17-1723525636779876885.json Pid:245 QuitTimeoutSeconds:60 CreateTs:1723525636 UpdateTs:1723526318}" requestId=435851e4-b5ff-437a-9930-14ae94b1dee7 service=HTTP_SERVER
+[GIN] 2024/08/13 - 05:18:38 | 200 |     2.65725ms |    192.168.65.1 | POST     "/ping"
+...
+
+ +The line starting with LogFile: is what we’re interested in. To view the log file, use the following command: + +
cat /tmp/astra/app-5ea31f2d5d7a5e48a9dbc583959c1b17-1723525636779876885.log
+
+ +You should then see entries like these: + +{% code title=">_Bash" overflow="wrap" %} +```bash +[SttMs] OnSpeechRecognized +2024/08/13 04:40:43.365841 INFO GetChatCompletionsStream recv for input text: [What's going on? ] sent sentence [Not much,] extension=OPENAI_CHATGPT_EXTENSION +2024/08/13 04:40:43.366019 INFO GetChatCompletionsStream recv for input text: [What's going on? ] first sentence sent, first_sentency_latency 876ms extension=OPENAI_CHATGPT_EXTENSION +2024/08/13 04:40:43.489795 INFO GetChatCompletionsStream recv for input text: [What's going on? ] sent sentence [ just here and ready to chat!] extension=OPENAI_CHATGPT_EXTENSION +2024/08/13 04:40:43.493444 INFO GetChatCompletionsStream recv for input text: [What's going on? ] sent sentence [ How about you?] extension=OPENAI_CHATGPT_EXTENSION +2024/08/13 04:40:43.495756 INFO GetChatCompletionsStream recv for input text: [What's going on? ] sent sentence [ What's on your mind?] extension=OPENAI_CHATGPT_EXTENSION +2024/08/13 04:40:43.496120 INFO GetChatCompletionsStream for input text: [What's going on? ] end of segment with sentence [] sent extension=OPENAI_CHATGPT_EXTENSION +``` +{% endcode %} + +When you see logs like this, it means the system is working correctly and logging each sentence you say.