Skip to content

Commit

Permalink
Merge branch 'dev/tsdb' of github.com:KunkaiLin1205/TEN-Agent into de…
Browse files Browse the repository at this point in the history 
…v/vision_tool_python
  • Loading branch information
@tomasliu-agora
tomasliu-agora committed Oct 19, 2024
2 parents 0a9b426 + 7c910a4 commit 5149dc6
Show file tree
Hide file tree
Showing 54 changed files with 1,496 additions and 201 deletions.
4 changes: 2 additions & 2 deletions .env.example
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ WORKERS_MAX=100
# Worker quit timeout in seconds
WORKER_QUIT_TIMEOUT_SECONDES=60

# Agora App ID and Agora App Certificate
# required: this variable must be set
# Agora App ID
# Agora App Certificate(only required if enabled in the Agora Console)
AGORA_APP_ID=
AGORA_APP_CERTIFICATE=

Expand Down
17 changes: 12 additions & 5 deletions .github/workflows/build-docker.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,23 @@ name: Build Docker
on:
push:
branches: [ "**" ]
# Publish semver tags as releases.
tags: ["v*.*.*"]
tags: [ "**" ]
paths-ignore:
- ".devcontainer/**"
- ".github/ISSUE_TEMPLATE/**"
- "images/**"
- "playground/**"
- ".github/**"
- "!.github/workflows/build-docker.yaml"
- ".vscode/**"
- "docs/**"
- "**.md"
pull_request:
branches: [ "main" ]
paths-ignore:
- ".devcontainer/**"
- ".github/**"
- "!.github/workflows/build-docker.yaml"
- ".vscode/**"
- "docs/**"
- "**.md"
workflow_dispatch:

env:
Expand Down
16 changes: 9 additions & 7 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,8 @@ And, if you choose to use OpenAI Realtime API, then the Vision and RAG won't be
### Prerequisites

#### Keys
- Agora [ App ID ](https://docs.agora.io/en/video-calling/get-started/manage-agora-account?platform=web#create-an-agora-project) and [ App Certificate ](https://docs.agora.io/en/video-calling/get-started/manage-agora-account?platform=web#create-an-agora-project)(certificate is not required)
- Azure [SST](https://azure.microsoft.com/en-us/products/ai-services/speech-to-text) and [TTS](https://azure.microsoft.com/en-us/products/ai-services/text-to-speech) API keys (feel free to use another provider)
- Agora [ App ID ](https://docs.agora.io/en/video-calling/get-started/manage-agora-account?platform=web#create-an-agora-project) and [ App Certificate ](https://docs.agora.io/en/video-calling/get-started/manage-agora-account?platform=web#create-an-agora-project)(certificate only required if enabled in the Agora Console)
- Azure [SST(ASR)](https://azure.microsoft.com/en-us/products/ai-services/speech-to-text) and [TTS](https://azure.microsoft.com/en-us/products/ai-services/text-to-speech) API keys
- [OpenAI](https://openai.com/index/openai-api/) API key

#### Installation
Expand All @@ -112,11 +112,11 @@ cp ./.env.example ./.env
```

#### 2. Setup API keys
Open the `.env` file and fill in the `keys` and `regions`. This is also where you can choose to use any different `extensions`:
Open the `.env` file and fill in the `keys` and `regions`.
```bash
# Agora App ID and Agora App Certificate
# Agora App ID
# Agora App Certificate(only required if enabled in the Agora Console)
AGORA_APP_ID=
# Leave empty unless you have enabled the certificate within the Agora account.
AGORA_APP_CERTIFICATE=

# Azure STT key and region
Expand All @@ -140,7 +140,6 @@ docker compose up
#### 4. Enter container and build agent
Open up a separate terminal window, enter the container and build the agent:
```bash

docker exec -it ten_agent_dev bash

make build
Expand All @@ -155,7 +154,10 @@ make run-server
### Finish and verify

#### TEN Agent
Open up [localhost:3000]( http://localhost:3000 ) in browser to play and test the TEN Agent.
Open up [localhost:3000]( http://localhost:3000 ) in browser to play the TEN Agent.

#### TEN Agent with OpenAI Realtime API
Open up [localhost:3002]( http://localhost:3002 ) in browser to play the TEN Agent with OpenAI Realtime API.

#### TEN Graph Designer

Expand Down
13 changes: 13 additions & 0 deletions agents/manifest-lock.json
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,19 @@
{
"version": 1,
"packages": [
{
"type": "extension",
"name": "default_extension_python",
"version": "0.2.0",
"hash": "e9a45737eb95cd9deca485f6d4274b46533a8346d59edf1e1b7c8cd29754a15b",
"dependencies": [
{
"type": "system",
"name": "ten_runtime_python"
}
],
"supports": []
},
{
"type": "system",
"name": "ten_runtime_go",
Expand Down
228 changes: 223 additions & 5 deletions agents/property.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,224 @@
"_ten": {
"log_level": 3,
"predefined_graphs": [
{
"name": "va.openai.azure.fashionai",
"auto_start": true,
"connections": [
{
"app": "localhost",
"data": [
{
"dest": [
{
"app": "localhost",
"extension": "interrupt_detector",
"extension_group": "default"
},
{
"app": "localhost",
"extension": "openai_chatgpt",
"extension_group": "chatgpt"
},
{
"app": "localhost",
"extension": "message_collector",
"extension_group": "transcriber"
}
],
"name": "text_data"
}
],
"extension": "agora_rtc",
"extension_group": "default"
},
{
"app": "localhost",
"cmd": [
{
"dest": [
{
"app": "localhost",
"extension": "azure_tts",
"extension_group": "tts"
},
{
"app": "localhost",
"extension": "fashionai",
"extension_group": "default"
}
],
"name": "flush"
}
],
"data": [
{
"dest": [
{
"app": "localhost",
"extension": "message_collector",
"extension_group": "transcriber"
},
{
"app": "localhost",
"extension": "fashionai",
"extension_group": "default"
}
],
"name": "text_data"
}
],
"extension": "openai_chatgpt",
"extension_group": "chatgpt"
},
{
"app": "localhost",
"audio_frame": [
{
"dest": [
{
"app": "localhost",
"extension": "agora_rtc",
"extension_group": "default"
}
],
"name": "pcm_frame"
}
],
"cmd": [
{
"dest": [
{
"app": "localhost",
"extension": "agora_rtc",
"extension_group": "default"
}
],
"name": "flush"
}
],
"extension": "azure_tts",
"extension_group": "tts"
},
{
"app": "localhost",
"data": [
{
"dest": [
{
"app": "localhost",
"extension": "agora_rtc",
"extension_group": "default"
}
],
"name": "data"
}
],
"extension": "message_collector",
"extension_group": "transcriber"
},
{
"app": "localhost",
"cmd": [
{
"dest": [
{
"app": "localhost",
"extension": "openai_chatgpt",
"extension_group": "chatgpt"
}
],
"name": "flush"
}
],
"extension": "interrupt_detector",
"extension_group": "default"
}
],
"nodes": [
{
"addon": "agora_rtc",
"app": "localhost",
"extension_group": "default",
"name": "agora_rtc",
"property": {
"agora_asr_language": "en-US",
"agora_asr_session_control_file_path": "session_control.conf",
"agora_asr_vendor_key": "$AZURE_STT_KEY",
"agora_asr_vendor_name": "microsoft",
"agora_asr_vendor_region": "$AZURE_STT_REGION",
"app_id": "$AGORA_APP_ID",
"channel": "astra_agents_test",
"enable_agora_asr": true,
"publish_audio": true,
"publish_data": true,
"remote_stream_id": 123,
"stream_id": 1234,
"subscribe_audio": true,
"token": "<agora_token>"
},
"type": "extension"
},
{
"addon": "interrupt_detector",
"app": "localhost",
"extension_group": "default",
"name": "interrupt_detector",
"type": "extension"
},
{
"addon": "openai_chatgpt",
"app": "localhost",
"extension_group": "chatgpt",
"name": "openai_chatgpt",
"property": {
"api_key": "$OPENAI_API_KEY",
"base_url": "",
"frequency_penalty": 0.9,
"greeting": "TEN Agent connected. How can I help you today?",
"max_memory_length": 10,
"max_tokens": 512,
"model": "gpt-4o-mini",
"prompt": "",
"proxy_url": "$OPENAI_PROXY_URL"
},
"type": "extension"
},
{
"addon": "azure_tts",
"app": "localhost",
"extension_group": "tts",
"name": "azure_tts",
"property": {
"azure_subscription_key": "$AZURE_TTS_KEY",
"azure_subscription_region": "$AZURE_TTS_REGION",
"azure_synthesis_voice_name": "en-US-JaneNeural"
},
"type": "extension"
},
{
"addon": "message_collector",
"app": "localhost",
"extension_group": "transcriber",
"name": "message_collector",
"type": "extension"
},
{
"addon": "fashionai",
"app": "localhost",
"extension_group": "default",
"name": "fashionai",
"property": {
"app_id": "$AGORA_APP_ID",
"channel": "astra_agents_test",
"stream_id": 12345,
"token": "",
"service_id": "agora"
},
"type": "extension"
}
]
},
{
"name": "va.openai.azure",
"auto_start": false,
Expand Down Expand Up @@ -59,7 +277,7 @@
"property": {
"azure_subscription_key": "${env:AZURE_TTS_KEY}",
"azure_subscription_region": "${env:AZURE_TTS_REGION}",
"azure_synthesis_voice_name": "en-US-JaneNeural"
"azure_synthesis_voice_name": "en-US-AndrewMultilingualNeural"
}
},
{
Expand Down Expand Up @@ -456,7 +674,7 @@
"property": {
"azure_subscription_key": "${env:AZURE_TTS_KEY}",
"azure_subscription_region": "${env:AZURE_TTS_REGION}",
"azure_synthesis_voice_name": "en-US-JaneNeural"
"azure_synthesis_voice_name": "en-US-AndrewMultilingualNeural"
}
},
{
Expand Down Expand Up @@ -1452,7 +1670,7 @@
"property": {
"azure_subscription_key": "${env:AZURE_TTS_KEY}",
"azure_subscription_region": "${env:AZURE_TTS_REGION}",
"azure_synthesis_voice_name": "en-US-JaneNeural"
"azure_synthesis_voice_name": "en-US-AndrewMultilingualNeural"
}
},
{
Expand Down Expand Up @@ -1682,7 +1900,7 @@
"property": {
"azure_subscription_key": "${env:AZURE_TTS_KEY}",
"azure_subscription_region": "${env:AZURE_TTS_REGION}",
"azure_synthesis_voice_name": "en-US-JaneNeural"
"azure_synthesis_voice_name": "en-US-AndrewMultilingualNeural"
}
},
{
Expand Down Expand Up @@ -1883,7 +2101,7 @@
"property": {
"azure_subscription_key": "${env:AZURE_TTS_KEY}",
"azure_subscription_region": "${env:AZURE_TTS_REGION}",
"azure_synthesis_voice_name": "en-US-JaneNeural"
"azure_synthesis_voice_name": "en-US-AndrewMultilingualNeural"
}
},
{
Expand Down
22 changes: 22 additions & 0 deletions agents/ten_packages/extension/fashionai/BUILD.gn
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#
#
# Agora Real Time Engagement
# Created by Wei Hu in 2022-11.
# Copyright (c) 2024 Agora IO. All rights reserved.
#
#
import("//build/feature/ten_package.gni")

ten_package("fashionai") {
package_kind = "extension"

resources = [
"__init__.py",
"manifest.json",
"property.json",
"src/__init__.py",
"src/addon.py",
"src/extension.py",
"src/log.py",
]
}
Loading

0 comments on commit 5149dc6

Please sign in to comment.