Add RTZR(ReturnZero) STT Plugin for LiveKit Agents

[kimdwkimdw]

RTZR(ReturnZero) STT Plugin for LiveKit Agents

This PR introduces a new ReturnZero (RTZR) plugin for the LiveKit Agents framework, enabling real-time speech recognition through the RTZR Open API.

ReturnZero is a leading speech AI provider in Korea, widely recognized for its superior Korean STT accuracy, as highlighted in Awesome Korean Speech Recognition. This integration allows developers to build high-quality, real-time Korean conversational agents on top of LiveKit.

Components

• stt.py – Primary implementation of the RTZR STT class, built on RTZROpenAPIClient. Handles WebSocket connection management, PCM (LINEAR16) audio streaming, endpoint detection, and transcript event emission.
• SpeechStream – Streaming recognition pipeline with interim/final transcript events, error handling, and server-side endpoint detection support.

Example Usage

We adapted the simplest voice AI example to use RTZR STT instead of Azure:

from dotenv import dotenv_values, load_dotenv

from livekit import agents
from livekit.agents import Agent, AgentSession, RoomInputOptions
from livekit.plugins import (
    noise_cancellation,
    openai,
    rtzr,
    silero,
)
from livekit.plugins.turn_detector.multilingual import MultilingualModel

load_dotenv(".env.local")
config = dotenv_values(".env.local")


class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(instructions="You are a helpful voice AI assistant.")


async def entrypoint(ctx: agents.JobContext):
    session = AgentSession(
        llm=openai.LLM.with_azure(
            model="gpt-4.1-mini",
            azure_deployment=config["AZURE_LLM_DEPLOYMENT"],
            azure_endpoint=config["AZURE_OPENAI_ENDPOINT"],
            api_version="2024-12-01-preview",
        ),
        stt=rtzr.STT(model="sommers_ko", language="ko"),
        tts=openai.TTS.with_azure(
            model="gpt-4o-mini-tts",
            voice="ash",
            azure_deployment=config["AZURE_TTS_DEPLOYMENT"],
            azure_endpoint=config["AZURE_OPENAI_ENDPOINT"],
            api_version="2025-03-01-preview",
        ),
        vad=silero.VAD.load(),
        turn_detection=MultilingualModel(),
    )

    await session.start(
        room=ctx.room,
        agent=Assistant(),
        room_input_options=RoomInputOptions(
            noise_cancellation=noise_cancellation.BVC(),
        ),
    )

    await session.generate_reply(instructions="Greet the user and offer your assistance.")


if __name__ == "__main__":
    agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint))

API Authentication

To use RTZR STT, you need to obtain client credentials from ReturnZero. Set them in your environment:

export RTZR_CLIENT_ID="your_client_id_here"
export RTZR_CLIENT_SECRET="your_client_secret_here"

Make sure to also include them in your .env.local file for local development.

Manual Tests Done

• Verified connection establishment with RTZR WebSocket API.
• Confirmed interim and final transcript events for Korean speech.
• Confirmed EOS handling and proper cleanup after session end.
• Compared results with Azure STT baseline – RTZR showed higher accuracy in Korean speech transcription, particularly in colloquial and conversational contexts.

References

• RTZR Developer Docs
• Awesome Korean Speech Recognition

[CLAassistant]

All committers have signed the CLA.

[kimdwkimdw]

@longcw @theomonnom Could you review this PR?

[longcw]

lgtm! something nit

[kimdwkimdw]

@longcw simplified logger with @utils.log_exceptions(logger=logger)