AIToolScan

Speech To Speech Overview

Speech To Speech by Hugging Face is a low-latency, fully modular voice-agent pipeline - VAD -> STT -> LLM -> TTS - exposed through an OpenAI Realtime-compatible WebSocket API. Every component is swappable, and the LLM slot speaks OpenAI-compatible protocols, so you can point it at a hosted provider, at HF Inference Providers, or at a vLLM or llama.cpp server on your own hardware for a fully local, fully open stack. This pipeline runs in production as the conversation backend for thousands of Reachy Mini robots.

Key Features

• Voice Activity Detection (VAD): Silero VAD v5 detects speech boundaries and turn-taking.
• Speech to Text (STT): Transcribes the user's turn with optional live partial transcripts - Parakeet TDT (default), Whisper, Faster Whisper, Lightning Whisper MLX, and Paraformer.
• Language Model (LLM): Generates the response, streaming text and tool calls - OpenAI-compatible Responses/Chat Completions APIs, Transformers, or mlx-lm on Apple Silicon.
• Text to Speech (TTS): Synthesizes audio and streams it back - Qwen3-TTS (default), Kokoro-82M, Pocket TTS, ChatTTS, and MMS TTS.
• OpenAI Realtime API: WebSocket and WebRTC support with live transcription, server VAD, interruption handling, and low-latency turn-taking.
• Fully Local: Serve the LLM with llama.cpp or vLLM and run every component on your own hardware for a 100% local, 100% open voice stack.
• Multi-Language: Parakeet TDT covers 25 European languages; Whisper and Qwen3-TTS add broad multilingual coverage with automatic language detection.

Run Modes

The pipeline supports four run modes: realtime (OpenAI Realtime protocol over WebSocket or WebRTC), local (direct microphone and speaker interaction), raw-websocket (raw PCM over WebSocket for minimal custom clients), and socket (raw PCM over TCP for remote server deployments).

Installation

Install with pip install speech-to-speech, then run speech-to-speech to start an OpenAI Realtime-compatible server at ws://localhost:8765/v1/realtime. Optional backends are installed with extras such as speech-to-speech[kokoro], speech-to-speech[pocket], speech-to-speech[chattts], and speech-to-speech[faster-whisper].