Voice assistants traditionally feel robotic and laggy due to the sequential nature of speech-to-text, LLM generation, and text-to-speech pipelines.
For the Santa Calling case study, launched for the 2023-2024 holiday season, any conversational lag over 1 second would break the magic and frustrate children. The challenge was building VoiceIQ as an AI-based real-time voice calling system that connects callers instantly to a custom-trained Santa.
The Challenge
Traditional voice systems require 2-4 seconds to respond
Maintaining a strict persona with consistent tone throughout long calls
Handling interruptions when children speak mid-sentence
Scaling to support thousands of concurrent calls on Christmas Eve
OUR SOLUTION
VoiceIQ Real-time Conversational Pipeline
Streaming Pipeline
We linked streaming transcription directly to token-by-token LLM generation, feeding chunked audio to TTS.
- WebSocket audio streaming
- Streaming LLM text generation
- Pre-emptive chunked TTS
Persona Customization
A robust prompt-engineering wrapper and vector storage engine ensure Santa remembers the child's details.
- Parent custom info injection
- Dynamic memory retrieval
- Holiday safety guardrails
Technical Latency Breakdown
Audio Capture
Audio is recorded on client and streamed in chunks using WebRTC.
ASR Processing
GPU-accelerated Whisper model performs streaming speech-to-text.
LLM Generation
GPT-4 Turbo streams answers. The TTS starts synthesizing before the sentence ends.
TTS Synthesis
ElevenLabs clone processes text into voice chunks, played instantly.