Deploying Cartesia State Space Models for Real-Time Voice AI on AWS

AWS highlighted how real-time voice AI partner Cartesia applies State Space Models (SSMs) to achieve low-latency conversational audio. Traditional transformer models incur quadratic computational growth over long sequences, whereas SSMs treat input sequences as continuous dynamical systems with constant memory footprints, making them suitable for streaming voice applications. Cartesia provides two core voice models: Sonic for text-to-speech, offering first-audio-byte generation in about 90ms across more than 44 languages, and Ink for streaming speech-to-text featuring context-aware end pointing and turn detection without separate voice activity detection models. To satisfy compliance and latency requirements, organizations can deploy Sonic within their own VPC using SageMaker AI endpoints subscribed via AWS Marketplace, or self-manage the container on EC2 GPU instances across Amazon ECS or EKS environments.

Key takeaways

• State Space Models maintain a constant memory footprint and linear recurrence to overcome the quadratic latency limitations of transformer-based audio processing.
• Sonic delivers text-to-speech with a first-audio-byte latency of roughly 90ms, voice cloning from 10-second samples, and coverage across over 44 languages.
• Ink provides streaming speech-to-text with native turn detection and semantic boundary recognition, enabling downstream LLMs to respond without waiting for turn completion.
• Deployments can be executed in an AWS VPC via SageMaker AI model packages from AWS Marketplace or self-hosted on EC2 GPU instances via ECS or EKS.