k2-fsa / OmniVoice

## Overview OmniVoice is an audio generation model from k2-fsa for zero-shot text-to-speech in 600+ languages. It uses a diffusion language model that iteratively fills masked acoustic tokens instead of speaking one frame at a time. That design keeps quality high while staying fast for long scripts. It outputs ready-to-use speech audio and removes the need to record a studio voice actor for every language. ## What you can build - Multilingual narration for videos, courses, and product demos - Cross-lingual voice cloning for localization with one reference speaker - Custom “voice personas” for agents, IVR menus, and in-app help - Audiobook-style long-form reading with consistent speaker identity - Expressive scripts with non-verbal cues like laughter or sighs ## Inputs - The text you want the model to speak, as plain text. You can include supported non-verbal tags like brackets around cues. - An optional output language choice. You can let the model auto-detect, or pick a specific language for better alignment. - An optional reference audio clip for voice cloning, uploaded as an audio file such as WAV, MP3, FLAC, or OPUS. Short clips work best. - An optional transcript of the reference audio, as plain text. If you don’t provide it, the model can auto-transcribe the reference clip. - An optional style description for voice design, written in natural language (for example: gender, age, pitch, accent, whisper). This is ignored if you provide a reference clip. - An optional quality vs speed control that changes how many iterative decoding steps the model runs. Fewer steps run faster with lower fidelity. - An optional text-adherence control (guidance). Higher values follow the text more strongly, but can add artifacts. - An optional fixed output duration in seconds (0 to 60). If set, the model adjusts pacing to fit exactly. - An optional speaking speed multiplier (0.5× to 2.0×). This is ignored when a fixed duration is set. ## Outputs - A single speech audio file in WAV format. - The audio is mono and sampled at 24,000 Hz. - The content is the synthesized speech for your full input text, using either: - an automatically chosen voice, - a cloned voice from your reference clip, or - a designed voice from your style description. ## Recommended settings - General text-to-speech: keep the default decoding steps and guidance, and leave duration unset. - Faster drafts: reduce decoding steps to around 16. - Voice cloning: use a clean 3 to 10 second reference clip and provide an accurate transcript when you can. - Cross-lingual cloning: expect the output accent to carry over from the reference clip’s language. - Fixed timing (ads, reels, timed slides): set a target duration, and don’t set speed. ## Limitations - Cross-lingual cloning can keep the reference accent even when you switch languages. - Voice design attributes are trained mainly on English and Chinese. Results can vary in low-resource languages. - Some languages have special input rules. For example, Hokkien output needs Tai-lo romanization, not Chinese characters. - Noisy, clipped, or heavily reverberant reference audio can reduce speaker similarity. - Bad transcripts hurt cloning. Auto-transcription can mishear names, numbers, and jargon. - Unstructured text can sound unnatural. Long number strings and missing punctuation often need normalization. ## Safety & compliance - Get explicit permission before cloning a real person’s voice. - Don’t use the model for impersonation, fraud, or deceptive content. - Disclose AI-generated speech when your users could reasonably assume it is human. - Check rights for any reference recordings you upload, including voices, music, and background audio.