Vevo: Controllable Zero-Shot Voice Imitation with
Self-Supervised Disentanglement

What can Vevo do?

Vevo can take either speech or text as input, and perform zero-shot imitation with controllable linguistic content (controlled by the source), style (controlled by the style reference), and timbre (controlled by the timbre reference) in a single forward pass. This inference pipeline can be adjusted for various zero-shot imitation tasks.

Given a source speech (\( \textcolor{blue}{U_i} \)) or text (\( \textcolor{blue}{T_i} \)), and a reference speech (\( \textcolor{red}{U_r} \)), we propose the following variants of Vevo according to the specific imitation task:

We will showcase the diverse capabilities of Vevo through the following examples.

Timbre Imitation and Voice Imitation (Conversion Task)

Vevo-Timbre v.s. Vevo-Voice

Vevo-Timbre and Vevo-Voice can both convert the speaker characteristics of speech. The key difference between them is that Vevo-Timbre only imitates the timbre of the reference speech, while effectively preserving the style of the source speech (e.g., prosody, emotion, accent). On the other hand, Vevo-Voice imitates both the timbre and style of the reference speech, resulting in higher speaker similarity to the reference. We will demonstrate the characteristics and distinctions between Vevo-Timbre and Vevo-Voice using samples from different domains.

For accented corpus, we observe that Vevo-Timbre retains the accent of the source speech, whereas Vevo-Voice mimics the accent of the reference speech:

For emotional corpus, we observe that Vevo-Timbre retains the emotion of the source speech, whereas Vevo-Voice mimics the emotion of the reference speech:

For general corpus, we observe that Vevo-Timbre maintains the prosody of the source speech, while Vevo-Voice can imitate the speech rate, stress, and rhythm of the reference speech:

Compared to Baselines

We select several baselines in zero-shot voice conversion, including HierSpeech++, LM-VC, UniAudio, and FACodec.

Style Imitation

We present the performance of Vevo-Style in the zero-shot style imitation task, focusing on widely studied styles such as accent and emotion. Notably, Vevo-Style employs a zero-shot manner (i.e., using just a few seconds of speech) to achieve style imitation, which is rarely seen in existing researches.

Accented Corpus

For accent imitation, we select baselines from the accent conversion field: ASR-AC, VoiceShop, and Conv-Speak. We use their demo website samples as our evaluation set. We also introduce a zero-shot style imitation model, Vevo-Style (ASR), for comparison. Its only difference compared to Vevo-Style is the use of ASR model rather than our proposed content tokenizer to extract content tokens (see Section 4.3 of our paper).

Emotional Corpus

For emotion imitation, we select a baseline from the emotion conversion field, Emovox. We use its demo website samples as our evaluation set.

Voice Imitation (Synthesis Task)

To evaluate the performance of Vevo-TTS in the zero-shot voice imitation (synthesis) task, we select the classic baselines of the zero-shot TTS filed, including the Non-AR model, Voicebox, and the AR models such as VALL-E and VoiceCraft, all of which are trained only on audiobook speech data. For comparison, we also include two stronger state-of-the-art models: CosyVoice and MaskGCT, which are trained on large-scale private corpus derived from in-the-wild video data, featuring highly diverse distributions.

Accented Corpus

Emotional Corpus

General Corpus

Ethics Statement

As with other powerful new AI innovations, we recognize this technology brings the potential for misuse and unintended harm. We will build a highly effective classifier that can distinguish between authentic speech and audio generated with Vevo to mitigate these possible future risks.