Speaker And Style Disentanglement Of Speech Based On Contrastive Predictive Coding Supported Factorized Variational Autoencoder

Abstract:

Speech signals encompass various information across multiple levels including content, speaker, and style. Disentanglement of these information, although challenging, is important for applications such as voice conversion. The contrastive predictive coding supported factorized variational autoencoder achieves an unsupervised disentanglement of a speech signal into speaker and content information by assuming speaker information to be temporally more stable than content-induced variations. Thus, a single utterance-level embedding vector is used to represent speaker information. This utterance-level feature, however, may consist of not only speaker information but also other temporally stable non-content information such as environment or emotion, which we call style. In this work, we propose a method to further disentangle non-content features into distinct speaker and style features, notably by leveraging readily accessible and well-defined speaker class labels without the necessity for style labels. Experimental results validate the proposed method effectiveness on extracting disentangled features, thereby facilitating speaker, style, or combined speaker-style conversion.

Note:

The following conversions may contain three original utterances from VOiCES devkit or LibriSpeech test set.
We use content utterance, speaker utterance and style utterance to denote the following:
Content utterance: The utterance from which the content should be preserved after conversion.
Speaker utterance: Utterance which provides the speaker identity for the conversion.
Style utterance: The utterance only influences the style of the conversion result.

Clean Voice Conversion:

The following demos are clean voice conversion.
Content utterances, speaker utterances and style utterances are all from LibriSpeech test set.
Content utterances and style utterances are the same, and only speaker utterances are changed during conversion.

Only Change Style

The following demos only change style embeddings.
Content utterances, speaker utterances are the same, from VOiCES_devkit test set.
Style utterances are from LibriSpeech test set.
'Clean content utterance' in below are the clean utterances without reverberation from VOiCES.

Change Style and Speaker

The following demos change style and speaker embeddings simultaneously.
Content utterances are from VOiCES_devkit test set.
Style utterances and speaker utterances are from LibriSpeech test set.