Voice Cloning: Replicating Voices through AI

Voice cloning is an area rapidly emerging thanks to generative AI. It involves replicating a person’s vocal stylings, including pitch, timbre, rhythms, mannerisms, and unique pronunciations, through technology.

Meta Platforms’ Audiobox: A Free Voice Cloning Program

Meta Platforms, the parent company of Facebook, Instagram, WhatsApp, and Oculus VR, has released its own voice cloning program called Audiobox. Developed by researchers at the Facebook AI Research (FAIR) lab, Audiobox is described as a “new foundation research model for audio generation” built upon their previous work in this field.

“It can generate voices and sound effects using a combination of voice inputs and natural language text prompts — making it easy to create custom audio for a wide range of use cases,” reads the Audiobox webpage.

Users can simply type in a sentence or description of a sound they want to generate, and Audiobox will do the rest. It also allows users to record their own voice and have it cloned by Audiobox.

Self-Supervised Learning and Audiobox’s Family of Models

Meta created a “family of models” for Audiobox, one for speech mimicry and another for generating ambient sounds and sound effects. These models are all based on the shared self-supervised model Audiobox SSL.

Self-supervised learning (SSL) is a deep learning technique where AI algorithms generate their own labels for unlabeled data. This approach is used because labeled data may not always be available or of high quality.

The researchers of Audiobox published a scientific paper explaining their SSL approach, stating, “because labeled data are not always available or of high quality, and data scaling is the key to generalization, our strategy is to train this foundation model using audio without any supervision.”

To train Audiobox, FAIR researchers relied on a vast amount of data, including 160K hours of speech, 20K hours of music, and 6K hours of sound samples. This data spans various languages and acoustic conditions to ensure fairness and representation.

• The speech portion includes audiobooks, podcasts, read sentences, talks, conversations, and real-world recordings.
• The music portion covers a wide range of musical genres and styles.
• The sound samples include different environmental sounds and effects.

However, the research paper does not disclose the exact sources of this data, raising questions about potential copyright issues. AI companies have faced legal challenges for training on copyrighted material without proper consent.

Meta has also released interactive demos to showcase the capabilities of Audiobox. Users can record their voice and hear it replicated, or they can input text prompts to generate cloned voice recordings. The demos also allow users to generate new voices based on text descriptions or restyle existing voices.

It’s important to note that the Audiobox demos come with a disclaimer stating that they are restricted for non-commercial use and limited to states outside of Illinois and Texas, where laws prohibit certain audio collection activities.

While Audiobox is not open source like Meta’s previous releases, advancements in AI technology suggest that commercial versions of voice cloning programs will likely become available in the near future.

With the rapid progress of AI, it is expected that voice cloning technology will continue to evolve, offering new possibilities for various industries and individuals alike.