Not the same job as speech synthesis

RVC stands for Retrieval-based-Voice-Conversion-WebUI, and the project describes itself as a simple and easy to use framework for voice timbre conversion and voice changing. What it does differs from speech synthesis: it does not generate speech from text, it takes a recording that already exists and replaces the timbre with another one.

Tools in this category used to carry a high barrier. RVC reduces it to two numbers: at least 10 minutes of speech data to train on, and 170 milliseconds of end to end latency for live voice changing.

The input is the voice itself, not a script. Hand it a recording and it returns the same content in the timbre of a target model. What changes is who is speaking, not what is being said.

The first feature in the project's own list is a top1 retrieval step that replaces source features with features drawn from the training set, which is how timbre leakage is prevented. The retrieved features come from the training data itself, so the output stays close to the target voice instead of carrying the accent and recording quality of the input along with it.

Training on ten minutes

The recommendation is to collect at least 10 minutes of low noise speech, and the project notes that a small amount of data still produces decent results. That scale means no studio is needed; a quiet indoor recording is basically enough.

The hardware demand is modest in the same way. The project states that training is fast even on a relatively weak graphics card. The base model was trained on the open, high quality VCTK corpus, close to 50 hours of it, and the project says there are no copyright concerns around it. A ckpt-merge option in the checkpoint tab allows blending models to shift the timbre further.

Installing: three dependency tracks

This branch targets 64-bit Python 3.12, with Ubuntu 24.04 recommended on that side. Dependencies come in three tracks. CPU, AMD and Intel graphics share one file. Nvidia splits by generation: RTX 50 series installs the CUDA 12.8 build of Torch first, anything before RTX 50 installs the CUDA 11.8 build, and the matching dependency file follows. The order matters, Torch before the dependency file. One command then prints the Torch and CUDA versions, so the environment is confirmed before going further.

AMD and Intel cards take the CPU dependency track: DirectML on Windows, plain CPU on Linux. It runs on non Nvidia platforms, but the compute gap has to be accepted.

Models have to sit in the right place

The web interface creates its working directory automatically, but the models are yours to download and place. The project requires a fixed layout: hubert_base for feature extraction, the rmvpe weights for pitch, pretrained and pretrained_v2 for training, pymss_weights for vocal separation, weights for user models and indices for the index files.

Downloading goes through a command line tool as well, grouped by purpose. One group is required for inference and feature extraction, another is required for v1 and v2 training, a third for vocal separation is optional, and a DirectML environment on Windows needs an extra pitch weight in onnx format. On Windows the two FFmpeg executables have to be dropped into the project root by hand.

Live voice changing and latency

Two entry points are provided, the web interface and a separate live voice changing interface. Latency is the number worth studying here: the project reports 170 milliseconds end to end, and 90 milliseconds with ASIO input and output devices, though it admits that this depends heavily on hardware driver support.

Pitch extraction is a key link in reaching that figure. The project uses the RMVPE algorithm from InterSpeech2023, described as eliminating the muted voice problem while being fast and light on resources. The interface can also switch back to pm, harvest or crepe.

Starting it and the licence

Startup is one command and the default port is 7865. An Ubuntu server without a desktop environment can pass a flag to skip opening a browser automatically. User models go in the weights directory and indices in the indices directory.

The licence is MIT, and the base model rests on the open VCTK dataset, which the project says raises no copyright concerns. One thing remains your own responsibility: voice conversion works on someone else's voice, so before publishing the result, confirm that permission exists.