top of page
Search

How to Translate Videos with AI-Powered Video Dubbing to English or ANY Other Language

  • locallab
  • Jul 1, 2024
  • 3 min read

Forget those cringeworthy robotic dubs that plague translated videos – we're diving into the world of SoniTranslate, an AI-powered tool that delivers natural-sounding, perfectly lip-synced translations in over 50 languages!



SoniTranslate: The Universal Translator We've Always Dreamed Of

Imagine this: you're watching your favorite YouTuber from Japan, but your Japanese skills are, shall we say, a little rusty. Now, with SoniTranslate, you can watch the video perfectly dubbed in your native language, complete with seamless lip-sync and realistic AI-generated voiceovers. It's like magic, only better – it's cutting-edge AI technology!



Features That Will Blow Your Mind

SoniTranslate isn't just another run-of-the-mill translation tool. It's packed with features that make it stand out from the crowd:

  • AI-powered voice conversion: Say goodbye to robotic dubs! SoniTranslate leverages advanced AI models like Coqui and Piper TTS to generate incredibly natural-sounding voiceovers. 🗣️

  • Multilingual support: With over 50 languages supported, you can enjoy content from all corners of the globe in your preferred language. 🌎

  • Customizable subtitles: Embed subtitles, create your own, or burn them directly into the video for maximum flexibility. ✏️

  • CPU mode: While a dedicated Nvidia GPU is recommended for faster processing, SoniTranslate also offers a CPU mode, allowing you to use it even without a powerful graphics card. 💻


If you want to watch the tutorial instead, watch here.





Two Ways to Unlock the Power of SoniTranslate

Ready to give it a try? You have two options:

1. The Easy Way: Google Colab

  • Head over to the SoniTranslate GitHub repository (link in the comments).

  • Click on the Google Colab link.

  • Hit the "Play" button on each cell in the Colab notebook to run the code.

  • Accept the license agreement for the model used by visiting the provided links and logging in to your Hugging Face account.

  • Obtain your Hugging Face access token (go to your Profile > Settings > Access Tokens).

  • Paste the token into the designated field in the Colab notebook and run that final cell.

  • Click the generated Gradio share link to access the SoniTranslate web interface.


2. The Hands-On Way: Manual Installation

  • Install Miniconda: If you don't have it already, download and install Miniconda from the Anaconda website.

  • Create a Conda Environment: Open your Anaconda Prompt (or Terminal for Linux/Mac users) and run the following command:


    conda create -n soni-translate python=3.10 -y


  • Activate the Environment: Activate the newly created environment using:


    conda activate soni-translate


  • Clone the Repository: Clone the SoniTranslate repository using git clone https://github.com/R3gm/SoniTranslate.git.

  • Navigate to the Directory: Use cd SoniTranslate to enter the project directory.

  • Install Dependencies: Run the three pip install commands listed in the GitHub README file to install the necessary packages.

  • Install FFmpeg: Download and install FFmpeg from their website.

  • Set Your Hugging Face Token: Use the following PowerShell command, replacing "PASTE YOUR TOKEN HERE" with your actual token:


    $env:YOUR_HF_TOKEN="PASTE YOUR TOKEN HERE"


  • Run SoniTranslate: Execute the following command: python app_rvc.py. To use CPU mode, add --cpu_mode at the end of the command.

  • Access the Web Interface: Copy and paste the local host URL displayed in your terminal into your browser.



Unleash the Multilingual Magic

Once you have SoniTranslate up and running, it's time to translate your video!

  1. Upload Your Video: Drag and drop your video file into the designated area or use the "Click to Upload" option.

  2. Choose the Source and Target Languages: Select the original language of the video and the language you want it translated to from the dropdown menus.

  3. Specify Speaker Count: Indicate how many speakers are present in the video.

  4. Select TTS Voices: Choose the AI-generated voices you want to use for each speaker.

  5. Adjust Advanced Settings: (Optional) Tweak the maximum audio acceleration to around 1.4 for improved performance. You can also enable the "Burn Subtitles" option to embed translated subtitles directly into your video.

  6. Hit Translate: Click the "Translate" button and let the magic happen! ✨



The Future of Video Translation is Here

With its powerful AI capabilities and user-friendly interface, SoniTranslate is a game-changer for video content creators, language learners, and anyone who wants to experience the world of video in their own language. It's a testament to the transformative potential of AI in breaking down language barriers and making information accessible to all.

Comments

bottom of page