Captions Launches Lipdub App For Dubbing Videos In 28 Languages

Video editing startup Captions has unveiled a new app called Lipdub, offering translation and dubbing capabilities in 28 different languages. The app, powered by artificial intelligence (AI), allows users to easily translate video clips and change lip movements to match the selected target language. Lipdub is available for free on the App Store and supports a range of languages including French, Hindi, Spanish, Italian, Portuguese, Japanese, and more.

User-Friendly Features and Functionality

Lipdub provides users with a simple and intuitive interface to translate videos of up to one minute in length. The app enables translations for both single-person dialogue and multi-person conversations, making it versatile for different types of video content. Users can then share the dubbing videos directly on various social media platforms, expanding their reach to a wider audience.

A noteworthy feature of Lipdub is its ability to apply specific language interpretations, such as Texas slang, Gen Z lingo, pirate speech, and even baby talk. The app adjusts the lip movements to match the chosen language, ensuring a more immersive and authentic experience for viewers.

While Lipdub offers impressive dubbing capabilities, it is important to note that there may be occasional delays between audio playback and lip movements. Despite this minor limitation, the app provides a valuable and accessible tool for content creators and enthusiasts.

The Power of AI in Video Editing

Captions’ Lipdub app builds on the success of its eponymous video editing application, which has garnered over 3 million creators and boasts more than 100,000 daily users. The company’s AI-powered video editing features have gained popularity, offering functionalities such as removing “hums” and “ahs,” enhancing speech clarity, and reducing background noise. Additionally, the Captions app includes an “AI Lipdub” feature, which allows users to modify lip movements during post-production by simply changing the transcript.

Gaurav Misra, the founder of Captions and former head of design engineering at Snap, established the startup in 2021. Captions recently secured $25 million in a Series B funding round led by Kleiner Perkins, with participation from Sequoia Capital, Andreessen Horowitz (a16z), and SV Angel. To date, Captions has raised a total of $40 million in funding, demonstrating investor confidence in the company’s innovative approach to video editing.

Expanding Opportunities for Language Localization

The introduction of Lipdub aligns with the industry’s growing interest in utilizing AI and translation technologies to reach diverse audiences. In June, YouTube unveiled plans to test an AI-powered tool that automatically dubs videos in multiple languages, while also focusing on improving lip-syncing accuracy. YouTube Studio, the platform’s content management system, is being integrated with AI-powered dubbing functionalities to streamline the process of translating videos into different languages directly on the platform.

Moreover, other AI-powered dubbing startups, including ElevenLabs, have recently entered the market with dubbing solutions supporting up to 29 languages. These advancements in AI-driven voice generation and dubbing technology not only benefit content creators but also provide opportunities for dubbing service providers to offer their expertise to popular YouTubers and content creators.