Apparate Labs launches PROTEUS, a new real-time AI video generation model

A few days ago, a research team from Stanford University and Apparate Labs released an innovative AI video generation model called Proteus, which can generate highly realistic and expressive virtual characters that laugh, rap, sing, blink, smile, and speak from a single image.

It can generate highly realistic and expressive virtual characters based on a single photo and voice, and these characters can show complex facial expressions and body movements in real-time videos. That is, it can accept voice input in real time to generate video streams, allowing users to interact with virtual characters through voice control.

PROTEUS is compatible with multimodal inputs of multiple large models and can process data in multiple forms such as voice, text, and images.

Proteus model highlights:

– Highly realistic: Using advanced AI technology, Proteus can generate virtual characters with highly realistic expressions and smooth movements from a single picture.

– Low latency generation: The Proteus model has low latency characteristics, can respond to voice input in real time, and generate video streams with more than 100 frames per second.

– Multimodal interaction: The model supports multiple input forms such as voice, text, and images to achieve natural and intuitive interaction with users.

Application areas:

The Proteus model has wide application potential in personalized virtual assistants, virtual pets, emotional support, customer service, education and training, game customization, film and television entertainment, marketing and social media.

Proteus has just been released, and a group of bigwigs sent “congratulatory letters”:

AI scientist Jia Yangqing praised that the quality of real-time artificial intelligence avatars is surprisingly good.

Nvidia scientist Jim Fan said that this project is impressive.

Early investor Brian Zhan wrote that the biggest problem with existing AI video tools, such as Runway and Pika, is that they will produce hallucinations, especially when generating videos containing humans. Apparate Labs has taken AI video generation to the next stage by solving problems such as temporal coherence and object constancy.

Outrageous! Lu Xun talked about tongue twisters

Proteus is a new generation of basic models for real-time expression generation of humans.

You know, even the most advanced and powerful generation models cannot fully achieve real-time generation of human expressions.

Existing models run slowly, fail to provide intuitive control over complex facial expressions and body movements of generated characters, and they still lack realism and expressiveness.

Proteus uses the state-of-the-art transformer architecture’s latent diffusion model. Its innovative latent space design ensures real-time high efficiency, and with the continuous optimization of architecture and algorithms, Proteus is able to achieve video streaming of more than 100 frames per second (100+ FPS).

In other words, with just a simple photo, Proteus can not only imitate human laughter, rap, singing, blinking, smiling, and conversation, but also perform more vivid expressions and movements.

According to the Proteus R&D team, they expect Proteus to become a voice-controllable visual avatar, providing an intuitive interactive interface for AI conversation entities, while being seamlessly compatible with many multimodal large language models, and providing customized services for various application scenarios.

Many netizens had a lot of ideas about this:

“You only need to fine-tune the large language model with Einstein’s data and add his vivid facial expressions to make the great Einstein a teaching assistant to teach physics in person. Teenagers no longer have to worry about not being able to learn science well.”

Another netizen said, “I love it so much. This year is definitely the year of AI video.”