Actual-time video AI developer Decart says it’s primed to remodel video advertising with the discharge of Lucy 2, an revolutionary new mannequin that’s in a position to seamlessly edit longform dwell streams by way of pure language prompts.
The open-source mannequin not solely generates high-quality video, however supplies the instruments wanted to dynamically edit that content material on the fly with out compromising its realism. It eliminates the necessity for AI video to be edited post-production, opening the door to new advertising makes use of in dwell streaming, digital try-ons, personalised product placement and extra, the corporate claims in a press launch.
Extra coherent edits and immediate adherence
What units Lucy 2 aside from competing video era fashions like Google’s Veo 3 and OpenAI’s Sora 3 is its skill to course of incoming video instantaneously with sub-second latency. Customers will be capable to apply stylistic modifications, manipulate objects and characters, modify the surroundings and extra – whereas the video is being generated.
Decart defined in a weblog put up that Lucy 2 integrates diffusion fashions which were optimised for temporal consistency and low latency to enhance the general coherence of its movies. It may possibly ingest dwell video from a number of sources, like a smartphone digicam, internet cam, or RTMP stream and apply AI processing to that footage in real-time to remodel it in an infinite variety of methods. The mannequin does this whereas operating at 1080p and 30 fps, producing fast outcomes with none interruptions to the dwell video stream or lack of high quality.
Creators will be capable to experiment with Lucy 2 in varied methods, utilizing easy prompts to “change the background with New York Metropolis” and make it look as in the event that they’re livestreaming from a wholly totally different location. Or, else they may command Lucy 2 to remodel the topic right into a cartoon character and utterly change the looks of the particular person on movie instantly.
In response to Decart, Lucy 2’s real-time efficiency advantages from a number of tweaks the corporate has made, like optimising it to run on Nvidia’s most superior GPUs and mannequin distillation methods.
Decart additionally talked in regards to the architectural modifications utilized to Lucy 2, which make it work very in another way from conventional video fashions. Notably, it integrates a diffusion pipeline that helps it to realize an unprecedented stage of coherence in frames. It outputs video repeatedly on a frame-by-frame foundation with out time constraints to remove the bizarre artefacts and flickering results that plagued earlier video fashions and obtain higher consistency in scenes.
The mannequin additionally advantages from enhanced immediate adherence, which permits it to interpret the person’s directions extra precisely. This implies creators will be capable to feed it with extra complicated descriptions. For example, somebody may say “add pink, blue and inexperienced fire-breathing dragons with out affecting the unique lighting,” and Lucy 2 will achieve this with out making errors.
A magic impact on advertising
Benchmark checks present that Lucy 2’s latency measures lower than 100 milliseconds per body, making it viable for livestreaming utility, and this will probably be of massive curiosity to entrepreneurs already utilizing AI-generated video.
For example, it has implications for livestreaming collaborations with social media influencers and different content material creators. Utilizing Decart’s Delulu Stream instruments, which at the moment are powered by Lucy 2, livestreamers on platforms like Twitch and TikTok can improve their video streams in varied methods, like by dynamically adjusting their look to appear like any character in any world.
Alternatively, streamers can apply thematic filters which might be extra attentive to real-time chat instructions. And, they’ll be capable to instantaneously insert manufacturers’ merchandise into their movies at any second to showcase alternative ways and conditions through which they will use them.
As a substitute of counting on recognised influencers, manufacturers can use Lucy 2 to create a digital ambassador that represents them completely. They’d be capable to construct complete “lives” for these digital ambassadors, displaying them in varied elements of the world, doing totally different actions to construct extra emotional connections with customers.
One benefit of digital model ambassadors is that they don’t have to sleep – working across the clock. Manufacturers may even livestream repeatedly, 24 hours a day, answering queries and offering recommendations on the best way to use merchandise with none fatigue.
Decart’s management additionally sees numerous potential for Lucy 2 on the subject of letting extra real looking digital try-ons for ecommerce. Some digital shops already make use of augmented actuality expertise to assist customers visualise how they may look whereas sporting a brand new jacket, or whether or not or not a brand new couch goes to match the decor of their dwelling, however Lucy 2 will deliver the realism of those experiences to a different stage.
Due to Lucy 2’s superior coherence, digital try-ons will probably be a lot much less distinguishable from truly attempting on a brand new shirt or pair of denims in a bodily retailer, they usually’ll be extra dynamic too. A client couldn’t solely strive on a pair of latest Levi’s, but in addition visualise themselves strolling round in them in a park or by a shopping center. Lucy 2 will mannequin the particular person’s movement precisely whereas synthesising how the material ought to react to motion to make it look as in the event that they’re actually sporting it.
Another choice is dwell product placement. As a result of Lucy 2 can edit livestreams on-the-fly and personalise them for every viewer, manufacturers will be capable to insert totally different merchandise in the identical video for thousands and thousands of various viewers, based mostly on every viewers member’s previous engagement with product pages.
A brand new period for world fashions
Decart’s CEO Dean Leitersdorf has described the launch of Lucy 2 because the “GPT-3 second for world fashions.” He mentioned it’s the primary occasion through which a world mannequin can now run dwell, in real-time, with none compromise by way of video high quality. “The shift doesn’t simply enhance video, it creates totally new markets, from dwell media and leisure to digital try-on, gaming and robotics,” he mentioned. “We’re assured these markets will probably be measured in billions.”
As an additional advantage, Lucy 2 can also be extraordinarily versatile and reasonably priced. Decart mentioned the mannequin has been optimised to run on Nvidia GPUs, and on Amazon and Google’s cloud infrastructure, giving clients a number of deployment choices.
Power-efficiency optimisations promise to cut back the price of AI-generated video dramatically, with Decart saying sustained real-time video era will price round $3 per hour.
