O/T - Mind-blowing AI - Could Be Applied to 3D Figures?
Fauvist
Posts: 2,152
in The Commons
Mind-blowing AI - Could Be Applied to 3D Figures?
https://youtu.be/mPU0WNUzsBo
Daz 3D is part of 
Connect
DAZ Productions, Inc.
7533 S Center View Ct #4664
West Jordan, UT 84084
Licensing Agreement | Terms of Service | Privacy Policy | EULA
© 2025 Daz Productions Inc. All Rights Reserved.
Comments
Just looks like a live deepfake to me, with all of its imperfections. The ability to deepfake a real face onto a 3D figure has existed for years, and is publicly available.
Is this deepfake 3D and moves as the human and matches the performance in real time? Could you tell me the name of the software?
deepfacelab is the pioneering deepfake program (and, I believe, where the term deepfake comes from). It uses machine learning to analyze footage/images of two faces to learn how to put one onto the other. That performance shows you the results of days or weeks of training the computer, and unless they've come up with some really amazing technological advancement, it's not something you'd be able to do on the fly.
Corridor Crew has a number of videos about deepfakes that are really educational about what deepfakes are, how they work, and their limitations.
You mean the singer performing on the stage wasn't spontaneously, in real time, being transformed into the video of Cowell singing on the big screen? That the singer was lipsyncing to something that was pre-created, and that weird camera was being used to essentially obsure the view of the live guy on stage?
The singer was singing, the camera was filming, and the screen was playing was playing the extrapolated and colated video with Cowell's face on it.
And, yes, it could be applied to 3D figures to various degrees of success.
there are iOS and Android apps that do this already
Here is the website as mentioned in the video.. https://metaphysic.ai/ If you look at what they have been able to do with regards to deepfake tech, it is quite impressive.. Here is another of what they have been able to do..
While it still has some uncanny valley elements to the video, it is impressive all the same, considering the age of the original footage..
Wow. I hope they do this with Billie Holiday too.
The last time I looked, the open source deepfake worked in 2d only, and it didn't do anything remotely in real-time. It took days and hours to process thousands of pictures.
This might be tangential, but I have sometimes seen those "AI creates likenesses of book characters" and think to myself that this figure or that figure would make for nice entries in my collection. lol
When I said it couldn't be done on the fly, what I meant is that the computer had been specifically trained on the singer's face and on Cowell's face. Once the training is done, the deepfake can be done in real-time, but you couldn't swap out a different performer or a different face and achieve the same effect.
They had to optimize this for Simon's face specifically, in order to do it. They did all the work before they went on stage. This will be a common technology soon. But its biggest fault is it only works well under certain conditions. They had a nice camera pointed straight at his face, and he never put his hands in his face or anything like that. He also never turned to the side. He stayed on target.
Actually, when I saw your thread title I thought you were talking about Metahuman newest feature. It makes it possible to use photogrammetry to more quickly create rigged and working 3D people without needing to manually rig the whole thing yourself. It still has its own issues. The examples I have seen had decent mesh results but the textures were lacking and killed the likeness IMO. But this is only the first iteration of this new feature, I would assume this is going to get more refined and improved with time, especially with how fast Epic is moving and shaking in this industry.
It works a lot like Facegen, just with way more photos. The more photos you take, the better. You could use a few photos, but it would be terrible. You need a lot of photos and they need to be all in the same setting.
On top of that, they also chose a performer who had a similar face shape to Simon, because that too can reveal the limitations of deepfakes. Since he never turned to the side, that also means that differences in the nose shape, for example, wouldn't cause problems.
Yeah, but I've never seen it done in real time before.
Does this mean they can extract the millions of movie film frames of Marilyn Monroe, and feed them into a computer and have the computer reconstruct a 3D duplicate of Monroe that can be animated by projecting it onto another performer - or is that a whole other kettle of fish?
It could be possible, they did something similar with Peter Cushing's character Grand Moff Tarkin in Starwars: Rogue One, so it would be possible just very complicated..
Adding a cutout mask so one can Overlay a different face onto another persons head is certainly within the realms of the user level PC desktop with a decent GPU..Especially for video. Still images are easier to spot as fakes because of improperly scale or alignment of the overlay.
Plenty of examples of this already exist on the internet.Just google for "celebrity deepfakes".Should find 10's of thousands of them. Some better than others.Obviously; with the time, appropriate hardware, and sufficient talent, some can get very good.
Here's a website that has a few links to apps to do it yourself on mobile phones...
https://www.dailydot.com/debug/impressions-deepfake-app/
Here's another discussing it...
https://www.esquire.com/it/news/attualita/a17246434/porno-falsi-celebrita/
There is a video on this one that adds a Nicholas Cage overlay on "Lois Lane" in a superman movie :) ( and other characters from other movies )
If one were going to process millions of frames from a film or video by hand it would become quite tedious and time consuming.Fortunalty, we have computers and software that can do it for us at incredible speed.
I think you're conflating a couple different things. Deep fakes are not 3D at all. Something like MetaHuman could certainly be used to generate a 3D a replica of Marilyn Monroe (or whomever else), and I don't think it even needs that much reference to work from. The Moff Tarkin and young Leia in Rogue One were entirely 3D (composited over live actors, if I remember correctly), though fans have gone in after the fact and deepfaked Carrie Fisher and Peter Cushing's faces over the CG versions in the movie to arguably better effect.
yeah they did that with ABBA too
The deepfake of Simon was OK, but at times it looked like a rubber mask of the type they put Russell Crowe in as an old John Nash in the movie 'A Beautiful Mind'.