Speech Recognition and Facial animation

Willy2

I bought Anilip2 to make my 3D characters speak.
This plugin works well, but the final result is far from perfect.
I notice that the lips move too mechanically.
Maybe by modifying the morphs, we could have a better result (which I have not tried)
https://youtu.be/yzqI6PrHn90

To make my characters speak in a more realistic way, I would like to try facial animation which is more complete, because we can add expressions.
There is the Face Mojo plugin, but it works with an iPhone.

I would like to know if some of you have experience in this field.

WendyLuvsCatz

there is PoseRecorder but it's not great

the lipsync plugin in the 32bit version is as good as anything IMO

Willy2

The problem with speech recognition is that the process is done in two steps:

• In the first step, the text or voice is converted into phonemes (this phase is always done very correctly)
• In a second step, the phonemes are converted into visemes (it's this phase that causes problems) Indeed, if each viseme is well build, on the other hand the succession of certain visemes leads to unrealistic sequences, which gives the impression of a mechanical and jerky movement of the mouth. We can see this jerky effect on the video presentation of Anilip2 (which is however well designed and remains the best plugin for DazStudio in its category)

I think Speech Recognition could be improved if we could easily correct unrealistic sequences, by modifying visemes locally. Currently, we can modify visemes but globally.
Another solution would be to smooth several successive vizemes in the timeline (however, this smoothing would be difficult to do since several parts of the face are concerned for each viseme)

In conclusion, I think that facial motion capture has more future than speech or text recognition (but it's necessary to have professional equipment)
Indeed, it's important to have enough points on the lips and around the lips during the motion capture, and also a powerful camera (which isn't the case with a plugin like Face Mojo which uses an iPhone)

Finally, another solution would be to combine the two technologies, to keep the best mouth animation sequences, by placing them in the animate2 timeline.

Padone

As Wendy notes, pose recorder doesn't require any expensive hardware.

https://www.daz3d.com/poserecorder

WendyLuvsCatz

the 32bit lipsync saved as a pose and rendered in the 64bit using Filament

 

 

 

but overlayed with facial aniblocks it is not great

Willy2

With WendyLuvsCatz's post, we see that the 32bit lipsync gives about the same results as Anilip2.
On the other hand, I am surprised by the video that Padone gives without his link. I didn't know PoseRecorder which seems very interesting.
On the promotional video, we notice that PoseRecorder gives very good results for mouth movements.

WendyLuvsCatz

PoseRecorder

 

 

Willy2

For characters with fleshy lips PoseRecorder gives an excellent result for the mouth (it's better quality than speech recognition which uses phonemes and visemes)
For characters who have a thin face (video with a character wearing a black beanie) there is not a big difference between PoseRecorder and Anilip2 (even if PoseRecorder is still better in this particular case)
In addition, PoseRecorder shows the asymmetry of the mouth, which gives even more realism to the video!
For the animation of the cat, in the third video, the result is stunning !

WendyLuvsCatz

Willy2 said:

For characters with fleshy lips PoseRecorder gives an excellent result for the mouth (it's better quality than speech recognition which uses phonemes and visemes)
For characters who have a thin face (video with a character wearing a black beanie) there is not a big difference between PoseRecorder and Anilip2 (even if PoseRecorder is still better in this particular case)
In addition, PoseRecorder shows the asymmetry of the mouth, which gives even more realism to the video!
For the animation of the cat, in the third video, the result is stunning !

actually it comes down to the video and capture subject not the characters I used, they all have different captures 

first one was an iPad video of me

second and third ones the videos shown inset

fleshy or thin lips etc is irrelevent

I need to render all 3 with the SAME character for a true represntation but I did not show you some of my hideous results like this

 

WendyLuvsCatz

the first one on other characters and showing capture

 

Willy2

The end of the last video, where we see on the right the source video, and on the left the result of the rendering, is really interesting. It gives a good idea of the possibilities of PoseRecorder.
Personally, only the deformations of the mouth interest me, because the movements of the head and eyes can be created in DazStudio in a very precise way.
In PoseRecorder can we indicate by a layer the position of the mouth, or is it detected automatically ?

 

WendyLuvsCatz

automatically sadly as I would prefer to choose a frame and set points having a crooked face (the last one was me in video, I used CC0 videos for the others)

Padone

As I understand it poserecorder works better if you keep your face in front of the camera without moving the head. That is, head movements confuse the tracker. Also, being a face mocap, the more expressivity extension your face has the better is the result. An old aged face is much less expressive than a child face for example, and will deliver different results.

It's also very important to start the video with a relaxed face that the tracker will use as reference. If you start the video with a smile for example, then the mouth will be shrinked in the mocap animation.

To help with face mocap one can do facial gymnastic and practice acting.