Profile Picture

Viseme and Facial Mocap

Posted By gordryd Last Year
You don't have permission to rate!
Author
Message
gordryd
gordryd
Posted Last Year
View Quick Profile
Distinguished Member

Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)

Group: Forum Members
Last Active: 7 hours ago
Posts: 264, Visits: 3.6K
I don't own a facial mocap device yet, but was curious how the movement of the lips is handled.  If the actor is talking (and you record the audio) is the motion good enough to use for lip sync?  Or do you need to replace w/viseme?  Can you mix/blend the 2, or is it only one or the other.  Thanks.
Kelleytoons
Kelleytoons
Posted Last Year
View Quick Profile
Distinguished Member

Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)

Group: Forum Members
Last Active: 7 hours ago
Posts: 6.8K, Visits: 14.1K
Well, *I* think it's good enough, but you should look at the MANY examples (including those I've posted on my channel) to make up your own mind before you invest.

You can capture audio at the same time but it will only be the tongue movement -- you can't mix visemes with the lips movement.



Alienware Aurora R7, Win 10, i7-8700k, 4.7GHz CPU, 32GB RAM, GTX Titan XP (12GB), Samsung 960 Pro 2TB M-2 SSD, TB+ Disk space
Mike "ex-genius" Kelley
gordryd
gordryd
Posted Last Year
View Quick Profile
Distinguished Member

Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)

Group: Forum Members
Last Active: 7 hours ago
Posts: 264, Visits: 3.6K
Same topic - different question(s):  Is the audio recorded by iClone sufficient quality to use in finished production?  (Not played back by iClone, but recorded into iClone during Facial Mocap).
My assumption would be that it records in whatever quality your input device uses (i.e., 44.1kHz, 16-bit), or does iClone require a specific audio format?  I couldn't find and technical specs re: audio in the iClone manual. 
Also, can you then export the audio file (.wav?) from iClone to a DAW for post-production and re-mixing back into a video editor?  How would one go about doing that?  Is it embedded in a MotionPlus file or available stand-alone in a directory somewhere?
Kelleytoons
Kelleytoons
Posted Last Year
View Quick Profile
Distinguished Member

Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)

Group: Forum Members
Last Active: 7 hours ago
Posts: 6.8K, Visits: 14.1K
It seems to be of sufficient quality, at least in the things I've done (and you can view my channel for many examples).  And you can export the audio separately out of iClone (it's render and you just choose "audio" and get a .WAV).

For critical things just use whatever audio capture you normally would and sync up in your video editor with the outputted audio (assuming the iClone capture isn't good enough for you).  



Alienware Aurora R7, Win 10, i7-8700k, 4.7GHz CPU, 32GB RAM, GTX Titan XP (12GB), Samsung 960 Pro 2TB M-2 SSD, TB+ Disk space
Mike "ex-genius" Kelley
animagic
animagic
Posted Last Year
View Quick Profile
Distinguished Member

Distinguished Member (20.7K reputation)Distinguished Member (20.7K reputation)Distinguished Member (20.7K reputation)Distinguished Member (20.7K reputation)Distinguished Member (20.7K reputation)Distinguished Member (20.7K reputation)Distinguished Member (20.7K reputation)Distinguished Member (20.7K reputation)Distinguished Member (20.7K reputation)

Group: Forum Members
Last Active: 2 hours ago
Posts: 11.9K, Visits: 20.8K
I always prepare audio for dialog outside of iClone with one file for each speaking person. Those files are only used for lip-syncing.

Any audio treatment such as reverb to simulate room acoustics is done separately. So I don't rely on iClone's audio export. I believe it's now 44.1k (used to be 32k), which I don't use as everything is standardized at 48k.


https://forum.reallusion.com/uploads/images/1a09220f-ab50-42ac-ad1a-33ec.pnghttps://forum.reallusion.com/Uploads/Images/d14339d0-cd32-4b35-88f9-40a0.png


gordryd
gordryd
Posted Last Year
View Quick Profile
Distinguished Member

Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)Distinguished Member (1.6K reputation)

Group: Forum Members
Last Active: 7 hours ago
Posts: 264, Visits: 3.6K
Understood about post-processing.  I guess my question is, if LiveFace is capturing motion and you want to record the audio at the same time, iClone can do that, but how good is it?  And would you be able to run an audio program simultaneously to LiveFace to capture the audio with 'better fidelity' and control?  Seems like the mocap would be pretty h/w intensive...
Kelleytoons
Kelleytoons
Posted Last Year
View Quick Profile
Distinguished Member

Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)Distinguished Member (21.8K reputation)

Group: Forum Members
Last Active: 7 hours ago
Posts: 6.8K, Visits: 14.1K
But you wouldn't need to do that -- most modern video editors can easily sync sound, so you can, for example, use a digital audio recorder for each actor, record them as you also capture the face, and then sync all up in your editor with the audio track exported from iClone.  And apply post processing as needed.  Exactly the way the pros do.



Alienware Aurora R7, Win 10, i7-8700k, 4.7GHz CPU, 32GB RAM, GTX Titan XP (12GB), Samsung 960 Pro 2TB M-2 SSD, TB+ Disk space
Mike "ex-genius" Kelley



Reading This Topic