Hey, You Guys!

As someone who does not have mocap yet, what would it take to lip-sync the Gospel books with pre-recorded audio? 

Would that mean the person lip-syncing would need to have the audio recordings memorized to be exactly in-sync with each of them? 

And how hard would that be? 

(There are over 200 different voice roles in this film production, BTW.  So the audio recordings themselves are being done by a wide variety of different people, of course.)

What would be the time and difficulty difference between lip-syncing with mocap in timing with pre-recorded audio vs iClone's lip-sync with Kelley-Toons lip-sync program improvement? 

Or are both mocap and the other used? 

Let's take a 30-second recording of JESUS speaking, for example, that needs to be lip-synced and animated out. 

What are the very best approaches for this both time-wise and also to end up with a very professional-looking product? 

Thanks! :-) 

~ Kimmie

www.TrackingJESUS.net 

--> More Information On This Bible Film Production Can Be Found At: 
www.TrackingJESUS.net/Productions 

--> Become An Early Subscriber To The 'Tracking JESUS' YouTube Channel At: 
https://www.youtube.com/channel/UCEMx91AySc3QuF-SS_3mGQQ

Hi kimmie

This is a short part of a bible project I 'm working on. I recorded the voices, And I made the lipsync reading the text and doing mocap with iphone X. It can be improved and it's very fast. Not perfect yet but it's not the final result




     

I'm sort of guessing, so hopefully someone else can either confirm or correct what I'm saying here...

I think it would be good to have the mocap and the audio "reasonably" in sync, so the basic head motions, and expressions align with the vocal track.  I don't think it matters which one you do first, though it seems to me it would be easiest to do the mocap first.

When you have both, then you'd you'd go into iClone and delete the existing viseme track and import the official audio track to generate new visemes (mouth "sound" actions).

Maybe I talk strangely, but I usally find myself deleting quite a number of visemes.


Independent of what I wrote above, my main tips would be:

• Keep the audio files short.  No more than a "paragraph" per file.  That will make it easier to adjust their placement in iClone.
• With so many voice actors, I'd be very concerned about consistency in the audio, especially the basic quality of the recordings.  Will some people have noisy backgrounds?  Or in a room with echo?  You don't want it so sound like John the Baptist is inside a tiled bathroom when he's outside at the river Jordan.  Unless they're being professionally recorded in a controlled environment, be prepared to put some effort into tuning the audio.  At the very least, some noise reduction will probably be necessary (I use Audacity, which works well for me and is free).
• After rendering, replace all the "iClone audio" with the clean audio tracks in your video editor (NLE).  Why?  This will give you, among other things, INDIVIDUAL VOLUME CONTROL for each speaker, and the ability to vary each person's volume over time.  For example, people farther away will be more quiet, and if a person terns away from the camera, lower the volume some.

Good luck,  I look forward to seeing some clips as you work on your epic production.


P.S.
@Charly - Nice.  Thanks for sharing.

---

iClone 7... Character Creator... Substance Designer/Painter... Blender... Audacity...
Desktop (homebuilt) - Windows 10, Ryzen 9 3900x CPU, GTX 1080 GPU (8GB), 32GB RAM, Asus X570 Pro motherboard, 2TB SSD, terabytes of disk space, dual  monitors.
Laptop - Windows 10, MSI GS63VR STEALTH-252, 16GB RAM, GTX 1060 (6GB), 256GB SSD and 1TB HDD

Charly, that is WONDERFUL! 

I wish I spoke French... I would love to know what was being said in that Bible scene! 

Very cool, man. :-)

~ Kimmie

www.TrackingJESUS.net 

--> More Information On This Bible Film Production Can Be Found At: 
www.TrackingJESUS.net/Productions 

--> Become An Early Subscriber To The 'Tracking JESUS' YouTube Channel At: 
https://www.youtube.com/channel/UCEMx91AySc3QuF-SS_3mGQQ

Thank you so much for that, 'Just A Viking'! :-) 

And yes, I agree on the clip lengths totally.  The very longest clips are about 30 seconds - and those ONLY because of longer-running sentences that cannot be broken up. 

But all in all, I have the whole layout of the clips kept as short as possible. 

In audio consistency, that has been a concern of ours too.  (It is something my son told me at the very, very beginning is so important.)

Virtually everyone is recording in a smaller back room of their house and with exactly matching mic sets and on the same settings.  

We are also using Audacity to clean up any 'white noise' the best we can. 

But it would take the LORD opening a very special door in order for most of the audio clips to be recorded in the same room (perhaps in a Church someplace). 
...And GOD just might do that... I sure won't put it past HIM. ;-) 

All of Mark's recordings are already done (from a small back room in his house). 

And Luke's recordings are well underway (also from a small back room at his own place). 

And in both cases, we are truly blessed to have guys who are computer geniuses and are also sensitive to sound tech stuff.  

But the more we can do all in the same place, the better, of course. 

~ Kimmie

www.TrackingJESUS.net 

--> More Information On This Bible Film Production Can Be Found At: 
www.TrackingJESUS.net/Productions 

--> Become An Early Subscriber To The 'Tracking JESUS' YouTube Channel At: 
https://www.youtube.com/channel/UCEMx91AySc3QuF-SS_3mGQQ

From an a quality perspective you want the voice recordings as "dry" as possible, so a church would not be very suitable. Computer ears are different from human ears.

To simulate various acoustic environments I have created a number of Reverb presets in Audacity that I then apply according to the situation (indoors, outdoors, etc.). The dry recording is used in iClone, the treated recording is used in the final production (i.e., the video editor).

---

Animagic: What is the difference between a small back room of a Church and a small back room of a house?

www.TrackingJESUS.net 

--> More Information On This Bible Film Production Can Be Found At: 
www.TrackingJESUS.net/Productions 

--> Become An Early Subscriber To The 'Tracking JESUS' YouTube Channel At: 
https://www.youtube.com/channel/UCEMx91AySc3QuF-SS_3mGQQ

I didn't catch that you meant a room in a church... In that case there's no difference. Heavy curtains or carpet would be good, bare walls not so.

---

Sorry, I meant the back room of a Church as a place to record most of the lines in vs many back rooms of many different houses. :-D 
So why not use any treated recordings in iClone?  Is it because of inaccuracy issues in iClone's lip sync program?
~ Kimmie

www.TrackingJESUS.net 

--> More Information On This Bible Film Production Can Be Found At: 
www.TrackingJESUS.net/Productions 

--> Become An Early Subscriber To The 'Tracking JESUS' YouTube Channel At: 
https://www.youtube.com/channel/UCEMx91AySc3QuF-SS_3mGQQ

Yes, Kimmie -- iClone's auto generation of visemes isn't very accurate.  In fact, it's pretty lousy.  The reason the Papagayo works so much better is that you type in the text so it can read the visemes that are normally generated by those words (I say "normally" because even there people have different speech patterns.  However, you can even account for this in Papagayo by phonetically spelling some things.  So you could type "you all" instead of "yawl").

Without the text all iClone can do is try and parse the audio which is a very difficult task.  In general it looks as about as accurate as the Japanese lip sync dialog you find in those monster movies where it's dubbed.  Now, you can work with this as a beginning process, and delete and add the visemes, but this is MUCH easier to do in the Papagayo interface.

---

Alienware Aurora R16, Win 11, i9-149000KF, 3.20GHz CPU, 64GB RAM, RTX 4090 (24GB), Samsung 870 Pro 8TB, Gen3 MVNe M-2 SSD, 4TBx2, 39" Alienware Widescreen Monitor
Mike "ex-genius" Kelley