We were playing around last night and realized I really have no clue hahaha

Traditionally we use a slate or clapper to help us sync the sound and video together when shooting a film. In this mocap process with iclone where the audio, body, and facial is done as a separate process what techniques are people using to sync up all their stuff?

Should one follow a traditional animation approach where audio is grabbed first so that the acting can be animated based on the audio?
Should the audio be captured at the same time as the mocap? If so how do you guys keep it in sync? Does iclone has a real time audio interface that can be used with the mocap?

I'd be interested in hearing from your experiences and how you'd solved some of this issues.

So far what i did is revert to a basic clap sound and gesture combo where the sound of the clap can be synced to the motion of the character clapping. pretty old school but works fine. is there a more efficient approach?









Ibis Fernandez   |   (available for hire)
----------------------------------------------------------------------------------------
Professional Animator, Filmmaker | Creator of the highest quality (modular) G2 rigs for cartoon animator and developer of Toon Titan and Puppet Producer 
Author of Flash Animation and Cartooning: A Creative Guide
>>> be sure to check out http://toontitan.com for professional grade assets, templates, and custom tools for Cartoon Animator and more.

The best thing to do is write yourself a script, and then write a shot list, and write a mocap list so you only capture what you need to.

A shot list is basically all of the camera angles that you are going to capture - then create a mocap list based on each item from the shot list. What are the actors doing, in each shot from the shot list. If one shot is only of the bartender, then only focus on the bartender. You basically want to know what shot each of your actors are going to be in. Wide angle of the entire bar? Then you'll need to record motion for each character. Use the shot list to determine whose in each scene. Use your mocap list to document what motions for each character you need to record for each shot.

Just slapped this together but you get the idea:

Script: A group of people enter the bar
Shot list: A medium shot of the bartender, slowly zooming in waving to the people entering the bar.
Mocap list: Bartender waves as the group enters the bar.
Mocap list: Bartender puts down a glass he was cleaning, then waves as the group enters the bar.
Mocap list: Bartender turns around, then waves to the people entering the bar.
Etc, etc.

How you were doing it in your video also works, but if you write the script, the shot list and have a mocap list it makes it a little easier.

IMDB: James Muia
Aviticus Dragon on YouTube

Computer Specs:
Intel® Core™ i7-7820X Processor (8x 4.30GHz/11MB L3 Cache)
Corsair Hydro Series H115i 280mm
Liquid CPU CoolerASUS ROG STRIX X299-E Gaming Motherboard
32 GB DDR4-3200 GSKILL Ripjaws V Memory
NVIDIA GeForce GTX 1080 Ti - 11GB - EVGA FTW3

It kind of depends on the software you are using.

The ipiSoft (which I use) and, I do believe the new RL mocap stuff also captures audio at the same time, so if you need audio you can just do both.  That's now the way most mocap works (performance mocap, like that Sekeris does).

But otherwise a clapboard makes a lot of sense.

---

Alienware Aurora R12, Win 10, i9-119000KF, 3.5GHz CPU, 128GB RAM, RTX 3090 (24GB), Samsung 960 Pro 4TB M-2 SSD, TB+ Disk space
Mike "ex-genius" Kelley

Really!? Why didn't I think of that!!! Did you even read the post?



Ibis Fernandez   |   (available for hire)
----------------------------------------------------------------------------------------
Professional Animator, Filmmaker | Creator of the highest quality (modular) G2 rigs for cartoon animator and developer of Toon Titan and Puppet Producer 
Author of Flash Animation and Cartooning: A Creative Guide
>>> be sure to check out http://toontitan.com for professional grade assets, templates, and custom tools for Cartoon Animator and more.

Good post. Any idea on how to make the workflow work with iClone using th eleast amount of 3rd party software possible? I mean that's kinda the point here. But i do see that with the hardware in place why would anyone really want to use iClone when you can just use motion builder and something else better suited for filmmaking?

Facial mocap with iclone as far as i know is not possible with iCmaybe something can be done with CrazyTalk 8 but would still leave the issue on how do people sync the facial and body etc. Do most people just eye ball it? Or is there some convention in place that allows for better planing of these things?



Ibis Fernandez   |   (available for hire)
----------------------------------------------------------------------------------------
Professional Animator, Filmmaker | Creator of the highest quality (modular) G2 rigs for cartoon animator and developer of Toon Titan and Puppet Producer 
Author of Flash Animation and Cartooning: A Creative Guide
>>> be sure to check out http://toontitan.com for professional grade assets, templates, and custom tools for Cartoon Animator and more.

What is the technique?



Ibis Fernandez   |   (available for hire)
----------------------------------------------------------------------------------------
Professional Animator, Filmmaker | Creator of the highest quality (modular) G2 rigs for cartoon animator and developer of Toon Titan and Puppet Producer 
Author of Flash Animation and Cartooning: A Creative Guide
>>> be sure to check out http://toontitan.com for professional grade assets, templates, and custom tools for Cartoon Animator and more.

I use the Neuron mocap plugin with a 18 Neurons kit. I use a wireless microphone on the performer recorded with audacity on the same machine. 
I start audacity first to naming the performance then I said "top!" when starting the record mocap. So the "top" on your audio file is the start of your mocap. The difference could be few frames witch can be adjusted on a video edit software.

I think you'll be much more accurate in general starting with the dialogue, as iClone stands now. That'll change with the future introduction of facial mocap.

By doing dialogue first, you can always be frame accurate by hitting record on your motion capture at frame 1 of the timeline. If your dialogue is in place already, it's always the same, take after take.

For conversations it may become valuable to record both speaking and moving together, but that gives you sync issues. Currently, iClone records and then inserts speaking. It's not real-time, so must be recorded externally as mtakerkart is doing. Speaking first gives you pretty good timing as well. It's much easier to add gesturing to speaking then speaking to gesturing! The worst is a canned conversation animation that has speaking added to it. There is almost always no correlation between gestures and speaking. 

An interesting interim is the auto-animation stuff in CT. I'm under the impression that it only exports the facial animation though, and not the upper body. Has anyone tested this? The upper body movement seems a pretty important part of it.

hmmm that's an interesting aproach.




Ibis Fernandez   |   (available for hire)
----------------------------------------------------------------------------------------
Professional Animator, Filmmaker | Creator of the highest quality (modular) G2 rigs for cartoon animator and developer of Toon Titan and Puppet Producer 
Author of Flash Animation and Cartooning: A Creative Guide
>>> be sure to check out http://toontitan.com for professional grade assets, templates, and custom tools for Cartoon Animator and more.

Yes, i realize this. I have compared the results of the data in terms of what iClone sees and what other programs see and there is a huge difference. The hands don't even work in the iClone captures most of the time. The thumbs, never do.

Ive actually switched to recording the raw data using the Axis Neuron software. I am thinking at sometime if it comes down to it i can use that data on other apps that can better handle it. To get the data from axis neuron into iclone or anything else all you have to do is hit play, and it gets broadcasted over to whatever plugin is set up to listen for it. Stick it doesn't address what the sync issues. This is something that has always been an issue even in regular live action film making. that why they came up with the clapper board. You see video of the object clapping, and you listen for the clap (or look at the waveform in the audio file) you match the visual with the audio and boom its synced. (by sliding the track).


Yes, this is basically what im doing now. Except I give myself a visual cue in the performance track to match my audio cue so as not to be aimlessly sliding around indefinitely.

It feels like there would have been some standardized technique by now. How the people in planet of the apes or avatar handle it? Seems like they just brute force synchronizing their stuff. I guess with  millions of dollars in budget you can afford to just brute force your way through anything hahaha




Ibis Fernandez   |   (available for hire)
----------------------------------------------------------------------------------------
Professional Animator, Filmmaker | Creator of the highest quality (modular) G2 rigs for cartoon animator and developer of Toon Titan and Puppet Producer 
Author of Flash Animation and Cartooning: A Creative Guide
>>> be sure to check out http://toontitan.com for professional grade assets, templates, and custom tools for Cartoon Animator and more.