Creating Facial and Lip-sync Animation in iClone for Blender

Greetings, my name is Peter Alexander, a 3D Generalist and freelancer who works heavily with iClone and Character Creator.
 
iClone's Facial Animation Technology includes facial expressions, lip-sync animation and facial mocap which provides real-time facial animation with facial capture equipment.
AccuLips provides a very intuitive way to generate procedural, editable lip-syncing animations, allowing users to easily create accurate, natural, and smooth talking performances. It detects and extracts text and viseme from audio, or imports a prepared script for precise lip-syncing which saves a massive amount of time when animating dialogue.

In this article, I'm going to run you through the process of using AccuLips to create lip-sync animation for my Laughing Buddha. I’m also going to show you how to export your animation from iClone to Blender for Eevee rendering.

To keep the focus on AccuLips, I've already done the character's movements using elements from a previous project. You can see the lip-sync animation tutorial in the following video:



Prepare Recorded Voice MP3 File Compatible with iClone.

For the character's audio track, I recorded my voice using Audacity, an audio manipulation application, and altered the audio's pitch and reverberation. After editing the track according to my needs, I save it as an MP3 file, which is compatible with AccuLips and iClone.



Import and Edit the Audio in iClone

In iClone, go to Animation and Create Script, choose AccuLips, and import the saved audio. Select Generate Text. AccuLips will analyze the audio and attempt to approximate the speech into text. The words that can’t be identified will be in red, much like a spell check. You can manually correct this for a better generation of visemes.



Adjust Text for Better Visemes and Modify Visemes

When you apply the AccuLips data to the animation, it will produce the various visemes to accompany the audio. I then go through the timeline, and if I want a sound to be emphasized or replaced with the lip-sync, I click on the viseme indicator and adjust it. The key point is that you want to emphasize visemes at certain points during the animation. If the dialog emphasizes a sound, the animation should as well. You can keep the lip editor open as you go through the timeline, to make things easier to adjust.



Export the Audio and Character from iClone

If you choose to render in Blender, you'll want to do two things. First, export the audio from the range of the video, then select the character and export the FBX from the same range. This will ensure that the audio and animation are in sync.





Import Character into Blender via Auto Setup

In Blender use the Blender Auto Setup, located in the CC3 tab, to import the FBX file as a character. In the CC3 tab, click the Range from Character option to set the proper animation range for the timeline.




Import the Audio in Blender

To import the audio, go to the Video Sequencer, press Shift+A, then select Sound from the menu. Or you can go to the Add menu at the top of the Video Sequencer section and add Sound from there. This will allow you to select the previously exported audio file. You can select the Playback panel on the timeline to enable scrubbing so you can hear your audio as you go through your animation timeline. If you don't hear the audio, open preferences, go to System, then check your Sound settings.






Save the Animation as a Video File

To save the animation as a video, go to the Scene tab, go to Output, and choose to save it in a video format rather than as an image. Also, remember to choose an audio encoding option. If you plan to do some compositing, you can set up a green screen or save this as a video with transparency (RGBA).



Rendering to Video File in Blender



Now with everything set up, choose Viewport Render Animation, rather than the full render (depending on your rendering preferences). This will render things as they appear in the Eevee Preview window.

Examples of other lip-syncs and facial animations:





If anyone has any questions, comments, or even suggestions, post them and I'll do my best to address them.

@Mythcons
Great , Many Thanks.👍👍
Did you render it with the Blender sequencer.?
Greetings rosuckmedia

Hi rosuckmedia,

I rendered it with ‘Viewport Render Animation’, which renders the audio along with the Eevee viewport. It is not a ‘full’ render, which requires a slightly different visual set up in my experience. The ‘Viewport Render Animation’ option can be found in the Search dialog, which I believe can be found through the F3 key.

Okay Many Thanks

Hi rosuckmedia,

Sorry about that speed issue, it is a consequences of certain time constraints. The normals issue occurs with some imported items from Character Creator and iClone. I'm not entirely sure what the origin of the issue is.

There are a few commands I use, depending on the issue. First select all vertices (Edit mode), then:

- Mesh > Normals > Recalculate Outside (I always type 'recalculate' to find this in the Search bar)
- Mesh > Normals > Average > Face Area (I always type 'average' to find this in the Search bar)
- Mesh > Normals > Set from Faces (Might have to search for the whole thing ['Set from Faces'] to narrow down this command)

Generally the first command listed will fix most normal issues in my experience.

Many Thanks👍
Greetings rosuckmedia

RAG DOLL COLLISION ANIMATIONS FOR ICLONE 8 & 7
---------------------------------------------------------------------------------------------------------------------

Ghost Origins

My latest Feature length film created with Iclone.

My Sci- Fi Graphic Novel on Amazon: https://a.co/d/9k3cwoY

My IMDB listing
Anabran Abdhul Quarnain - IMDb

Hi there, my understanding is that you want your rendering to be at 60 FPS, whether its video or an image sequence. My experience in this is only through errors I've made in accidentally changing the FPS to 30, and the audio going out of sync.
You might want to do a quick test on a small batch of rendered images just to make sure. I wish I had a firm answer, but I don't do a lot of rendering. I'm more of a content creator.

I hope that helps a little.