Open Beta - Export Omniverse Audio2Face Animations to iClone

Hello everyone,

When we first introduced Audio2Face in late July, we mentioned that we are working on the solution to bring Audio2Face animation back to iClone. Now it's time for everyone to experience it!

What is Audio2Face?

Omniverse Audio2Face is an AI-powered application that generates expressive facial animation from just an audio source. Audio2Face 2021.2.6 supports Character Creator CC3 Base+ and Game Base presets, which largely simplifies the wrap process for facial and lip-sync animation creation.

In ver. 2021.3.0, Audio2Face supports exporting animation as blendshapes (.json). This guide will also offer a solution for loading back the json data onto CC characters in iClone.

With the ability to bake Audio2Face blendshapes and export it back to iClone, and in combination with iClone's native facial animation tools, users now have another option for lip-syncing besides AccuLips. On top of this, comes the ability to export the character animations to other mainstream 3D engines such as Blender, Unreal Engine, Unity, etc.
For Omniverse users, here is a total solution for full facial and body animation in combination with Audio2Face animation. Users can send the result to Omniverse via the upcoming iClone Omniverse Connector.

Please note that this is a Beta solution for sneak preview. The official solution will be released with iClone 8.

Required Software & Files

Character Creator 3.44 or above
iClone 7.93 or above
iClone Python script for loading Audio2Face blendshape JSON
*Script is updated on Nov 4th for UI optimization.
Omniverse Audio2Face 2021.3.0
Omniverse USD Sample Model (CC3+ Neutral Base) for generating animation and blendshape baking
*This file is no longer needed while using CC version 3.44.
*In the previous CC version 3.43, the blendshapes of the exported A2F head mesh are not named clearly, which makes it difficult to be identified in A2F. Before CC v3.44 releases, you can use the USD file that we provide as the sample model used in A2F, for generating animation and blendshape baking.

Workflow

Current version - iClone 7.93 has native USD export function: (3DXChange 7 Pipeline required)
75% of original size (was 672x19) - Click to enlarge

Previous version:
75% of original size (was 672x19) - Click to enlarge

Video Demo (no dubbing)

*Recorded by using of CC3.43 and iClone 7.92:

Part 1: Audio2Face Character Transfer

Part 2: Audio2Face Blendshape Conversion (Viking)

Part 3: Using Sample Model for Blendshape Export (JSON)

Part 4: Importing JSON, and Adjusting Strength Smooth in iClone

Step-by-Step Guide

Please check THIS DOCUMENT for details.

Reallusion Team



0 active, 0 guests, 0 members, 0 anonymous.
No members currently viewing this topic!

0 active, 0 guests, 0 members, 0 anonymous.

No members currently viewing this topic!

Author	Message
Miranda (RL)	Miranda (RL) Posted 4 Years Ago
Distinguished Member Group: Administrators Last Active: 3 Years Ago Posts: 1.8K, Visits: 11.2K	Hello everyone, When we first introduced Audio2Face in late July, we mentioned that we are working on the solution to bring Audio2Face animation back to iClone. Now it's time for everyone to experience it! What is Audio2Face? Omniverse Audio2Face is an AI-powered application that generates expressive facial animation from just an audio source. Audio2Face 2021.2.6 supports Character Creator CC3 Base+ and Game Base presets, which largely simplifies the wrap process for facial and lip-sync animation creation. In ver. 2021.3.0, Audio2Face supports exporting animation as blendshapes (.json). This guide will also offer a solution for loading back the json data onto CC characters in iClone. With the ability to bake Audio2Face blendshapes and export it back to iClone, and in combination with iClone's native facial animation tools, users now have another option for lip-syncing besides AccuLips. On top of this, comes the ability to export the character animations to other mainstream 3D engines such as Blender, Unreal Engine, Unity, etc. For Omniverse users, here is a total solution for full facial and body animation in combination with Audio2Face animation. Users can send the result to Omniverse via the upcoming iClone Omniverse Connector. Please note that this is a Beta solution for sneak preview. The official solution will be released with iClone 8. Required Software & Files Character Creator 3.44 or above iClone 7.93 or above iClone Python script for loading Audio2Face blendshape JSON Script is updated on Nov 4th for UI optimization. Omniverse Audio2Face 2021.3.0 Omniverse USD Sample Model (CC3+ Neutral Base) for generating animation and blendshape baking This file is no longer needed while using CC version 3.44. In the previous CC version 3.43, the blendshapes of the exported A2F head mesh are not named clearly, which makes it difficult to be identified in A2F. Before CC v3.44 releases, you can use the USD file that we provide as the sample model used in A2F, for generating animation and blendshape baking. Workflow* Current version - iClone 7.93 has native USD export function: (3DXChange 7 Pipeline required) 75% of original size (was 672x19) - Click to enlarge Previous version: 75% of original size (was 672x19) - Click to enlarge Video Demo (no dubbing) Recorded by using of CC3.43 and iClone 7.92: Part 1: Audio2Face Character Transfer Part 2: Audio2Face Blendshape Conversion (Viking) Part 3: Using Sample Model for Blendshape Export (JSON) Part 4: Importing JSON, and Adjusting Strength Smooth in iClone Step-by-Step Guide* Please check THIS DOCUMENT for details. Reallusion Team

Open Beta - Export Omniverse Audio2Face Animations to iClone

Open Beta - Export Omniverse Audio2Face Animations to iClone

Threaded View

Reading This Topic