Profile Picture

Open Beta - Export Omniverse Audio2Face Animations to iClone

Posted By Miranda (RL) Last Year
Rated 5 stars based on 1 vote.

Open Beta - Export Omniverse Audio2Face Animations to iClone

Author
Message
Miranda (RL)
Miranda (RL)
Posted Last Year
View Quick Profile
Distinguished Member

Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)

Group: Administrators
Last Active: 5 Months Ago
Posts: 1.8K, Visits: 11.2K
Hello everyone,

When we first introduced Audio2Face in late July, we mentioned that we are working on the solution to bring Audio2Face animation back to iClone. Now it's time for everyone to experience it!




What is Audio2Face?

Omniverse Audio2Face is an AI-powered application that generates expressive facial animation from just an audio source. Audio2Face 2021.2.6 supports Character Creator CC3 Base+ and Game Base presets, which largely simplifies the wrap process for facial and lip-sync animation creation.

In ver. 2021.3.0, Audio2Face supports exporting animation as blendshapes (.json). This guide will also offer a solution for loading back the json data onto CC characters in iClone.
  • With the ability to bake Audio2Face blendshapes and export it back to iClone, and in combination with iClone's native facial animation tools, users now have another option for lip-syncing besides AccuLips. On top of this, comes the ability to export the character animations to other mainstream 3D engines such as Blender, Unreal Engine, Unity, etc. 
  • For Omniverse users, here is a total solution for full facial and body animation in combination with Audio2Face animation. Users can send the result to Omniverse via the upcoming iClone Omniverse Connector.
Please note that this is a Beta solution for sneak preview. The official solution will be released with iClone 8.


Required Software & Files

  • Character Creator 3.44 or above
  • iClone 7.93 or above
  • iClone Python script for loading Audio2Face blendshape JSON
    *Script is updated on Nov 4th for UI optimization.
  • Omniverse Audio2Face 2021.3.0
  • Omniverse USD Sample Model (CC3+ Neutral Base) for generating animation and blendshape baking
    *This file is no longer needed while using CC version 3.44.
    *In the previous CC version 3.43, the blendshapes of the exported A2F head mesh are not named clearly, which makes it difficult to be identified in A2F. Before CC v3.44 releases, you can use the USD file that we provide as the sample model used in A2F, for generating animation and blendshape baking.


Workflow

Current version - iClone 7.93 has native USD export function: (3DXChange 7 Pipeline required)
https://forum.reallusion.com/uploads/images/955de6f4-ba0e-42a1-998a-8651.png



Previous version:
https://forum.reallusion.com/uploads/images/c40dcfc6-9b50-48e0-9db1-e95d.png


Video Demo (no dubbing)

*Recorded by using of CC3.43 and iClone 7.92:
Part 1: Audio2Face Character Transfer


Part 2: Audio2Face Blendshape Conversion (Viking)


Part 3: Using Sample Model for Blendshape Export (JSON)


Part 4: Importing JSON, and Adjusting Strength Smooth in iClone



Step-by-Step Guide

Please check THIS DOCUMENT for details.


Reallusion Team
Edited
Last Year by Miranda (RL)
Jeasy
Jeasy
Posted Last Year
View Quick Profile
Veteran Member

Veteran Member (982 reputation)Veteran Member (982 reputation)Veteran Member (982 reputation)Veteran Member (982 reputation)Veteran Member (982 reputation)Veteran Member (982 reputation)Veteran Member (982 reputation)Veteran Member (982 reputation)Veteran Member (982 reputation)

Group: Forum Members
Last Active: 2 days ago
Posts: 137, Visits: 1.7K
This is amazing news Miranda (RL), can't wait to check it out Smile

Thanks!

Edited
Last Year by Miranda (RL)
GARCIAGA
GARCIAGA
Posted Last Year
View Quick Profile
Junior Member

Junior Member (157 reputation)Junior Member (157 reputation)Junior Member (157 reputation)Junior Member (157 reputation)Junior Member (157 reputation)Junior Member (157 reputation)Junior Member (157 reputation)Junior Member (157 reputation)Junior Member (157 reputation)

Group: Forum Members
Last Active: 2 Months Ago
Posts: 9, Visits: 201
It looks great, wich one will be better Accu-Lips or Audio2Face??

mtakerkart
mtakerkart
Posted Last Year
View Quick Profile
Distinguished Member

Distinguished Member (13.9K reputation)Distinguished Member (13.9K reputation)Distinguished Member (13.9K reputation)Distinguished Member (13.9K reputation)Distinguished Member (13.9K reputation)Distinguished Member (13.9K reputation)Distinguished Member (13.9K reputation)Distinguished Member (13.9K reputation)Distinguished Member (13.9K reputation)

Group: Forum Members
Last Active: Yesterday
Posts: 2.9K, Visits: 25.3K
I would like some help to test Audio2face with the french language. Reallusion's example offers 6 languages but not French ...
I don't have an rtx graphics card which is mandatory to use it. If it is accurate I would buy the smaller rtx card because it does not require resources.
I put below an audio file of 20 seconds for the test. can you take a screenshot in video?
ttamd
ttamd
Posted Last Year
View Quick Profile
New Member

New Member (25 reputation)New Member (25 reputation)New Member (25 reputation)New Member (25 reputation)New Member (25 reputation)New Member (25 reputation)New Member (25 reputation)New Member (25 reputation)New Member (25 reputation)

Group: Forum Members
Last Active: 2 Months Ago
Posts: 3, Visits: 52
Amazing, thanks, will digg into this.
Miranda (RL)
Miranda (RL)
Posted Last Year
View Quick Profile
Distinguished Member

Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)

Group: Administrators
Last Active: 5 Months Ago
Posts: 1.8K, Visits: 11.2K
Hi GARCIAGA,

They are developed by using different algorithms. In my personal opinion, AccuLips provides flexibility in generating and fine tuning (e.g. word dictionary), while Audio2Face supports multi-languages, and more automatic functions will be introduced in the future. When Audio2Face is getting mature, it's definitely a great tool to work with.

GARCIAGA (9/20/2021)
It looks great, wich one will be better Accu-Lips or Audio2Face??



Miranda (RL)
Miranda (RL)
Posted Last Year
View Quick Profile
Distinguished Member

Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)

Group: Administrators
Last Active: 5 Months Ago
Posts: 1.8K, Visits: 11.2K
@Jeasy, ttamd,

Welcome to show what you come up with. Smile

@Mark,

This is French. Doesn't sound like? Tongue
https://youtu.be/Q64C4CWOV8s?t=17
https://forum.reallusion.com/uploads/images/f9d92ad6-cc5a-4515-91e2-9bff.png
rosuckmedia
rosuckmedia
Posted Last Year
View Quick Profile
Distinguished Member

Distinguished Member (7.1K reputation)Distinguished Member (7.1K reputation)Distinguished Member (7.1K reputation)Distinguished Member (7.1K reputation)Distinguished Member (7.1K reputation)Distinguished Member (7.1K reputation)Distinguished Member (7.1K reputation)Distinguished Member (7.1K reputation)Distinguished Member (7.1K reputation)

Group: Forum Members
Last Active: 3 days ago
Posts: 2.4K, Visits: 3.7K
I see that the German language file is also missing.😢😢
Will it be added later???
Greetings rosuckmedia
Miranda (RL)
Miranda (RL)
Posted Last Year
View Quick Profile
Distinguished Member

Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)Distinguished Member (46.2K reputation)

Group: Administrators
Last Active: 5 Months Ago
Posts: 1.8K, Visits: 11.2K
Hi rosuckmedia,

Do you have a chance to try it?
Audio2Face shall work with any languages, and Omniverse Team continually updating with more and more languages. Smile

rosuckmedia (9/22/2021)
I see that the German language file is also missing.😢😢
Will it be added later???
Greetings rosuckmedia


rosuckmedia
rosuckmedia
Posted Last Year
View Quick Profile
Distinguished Member

Distinguished Member (7.1K reputation)Distinguished Member (7.1K reputation)Distinguished Member (7.1K reputation)Distinguished Member (7.1K reputation)Distinguished Member (7.1K reputation)Distinguished Member (7.1K reputation)Distinguished Member (7.1K reputation)Distinguished Member (7.1K reputation)Distinguished Member (7.1K reputation)

Group: Forum Members
Last Active: 3 days ago
Posts: 2.4K, Visits: 3.7K
Hi Miranda,
Thanks for your quick reply.😊
I need to reinstall Omniverse first, then I will try it. I have a RTX 2080 should actually work.
Greetings rosuckmedia



Reading This Topic