Open Beta - Export Omniverse Audio2Face Animations to iClone


https://forum.reallusion.com/Topic494983.aspx
Print Topic | Close Window

By Miranda (RL) - 3 Years Ago
Hello everyone,

When we first introduced Audio2Face in late July, we mentioned that we are working on the solution to bring Audio2Face animation back to iClone. Now it's time for everyone to experience it!




What is Audio2Face?

Omniverse Audio2Face is an AI-powered application that generates expressive facial animation from just an audio source. Audio2Face 2021.2.6 supports Character Creator CC3 Base+ and Game Base presets, which largely simplifies the wrap process for facial and lip-sync animation creation.

In ver. 2021.3.0, Audio2Face supports exporting animation as blendshapes (.json). This guide will also offer a solution for loading back the json data onto CC characters in iClone.
  • With the ability to bake Audio2Face blendshapes and export it back to iClone, and in combination with iClone's native facial animation tools, users now have another option for lip-syncing besides AccuLips. On top of this, comes the ability to export the character animations to other mainstream 3D engines such as Blender, Unreal Engine, Unity, etc. 
  • For Omniverse users, here is a total solution for full facial and body animation in combination with Audio2Face animation. Users can send the result to Omniverse via the upcoming iClone Omniverse Connector.
Please note that this is a Beta solution for sneak preview. The official solution will be released with iClone 8.


Required Software & Files

  • Character Creator 3.44 or above
  • iClone 7.93 or above
  • iClone Python script for loading Audio2Face blendshape JSON
    *Script is updated on Nov 4th for UI optimization.
  • Omniverse Audio2Face 2021.3.0
  • Omniverse USD Sample Model (CC3+ Neutral Base) for generating animation and blendshape baking
    *This file is no longer needed while using CC version 3.44.
    *In the previous CC version 3.43, the blendshapes of the exported A2F head mesh are not named clearly, which makes it difficult to be identified in A2F. Before CC v3.44 releases, you can use the USD file that we provide as the sample model used in A2F, for generating animation and blendshape baking.


Workflow

Current version - iClone 7.93 has native USD export function: (3DXChange 7 Pipeline required)
https://forum.reallusion.com/uploads/images/955de6f4-ba0e-42a1-998a-8651.png



Previous version:
https://forum.reallusion.com/uploads/images/c40dcfc6-9b50-48e0-9db1-e95d.png


Video Demo (no dubbing)

*Recorded by using of CC3.43 and iClone 7.92:
Part 1: Audio2Face Character Transfer


Part 2: Audio2Face Blendshape Conversion (Viking)


Part 3: Using Sample Model for Blendshape Export (JSON)


Part 4: Importing JSON, and Adjusting Strength Smooth in iClone



Step-by-Step Guide

Please check THIS DOCUMENT for details.


Reallusion Team
By Jeasy - 3 Years Ago
This is amazing news Miranda (RL), can't wait to check it out Smile

Thanks!

By GARCIAGA - 3 Years Ago
It looks great, wich one will be better Accu-Lips or Audio2Face??

By mtakerkart - 3 Years Ago
I would like some help to test Audio2face with the french language. Reallusion's example offers 6 languages but not French ...
I don't have an rtx graphics card which is mandatory to use it. If it is accurate I would buy the smaller rtx card because it does not require resources.
I put below an audio file of 20 seconds for the test. can you take a screenshot in video?
By ttamd - 3 Years Ago
Amazing, thanks, will digg into this.
By Miranda (RL) - 3 Years Ago
Hi GARCIAGA,

They are developed by using different algorithms. In my personal opinion, AccuLips provides flexibility in generating and fine tuning (e.g. word dictionary), while Audio2Face supports multi-languages, and more automatic functions will be introduced in the future. When Audio2Face is getting mature, it's definitely a great tool to work with.

GARCIAGA (9/20/2021)
It looks great, wich one will be better Accu-Lips or Audio2Face??



By Miranda (RL) - 3 Years Ago
@Jeasy, ttamd,

Welcome to show what you come up with. Smile

@Mark,

This is French. Doesn't sound like? Tongue
https://youtu.be/Q64C4CWOV8s?t=17
https://forum.reallusion.com/uploads/images/f9d92ad6-cc5a-4515-91e2-9bff.png
By rosuckmedia - 3 Years Ago
I see that the German language file is also missing.😒😒
Will it be added later???
Greetings rosuckmedia
By Miranda (RL) - 3 Years Ago
Hi rosuckmedia,

Do you have a chance to try it?
Audio2Face shall work with any languages, and Omniverse Team continually updating with more and more languages. Smile

rosuckmedia (9/22/2021)
I see that the German language file is also missing.😒😒
Will it be added later???
Greetings rosuckmedia


By rosuckmedia - 3 Years Ago
Hi Miranda,
Thanks for your quick reply.😊
I need to reinstall Omniverse first, then I will try it. I have a RTX 2080 should actually work.
Greetings rosuckmedia
By mtakerkart - 3 Years Ago
@Miranda

Thank you . Smile
By rosuckmedia - 3 Years Ago
Hallo all,
I need your help
Do I need to enter anything here, I don't have a localhost yet.
Do you need a localhost at all??
https://forum.reallusion.com/uploads/images/633a1a39-4b82-4658-8297-b92d.png

But I can start Audio2Face
Greetings rosuckmedia

https://forum.reallusion.com/uploads/images/322ce9dc-8217-45a5-8248-26be.png
By rosuckmedia - 3 Years Ago
.Okay, I made a little video of my problem.
What am I doing wrong, do I have to do that at all. ??
Greetings rosuckmedia

By Bassline303 - 3 Years Ago
Wow Rob , where did you learn english so quick? BigGrin

Why do you need Nucleus Service?
As far as I understand it , it is only for working on a unix server with a couple of artists at the same project...or am I wrong?
By rosuckmedia - 3 Years Ago
Hi Bassline,
I took a quick course😁😁😁

I thought I needed this, but I need a localhost and do not know how to do it.
https://forum.reallusion.com/uploads/images/a5b3c3bf-ff8d-49a5-b5c8-ac27.png
Maybe you can help me.
Greetings rosuckmedia
By 3dtester - 3 Years Ago
Hi Robert.
Yes, you need to install a Nucleus Collaboration Service.



I've done that a few days ago, using this manual:
https://docs.omniverse.nvidia.com/prod_nucleus/prod_nucleus/installation/workstation.html

By rosuckmedia - 3 Years Ago
Hello 3dtester
Thank you, πŸ˜Š
I have also seen this instruction.
Question why do we need this.???

I have now managed to unlock localhost.
There are all the samples in there that you need.
https://forum.reallusion.com/uploads/images/68333070-81d2-422a-8b1e-0fe4.png
I have enough of Omniverse AudioFace2 for today.😁
Greetings Robert
By Bassline303 - 3 Years Ago
Ok , good to know !  
Didn't test it yet ... 
By 3dtester - 3 Years Ago
Here is already the first problem Wink
I have opened the file "Viking_Mark_Transfered.usd", hid the "mark_model", and unhid the "CC_Base_Body" mesh.
But it appears on top of the other mesh. In the tutorial video #2 it is side-by-side.

https://forum.reallusion.com/uploads/images/8fce3f31-7da5-45ca-bf30-fa49.jpg

https://forum.reallusion.com/uploads/images/f8e890e6-18d3-437b-a9c2-21e1.jpg

Crying
By rosuckmedia - 3 Years Ago
Hi 3dtester
I have the same Problem😒
Greetings Robert
By adamlan - 3 Years Ago
1, localhost is not necessary, but the sample files are download to localhost automatically.\

2,The translation gizmo of Viking is on word space (0,0,0)
You can pitch the camera down to find the translation gizmo
Once you draged the gizmo, the transform section of property window may show out.
By rosuckmedia - 3 Years Ago
@adamlan,
Many thx for information.πŸ‘
Greetings rosuckmedia
By rosuckmedia - 3 Years Ago
Hi all.
In the first tutorial after the post wrap,
Both characters have a viseme.
https://forum.reallusion.com/uploads/images/f4820fd2-70ad-4048-bea1-9ba8.png

In my case, only one character, see video.
I did everything as in the tutorial.
Can this be due to my graphics card RTX 2080 ??
Greetings Robert

By Miranda (RL) - 3 Years Ago
It seems missing the step of Confirm Mesh Attachment.
at 3:00 in this video:

By adamlan - 3 Years Ago
This situation happens sometimes...
Please open OmniGraph -> Editor....
And check if the graph like this:
https://forum.reallusion.com/uploads/images/b89bafda-bbd5-40b4-bc8f-c75d.png

If it is.
Save and reopen this USD file can probably solve it 
If still not function, maybe share your file let us check it out. 
By rosuckmedia - 3 Years Ago
Hello Miranda and adamlan.
Thanks for the information, now it worked for me.πŸ˜€
Greetings rosuckmedia
https://forum.reallusion.com/uploads/images/eee2aa3b-f1b2-4e4e-b674-bea9.gif
By rosuckmedia - 3 Years Ago
Hi all,
My fisrt Beginner Test Omniverse Audio2Face Beta.
Greetings rosuckmediaπŸ˜€


By Bassline303 - 3 Years Ago
Cool Rob , go ahead , I've to learn it too  BigGrin
By 3dtester - 3 Years Ago
Hi, i got it working too Smile

Made that clip just 4 fun.
It is based on a voice track which is part of the Audio2Face samples.
The character is from the free 2021 Lip-Sync Content Pack BigGrin



By Miranda (RL) - 3 Years Ago
Great to know that you both got it work!
Looking forward to seeing more language lip sync animations. Smile
By rosuckmedia - 3 Years Ago
Nvidia Omniverse AudioFace2 Beta for Iclone 7 , First German Test.

English translation from video
Hello all,
I am currently testing the beta version of Omniverse Audio 2 Face.
This is an alternative to Iclone 7 Acculipsync.
With this you can probably use all languages.
Iclone 7 Acculipsync is no longer developing for other languages.
I am only at the beginning, and must work myself in here first.
You need a RTX graphics card for software, which is unfortunately not available to everyone.
With kind regards rosuckmedia😊

By Bassline303 - 3 Years Ago
hahaha great Rob !   Cool I can learn from you BigGrin
By rosuckmedia - 3 Years Ago
Hi Bassline,
Thx😊
You can do it better than me, I'm just starting out😁😁😁
Greetings Robert
By 3dtester - 3 Years Ago
Very good, Robert Cool

For some reason this weird idea came to my mind w00t


Having fun too BigGrin
By Bassline303 - 3 Years Ago
hahaha ...very good !  That sounds really like a stupid German haha  BigGrin
Cool how it looks like , when I have the time I will join the Audio2Face party πŸ€ͺ
By rosuckmedia - 3 Years Ago
Hi 3dtester
Thx😊
Hahaha very Cool.😁
you load the preset before ??, I've never used it before.
Greetings Robert
By charly Rama - 3 Years Ago
well done 3Dtester well done BigGrin
By 3dtester - 3 Years Ago
Thanks Wink

@Robert
Basically, the preset has to be loaded before the export.
But if you save and reload your project, these settings are already there.
So it has to be only done once. Despite if you change your target character to game mode and vice versa, or if you start a new project.
We can also save and load our own presets, but i currently don't see a reason for that...
By rosuckmedia - 3 Years Ago
@3dtester,
Many Thanks.😊
If I create my own preset, you shouldn't actually have to import the samples, you can do a little Tuto,
so that I can see
Greetings Robert
By 3dtester - 3 Years Ago
Unfortunately i currently can't do a tutorial Crazy
This is a highly complicated setup. I basically understand what it does, but i couldn't create these projects from scratch Sad
But it works very well using the sample files RL provides, and following the existing video tutorials Smile
BTW: The blendshape conversion presets which RL has created, are just perfect. So i think there is no need to adjust these setting by yourself Wink




By rosuckmedia - 3 Years Ago
Hello 3tester,
Okay, thanks,😊
I wish there was a tutorial from RL from start to finish.
I will stop testing for now until a complete tutorial appears.
Greetings Robert
By 3dtester - 3 Years Ago
Okay. But i will continue to use my saved projects for creating different languague speech and testing them in iClone.
This is a very helpflul workflow to say the least Wink

Stupid project quickly done again using Audio2Face

By Bassline303 - 3 Years Ago
3dtester (10/3/2021)
Okay. But i will continue to use my saved projects for creating different languague speech and testing them in iClone.
This is a very helpflul workflow to say the least Wink

Stupid project quickly done again using Audio2Face



Cool ! That looks great 3dtester !
By Bassline303 - 3 Years Ago
So , starting up now : 

https://forum.reallusion.com/uploads/images/dcc5a876-79ec-4eb4-a353-6eab.jpg


By Bassline303 - 3 Years Ago
BTW , there is a NVIDIA Omniverse Contest going on :

https://forum.reallusion.com/uploads/images/7b4961fa-b388-466b-8684-752c.png


By rosuckmedia - 3 Years Ago
I've resumed my omniverse tests
This time I also loaded the RL samples.
Englisch and German Test
Will try to refine everything.
Greetings Robert😁


By Bassline303 - 3 Years Ago
That is coming in Omniverse :
AIAudioToGesture ...a.s.o

By Miranda (RL) - 3 Years Ago
Hi everyone!

iClone Omniverse Connector is around the corner!
When it officially launches, we will move this post to the sub-forum of iClone Omniverse Connector.

The URL will be: (it's not accessible at the moment)
https://forum.reallusion.com/iClone/Plugins-Others/Omniverse-Connector

See you soon! Smile
By Smit_R - 3 Years Ago
After seeing the demo video (rapper) I thought I would give it a try, but I can't seem to find that lower face blendshape preset that is used.
Only the preset of the Viking and a female (which I'm using, because the Viking one is giving me an error).

I mean, I'ts looking fine
in audio2face before applying, but once imported in iClone I got no movement.
There must be something I'm missing. Can someone point me in the right direction? I really want to try this stuff out.

By 3dtester - 3 Years Ago
Hi @Smit_R

The blendshape presets only uncheck a bunch of entries in the blendshape list.
You can compare the list after loading the preset.
BTW: i have used the Viking blendshape preset as this is for the CC3+ base.
The other one is for GameBase characters.

https://forum.reallusion.com/uploads/images/b8fdff54-eb75-44c1-b416-93d4.jpg

https://forum.reallusion.com/uploads/images/4617a93e-fc68-47ba-a042-79eb.jpg

I find it unlikely that the problem should have something to do with the blendshape preset.
You say you can see the animation in Audio2Face, but not in iClone.
If you want you can send me your export file and i can check if it works here.

BTW, i created my own Audio2Face project based on the sample files.
It uses the "Mark" heads from Omniverse and the CC3 sample head from RL.

https://forum.reallusion.com/uploads/images/76e7b84e-505f-451b-92c5-21c0.jpg

By Smit_R - 3 Years Ago
@3dtester
Thanks for the swift reply and answer.

I'm definately gonna check it out and let you know when it fails on me (again).
It's also a good idea for having a preset Project.

By Miranda (RL) - 3 Years Ago
Hi everyone,

Python plug-in could be downloaded from Marketplace now.
https://marketplace.reallusion.com/omniverse-audio2face-plug-in


Miranda
By 3dtester - 3 Years Ago
Watch this. Very insightful and inspiring Cool


By rosuckmedia - 3 Years Ago
I did a little test, but in English. The new plugin works very well.πŸ‘
Greetings Robert

By toystorylab - 3 Years Ago
rosuckmedia (11/16/2021)
I did a little test, but in English. The new plugin works very well.πŸ‘
Greetings Robert


Nice!
By rosuckmedia - 3 Years Ago
Hi Pieter,
Thanks, I sang that😁😁😁😁😁
By Bassline303 - 3 Years Ago
Rob da Robber ahh Rapper BigGrin Cool!
By animagic - 3 Years Ago
Just starting out with this as well. For some reason I'm missing the RL blendshape presets...Unsure
By rosuckmedia - 3 Years Ago
@Bassline
thx😁
@animagic
You can find the presets here Game Base and CC3_Plus

localhost-Nvidia-Assets-Characters-Reallusion-Audio2Face_Preset_ExamplesDebra_AF"_CC_GameBase-blendshape preset

localhost-Nvidia-Assets-Characters-Reallusion-Audio2Face_Preset_ExamplesViking_A2F"_CC3_Plus-blendshape preset

Greetings Robert😊
By animagic - 3 Years Ago
rosuckmedia (11/17/2021)
@Bassline
thx😁
@animagic
You can find the presets here Game Base and CC3_Plus

localhost-Nvidia-Assets-Characters-Reallusion-Audio2Face_Preset_ExamplesDebra_AF"_CC_GameBase-blendshape preset

localhost-Nvidia-Assets-Characters-Reallusion-Audio2Face_Preset_ExamplesViking_A2F"_CC3_Plus-blendshape preset

Greetings Robert😊

Thanks, Robert, I've found them.
By planetstardragon - 3 Years Ago
@RL -  I saw Turbosquid just added an omniverse section  https://www.turbosquid.com/Search/3D-Models/omniverse

I did a little research,  and I see where you are going with this -  smart,  kudos to the R&D department. Wink

@nvidia -  ease up on videocard requirements please  - inflation and covid are making the cost of living a priority, you'll have a larger audience making space for freelancers - rich kids don't need to freelance. \Pinch/

By alerender - 3 Years Ago
rosuckmedia (11/17/2021)
@Bassline
thx😁
@animagic
You can find the presets here Game Base and CC3_Plus

localhost-Nvidia-Assets-Characters-Reallusion-Audio2Face_Preset_ExamplesDebra_AF"_CC_GameBase-blendshape preset

localhost-Nvidia-Assets-Characters-Reallusion-Audio2Face_Preset_ExamplesViking_A2F"_CC3_Plus-blendshape preset

Greetings Robert😊


Hello, I cannot access the Local Host. Any idea how I could fix it? ThanksSOLVED  I was able to solve it by installing Nucleus

By 3dtester - 3 Years Ago
Finally i am able to create Audio2Face Setups "from scratch", and made a small tutorial video.
I'm not 100% sure if i am doing everything correctly, but at least i think i figured how it basically works.


By hobbyuser3d - 3 Years Ago
thanks for posting your test..im curious to try but omniverse apps is going to require me upgrading my windows 10 to newer build.
i think you can get close results with hand puppet..possibly even better in some cases because its not computer generated
and has some timing variance..
By 3dtester - 3 Years Ago
hobbyuser3d (12/29/2021)
thanks for posting your test..im curious to try but omniverse apps is going to require me upgrading my windows 10 to newer build.
i think you can get close results with hand puppet..possibly even better in some cases because its not computer generated
and has some timing variance..


Yes, Omniverse requires at least Windows 1909.
The In-Place Upgrade of Windows works very well.
Of course, always make a backup of your system before you do that.

Audio2Face is multi-lingual.
I agree it is possible with iClone AccuLips to achive very good results, but it would be even better if it would recoginze different languages.
It depends on your project. For example if you have a lot of non-english recorded speech, you can easily get lip-sync with one click using Audio2Face.
With AccuLips that might be a lot of more work.
By toystorylab - 3 Years Ago
3dtester (12/28/2021)
Finally i am able to create Audio2Face Setups "from scratch", and made a small tutorial video.

Thanx Wink
By hobbyuser3d - 3 Years Ago
yeah i agree its very easy to lipsynch anything familiar own language, songs you know by heart
thats why i suggested the viseme overlay feature scrolling it kind of addresses some issues with timing
and (language doesnt matter as its a letter representation of sound they mean nothing read) as you then have visual cues when speech begins..especially useful if your working with music a band gives you
a track and wants a animated video. Your own voice mocap or voice on a seperate track is less of an issue and more basic and easy to work with.

You wont be able to use viseme in such cases as the algorithm will pick up instruments like drums etc but the visememe data is useful as a visual
cue when hand puppetting the interface as it is now is not designed for it ..it should be an overlay in top screen bottom screen options and in the viewport
right in front of your eyes and close to the work.


By 3dtester - 2 Years Ago
You are right, it is very helpful to have the speech in the timeline.
Although in some cases (like someone singing a song), it might be an option.
I already have ideas to try out BigGrin


By alerender - 2 Years Ago
https://forum.reallusion.com/uploads/images/a69ac3f9-1e24-425a-9498-bf39.jpg

it gives me this error when I try to import the viking preset . ...and if I try with the game_base ... in iclone the facial animation is not produced. Any idea how to solve it?

and this error is when I just select that .json ...

https://forum.reallusion.com/uploads/images/5feb458b-3fbf-454b-b7ee-c66b.jpg
By alerender - 2 Years Ago
Sorry, I don't know how to delete this post
By Boris Goncharuk - 2 Years Ago
Thanks a lot for the detailed description of the pipeline!
Someone wrote above about the difficulty of working with audio2face when animating songs when the vocals are superimposed on the instrumental accompaniment. It may be useful to someone - I found an online program for isolating vocals here: https://vocali.se/en  It works fine, but still leaves artifacts, especially in the voice pauses; I personally fix that with a free program I found here: https://www.audacityteam.org/
Of course, the isolated voice is only fed to the audio2face input, you can load the original track into iClone
By Data Juggler - 2 Years Ago
adamlan (9/24/2021)
This situation happens sometimes...
Please open OmniGraph -> Editor....
And check if the graph like this:
https://forum.reallusion.com/uploads/images/b89bafda-bbd5-40b4-bc8f-c75d.png

If it is.
Save and reopen this USD file can probably solve it 
If still not function, maybe share your file let us check it out. 


Can you explain what those node looking things are? I see those in several software applications, and videos and people use them and they all act like we know what those little boxes are / do / and why?

It's funny I had to watch a YouTube video to find this.

By SCE - 2 Years Ago
When can we look forward to the iClone 8 Audio2Face connector? The Audio2Emotion update coming in a few weeks will be a huge help, and I really hope it can be released around the same time!
By Beansupreme - 2 Years Ago


Does anyone else have this problem when selecting the cc3 plus profile? I haven't had any luck importing a cc3/4 character into audio2face. I had to strip everything out in blender except the main body mesh and manually place the markers for it to work. Just curious if there's a specific workflow other than what is posted in the main videos?


By Beansupreme - 2 Years Ago
I'm an idiot. I watched the video a million times but never noticed that you have to load a specific exported audio2face sub usd file. derp!
By faradays.infinity - 2 Years Ago
WHAT am i doing wrong??????????
as far as i can tell, i have installed the downloaded plugin for importing A2F in the correct folder yet i cannot get the importer to show in plugins
https://cdn.discordapp.com/attachments/828737244335833148/982955752496975922/unknown.png
https://cdn.discordapp.com/attachments/828737244335833148/982955888098824192/unknown.png

https://cdn.discordapp.com/attachments/828737244335833148/982956349417734154/unknown.png
By Peter (RL) - 2 Years Ago
Unfortunately Python plugins for iClone 7 will not work in iClone 8. We are currently working on updating our own plugins. We have also asked our developers to update their plugins for iClone 8 as soon as they can. Hopefully it will not be too long before the iClone 7 plugins can be used in iClone 8.
By Beansupreme - 2 Years Ago
Would be great to have audio2face plugin sooner than later. Is there an estimated time frame? I accidently updated my cc3 to cc4 and now only works with iclone 8. So i'm kind of stuck until that plugin is updated.
By Beansupreme - 2 Years Ago
*double post
By faradays.infinity - 2 Years Ago
I got direct word from the developer it seems, sadly i cannot wait until then so i will be asking help for some friends or finding another solution entirely

thank you for the responses it makes me feel not alone shouting into the void

By Beansupreme - 2 Years Ago
Great, glad to have a timeline to work with. Thanks for the update Smile
By e0175511 - 2 Years Ago
is there anyway to batch load Json file into this audio2face plugin and export the fbx out?
I wish to do this through scripting in python, cuz imagine if u hv 1000+ audio2face json file to process, impossible we load it one by one juz like the tutorial video.
Any guide or suggestion would be useful, appreciate for the help!
By Bassline303 - 2 Years Ago
By manuel.hendry - 2 Years Ago
Audio2Face 2022.1 is out. Can we please have an update on the release date for the Audio2Face plugin for iClone 8? Very much looking forward to this. Thanks!
By 3dtester - 2 Years Ago
🌟Bump🌟
By bad_character - 2 Years Ago
It’s been out for a while. I used it with the latest audio2face emotion release just the other day.
By Sepiasiren2021 - 2 Years Ago
Been playing with it and love the ease of use, but I can't seem to 1) stop it from loading the full body and 2) get it to map the face so it speaks.  I have to mask the clothing once I pull it in.  I have followed Solomon's tutorial and the methodology shown in the videos. I am uncertain as to what I need to do because I am following the instructions.  I will attach a screenshot here.https://forum.reallusion.com/uploads/images/fe1575b4-7901-4b00-a639-5d78.png 
By Bassline303 - 2 Years Ago
Awesome , made with IClone & Omniverse :