|
By Miranda (RL) - 4 Years Ago
|
Hello everyone,
When we first introduced Audio2Face in late July, we mentioned that we are working on the solution to bring Audio2Face animation back to iClone. Now it's time for everyone to experience it!
What is Audio2Face?
Omniverse Audio2Face is an AI-powered application that generates expressive facial animation from just an audio source. Audio2Face 2021.2.6 supports Character Creator CC3 Base+ and Game Base presets, which largely simplifies the wrap process for facial and lip-sync animation creation.
In ver. 2021.3.0, Audio2Face supports exporting animation as blendshapes (.json). This guide will also offer a solution for loading back the json data onto CC characters in iClone. - With the ability to bake Audio2Face blendshapes and export it back to iClone, and in combination with iClone's native facial animation tools, users now have another option for lip-syncing besides AccuLips. On top of this, comes the ability to export the character animations to other mainstream 3D engines such as Blender, Unreal Engine, Unity, etc.
- For Omniverse users, here is a total solution for full facial and body animation in combination with Audio2Face animation. Users can send the result to Omniverse via the upcoming iClone Omniverse Connector.
Please note that this is a Beta solution for sneak preview. The official solution will be released with iClone 8.Required Software & Files - Character Creator 3.44 or above
- iClone 7.93 or above
- iClone Python script for loading Audio2Face blendshape JSON
*Script is updated on Nov 4th for UI optimization. - Omniverse Audio2Face 2021.3.0
- Omniverse USD Sample Model (CC3+ Neutral Base) for generating animation and blendshape baking
*This file is no longer needed while using CC version 3.44.
*In the previous CC version 3.43, the blendshapes of the exported A2F head mesh are not named clearly, which makes it difficult to be identified in A2F. Before CC v3.44 releases, you can use the USD file that we provide as the sample model used in A2F, for generating animation and blendshape baking.
Workflow
Current version - iClone 7.93 has native USD export function: (3DXChange 7 Pipeline required)
 Previous version:
 Video Demo (no dubbing)*Recorded by using of CC3.43 and iClone 7.92: Part 1: Audio2Face Character Transfer Part 2: Audio2Face Blendshape Conversion (Viking)
Part 3: Using Sample Model for Blendshape Export (JSON) Part 4: Importing JSON, and Adjusting Strength Smooth in iClone Step-by-Step Guide
Please check THIS DOCUMENT for details. Reallusion Team
|
|
By Jeasy - 4 Years Ago
|
This is amazing news Miranda (RL), can't wait to check it out :)
Thanks!
|
|
By GARCIAGA - 4 Years Ago
|
It looks great, wich one will be better Accu-Lips or Audio2Face??
|
|
By mtakerkart - 4 Years Ago
|
|
I would like some help to test Audio2face with the french language. Reallusion's example offers 6 languages but not French ...
I don't have an rtx graphics card which is mandatory to use it. If it is accurate I would buy the smaller rtx card because it does not require resources. I put below an audio file of 20 seconds for the test. can you take a screenshot in video?
|
|
By ttamd - 4 Years Ago
|
Amazing, thanks, will digg into this.
|
|
By Miranda (RL) - 4 Years Ago
|
Hi GARCIAGA,
They are developed by using different algorithms. In my personal opinion, AccuLips provides flexibility in generating and fine tuning (e.g. word dictionary), while Audio2Face supports multi-languages, and more automatic functions will be introduced in the future. When Audio2Face is getting mature, it's definitely a great tool to work with.
GARCIAGA (9/20/2021)
It looks great, wich one will be better Accu-Lips or Audio2Face??
|
|
By Miranda (RL) - 4 Years Ago
|
@Jeasy, ttamd,
Welcome to show what you come up with. :)
@Mark,
This is French. Doesn't sound like? :P
https://youtu.be/Q64C4CWOV8s?t=17
|
|
By rosuckmedia - 4 Years Ago
|
|
I see that the German language file is also missing.😢😢 Will it be added later???
Greetings rosuckmedia
|
|
By Miranda (RL) - 4 Years Ago
|
Hi rosuckmedia,
Do you have a chance to try it?
Audio2Face shall work with any languages, and Omniverse Team continually updating with more and more languages. :)
rosuckmedia (9/22/2021)
I see that the German language file is also missing.😢😢 Will it be added later???
Greetings rosuckmedia
|
|
By rosuckmedia - 4 Years Ago
|
|
Hi Miranda, Thanks for your quick reply.😊 I need to reinstall Omniverse first, then I will try it. I have a RTX 2080 should actually work.
Greetings rosuckmedia
|
|
By mtakerkart - 4 Years Ago
|
@Miranda
Thank you . :)
|
|
By rosuckmedia - 4 Years Ago
|
Hallo all,
I need your help
Do I need to enter anything here, I don't have a localhost yet. Do you need a localhost at all??
But I can start Audio2Face
Greetings rosuckmedia
|
|
By rosuckmedia - 4 Years Ago
|
|
.Okay, I made a little video of my problem. What am I doing wrong, do I have to do that at all. ??
Greetings rosuckmedia
|
|
By StyleMarshal - 4 Years Ago
|
Wow Rob , where did you learn english so quick? :D
Why do you need Nucleus Service?
As far as I understand it , it is only for working on a unix server with a couple of artists at the same project...or am I wrong?
|
|
By rosuckmedia - 4 Years Ago
|
|
Hi Bassline, I took a quick course😁😁😁
I thought I needed this, but I need a localhost and do not know how to do it.
Maybe you can help me.
Greetings rosuckmedia
|
|
By 3dtester - 4 Years Ago
|
Hi Robert.
Yes, you need to install a Nucleus Collaboration Service.
I've done that a few days ago, using this manual:
https://docs.omniverse.nvidia.com/prod_nucleus/prod_nucleus/installation/workstation.html
|
|
By rosuckmedia - 4 Years Ago
|
|
Hello 3dtester Thank you, 😊
I have also seen this instruction. Question why do we need this.???
I have now managed to unlock localhost. There are all the samples in there that you need.  I have enough of Omniverse AudioFace2 for today.😁 Greetings Robert
|
|
By StyleMarshal - 4 Years Ago
|
Ok , good to know !
Didn't test it yet ...
|
|
By 3dtester - 4 Years Ago
|
Here is already the first problem ;)
I have opened the file "Viking_Mark_Transfered.usd", hid the "mark_model", and unhid the "CC_Base_Body" mesh.
But it appears on top of the other mesh. In the tutorial video #2 it is side-by-side.
:crying:
|
|
By rosuckmedia - 4 Years Ago
|
Hi 3dtester
I have the same Problem😢
Greetings Robert
|
|
By adamlan - 4 Years Ago
|
1, localhost is not necessary, but the sample files are download to localhost automatically.\
2,The translation gizmo of Viking is on word space (0,0,0)
You can pitch the camera down to find the translation gizmo
Once you draged the gizmo, the transform section of property window may show out.
|
|
By rosuckmedia - 4 Years Ago
|
@adamlan,
Many thx for information.👍
Greetings rosuckmedia
|
|
By rosuckmedia - 4 Years Ago
|
|
Hi all.
In the first tutorial after the post wrap, Both characters have a viseme.
In my case, only one character, see video. I did everything as in the tutorial. Can this be due to my graphics card RTX 2080 ??
Greetings Robert
|
|
By Miranda (RL) - 4 Years Ago
|
It seems missing the step of Confirm Mesh Attachment.
at 3:00 in this video:
|
|
By adamlan - 4 Years Ago
|
This situation happens sometimes...
Please open OmniGraph -> Editor....
And check if the graph like this:
If it is.
Save and reopen this USD file can probably solve it
If still not function, maybe share your file let us check it out.
|
|
By rosuckmedia - 4 Years Ago
|
|
Hello Miranda and adamlan. Thanks for the information, now it worked for me.😀 Greetings rosuckmedia  |
|
By rosuckmedia - 4 Years Ago
|
Hi all,
My fisrt Beginner Test Omniverse Audio2Face Beta.
Greetings rosuckmedia😀
|
|
By StyleMarshal - 4 Years Ago
|
|
Cool Rob , go ahead , I've to learn it too :D
|
|
By 3dtester - 4 Years Ago
|
Hi, i got it working too :)
Made that clip just 4 fun.
It is based on a voice track which is part of the Audio2Face samples.
The character is from the free 2021 Lip-Sync Content Pack :D
|
|
By Miranda (RL) - 4 Years Ago
|
Great to know that you both got it work!
Looking forward to seeing more language lip sync animations. :)
|
|
By rosuckmedia - 4 Years Ago
|
Nvidia Omniverse AudioFace2 Beta for Iclone 7 , First German Test.
English translation from video Hello all, I am currently testing the beta version of Omniverse Audio 2 Face. This is an alternative to Iclone 7 Acculipsync. With this you can probably use all languages. Iclone 7 Acculipsync is no longer developing for other languages. I am only at the beginning, and must work myself in here first. You need a RTX graphics card for software, which is unfortunately not available to everyone. With kind regards rosuckmedia😊
|
|
By StyleMarshal - 4 Years Ago
|
|
hahaha great Rob ! Cool I can learn from you :D
|
|
By rosuckmedia - 4 Years Ago
|
|
Hi Bassline,
Thx😊 You can do it better than me, I'm just starting out😁😁😁
Greetings Robert
|
|
By 3dtester - 4 Years Ago
|
Very good, Robert :cool:
For some reason this weird idea came to my mind :w00t:
Having fun too :D
|
|
By StyleMarshal - 4 Years Ago
|
hahaha ...very good ! That sounds really like a stupid German haha :D
Cool how it looks like , when I have the time I will join the Audio2Face party 🤪
|
|
By rosuckmedia - 4 Years Ago
|
Hi 3dtester
Thx😊
Hahaha very Cool.😁
you load the preset before ??, I've never used it before.
Greetings Robert
|
|
By charly Rama - 4 Years Ago
|
|
well done 3Dtester well done :D
|
|
By 3dtester - 4 Years Ago
|
Thanks ;)
@Robert
Basically, the preset has to be loaded before the export.
But if you save and reload your project, these settings are already there.
So it has to be only done once. Despite if you change your target character to game mode and vice versa, or if you start a new project.
We can also save and load our own presets, but i currently don't see a reason for that...
|
|
By rosuckmedia - 4 Years Ago
|
@3dtester,
Many Thanks.😊
If I create my own preset, you shouldn't actually have to import the samples, you can do a little Tuto, so that I can see
Greetings Robert
|
|
By 3dtester - 4 Years Ago
|
Unfortunately i currently can't do a tutorial :crazy:
This is a highly complicated setup. I basically understand what it does, but i couldn't create these projects from scratch :(
But it works very well using the sample files RL provides, and following the existing video tutorials :)
BTW: The blendshape conversion presets which RL has created, are just perfect. So i think there is no need to adjust these setting by yourself ;)
|
|
By rosuckmedia - 4 Years Ago
|
|
Hello 3tester, Okay, thanks,😊 I wish there was a tutorial from RL from start to finish. I will stop testing for now until a complete tutorial appears.
Greetings Robert
|
|
By 3dtester - 4 Years Ago
|
Okay. But i will continue to use my saved projects for creating different languague speech and testing them in iClone.
This is a very helpflul workflow to say the least ;)
Stupid project quickly done again using Audio2Face
|
|
By StyleMarshal - 4 Years Ago
|
|
3dtester (10/3/2021)
Okay. But i will continue to use my saved projects for creating different languague speech and testing them in iClone.
This is a very helpflul workflow to say the least ;)
Stupid project quickly done again using Audio2Face
Cool ! That looks great 3dtester !
|
|
By StyleMarshal - 4 Years Ago
|
So , starting up now :
|
|
By StyleMarshal - 4 Years Ago
|
BTW , there is a NVIDIA Omniverse Contest going on :
|
|
By rosuckmedia - 4 Years Ago
|
I've resumed my omniverse tests This time I also loaded the RL samples. Englisch and German Test Will try to refine everything.
Greetings Robert😁
|
|
By StyleMarshal - 4 Years Ago
|
That is coming in Omniverse :
AIAudioToGesture ...a.s.o
|
|
By Miranda (RL) - 4 Years Ago
|
Hi everyone!
iClone Omniverse Connector is around the corner!
When it officially launches, we will move this post to the sub-forum of iClone Omniverse Connector.
The URL will be: (it's not accessible at the moment)
https://forum.reallusion.com/iClone/Plugins-Others/Omniverse-Connector
See you soon! :)
|
|
By Smit_R - 4 Years Ago
|
After seeing the demo video (rapper) I thought I would give it a try, but I can't seem to find that lower face blendshape preset that is used.
Only the preset of the Viking and a female (which I'm using, because the Viking one is giving me an error).
I mean, I'ts looking fine in audio2face before applying, but once imported in iClone I got no movement.
There must be something I'm missing. Can someone point me in the right direction? I really want to try this stuff out.
|
|
By 3dtester - 4 Years Ago
|
Hi @Smit_R
The blendshape presets only uncheck a bunch of entries in the blendshape list.
You can compare the list after loading the preset.
BTW: i have used the Viking blendshape preset as this is for the CC3+ base.
The other one is for GameBase characters.
I find it unlikely that the problem should have something to do with the blendshape preset.
You say you can see the animation in Audio2Face, but not in iClone.
If you want you can send me your export file and i can check if it works here.
BTW, i created my own Audio2Face project based on the sample files.
It uses the "Mark" heads from Omniverse and the CC3 sample head from RL.
|
|
By Smit_R - 4 Years Ago
|
@3dtester
Thanks for the swift reply and answer.
I'm definately gonna check it out and let you know when it fails on me (again).
It's also a good idea for having a preset Project.
|
|
By Miranda (RL) - 4 Years Ago
|
Hi everyone,
Python plug-in could be downloaded from Marketplace now.
https://marketplace.reallusion.com/omniverse-audio2face-plug-in
Miranda
|
|
By 3dtester - 4 Years Ago
|
Watch this. Very insightful and inspiring :cool:
|
|
By rosuckmedia - 4 Years Ago
|
|
I did a little test, but in English. The new plugin works very well.👍 Greetings Robert
|
|
By toystorylab - 4 Years Ago
|
rosuckmedia (11/16/2021)
I did a little test, but in English. The new plugin works very well.👍 Greetings Robert
Nice!
|
|
By rosuckmedia - 4 Years Ago
|
Hi Pieter,
Thanks, I sang that😁😁😁😁😁
|
|
By StyleMarshal - 4 Years Ago
|
Rob da Robber ahh Rapper :D Cool!
|
|
By animagic - 4 Years Ago
|
|
Just starting out with this as well. For some reason I'm missing the RL blendshape presets...:unsure:
|
|
By rosuckmedia - 4 Years Ago
|
@Bassline
thx😁
@animagic
You can find the presets here Game Base and CC3_Plus
localhost-Nvidia-Assets-Characters-Reallusion-Audio2Face_Preset_ExamplesDebra_AF"_CC_GameBase-blendshape preset
localhost-Nvidia-Assets-Characters-Reallusion-Audio2Face_Preset_ExamplesViking_A2F"_CC3_Plus-blendshape preset
Greetings Robert😊
|
|
By animagic - 4 Years Ago
|
rosuckmedia (11/17/2021)
@Bassline
thx😁
@animagic
You can find the presets here Game Base and CC3_Plus
localhost-Nvidia-Assets-Characters-Reallusion-Audio2Face_Preset_ExamplesDebra_AF"_CC_GameBase-blendshape preset
localhost-Nvidia-Assets-Characters-Reallusion-Audio2Face_Preset_ExamplesViking_A2F"_CC3_Plus-blendshape preset
Greetings Robert😊
Thanks, Robert, I've found them.
|
|
By planetstardragon - 4 Years Ago
|
@RL - I saw Turbosquid just added an omniverse section https://www.turbosquid.com/Search/3D-Models/omniverse
I did a little research, and I see where you are going with this - smart, kudos to the R&D department. ;)
@nvidia - ease up on videocard requirements please - inflation and covid are making the cost of living a priority, you'll have a larger audience making space for freelancers - rich kids don't need to freelance. \:pinch:/
|
|
By alerender - 4 Years Ago
|
rosuckmedia (11/17/2021)
@Bassline
thx😁
@animagic
You can find the presets here Game Base and CC3_Plus
localhost-Nvidia-Assets-Characters-Reallusion-Audio2Face_Preset_ExamplesDebra_AF"_CC_GameBase-blendshape preset
localhost-Nvidia-Assets-Characters-Reallusion-Audio2Face_Preset_ExamplesViking_A2F"_CC3_Plus-blendshape preset
Greetings Robert😊
Hello, I cannot access the Local Host. Any idea how I could fix it? ThanksSOLVED I was able to solve it by installing Nucleus
|
|
By 3dtester - 4 Years Ago
|
Finally i am able to create Audio2Face Setups "from scratch", and made a small tutorial video.
I'm not 100% sure if i am doing everything correctly, but at least i think i figured how it basically works.
|
|
By hobbyuser3d - 4 Years Ago
|
thanks for posting your test..im curious to try but omniverse apps is going to require me upgrading my windows 10 to newer build.
i think you can get close results with hand puppet..possibly even better in some cases because its not computer generated
and has some timing variance..
|
|
By 3dtester - 4 Years Ago
|
|
hobbyuser3d (12/29/2021)
thanks for posting your test..im curious to try but omniverse apps is going to require me upgrading my windows 10 to newer build.
i think you can get close results with hand puppet..possibly even better in some cases because its not computer generated
and has some timing variance..
Yes, Omniverse requires at least Windows 1909.
The In-Place Upgrade of Windows works very well.
Of course, always make a backup of your system before you do that.
Audio2Face is multi-lingual.
I agree it is possible with iClone AccuLips to achive very good results, but it would be even better if it would recoginze different languages.
It depends on your project. For example if you have a lot of non-english recorded speech, you can easily get lip-sync with one click using Audio2Face.
With AccuLips that might be a lot of more work.
|
|
By toystorylab - 4 Years Ago
|
|
3dtester (12/28/2021)
Finally i am able to create Audio2Face Setups "from scratch", and made a small tutorial video.
Thanx ;)
|
|
By hobbyuser3d - 4 Years Ago
|
yeah i agree its very easy to lipsynch anything familiar own language, songs you know by heart
thats why i suggested the viseme overlay feature scrolling it kind of addresses some issues with timing
and (language doesnt matter as its a letter representation of sound they mean nothing read) as you then have visual cues when speech begins..especially useful if your working with music a band gives you
a track and wants a animated video. Your own voice mocap or voice on a seperate track is less of an issue and more basic and easy to work with.
You wont be able to use viseme in such cases as the algorithm will pick up instruments like drums etc but the visememe data is useful as a visual
cue when hand puppetting the interface as it is now is not designed for it ..it should be an overlay in top screen bottom screen options and in the viewport
right in front of your eyes and close to the work.
|
|
By 3dtester - 3 Years Ago
|
You are right, it is very helpful to have the speech in the timeline.
Although in some cases (like someone singing a song), it might be an option.
I already have ideas to try out :D
|
|
By alerender - 3 Years Ago
|
it gives me this error when I try to import the viking preset . ...and if I try with the game_base ... in iclone the facial animation is not produced. Any idea how to solve it?
and this error is when I just select that .json ...
|
|
By alerender - 3 Years Ago
|
|
Sorry, I don't know how to delete this post
|
|
By Boris Goncharuk - 3 Years Ago
|
|
Thanks a lot for the detailed description of the pipeline! Someone wrote above about the difficulty of working with audio2face when animating songs when the vocals are superimposed on the instrumental accompaniment. It may be useful to someone - I found an online program for isolating vocals here: https://vocali.se/en It works fine, but still leaves artifacts, especially in the voice pauses; I personally fix that with a free program I found here: https://www.audacityteam.org/Of course, the isolated voice is only fed to the audio2face input, you can load the original track into iClone
|
|
By Data Juggler - 3 Years Ago
|
adamlan (9/24/2021)
This situation happens sometimes... Please open OmniGraph -> Editor.... And check if the graph like this: If it is. Save and reopen this USD file can probably solve it If still not function, maybe share your file let us check it out.
Can you explain what those node looking things are? I see those in several software applications, and videos and people use them and they all act like we know what those little boxes are / do / and why?
It's funny I had to watch a YouTube video to find this.
|
|
By SCE - 3 Years Ago
|
|
When can we look forward to the iClone 8 Audio2Face connector? The Audio2Emotion update coming in a few weeks will be a huge help, and I really hope it can be released around the same time!
|
|
By Beansupreme - 3 Years Ago
|
Does anyone else have this problem when selecting the cc3 plus profile? I haven't had any luck importing a cc3/4 character into audio2face. I had to strip everything out in blender except the main body mesh and manually place the markers for it to work. Just curious if there's a specific workflow other than what is posted in the main videos?
|
|
By Beansupreme - 3 Years Ago
|
|
I'm an idiot. I watched the video a million times but never noticed that you have to load a specific exported audio2face sub usd file. derp!
|
|
By faradays.infinity - 3 Years Ago
|
WHAT am i doing wrong??????????
as far as i can tell, i have installed the downloaded plugin for importing A2F in the correct folder yet i cannot get the importer to show in plugins
|
|
By Peter (RL) - 3 Years Ago
|
|
Unfortunately Python plugins for iClone 7 will not work in iClone 8. We are currently working on updating our own plugins. We have also asked our developers to update their plugins for iClone 8 as soon as they can. Hopefully it will not be too long before the iClone 7 plugins can be used in iClone 8.
|
|
By Beansupreme - 3 Years Ago
|
|
Would be great to have audio2face plugin sooner than later. Is there an estimated time frame? I accidently updated my cc3 to cc4 and now only works with iclone 8. So i'm kind of stuck until that plugin is updated.
|
|
By Beansupreme - 3 Years Ago
|
|
*double post
|
|
By faradays.infinity - 3 Years Ago
|
I got direct word from the developer it seems, sadly i cannot wait until then so i will be asking help for some friends or finding another solution entirely
thank you for the responses it makes me feel not alone shouting into the void
|
|
By Beansupreme - 3 Years Ago
|
|
Great, glad to have a timeline to work with. Thanks for the update :)
|
|
By e0175511 - 3 Years Ago
|
is there anyway to batch load Json file into this audio2face plugin and export the fbx out?
I wish to do this through scripting in python, cuz imagine if u hv 1000+ audio2face json file to process, impossible we load it one by one juz like the tutorial video.
Any guide or suggestion would be useful, appreciate for the help!
|
|
By StyleMarshal - 3 Years Ago
|
|
|
|
By manuel.hendry - 3 Years Ago
|
|
Audio2Face 2022.1 is out. Can we please have an update on the release date for the Audio2Face plugin for iClone 8? Very much looking forward to this. Thanks!
|
|
By 3dtester - 3 Years Ago
|
|
🌟Bump🌟
|
|
By bad_character - 3 Years Ago
|
|
It’s been out for a while. I used it with the latest audio2face emotion release just the other day.
|