Profile Picture

CC4 and iC8 ARKit to other ARKit

Posted By bastion_100711 5 Months Ago
You don't have permission to rate!
Author
Message
bastion_100711
bastion_100711
Posted 5 Months Ago
View Quick Profile
Junior Member

Junior Member (164 reputation)Junior Member (164 reputation)Junior Member (164 reputation)Junior Member (164 reputation)Junior Member (164 reputation)Junior Member (164 reputation)Junior Member (164 reputation)Junior Member (164 reputation)Junior Member (164 reputation)

Group: Forum Members
Last Active: 2 Months Ago
Posts: 26, Visits: 168
Can someone tell me what I'm missing?  If CC4 and iC8 use ARKit why do I have issues when I export their animations to other non-Reallusiion Characters using ARKit?
seencapone
seencapone
Posted 2 Months Ago
View Quick Profile
New Member

New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)

Group: Forum Members
Last Active: Last Month
Posts: 5, Visits: 25
More info is needed. Exactly what problems are you having? Reallusion definitely has its own ideas about incorporating ARKit into its characters.
For starters, they randomly decided to internally rename the standard ARKit expressions, which makes retargeting an exported character in a 3rd party program a big pain in the ass. The CC4 Extended facial profile has lots of additional uniquely-named expressions beyond the standard 54 ARKit expressions.
I wish Reallusion would provide a Facial Profile template which ONLY has the standard ARKit expressions named correctly, but you can create your own 'Custom' set in the Facial Profile Editor>Edit Expressions in about half an hour & save as a custom profile. Start by loading the CC3+ Traditional facial profile which will load ARKit expressions into a 'Custom' set within the Facial Profile Editor (although you'll have to rename them individually) and as an extra step, deactivate all the Traditional expressions unless you want them later.

The *biggest* problem is that CC4 expressions use a combination of mesh deformation (otherwise known as a blend shape) AND bone/joint transformation with the 'Eyes Look Up/Down/In/Out' and the 'Jaw Open/Mouth Open' expressions.
For example, in CC4 & iClone, the 'Eye Look Up' expression slider works as expected: the eyeball rotates upward and the eyelid/eye muscle mesh deforms accordingly. However, when you export the character to use in another 3D program, the eyeball motion is NOT included in the blendshape action, because CC4 uses joint rotation for the eyeball. The eyelid facial geometry will 'look up' but the eyeball geometry remains fixed.

So obviously, when using the iPhone for face capture, it doesn't know about bones or joints, it only cares about facial geometry. You cannot retarget eyeball transforms to an eye joint because that data doesn't exist within ARKit. (It depends on which ARKit app you use, but I think most of them output eyeball tracking motion as part of the blendshape rather than as rotational data.)
Reallusion hasn't offered any solution for this, which is profoundly weird, because eye tracking is the most important aspect of face capture besides mouth & lip synch.

For the jaw/mouth expression, which also uses a combo of mesh and bone, fortunately there is a solution. When you export the character as .fbx, in the options panel (click the gear icon at the top of the Export FBX window to open) you will see a checkbox "Mouth Open As Morph" which will create a new blendshape called "Mouth_Merged" or something like that, which you can set as the target for ARKit's "jawOpen" shape.

I don't know if this helps solve your problems without knowing more. I've been trying to figure out a good pipeline for ages, so if anyone has any tips or workarounds, I'd appreciate it.

Rampa
Rampa
Posted 2 Months Ago
View Quick Profile
Distinguished Member

Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)

Group: Forum Members
Last Active: 10 hours ago
Posts: 8.2K, Visits: 62.3K
Try checking the box labled "Convert Skinned Expressions to Morphs" that is right below "Mouth Open as Morph". It is a newer option that covers not only the eyes, but also the mouth and tongue.
seencapone
seencapone
Posted 2 Months Ago
View Quick Profile
New Member

New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)

Group: Forum Members
Last Active: Last Month
Posts: 5, Visits: 25
Thanks. I played around with that export option after spotting it, but couldn't find any documentation on the workflow or mention of it online. The results were NOT a good solution to the problem.

'Convert Skinned Expressions to Morphs' sounds ideal, but what it does is create an unholy mess upon import into Maya -- dividing up the character into individual geometry and adding multiple unnecessary, redundant blendshape groups to each one, bloating the file size and making navigation to the appropriate expression very difficult. And what's more, it does not fix the problem, which is simply to optimize the character for use with ARKit face capture. It actually compounds the problem by further splitting the model's blendshapes apart from each other. It also breaks the mouthOpen/jawOpen and disconnects everything, so that the teeth and eyes are left floating when the head moves, and so on. This makes any potential motion capture retargeting a nightmare.
 Do you know what this option was originally designed to do? It seems to make things more complicated, not less.

***What is needed is a simple 'EyeLook to Morph' checkbox which elegantly unifies the eyeball rotation to its respective blendshape, the same way that 'Mouth Open as Morph' does for the jaw & teeth. ***

I have been struggling for days to detach the eyeball morph from its connection to the eye bone, but it seems hardwired into CC4 and cannot be modified or tweaked within the Facial Profile editor by any approach. Perhaps if the 'Convert Skinned Expressions to Morphs' option allowed you to further select WHICH expressions you wanted to convert (namely the eyeLook morphs) it would be more useful than its current leafblower approach.
seencapone
seencapone
Posted 2 Months Ago
View Quick Profile
New Member

New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)

Group: Forum Members
Last Active: Last Month
Posts: 5, Visits: 25
Ok. I went back in to try the 'Skinned Expression as Morphs' output option again, using a fresh base CC4+ Kevin character, and exported it into Maya.
I cannot possibly imagine the intended use of this feature!!
It breaks all the unity of the blendshapes completely. To get it to work with motion capture retargeting, for example, you would have to go in and manually reconnect the jawOpen blendshape across the base Body, Tongue, and Teeth individually just to get a simple mouth to open.
To solve my original problem (prepping the eyeLook expressions for use with ARKit), you would still have to go in and manually connect the base_Body's 'Eye Look <direction>' blendshapes to drive the blendshapes on the base_Eye geometry, which is the opposite solution.
That would be a pain enough, but now the base_Eye geometry contains a whole set of useless blendshapes for EVERY facial expression. Why the heck does the Eye geometry need all the mouth and tongue blendshapes and visemes spooged into it? Same goes for the teeth and tongue, which are now bloated with all the eye and brow blendshapes for no reason.

This export option could potentially solve some problems but it needs to be a LOT more refined!!
--

Rampa (9/25/2024)
Try checking the box labled "Convert Skinned Expressions to Morphs" that is right below "Mouth Open as Morph". It is a newer option that covers not only the eyes, but also the mouth and tongue.


Rampa
Rampa
Posted 2 Months Ago
View Quick Profile
Distinguished Member

Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)

Group: Forum Members
Last Active: 10 hours ago
Posts: 8.2K, Visits: 62.3K
Here is another possibility you can try. Found this searching Youtube for alternate methods that users may have found. This is from before that option was available.


seencapone
seencapone
Posted 2 Months Ago
View Quick Profile
New Member

New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)New Member (22 reputation)

Group: Forum Members
Last Active: Last Month
Posts: 5, Visits: 25
Thanks! I also found this tutorial last night and spent a few hours trying it out. I exported the .obj files and baked them back onto custom expression sliders that matched the ARKit Eye_Look blendshapes. It worked great inside CC4 and I thought my problem was solved. However, every time I exported the character as an .fbx and brought it into Maya, the results were always random and broken. It was weird. Sometimes the eyeball rotations would come in on a few of the blendshapes, sometimes on just one of them, sometimes none at all. There was no rhyme or reason or pattern to it, I traced my steps very carefully.

There's something going on internally, I guess -- relationships which are hardwired into the CC4 characters which doesn't let you screw with the expressions or skin/bone weighting. I don't know.

Unfortunately the author of this tutorial didn't test these baked morphs as an export into a third party 3D program -- she kept her workflow entirely between CC4 and iClone, where it works fine either way, bones or not -- so I'm not sure what the point of it was. But, I'm going to mess with this workflow some more, starting with a fresh character, and I'll let you know how it goes.
Rampa
Rampa
Posted 2 Months Ago
View Quick Profile
Distinguished Member

Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)Distinguished Member (37.4K reputation)

Group: Forum Members
Last Active: 10 hours ago
Posts: 8.2K, Visits: 62.3K
Did you upgrade to CC4.5? If you did, send a CC4support ticket for an updated version. There were some issues that may be affecting your export.



Reading This Topic