I purchased an iPhone 12 Pro Max yesterday (my first iPhone since the 4 I think) and finally got to play with facial mocap. This is the first time I've played with mocap and I'm having a blast so far. In one of the tutorial videos it is mentioned that lighting, internet connection, etc can affect the quality of the mocap data. I used wifi last night and am picking up an ethernet adapter tonight.

Have any of you experimented with different lighting? Was there a noticeable difference in data quality between lighting? What lighting would you recommend?

Are there any other tips and tricks you all know about to improve the quality of the mocap data?

I would argue that lighting plays almost zero influence on the Live Face capture as it can capture in near darkness (unlike Faceware - my guess is the person doing the tutorial was getting it confused with that.  Or he was just wrong).

Unless someone can show me something that demonstrates otherwise that is my position.

---

Alienware Aurora R16, Win 11, i9-149000KF, 3.20GHz CPU, 64GB RAM, RTX 4090 (24GB), Samsung 870 Pro 8TB, Gen3 MVNe M-2 SSD, 4TBx2, 39" Alienware Widescreen Monitor
Mike "ex-genius" Kelley

Thank you Kellytoons! The Live Face capture is really impressive. I had poor lighting last night and it worked pretty great out of the box. So, it sounds like I shouldn't overthink my lighting setup.

Here is the tutorial I was talking about (10:53):
Master Class - Getting The Best iPhone Mocap in iClone - by 3DTest: https://www.youtube.com/watch?v=g1GKgaexCDg

He says that "... data quality is variable depending on many factors. Not least the users face, lighting, PC connection and lots more." But that is all he says and it doesn't look like he had great lighting in that video.

He's certainly had the most experience of anyone (I know he was beta and I know he's created nearly all the tutorials on the process).  However, I have disagreed with him in the past on certain things (with Faceware it was frame rate - he was a big believer in a high framerate and I STILL think the process is better with a lower framerate, like around 30fps instead of the 60 or more he recommended).

But - anyone can say anything (including me <g>.  If you have any serious doubts or questions you can always test things out for yourself.  I really think that's best anyway, and if something isn't working for you then change it (conversely, don't fix what isn't broken).

---

Alienware Aurora R16, Win 11, i9-149000KF, 3.20GHz CPU, 64GB RAM, RTX 4090 (24GB), Samsung 870 Pro 8TB, Gen3 MVNe M-2 SSD, 4TBx2, 39" Alienware Widescreen Monitor
Mike "ex-genius" Kelley

You make a very good point about testing for myself. I was already picking up an ethernet adapter so I grabbed a cheap 6in ring light to quickly test. The light has 10 different brightness levels and three different colors. I found that lighting does have a small affect on the noise levels. Now how much of a difference lighting makes after doing a zero pose and turning on smoothing, I can't say yet.

I suspect having even lighting across your face does help some, but it was probably more of a factor with earlier tech. As you said, Live Face can capture in near darkness. It seems to me that the apps AI is good enough to compensate for most situations. I doubt the average user needs to worry about lighting. But I'm also new to this and talking out of my butt. lol Still for $30-40, its pretty affordable for people to test for themselves. Plus if you ever do any live streaming or recording showing your face, its nice to have a ring light.

I'm going to do some more testing and if I see any significant value in using the ring light vs my crappy overhead light, I'll post my findings here.

The TrueDepth camera works with infra-red (it projects 30,000 infrared dots on the face) to be able to do face recognition under any light conditions. Face recognition is used to allow you to unlock your phone with your face (Face ID).

I got this from this article: https://support.apple.com/en-us/HT208108. 

So my guess is that very bright lighting may actually impact the recognition in a negative way, and that any other lighting is just for your benefit, not so much for the phone.

---

Thank you animagic!

I did notice that on the brightest settings there was a lot more noise (and it was kind of blinding me). Different settings removed noise from one area but would add noise to another. When I used the ring light at lower levels, having the even lighting on my face seemed to slightly reduce the amount of noise over all compared to the light from my ceiling fan. But after reading the article you shared and finding the one below, Live Face uses the heat signatures reflecting off your face, so of course different lighting would affect the heat signature and cause different amounts of noise. The phone also has sensors that help the camera adjust to the the correct amount of illumination.

So, lighting doesn't really matter. If you use the same data profile as a starting point across multiple characters and like to fine tune each data setting, maybe using the same lighting will speed up your workflow slightly. But it seems the phone/app does all the heavy lifting, so we can get to creating.

The below information was pulled from
https://www.forbes.com/sites/jvchamary/2017/09/16/how-face-id-works-apple-iphone-x/?sh=23118f6b624d

1. The proximity sensor and ambient light sensor help the TrueDepth camera system determine how much illumination will be needed for face recognition;
2. The flood illuminator produces infrared (IR) light, part of the electromagnetic spectrum that's invisible to the naked eye, to illuminate your face;
3. The dot projector produces more than 30,000 dots of invisible IR light to create a three-dimensional map (for area and depth) of your facial landscape;
4. The infrared camera captures images of the dot pattern and the IR light (a heat signature) that's been reflected back from your face

Ah, you guys with all your facts and everything <g>.

To me it was just a gut feeling but, as I said, doing your own tests always is best because then it tells YOU what is important.  I suspect had I been more diligent I would have fixed my lighting issues with Faceware a long time ago (as it was, though, I had more issues with my glasses wearing, something which Live Face doesn't seem to care about at all).

---

Alienware Aurora R16, Win 11, i9-149000KF, 3.20GHz CPU, 64GB RAM, RTX 4090 (24GB), Samsung 870 Pro 8TB, Gen3 MVNe M-2 SSD, 4TBx2, 39" Alienware Widescreen Monitor
Mike "ex-genius" Kelley

This is my first iPhone ever so I like to learn...

It is also a good explanation why it is not trivial to get the same results on an Android phone, unless it includes the same setup. I'm not an Apple fan but one has to be realistic.

---