|
By tbseo - 5 Years Ago
|
It would be nice if there was a way to take a 2d clip of someone talking and crop the head and then use markers to mark the head, eyes nose and mouth (similar to deepfacelab) and then track the head and facial movements and apply it to the character.
Even if it's nowhere near perfect I think it would make the animation way more realistic if I'm using the same speech track from the 2d movie.
|
|
By Kelleytoons - 5 Years Ago
|
Faceware can do this.
|
|
By tbseo - 5 Years Ago
|
Thanks. It looks useful. I tried the trial but there doesn't seem to be any way to record anything without buying.
If I wanted to use this for my purpose can I just buy the one iclone plugin?
I browsed the website a little and I'm getting the impression I'm going to have to buy multiple programs just to do this one specific thing.
|
|
By Kelleytoons - 5 Years Ago
|
|
The total cost is around $1300 -- you will need Motion Live, the Faceware plugin, and the Faceware piece, but the last two are bundled together (actually, I think you can get all three together).
|
|
By tbseo - 5 Years Ago
|
Ya that sounds about right. Lol.
For some reason when you're just a hobbyist and want to do one specific thing its always insanely expensive.
With free deep learning apps coming out I can't see that business model lasting much longer.
Are there any alternative programs that can at least make facial animation easier and more realistic? I find it difficult in iclone for some reason. I'm going to try doing the face and speech first with only my character in the scene in the default pose and then try and save it out and see if it's easier with less clutter.
|
|
By Kelleytoons - 5 Years Ago
|
While it's true it's very expensive for a hobbyist, it's still incredibly cheap compared to what it used to be (I spent $3K on the DOS version of Max about 30 years ago, back when that kind of money could buy you a brand new car).
Elsewhere on this forum I made a thread about Papagayo, which enables you to do excellent lip sync easily and quickly (and free). Start with that, add in some facial expressions (which you can do quickly), and you've already got more than we had for many, many decades.
|
|
By tbseo - 5 Years Ago
|
Thanks I'll look into it.
Does iclone have a way of combining text and the audio track for better results? I think I used to be able to do that with Mimic or some other program from years ago.
All I can seem to do in iclone is type the text and it adds that text with a generic voice I don't want and I can't seem to seperate that audio track and delete it.
The text file gives much better results and I wanted to just reposition those vizemes under my own audio track.
I tried doing the text file and then my audio file after on the timeline and then delete the vizemes my audio created and cut and paste the text vizemes and try to sync them but iclone does not like that at all.
I have to paste the vizemes and then click on each one and reselect it or it won't register. Then at some point it always crashes.
Thanks again.
|
|
By Kelleytoons - 5 Years Ago
|
Look at my thread on this (which has an improved script for this process).
|
|
By charly Rama - 5 Years Ago
|
It's a pity because I've bought faceware, faceware plugin for Iclone , and I do no more use it , I use only live face with my iphone X. If transfer licence is possible, Y can sell it to you for a price less less expensive but I 'm not sure it's possible.
|
|
By tbseo - 5 Years Ago
|
Thanks for the tutorial. I'm going to give it a try.
You mention using the number 42 for 24 frames per second. I have no programming experience but do I need to use different numbers for 30, 50, and 60 fps?
Hopefully I can figure out how to do this.
I'm wondering if papagayo used to be that mimic program I used with poser.
|
|
By Kelleytoons - 5 Years Ago
|
Mimic is/was a separate program. Papagayo was a freeware program written by Mike Clifton, the author of Moho, to work with that program.
There are other threads about my program -- just search here for "Papagayo" and you'll see. And lots of folks to help you if you do need help.
Audio actually doesn't care what fps you are using (audio isn't synced to fps).
|
|
By Rampa - 5 Years Ago
|
You can adjust individual visemes within iClone.
|
|
By tbseo - 5 Years Ago
|
I think I got it working but I end up with an animation that needs some work but the vizemes do not show. If I add a vizeme on the empty track I lose all the other ones.
I noticed in the video you were expecting the same thing but then the vizemes miraculously appeared.
|
|
By Kelleytoons - 5 Years Ago
|
|
They should appear after you hit play.
|
|
By tbseo - 5 Years Ago
|
That's weird. I'm using pagayo2.py.
I load the model then add an audio file. The delete the visemes and run the script (with no editing). I have an Exports folder in c:/documents with the files. I load them and get mouth movement but no visemes to edit even after playing through.
The only thing I can think of is that I have iclone installed on my B drive. ?
I am getting lots of "no viseme at frame 78 AI (rest, etc) in the console window.
|
|
By Kelleytoons - 5 Years Ago
|
You don't need to worry about those error messages.
So the mouth moves? Then the visemes HAVE to be loaded. You might try saving the file and then reloading. And make sure you've got the avatar selected AND the viseme track opened up.
|
|
By tbseo - 5 Years Ago
|
I've tried about 40 times with different models and different versions of the script. Different locations on my drives.
I think there was one time in the beginning I saw the vizemes appear but it was a test so I tried again with a better avatar and lost it. I've been getting either no mouth movement or mouth movement but no viseme track ever since.
I'm perplexed.
|
|
By Kelleytoons - 5 Years Ago
|
|
I'd have to see a video of what you are doing (you have the video of what *I* did).
|
|
By tbseo - 5 Years Ago
|
I might try and do that tomorrow if I have time. I've never recorded my screen before. Does windows have a built in screen recorder?
I'm thinking it might have something to do with the mess ive made of my files.
Ive had little problems here and there and moved files around trying to fix it over the last year or 2.
Although it is strange the visemes are added fine every time when I just use iclone.
|
|
By Kelleytoons - 5 Years Ago
|
I use the freeware OBS studio (fully functioned web broadcaster -- might be overkill but I used to do a lot of Twitch stuff). I *think* there IS a native Win 10 recorder but I've never used it.
Failing that, we might just go through the steps one at a time and you can send me your files (like your audio and the .DAT file you created) and I can see if they load in properly on this end. I know we can figure out the issue (unfortunately my code isn't designed to be robust but only a proof of concept, although others have said it worked fine for them).
|
|
By tbseo - 5 Years Ago
|
Unique problems seem to be my specialty. I just tried a couple more times quickly and now when I select the viseme track and hit delete the whole character deletes. Tried twice with 2 different characters. I ended up having to select the viseme's and "cut".
Is there anything in the code I can change just as a desperate hail Mary?
Aside from having iclone on my B drive I also replaced my graphics card with a 1660 ti. I can't see how that would make a difference but who knows.
I just realized I actually have iclone on my C drive but my template folder is on the B drive. Another issue I have is if I load some characters (like Gwynn) and then try to run physics on the hair it just falls to the floor. When I check the weight map it's just a white square. The diffuse and opacity are there though.
If I recall this started happening when I did something with my directories.
I just loaded up Gwynn and tried your script on her and same problem with the mouth not moving and no visemes.
If the results were at least consistent Id have some hope.
If I get some time tonight I'll try and figure out a way to do a demo.
Thanks for all the help.
|
|
By Kelleytoons - 5 Years Ago
|
That graphics card worries me -- not about the viseme process, but even just running iClone. But if it loads and runs don't think about it.
I'm exhausted from tennis but we can walk through this. I need to check the current Python implementation and see if perhaps if my script that was improved by others here will run under the new version. That might be the answer (because IIRC it doesn't require an existing viseme track). That thread can be found here if you search "Papagayo" but I'll try it myself as soon as I have the energy (not a good day for me -- my wife fell down and severely hurt her shoulder and now I have to be nursemaid until we see an orthopod on Monday -- but I had already worn myself out on the court this morning).
We'll get it working -- don't give up (assuming that by looking at my video it seems like something you'd want).
|
|
By tbseo - 5 Years Ago
|
I'm doing lots of running around today with work but I'll hopefully get some time to tackle this with a clear head. Or at least tackle this...
Why does the graphics card worry you? I had a GTX 970 and never had a problem with iclone and then upgraded to the GTX 1660 ti a couple months ago.
Aside from my power supply starting to sound like a diesel engine everything is running smooth.
Hope you're wife is recouperating. Too bad dr's don't make weekend house calls. Unless you're Michael Jackson.
|
|
By tbseo - 5 Years Ago
|
|
I screen recorded with the windows Xbox app. It only shows the apps that you're working in but I'll edit it together and post it.
|
|
By tbseo - 5 Years Ago
|
Here's the video. It's a little weird without seeing the pop up windows etc but I added text that will hopefully make everything clear. I was really tired so I hope I didnt make any stupid mistakes.
https://www.sendspace.com/file/fgcvn8
|
|
By Kelleytoons - 5 Years Ago
|
|
I'll look at this this evening (it's been a very trying couple of days -- my wife broke her arm and we've been dealing with a lot of doctors).
|
|
By Kelleytoons - 5 Years Ago
|
|
Okay, first of all you HAVE to have the Console open while you run ANY Python scripts. That window will show you exactly what is happening (and here most likely tells you the error). So try again running the script with it open and see what it says. If you can't figure it out, put a screen shot here of it and that will tell us all we need to know.
|
|
By tbseo - 5 Years Ago
|
OK I tried it with the console open. I was hoping for an error message but it doesn't look like there was anything in the log. I uninstalled all the faceware stuff wondering if that might have been the issue since I installed it just a few days ago for the trial period but that didn't seem to make a difference either.
It sounds like you have your hands full at the moment so don't no rush. I've been making due with iclones viseme generation for now. It's just frustrating. Usually I can find my own little ways of doing things that end up looking realistic but there doesn't seem to be any wiggle room with iclone.
C:/Users/User/Desktop/one/Voice 1.dat
Start
No viseme at frame 1 rest
No viseme at frame 6 rest
No viseme at frame 5 AI
No viseme at frame 7 etc
No viseme at frame 8 O
No viseme at frame 9 MBP
No viseme at frame 10 etc
No viseme at frame 12 U
No viseme at frame 13 etc
No viseme at frame 14 FV
No viseme at frame 15 rest
No viseme at frame 17 rest
No viseme at frame 16 AI
No viseme at frame 18 etc
No viseme at frame 20 AI
No viseme at frame 21 L
No viseme at frame 23 E
No viseme at frame 24 etc
No viseme at frame 26 U
No viseme at frame 28 etc
No viseme at frame 33 AI
No viseme at frame 36 MBP
No viseme at frame 39 etc
No viseme at frame 41 AI
No viseme at frame 42 etc
No viseme at frame 46 rest
Finished
|
|
By Kelleytoons - 5 Years Ago
|
Wait -- what version of my program are you running? I think you did say Papagayo2 but you don't say otherwise.
My program absolutely requires your .WAV file to be in your \exports directory (from Documents) or it will throw an error. But it didn't so I assume you did put the .WAV file there. It also requires the .DAT file to be there (but it won't throw an error in that case). If you don't have both of those files there it won't run properly and that first line it's showing doesn't look like it (indeed, it doesn't even look like my code -- my code, again, throws errors and tells you why).
Are you sure that's not a very early version? Here's (always) the latest:
https://www.dropbox.com/s/g6kv4qynxo2bcv8/Pagayo2.py?dl=0
Run that and tell me what IT says to you.
|
|
By tbseo - 5 Years Ago
|
I downloaded the script several times and from several links so I guess I ended up losing track of the one that was giving me actual mouth movement. I did edit the script for the versions that required it.
I downloaded a fresh one from the link and put everything in the exports folder and tried again and got mouth movement but no visemes. I hit play and they don't appear. All attempts after that first one have given me no mouth movement or visemes.
I'm going to try a few more times and check something. I took a closer look at the mouth movements and they don't even come close to matching the audio even though it was only one sentence and I did move everything into position. I wish I had saved that first attempt where I actually got mouth movement because now I'm starting to think those visemes may have been from the iclone import even after I erased them. The iclone generated visemes for that dialog are pretty bad.
I also tried saving the empty viseme track as a motionplus file and loaded it from the library and I get the same thing
This is the log.
C:/Users/User/Documents/exports/Voice 1.dat
Start
No viseme at frame 1 rest
No viseme at frame 6 rest
No viseme at frame 5 AI
No viseme at frame 6 rest
No viseme at frame 8 rest
No viseme at frame 7 etc
No viseme at frame 8 O
No viseme at frame 10 MBP
No viseme at frame 11 etc
No viseme at frame 13 U
No viseme at frame 14 etc
No viseme at frame 15 FV
No viseme at frame 16 AI
No viseme at frame 18 etc
No viseme at frame 20 AI
No viseme at frame 21 L
No viseme at frame 23 E
No viseme at frame 24 etc
No viseme at frame 25 U
No viseme at frame 27 etc
No viseme at frame 31 AI
No viseme at frame 33 MBP
No viseme at frame 36 etc
No viseme at frame 39 AI
No viseme at frame 42 etc
No viseme at frame 47 rest
Finished
|
|
By tbseo - 5 Years Ago
|
I got the latest script and and it's now working for me. I don't know what that was all about but problem solved. Lol.
I did only test it once though so maybe I shouldn't pop open the champagne just yet.
|