Tutorial: Sound-controlled animation


https://forum.reallusion.com/Topic276724.aspx
Print Topic | Close Window

By RobertoColombo - 9 Years Ago
Hello,

here is my new tutorial, which shows how to generate a sound-controlled animation, i.e. an animation driven by an audio file.

Cheers

  Roberto



Here is the Phyton script:

==============================================================
import bpy

bpy.context.area.type = 'GRAPH_EDITOR'
bpy.ops.anim.keyframe_insert_menu(type = 'BUILTIN_KSI_VisualScaling')
bpy.ops.graph.sound_bake(filepath=" Write here the directory path & file name!!! ")
==============================================================

Note, you can substitute the 'BUILTIN_KSI_VisualScaling' transformation key type with any of these values:

Location
Rotation
Scaling
BUILTIN_KSI_RotScale
BUILTIN_KSI_LocRot
BUILTIN_KSI_LocScale
BUILTIN_KSI_LocRotScale
BUILTIN_KSI_VisualLoc
BUILTIN_KSI_VisualRot
BUILTIN_KSI_VisualScaling
BUILTIN_KSI_VisualRotScale
BUILTIN_KSI_VisualLocRot
BUILTIN_KSI_VisualLocScale
BUILTIN_KSI_VisualLocRotScale
BUILTIN_KSI_DeltaLocation
BUILTIN_KSI_DeltaRotation
BUILTIN_KSI_DeltaScale
BUILTIN_Available



By Pollux - 9 Years Ago
Thank you so much Roberto. I love your tutorials.
By pumeco - 9 Years Ago

I had no idea 'The Great Roberto' was a scripting god as well as a guitar god :w00t::D

That was great, I really enjoy your style of tutorials and I'm seriously jealous of your dungeon!
I'm getting one of those myself so I can have lots of fun with my Peasant Girls!

BTW, Roberto, just an idea, but have you tried using an EQ in your DAW to split the audio into three?  What I mean is you could add an EQ to the original audio, and for the first file you export, set the EQ so that you only hear the Bass tones.  With the second, export the audio so that you only hear the Mid, and on the third, export where you can only hear the Treble.  That would allow you do do each animation seperately so you'd have a 3-band spectrum display like on a Hi-Fi.  Haven't tried it yet, but it absolutely should work using your script!

Anyway, thanks for this, it's handy for both iClone and Blender, very cool!


By RobertoColombo - 9 Years Ago
Hi Pumeco,

thanks for the nice words. I really appreciate. :)
In reality I know how to program in Assembler (for various ST and ARM CPUs), C, C++, VB.NET, Javascript, Phyton, ASP, DHTML... and I regret I never seriously learnt Java :-)

Yes, what you suggested popped up in my mind and it is surely possible.
Actually, I was thinking to look whether it exists any audio plug in able to distinguish the note frequencies and create multiple WAV where only the selected frequencies are played, out of the source file.
Obviously if you work with MIDI, you can always filer and create multiple audio.
Then, with a simple Phyton "for" loop you can upload everything in one shot.
As I said in the tutorial: this is just the basic example, then human being's fantasy and inventive does the rest ;)

Cheers

  Roberto
By pumeco - 9 Years Ago

Sounds cool Roberto, would be pretty neat to have it separate the frequencies automatically like that.  I can't advise on the scripting cause I'm not a coder, but from an audio point of view, the easiest way to separate the frequencies might be a network of filters so that basically, you decide how many bands you want to analyse, and set them up so that they're offset equally from each other.  Once you have that (basically a filter bank), you could maybe assign a wav name to each filter output.  Like if you had three for example, the outputted files could be named :

filename_bass
filename_mid
filename_treble

As for the design of the filters themselves, just go for a DJ style filter, something that can cut all frequencies other than the band selected.
Doing a full-on spectrum analyser is basically the same sort of thing, but with a lot more bands.


By animagic - 9 Years Ago
Thanks, Roberto! This will actually inspire me to open Blender and do something with it.
By RobertoColombo - 9 Years Ago
Hi Job,

definitively!
It can take a while before you become familiar, and the quantity of features is so big that can be scaring...
But once you start moving the 1st steps and get the habit of the interface, then things start to make sense and you will see that is it not that difficult to do some nice stuff.
Of course, I am light years away from having a deep know-how about it, I move step by step, according to what I need and according with my free time.
I also use it to have a pause from iClone when I finish a video :-)
The great thing is the huge quantity of tutorials out there online, so whenever i need to do something, I can find some website explaining how to do it.

Cheers

  Roberto

By action_2009 - 9 Years Ago
I have created for iClone 5 a simple sound controller from this tut:

LINK

https://forum.reallusion.com/uploads/images/b9d833aa-7b51-462d-8ddc-6ea0.gif

It's not perfect but you can make animations with it.







By RobertoColombo - 9 Years Ago
Hi Action,

great job!!! I am happy to see that the method has been now used to create something cool :)

Hey sw00000p, thanks a lot!!!!
Nice words from a "Guru" are always more than welcomed ;) :w00t:
Yes, surely it is possible to customize and go beyond this.... but now I started the next iClone music video, so I will go on with Blender only from time to time... mmmh... well...in reality I will use it soon, because I need to create something that probably is achievable only with Blender... but I have to study a bit in order to understand how to do that.
Once I manage to do it, I will probably make another tutorial.

Cheers

  Roberto


By RobertoColombo - 9 Years Ago
sw00000p (4/2/2016)
RobertoColombo (4/2/2016)
...mmmh... well...in reality I will use it soon, because I need to create something that probably is achievable only with Blender... but I have to study a bit in order to understand how to do that.

May I ask, what that  is? :Whistling:
...just curious, I have no intention of interfering. :)


Hi sw00000p,

"just" creating some props (with Blender, specifically some painting "classic-style" frames) and then create the high and normal maps representing the typical classic-style frame decoration and finally use the tassellation to get the 3D deepness.
Maybe nothing incredibly difficult, I don't know: I haven't done anything like that so far, so I need to figure out the right flow and methodology.

Cheers

  Roberto

By RobertoColombo - 9 Years Ago
Hi sw00000p,

thanks for the suggestions.
A painting frame is a simple, squared object, so in Blender unwrapping it is quite easy.
In Blender you can also adjust the UV map (in order to maximize the parts you want, create island with seams etc. etc.), save the unwrap reference image, creates the various maps in a photo/paint editor (I use GIMP) etc. etc.
I plan to create and use a gray-scale Displacement map, based on the UV unwrapped image, because in my next music video (which you can guess it is based into a painting gallery), those frames are just secondary objects, so I do not need a huge quality: just some kind of raw but realistic 3D deepness.
So, the idea is simply to overlay some B&W patterns (resembling a frame decoration) in the unwrapped UV map to create the necessary maps.
It should be simple and I will probably create a tutorial once I have finished.

.. and, btw, I am building the best gallery in the world: imagine to put the most famous paintings in the world  into one single gallery!!
Once finished I will probably start an online ticket sales for those who want to visit it  :D :P

Cheers

  Roberto

By RobertoColombo - 9 Years Ago
Hi sw00000p, 

yes, modelling the the frame is super-easy, but creating and applying the right displacement map for the tassellation can require some tricks here and there.
I am working on it...

Cheers

  Roberto
By RobertoColombo - 9 Years Ago
Hi sw00000p,

thanks.
This is clear: the tricks I mentioned is to solve unexpected things.
I mean... if the final result in iClone is not how I expected it to be (the first experiment was not that successful :ermm:).

Also, I am not going to manually recreate all the frame decorations in B&W.
I downloaded several frame jpeg and from them I will create the heighmaps: not a push-button thing because often those frame jpeg have different lighting around the perimeter.
In fact, this forces me to go also a bit more inside GIMP techniques  :-)

Cheers

  Roberto