Hello all,
I am a veteran PPro/AE user and beginner programmer attempting to write a script to automate the lip-syncing tasks I do within an animation series.
Workflow
This project involves video clips that can stretch to 30+ minutes, making placing the required mouth images over each frame of footage a very numbing, tedious process. In short, I'm given a Premiere Project that contains near complete footage, with the only thing missing being the mouths for each character onscreen. We are provided the correct mouth images - in .PNG format - within a separate folder.
I then go through and place the correct mouth images over each character in my footage, with the particular image used being dictated by whether the character is speaking. Luckily, I don't need to move the mouths around using any of the Transform properties or keyframe anything...they are already saved with the correct dimensions and placement, so all I'm doing is dragging-and-dropping the correct mouth image over my footage, one frame at a time. However, placing images of one frame length for each and every frame can quickly get tedious when you're working with shots that can be 30+ seconds, in an animation project that runs over 30min.
As far as the variety of our mouthsets goes, we use three main images to cover each character speaking. These images are divided as such:
- Closed Mouth (character isn't speaking) - usually designated as FILENAME-C.PNG
- Mid Mouth (character's mouth is halfway open) - usually designated as FILENAME-M.PNG
- Open Mouth (character's mouth is fully open) - usually designated as FILENAME-O.PNG
Quick visual example of this:
Scripting Solution
Now, for the scripting goodness...I'm trying to write a script that would automate this workflow. The idealized, script-assisted workflow would look as such:
1) I set In and Out points on the timeline over the footage I want to automate mouth placement. This keeps it contained to just one shot and one corresponding mouthset, instead of the entire timeline.
2) I launch the UI/panel for my script
3) I drag and drop the three main images I need placed into the panel. The script would ideally know which mouth to use based on the filename (FILENAME-C.PNG to designate a mouth that's closed, FILENAME-M.PNG for a mouth that's midway open, FILENAME-O.PNG for a mouth that's fully open, etc).
4) I hit Run.
5) The script looks through the specific frames I've selected via In and Out points. For each frame where the underlying audio is at a minimum (-260dB to -30dB), it pastes the closed mouth file (FILENAME-C.PNG). For each frame where the audio is between -30dB and -10dB, it pastes the mid mouth file (FILENAME-M.PNG). For each frame where the audio is between -10dB and 0dB, it pastes the fully open mouth file (FILENAME-O.PNG).
Now, as far as how to specify this in coding?
I have absolutely no clue. This is where I turn to the community for guidance and suggestions.
My understanding of Javascript is pretty minimal but I can crunch through the basic coding and tutorials and learn it quickly (I did some programming in C/C++ many years ago; it wasn't too hard for me to pick up, nor have I had difficulty with HTML or CSS).
From my searches, I believe that I would need to build the basic panel in HTML 5 with some buttons that trigger the scripting. Once I have a working panel/UI, I can start to communicate with Premiere using ExtendScript.
I've seen examples of this done in After Effects, so I know it can be done (see mamoworld's very impressive Auto Lip-Sync ), but I have no idea how to pull this off inside of Premiere. Most importantly, the part where I tell the program to look at the underlying audio and paste a one-frame image file based on the audio clip's volume is one thing I'm very uncertain about.
Any ideas, help, suggestions or advice about what to do?
