Version 5.0 works with latest Comfy Handy Nodes 1.24
manually update if you dont see it in Manager by searching for "Handy Nodes"
cd <your custom nodes folder>
git clone https://github.com/trashkollector/TKNodes
Up to 4 prompts/images.
***** Works for Singing or Talking *****
How this works, supply an audio file, size does not matter as the workflow will chunk the audio.
It will then use alternating photos to keep video dynamic.
Audio now is chunked when there is silence in the audio for better transtion.
Please get Handy Nodes 1.1.5 from Manager.
Anyone with low VRAM change this. this tells the workflow to try use 12 second chunks.
Audio driven video using alternating speakers. See Podcast demo.
Please make sure to get the newest TK Handy Nodes version 1.1.0
Description
Version 1. LTX2 LIP SYNC
FAQ
Comments (15)
Is there a workflow to lip-sync an audio file to an actual video, rather than just a static image?
I read that all you had to do is bypass the image to video node, and it should work like you want.
@artificialotaku Thx man
@artificialotaku any workflow example please looking for that too urg
The first node where I think size and frames, etc are entered doesn't look to be available any longer in ComfyUI Manager. Do you know where it can be gotten from?
I have the same problem. The node "TKVideoUserInputs" is in the Custom node-group "comfyui-handynodes-kt". And that group doesn't exist any more.
@Mikel007 Found them. You need Handy Nodes by Trash Kollector from Comfy Manager. There will be 2 (one from Trash Kollector and another) so make sure to install the correct one.
Then I got a "mat1 and mat2 shapes cannot be multiplied (1024x3840 and 1920x4096)" error on running. I've gotten it on other LTX flows as well recently. I think some node I've installed is breaking things... It happens on the negative prompt in one of the subgraphs.
Thanks for finding this.
Do people make money including these weird nodes in their workflows somehow? It's like everyone on CivitAI has some odd node that you have to track down to get anything to work.
Also, the comfyui_smznodes might be the reason for your other problem, try deleting that folder if you have it, it apparently messes with the LTX2 models somehow.
@eilers333 Yeah, I'm really beginning to dislike ComfyUI with a passion. There are so many different nodes that look to do similar things and everyone uses different ones for their flows. Getting a bit tired of needing multiple copies of Comfy as I never know if trying a new flow and it's nodes is going to break a flow that was working fine.
Using links for models helps with storage but is still a pain overall.
@eilers333 Thanks for the tip, but I don't have that installed. I think it might have something to do with the models I'm using. I spent hours the other day trying to figure it out and looked to get one flow where I got the error working, but then the output didn't work for some reason.
I have started to sometimes copy the custom nodes folder and then pretty much delete everything in it so nodes needed for a flow I'm testing redownload, but a lot of the time I don't end up keeping the flow as it wasn't quite what I was looking for.
I should stop ranting. Lol.
@jonk999 I spent all day trying to rebuild all my LTX2 gguf workflows that only generate static now, so I sympathize. I updated yesterday and nothing has worked properly since. I SHOULD have duplicated my setup, but was in a hurry. What a waste of time.
@eilers333 I have a 4TB SSD that I keep most of my models and ComfyUI (as well as Forge and a couple of other things on). I just counted that have 9 installs of Comfy on it. Lol. I've ended up separating them into Image Gen, Video Gen, LTX Gen and Video & Audio Gen. As well as a couple of test copies that I play around in so it isn't a huge deal if things break in those...
I also didn't have much luck with this workflow. I got all the errors fixed I was getting and when I tried a couple of generations I couldn't get the lips to move. The audio was there, just without any face movement.
@jonk999 are you trying with non-human? non-human is very finicky to work. Human should work smoothly. Also you can change the compression rate from 33 to 42, this can help if video becomes static. But honestly, I have never had an issue with human becoming static, but have had issues with non-human going static.
@trashkollector175 Hi. Was human. Not sure why couldn't get mouth moving. Will have another look when I have time.