Working on a simplified version to help you all out. Will include any improvements i have made and a few custom sigmas that may help.
I use DPMpp_SDE for first stage, and eular_ancestral_cfg for upscales
Few custom sigmas that work (bong tangent custom is the same as if you downloaded the node that offers it)
8 step/7..
Bong tangent sigmas - 1, .971, .929, .863, .744, .500, .213, .07, 0 - Stable and my most used sigma...
My sigmas, they drop off a cliff at step 2-3, keeping more of the motion of loras and prompts.
My custom sigma - 1, .97, .9, .62, .48, .42, .38, .34
My custom sigma more movement/stronger loras - 1, .857, .5, .42, .38, .28, .22, .18
My custom may help get more motion, but will be less stable, use only if nothing else is allowing the motion you want.
LTX cooks videos in about 2-3 steps, using high denoise beyond that nullifies motion and can limit lora strengths. Using the custom uses more of the motion from loras or from prompt but can be harder to control. The second and third sigma numbers are key to affecting this.
SGM Uniform is high denoise throughout, makes stable videos but can limit the motion you get.
Upscale sigmas - This is how i upscale. 1 step is good enough most of the time, 2 steps helps if teeth are an issue, and 3 step is only needed if your base video is bad and needs to actually be fixed. I also use RTX upscale instead of the latent upscale. Both work but RTX seems to hold teeth better than latent. Latent if viewed directly after latent upscale you can see why it has issue with teeth as the mouth gets very wonky in the latent upscale.
Sigmas
1 step - .65, 0
2 step - .65,.44, 0
3 step - .85, .52, .22, 0
This is fairly fast as well.
Second stage upscale can be used as detailer without upscaling more, using 19b detailer lora at 0.3-0.4. It can help clear faces and mouth if needed. Using 1 step sigmas above.
LTX 2.3
V1.1
No changes to the main flow
Added 3 example flows with example files and instructions
2 frame interpolation example
Img2vid with motion transfer example
remake video adding sync or sound example
All have their own json flow with everything set up for the task and instructions.
All have the needed files, video ref, image refs, audio files, etx.
This should show you how to use it for the most part.
1-3 stages
1. Basic image to video
2. First/last frame interpolation
3. Multi frame interpolation
4. Video motion transfer with guide frames
5. Video insertion as a mid video ref.
6. Video reproduction adding sound and sync
Audio
1. Custom voice ref to guide the generated voice
2. Custom audio can be synced
All options are optional, if no video or image ref is used it does regular text to video.
All options can be used together when needed.
Included the images i used for interpolation for you to test with. She is an ai girl.
Notes everywhere, read. Enjoy :)
Description
No changes to the main flow
Added 3 example flows with example files and instructions
2 frame interpolation example
Img2vid with motion transfer example
remake video adding sync or sound example
All have their own json flow with everything set up for the task and instructions.
All have the needed files, video ref, image refs, audio files, etx.
This should show you how to use it for the most part.
FAQ
Comments (7)
thanks alot
That looks very promising. Thank you! Cannot find hqtalk lora anywhere. Is it original file name in the workflow?
Sorry :)
hq i speak of is
https://huggingface.co/AviadDahan/LTX-2.3-ID-LoRA-CelebVHQ-3K
The other choice is this
https://huggingface.co/AviadDahan/LTX-2.3-ID-LoRA-TalkVid-3K
@sy0ww4bb1984 thank you so much. Whatever I try, I cannot make the sound work, though. It's just high pitch noise, if i remove the previewer from SamplerCustomAdvanced, Video Combine fails with error, that audio frames are not provided.
@comthumb The audio is probably the most confusing thing about this flow tbh. I gave a few different ways to use audio and did not properly explain what each did and how to combine or not combine them. I still have much to do with this flow to make it easier, i just threw it out here because the video copy was actually cool and i did not see others doing it.
I will try to explain the what happens to the audio
First with no options enabled (audio ref bypassed and custom audio bypassed) it uses an empty audio latent by default allowing normal LTX audio gen.
When you enable the audio ref group and give it a ref voice, using the HQtalk lora it will attempt to change the voice of the generation to the one provided.
When you enable custom audio this removes all need for LTX to do audio. It takes the audio you give it and uses it to form the video using its tone, speed, etc to drive motion and sync.
You cant use custom audio and audio ref together. They do different things.
Now when using custom audio you have a mask, this is set at 0 to not change the audio. Custom audio group has this mask, as well as 2 small groups between the samplers to stop/start editing the audio.
EDIT - also when using the audio ref and HQ lora you need a specific prompt style i think i included and example
What are you trying to accomplish? I could probably explain better if i knew what you were trying to do. Easier than just telling you every possibility,
@sy0ww4bb1984 Thank you very much for the explanation about audio. I think there was compatibility issues with comfyui, as LTX's own sample workflows did not produce correct audio for me, but latest update fixed the problem. So far your workflow is the best I've tried. Great job.
@comthumb Thx for the comment tbh. None of my workflows are currently counting or showing any stats so i didnt really know if anyone is using any of them. I cant see likes, downloads or anything for over a week now.