** Latest version changes:
V1.2.2 adds automatic toggle for landscape/portrait derived from input image.
**
Based on WAN-KR this is a multi-clip workflow that can combine up to 4 clips.
Being that this is currently a simple feed of the last frame into the next clip it has the known drawback of losing context if something is covered up in the former clip. See the example clip where Hades Model covers her necklace and it changes form. Also sometimes the sudden change of movement between clips is noticeable because the AI as no memory of the movement vector - Still, often it does work pretty well.
HOW TO:
Generate your init clip. If satisfied enable CLIP_2, add your prompt, generate.
If not satisfied with clip 2, change the prompt or just the seed and regenerate clip 2, clip 1 remains untouched and needs not to be redone.
( Important: Do not start Comfy with "cache none", as this will disable the node cache too and this is the whole point of the workflow.)
If satisfied with clip 2 and the combination 1+2 preview, move on to CLIP_3, repeat procedure. Clip 1 and 2 remain untouched due to locked seed.
Optionally move on to clip_4, although you can always stop combining after 2 or 3 clips.
As final step, combine and optionally interpolate and upscale the result.
To use the resolution preset for Kijai selector node put the JSON with the custom dimensions into the ComfyUI\custom_nodes\comfyui-kjnodes\ folder. If you don't need all those resolutions just edit the file down.
If you have your prompts/storyboard nailed down and want to generate some complete clips without checking the transitions, you can simply set the initial seed to random/increment, keep the seed on 2+3+4 locked, enable clip2-4 as well as combine clip and then you can pipe several runs into ComfyUI pipeline.
Navigating this behemoth can be done easily with the Bookmarks/quick keys 1-9, I and K which can be customized. Also there are locked yellow group bypasser to jump back to control panel.
Depending on your machine you need to change the GGUF checkpoint from Q8 down to Q6, 5 or even 4. If you want the fp8 for whatever reason just swap out the GGUF loader to the normal one.
Q: Do I need all those Lightning/LX2V LORAs?
A: No. But if you ask 5 people which are best, you'll get 5 answers, like A,B, C or mix of A+B or whas it A+C?, so during testing I implemented the enabler/mixer for convenience.
If in doubt, just stick with either Kijai's latest or the seko variant from the Original team (all links in WF) - for now, until the next best thing comes along.
Upscaling a long and hi-res clip can be tricky on memory/VRAM. I would advise to not use a 4x model in that case. Even with 96Gb I got some swapping in some cases.
Description
Added Bookmarks/Keyboard shortcuts (thanks to suggestion by @ArtificeAI)
Removed "Interpolate Init". If you just render the 1st clip and "combine" it you can interpolate it as it is. No need for redudancy
Update notes and disabled clips/combine for default wf template
FAQ
Comments (4)
Wait, this is not using sage attention and torch patch?
It is not patching on the fly in the workflow. But a patch node could easily be added I presume. I am running my Comfy usually via batch file with "--use-sage-attention --fast fp16_accumulation" parameter. If I don't want to use sage (for whatever reason) or just sage without fp16 accu I start with a different batch file.
I really like the simplicity and effectiveness of this one. Is it possible to modify this to set loras per clip rather than loading and applying them at the start?
Sorry for the late answer, yes it is possible. In fact my local 1.0 release candidate has this already. I was evaluating tweaks for the clip transitions, quality degrade mitigation, color match, and especially the upscale that I kind got lost in the zone. And also I exported the lx2v/lightning into its own lora nodes. I'll try to wrap things around in the next few days. If you want to try it yourself you need to duplicate the lora loaders and shift/sample/(torch compile) nodes into each clip. the lora loaders then need to be connected to the model/clip load at the very beginning (like for the initial clip). The chain for the model/clip connect is: model/clip --> lora loader hi+low (clip only to hi) --> shift/sample hi+low --> KSamplers of the clip. Also the clip of the hi lora loader should be connected to the CLIP text encode (positive prompt) of the movie clip. If you use Torchcompile, these nodes need to be copied too and put as last node before each Ksampler of the clip after the model shift/sample nodes, again like it is done for the initial clip. Finally, either wire the model shift slider to the shift value of all the new shift nodes, or punch in the setting manually. Frankly I haven't touched shift for a long time myself.