<h3 id="workflow-overview">Workflow Overview</h3>This is a sophisticated ComfyUI workflow designed for high-quality, controllable video generation using the powerful Wan2.2 5B Fun model. It leverages ControlNet (via Canny edge detection) to transform a driving motion video and a starting reference image into a stunning, coherent animated sequence. Perfect for creating dynamic character animations with consistent style and precise motion transfer.Core Concept: Use a "control video" (e.g., a person dancing) to guide the motion, and a "reference image" (e.g., a character design) to define the style and subject. The workflow intelligently merges them into a new, AI-generated video.<hr /><h3 id="key-features-and-highlights">Key Features &amp; Highlights</h3><ul><li>🚀 State-of-the-Art Model: Utilizes the <code>Wan2.2-Fun-5B-Control-Q8_0.gguf</code> quantized model for a balance of incredible quality and manageable hardware requirements.</li><li>🎨 Precision Control: Implements a Canny Edge ControlNet. The workflow extracts edges from your input video, ensuring the generated animation perfectly follows the original motion.</li><li>⚡ Optimized for Speed: Integrates a custom LoRA (<code>Wan2_2_5B_FastWanFullAttn</code>), allowing for high-quality results in just 8 sampling steps without significant quality loss.</li><li>🧠 Efficient LLM Inference: Uses a separate, quantized <code>umt5-xxl-encoder</code> CLIP model for text encoding, reducing VRAM load on your GPU.</li><li>🔧 Complete Pipeline: Everything from model loading, video preprocessing, conditioning, sampling, to final video encoding is included in one seamless, organized graph.</li><li>📁 Ready-to-Use: Pre-configured with optimal settings, including a detailed positive/negative prompt. Just load your own image and video to start creating.</li></ul><hr /><h3 id="workflow-structure">Workflow Structure</h3>The workflow is neatly grouped into logical sections for easy understanding and customization:<ol><li><code>Step1 - Load models</code>: Loads the main Wan2.2 5B model, its VAE, the CLIP text encoder, and the FastWan LoRA.</li><li><code>Step 2 - Start_image</code>: Loads your initial reference image. This defines the character and style for the first frame.</li><li><code>Step 3 - Control video and video preprocessing</code>: Loads your motion video and processes it through the Canny node to extract edge maps.</li><li><code>Step 4 - Prompt</code>: Where you input your positive and negative prompts to guide the generation.</li><li><code>Step 5 - Video size &amp; length</code>: The <code>Wan22FunControlToVideo</code> node packages everything, setting the output video dimensions and length based on the control video.</li><li>Sampling &amp; Decoding: The KSampler runs for 8 steps with UniPC, and the VAE decodes the latents into final images.</li><li>Video Output: The <code>VHS_VideoCombine</code> node encodes the image sequence into an MP4 video file.</li></ol><hr /><h3 id="how-to-use-this-workflow">How to Use This Workflow</h3><ol><li>Download &amp; Install:<ul><li>Ensure you have ComfyUI Manager to easily install missing custom nodes.</li><li>Required Custom Nodes: <code>ComfyUI-VideoHelperSuite</code>, <code>ComfyUI-GGUF</code> (for loading the .gguf models).</li><li>Download the <code>.json</code> file from this post.</li></ul></li><li>Load the Models:<ul><li>Main Model: Place <code>Wan2.2-Fun-5B-Control-Q8_0.gguf</code> in your <code>ComfyUI/models/gguf/</code> folder.</li><li>CLIP Model: Place <code>umt5-xxl-encoder-q4_k_m.gguf</code> in the same <code>gguf/</code> folder.</li><li>VAE: The workflow points to <code>Wan2.2_VAE.safetensors</code>. Ensure it's in your <code>models/vae/</code> folder.</li><li>LoRA: Place <code>Wan2_2_5B_FastWanFullAttn_lora_rank_128_bf16.safetensors</code> in your <code>models/loras/</code> folder. Adjust the path in the LoraLoader node if yours is in a subfolder (e.g., <code>wan_loras/</code>).</li></ul></li><li>Load Your Assets:<ul><li>Reference Image: In the <code>LoadImage</code> node, change the image name to your own file (e.g., <code>my_character.png</code>).</li><li>Control Video: In the <code>LoadVideo</code> node, change the video name to your own motion clip (e.g., <code>my_dance_video.mp4</code>).</li></ul></li><li>Customize Your Prompt:<ul><li>Edit the text in the Positive Prompt node to describe your desired character and scene.</li><li>The provided negative prompt is already comprehensive, but you can modify it as needed.</li></ul></li><li>Run the Workflow:<ul><li>Queue the prompt in ComfyUI. The final video will be saved to your <code>ComfyUI/output/video/</code> folder.</li></ul></li></ol><hr /><h3 id="tips-for-best-results">Tips for Best Results</h3><ul><li>Control Video: Use a video with clear, strong motion and good contrast for the Canny detector to work best. Silhouettes or videos with a plain background work excellently.</li><li>Reference Image: The first frame of your output will closely match this image. Use a high-quality image of your character in a pose similar to the first frame of your control video.</li><li>Length: The <code>length</code> in <code>Wan22FunControlToVideo</code> is set to <code>121</code> based on the original video. If your video is a different length, you must update this value to match the number of frames.</li><li>Experiment: Try adjusting the LoRA strength (e.g., between <code>0.4</code> - <code>0.7</code>) or the Canny thresholds to fine-tune the balance between motion fidelity and creative freedom.</li></ul><hr /><h3 id="required-models-(download-links)">Required Models (Download Links)</h3><ol><li>Wan2.2-Fun-5B-Control-Q8_0.gguf: <a target="_blank" rel="ugc" href="https://huggingface.co/QuantStack/Wan2.2-Fun-5B-Control-GGUF">https://huggingface.co/QuantStack/Wan2.2-Fun-5B-Control-GGUF</a></li><li>umt5-xxl-encoder-q4_k_m.gguf: <a target="_blank" rel="ugc" href="https://huggingface.co/city96/umt5-xxl-encoder-gguf/tree/main">https://huggingface.co/city96/umt5-xxl-encoder-gguf/tree/main</a></li><li>Wan2.2_VAE.safetensors: <a target="_blank" rel="ugc" href="https://huggingface.co/QuantStack/Wan2.2-Fun-5B-InP-GGUF/tree/main/vae">https://huggingface.co/QuantStack/Wan2.2-Fun-5B-InP-GGUF/tree/main/vae</a></li><li>Wan2_2_5B_FastWanFullAttn_lora_rank_128_bf16.safetensors: <a target="_blank" rel="ugc" href="https://huggingface.co/Kijai/WanVideo_comfy/blob/main/FastWan/Wan2_2_5B_FastWanFullAttn_lora_rank_128_bf16.safetensors">https://huggingface.co/Kijai/WanVideo_comfy/blob/main/FastWan/Wan2_2_5B_FastWanFullAttn_lora_rank_128_bf16.safetensors</a></li></ol><hr /><h3 id="conclusion">Conclusion</h3>This workflow demonstrates the powerful synergy between the Wan2.2 model, ControlNet, and efficient LoRAs. It abstracts away the complexity, providing you with a robust, one-click solution for creating amazing AI-powered animations. Enjoy creating!If you use this workflow, please share your results! I'd love to see what you create.

Wan2.2 5B Fun Control - Fast Video ControlNet

Updated workflow for better quality.

Files with SHA256: 33560f05a209f1305aa62139d67e48258d68a5edd1ce483981354682ab654aba