<h1 id="sparsectrl-animation-comfyui-workflow-guide-66uf991do">SparseCtrl Animation Comfyui Workflow Guide</h1>This AnimateDiff LCM Animation Workflow is designed to make subtle animations easy and compatible with low VRAM setups. This workflow integrates a range of models, control nets, and custom nodes to provide refined, high-quality animations using Stable Diffusion.<h3 id="getting-started:-initial-setup-and-downloads-fjeiidbux">Getting Started: Initial Setup and Downloads</h3>You’ll need several model files and custom nodes to replicate this setup. Here’s a list to get started:<ol><li>Recommended Checkpoints: <a target="_new" rel="ugc" href="https://civarchive.com/">AniVerse - V5.0</a>, <a target="_blank" rel="ugc" href="https://civarchive.com/models/56680/imp">IMP</a>, <a target="_blank" rel="ugc" href="https://civarchive.com/models/24350/perfectdeliberate">PerfectDeliberate</a> and <a target="_blank" rel="ugc" href="https://civarchive.com/models/306814/photon-lcm">PhotonLCM</a> high-quality Stable Diffusion checkpoints.</li><li>AnimateDiff Motion Model &amp; LoRA: <a target="_new" rel="ugc" href="https://huggingface.co/wangfuyun/AnimateLCM/blob/main/AnimateLCM_sd15_t2v.ckpt">AnimateLCM by wangfuyun</a> – Includes both the motion model and LoRA for smooth frame-by-frame animations.</li><li>Advanced Control Nets: <a target="_new" rel="ugc" href="https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet">Advanced ControlNet by Kosinkadink</a> – Adds scheduling, masking capabilities, and sliding context support for refined animation control.</li><li>SparseCtrl Control Net: <a target="_new" rel="ugc" href="https://huggingface.co/guoyww/animatediff/blob/main/v3_sd15_sparsectrl_rgb.ckpt">SparseCtrl</a> – Provides sparse control for precise motion guidance.</li><li>QRcode Monster Control Net: <a target="_new" rel="ugc" href="https://huggingface.co/monster-labs/control_v1p_sd15_qrcode_monster/tree/main">QRcode Monster</a> – this is responsible for the morphing animations.</li><li>Clip Vision Model: <a target="_new" rel="ugc" href="https://huggingface.co/shiertier/clip_vision/blob/main/SD15/model.safetensors">Clip Vision</a> is essential for enhancing image conditioning.</li><li>Upscaler Model: <a target="_new" rel="ugc" href="https://huggingface.co/uwg/upscaler/blob/main/ESRGAN/4x_NMKD-Siax_200k.pth">4x NMKD Siax Upscaler</a> – Used for the final upscale pass to ensure your animation looks sharp and high-quality.</li></ol>Tip: Make sure to place these models in their appropriate directories within ComfyUI (e.g., <code>./models/checkpoints</code>, <code>./models/controlnets</code>, <code>./models/upscalers</code>).<h3 id="input-steps-for-creating-animations-zqk65cxpx">Input Steps for Creating Animations</h3>Configure the Initial Setup:<ul><li>Checkpoint: Select your Stable Diffusion checkpoint, such as AniVerse V5.0.</li><li>Required LoRA: AnimateLCM_sd15_t2v_lora @ strength of 0.5 - 1.0</li><li>Optional LoRAs: Include any LoRA models you’d like to enhance specific features.</li><li>VAE, Seed, Prompt: Set your VAE for image quality, choose a seed for consistency</li><li>Models for Additional Control: Select the CLIP Vision and SAM models for added conditioning and segmentation used by the detailer nodes.<img src="https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/4924c346-f20e-4f30-a611-89c79749bb06/width=525/4924c346-f20e-4f30-a611-89c79749bb06.jpeg" />Craft Prompt:</li></ul><ul><li>Simple is better. You can use prompt travel to change the prompt on specific animation frames.</li><li><img src="https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/e4325a0f-c32b-4707-a1be-e39df27c9d19/width=525/e4325a0f-c32b-4707-a1be-e39df27c9d19.jpeg" />Prepare the Input Image:<ul><li>Resize and Adjust: Upload an input image that serves as a base for the animation. Resize it to ensure compatibility with IPAdapter and SparseCtrl Control Net.</li><li>Quality Check: Optimizing the image dimensions helps IPAdapter and SparseCtrl process it more effectively, creating smoother animations.<img src="https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/60c51043-0237-48ef-9ac5-0cc46a14d5e0/width=525/60c51043-0237-48ef-9ac5-0cc46a14d5e0.jpeg" />(tip: use right-click "queue group output nodes (rgthree)" to view your adjustments to the image resize and crop before queuing up the entire workflow.</li></ul></li><li>AnimateDiff Configuration:<ul><li>AnimateDiff Model: Load the AnimateDiff LCM model, specifically designed for seamless motion.</li><li>Set the Scheduler and Looping: Use the beta scheduler to control frame variation and enable looping for continuous animation. (models are trained on a context_length of 16 and a context_overlap of 4. You can adjust it, but your results may vary). Select if you want a loop or not with closed_loop.<img src="https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/4ff699dc-e403-463c-aacb-1e06e1cf1722/width=525/4ff699dc-e403-463c-aacb-1e06e1cf1722.jpeg" /></li></ul></li><li>Control Net Configuration:<ul><li>Adjust Control Net Strength: The SparseCtrl and QRcode Monster control nets should work well with default settings, but adjusting the strength can fine-tune the animation’s motion.</li><li>Optional Customization: To influence the animation’s style more, experiment with the control net parameters to match your desired effect.<img src="https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/0db90377-7b93-46fa-9001-7b9d1b6ee37e/width=525/0db90377-7b93-46fa-9001-7b9d1b6ee37e.jpeg" /></li></ul></li></ul><hr /><h3 id="processing-steps-for-refining-animation-quality-qgp2f1p03">Processing Steps for Refining Animation Quality</h3><ol><li>Initial Rough Sample:<ul><li>This step creates a rough animation pass to establish the movement, though it may appear choppy initially. It’s a foundational pass intended for further refinement.</li></ul></li><li>Latent Upscale and Resampling (Highres Fix):<ul><li>Apply a high-resolution fix to enhance image quality. This latent upscale reduces artifacts and improves clarity, particularly useful for animations generated on lower VRAM setups.</li></ul></li><li>Face and Eye Detailing:<ul><li>Use the face and eye detailer to reduce common artifacts for close-up animations. This optional step can refine facial details for smoother, more natural animations.</li><li>If you experience VRAM limitations, consider bypassing this step to optimize processing speed.</li></ul></li><li>Final Upscale and Interpolation to 30 FPS:<ul><li>The final upscale pass uses the 4x NMKD Siax Upscaler model to enhance the resolution.</li><li>30 FPS Interpolation: This final step smooths motion by interpolating frames, delivering a polished animation with fluid motion at 30 frames per second.</li></ul></li></ol><hr /><h3 id="tips-and-recommendations-zg91oiwtl">Tips and Recommendations</h3><ul><li>Experiment with Settings: Adjusting parameters like checkpoint model, image size, control net strength, and IPAdapter configuration can significantly impact animation quality.</li><li>Handling VRAM Constraints: This workflow can work well even on lower VRAM systems. If you encounter memory issues, try simplifying by bypassing the face detailer step or reducing image resolution.</li><li>Custom Nodes: This setup uses several custom nodes in ComfyUI’s manager, including MTB, Efficiency, AnimateDiff, RIFE, and Face Detailer. Ensure all custom nodes are installed and up-to-date.</li></ul><h3 id="final-notes-gd15ugnae">Final Notes</h3>This SparseCtrl Animation Workflow offers a straightforward way to create expressive animations, combining powerful movement, quality, and detail tools. If you’re new to Stable Diffusion animations, this guide provides a solid starting point.Enjoy experimenting, and don’t hesitate to share feedback or troubleshooting questions. Posting images to this page is appreciated for the buzz ⚡⚡⚡😊

AnimateDiff LCM Workflow - Img2Vid with SparseNet

Added anything everywhere nodes to clean up pipes and simplified workflow to make it more user-friendly.

Files with SHA256: e3086ed72325d4e499a72a2ae84fe9efde44c7a1d57767199755d30eac548fd5