IntroductionThis workflow makes use of AnimateDiff, IPAdapter and Controlnet to make clouds dance. The key innovation in this workflow is that I VAE encode the depth map and adjust the denoise such that the depth map gets denoised into clouds. This avoids the video morphing every 16 frames due to the context length of AnimateDiff.Models<ol><li>SD1.5 LCM Checkpoint. I use Dreamshaper 8 LCM: <a target="_blank" rel="ugc" href="https://civarchive.com/models/4384?modelVersionId=252914">https://civarchive.com/models/4384?modelVersionId=252914</a></li><li>AnimateLCM_sd15_t2v.ckpt: Download from <a target="_blank" rel="ugc" href="https://huggingface.co/wangfuyun/AnimateLCM/tree/main">https://huggingface.co/wangfuyun/AnimateLCM/tree/main</a> and put in models/animatediff_models</li><li>depth_anything_v2_vitl.pth and control_v11f1p_sd15_depth_fp16.safetensors. Download using Manager.</li><li>IPAdapter PLUS: download ip-adapter-plus_sd15.safetensors using ComfyUI manager. Also download ViT-H and ViT-G clip_vision models using the ComfyUI manager.</li></ol>Custom NodesInstall with Manager:<ol><li>ComfyUI-VideoHelperSuite</li><li>KJNodes for ComfyUI</li><li>ComfyUI's ControlNet Auxiliary Preprocessors</li><li>AnimateDiff Evolved</li><li>ComfyUI_IPAdapter_plus</li><li>ComfyUI Frame Interpolation</li></ol>

Make Clouds Dance!

Due to comments that people do not see the dancing. I decided to update the workflow to make the dancer more apparent. 1) Use 2 IPAdapters: Once for the dancer and one for the background2) Added a depth controlnet with 0.3 end_percent

Files with SHA256: bd2e78e7dc86d51f8df0c438bf0b016fa90835f980ed99ffb6b50b40a65fb642