This controlnet is specifically finetuned for animatediff whose purpose is to keep the initial image appearance in the process of animation. It performance is unstable when compared to the tile controlnet, because the train dataset is less than tile controlnet. When you use it for animation, I recommend you use the less motion version for its stable performance. For img2video:<a target="_blank" rel="ugc" href="https://github.com/crystallee-ai/controlGIF">https://github.com/crystallee-ai/controlGIF</a> could be a reference for how to use these checkpoints. For vid2vid: you could see this <a target="_blank" rel="ugc" href="https://discord.com/channels/1076117621407223829/1149372684220768367/threads/1192162917395730635">https://discord.com/channels/1076117621407223829/1149372684220768367/threads/1192162917395730635</a>, which is a creative and effective workflow. <a target="_blank" rel="ugc" href="https://huggingface.co/crishhh/animatediff_controlnet/discussions/2#657d5a203e3b5bea66facb65">crishhh</a> on <a target="_blank" rel="ugc" href="https://huggingface.co/crishhh/animatediff_controlnet">huggingface</a>A mirror for the animatediff controlnet models by crishhh on huggingface, original repository can be found <a target="_blank" rel="ugc" href="https://huggingface.co/crishhh/animatediff_controlnet">here</a>. As of now only SD1.5 is supported, but crishhh is working on an SDXL model. I can't explain how it works exactly and what settings to use for best results, as there is zero documentation sadly. Use the force of Trial and Error, as usual :)

Animatediff Controlnet Models

animatediffControlnet_sd15LessMotionFP16.safetensors

animatediffControlnet_sd15LessMotionFP16.ckpt

SD1.5 - less motion FP16

Files with SHA256: f28d8e7c412bf44ecea5fb82f7135554152d1e415274fec4507331f7b2496299