Support me for update
2025/2/8
update v1.5, add word cloud transfer
2024/9/22
Rewrote the edge detection function
and removed the #include "lygia/space/ratio.glsl" line
2024/9/4
update v1.4, automatic edge detection and replacement
2024/8/11
update v1.3. add SDXL upscale and more stable replace bg
2024/7/29
Simple Auto Mask For Videos @SouthbayJa
2024/7/14
update v1.2. Dance References
2024/7/12
For new comfyUI users, check your input model, like Lora. checkpoint, controlnet, aniamtediff, and upscale model, and don't forget to update your comfyUI all. There is a screenshot of the node in the download file for reference.
2024/7/9
Don’t let my reference examples stop you from exploring. In fact, any object can dance, such as water, rocks, clothes, cars, particles, etc.
2024/7/8
Updated download file-1.3, add ColorMod node, modified AnimateDiff node, and added a reference image
2024/7/7
I created a flower collection that might be helpful to you.
I used AE to cut out the character dance and turn it into a black and white mask. If you need skills in this area, just ask ChatGPT. He will teach you how to use AE to turn a video into a black and white mask. This is what I did.
Tips below from:https://civarchive.com/models/372584/ipivs-morph-img2vid-animatediff-lcm-hyper-sd
❗If you are getting an error message CLIP Vision Model not found: /ComfyUI/models/clip_vision folder
CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors download and rename
CLIP-ViT-bigG-14-laion2B-39B-b160k.safetensors download and rename
❗If you are getting an error message IPAdapter model not found:
You are likely missing the IPAdapter model. In ComfyUI Manager Menu click Install Models - Search for ip-adapter_sd15_vit-G.safetensors and click Install.
If installing through Manager doesn't work for some reason you can download the model from Huggingface and drop it into \ComfyUI\models\ipadapter folder.
2024/7/6
Use QRCode Controlnet to control appearance and action, use IPAdapter to control the content and style of conversion.
Description
FAQ
Comments (85)
you should replace upload image
I was going to also mention that for AnimateDiff, you must used Advanced ControlNet loader and Apply. They also need to match otherwise you may encounter this or a similar error. Sorry I can't reproduce this easily to fact check. But look at that too.
@HuBaYi81 Yes, you can replace your usual controlnet nodes, that's not a problem
@HuBaYi81 My mistake, I forgot to connect the image node to controlnet, the file has been re-uploaded
I've got the same error, even after updating everything on the workflow, LCMs, checkpoints, loras, everything, of course updated the images and used the provided footage of the dancer. Still the same error. I dont know if this is correct, but I've managed to overcome this error this way: https://drive.google.com/file/d/1dehBkbl7TOCq8yS52v7BYp8fddkGUSs7/view and now the workflow works.
@deliankristoff696 I can't see your image, please access in your gmail
@downpink I'm sorry, it should work now. But its the same thing, just reconnected the "image". You've updated the workflow :) Thank you for this awesome contribution! Next step: to figure out how to add consistent background. 🤔
@downpink Any image...Does it have a specific size?
@HASZD25
It depends on your needs. There is no size limit. If you want to be more controllable, you can first generate a picture from Controlnet Sampler as a reference, which will be more detailed.
I love how the Civitai community shares everything with eachother, probably not gonna use it, but awesome to share it! 💕
Hi there, Is anyone getting this error please "Error occurred when executing Efficient Loader: 'NoneType' object has no attribute 'lower'"? Thanks.
be sure that you have all the necessary loras, checkpoints, vaes, etc installed and you have manually selected them in every node.
it's for your lora, check that and it will be fine
thanks for the workflow, love that
can you please share the picture used for one of your works as an example?
I found pictures on pinterest for reference, https://ph.pinterest.com/pin/2533343536112953/
Regardless of this workflow there is not enough instruction on how to do and what to do, when will you provide proper instructions?
I've updated some usage tips
In the future it may help if you instead say "I could not get this to work with the available instructions, please provide more detailed written instructions for this process." That way you don't make yourself look like an entitled brat.
@BePatientImOnA1080Ti Solving problems is my first principle
You can work harder, the stupid bird flies first, fly more, and there will be no such problems
@BePatientImOnA1080Ti You're not only cute, but you're also very humorous, and if you were more accommodating, wouldn't it be better if you gave us stupid birds more time?
@rong028 Thank you but I think some of what I've said has been lost in translation. My comments were towards loneillustrator, not downpink. I think what downpink has done is great and worth of praise. I'm not against giving clear instructions and helping everybody to succeed, but I am against the way loneillustrator said this as a demand and not a request asking for additional help.
@BePatientImOnA1080Ti I agree with you
where to get the Ghibli_v6 Lora?
You can replace it, or search for it on Civitai
check your Controlnet model
@downpink Error occurred when executing KSampler (Efficient): 0 https://imgur.com/a/jmoa1qH
@lukethighwalker Have you generated videos before? You can update all your ComfyUI, or you can rebuild the Ksampler node. Remember to select run_nvidia_gpu.bat when starting ComfyUI.
@downpink thank you... forgot to update nodes after installing missing ones.. A1111 user moment..
@lukethighwalker hhh, enjoy your journey
still cant find this lora
@justicemanhero371 https://civitai.com/models/54233/ghiblibackground
Super cool workflow! Can we change background ?
I have tried different object but somehow results are not good. Why only flowers result come super nice?
You can try more. It’s not just about flowers. I’ve tried making particle effects which are also good.https://civitai.com/images/17796816
博主好，有一个节点始终找不到，ComfyUI 节点：RIFE VFI（推荐 rife47 和 rife49）实在太喜欢这个工作流，希望博主提供一下，谢谢。
Hello blogger, there is a node that has never been found, ComfyUI node: RIFE VFI (rife47 and rife49 recommended) really like this workflow, I hope the blogger provides it, thank you.
you can update all your comfyUI.
@downpink 感谢博主在百忙中的回复，我已经更新了，刚刚这个问题没有了，又有新的问题，这个问题怎么解决啊？执行 Efficient Loader 时出错：'NoneType' 对象没有属性 'lower' 文件“D:\SDAI\ComfyUI-aki-v1.3\execution.py”，第 151 行，在 recursive_execute output_data 中，output_ui = get_output_data(obj, input_data_all) 文件“D:\SDAI\ComfyUI-aki-v1.3\execution.py”，第 81 行，在 get_output_data 中 return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) 文件“D:\SDAI\ComfyUI-aki-v1.3\execution.py”，第 74 行，在 map_node_over_list 中 results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) 文件“D:\SDAI\ComfyUI-aki-v1.3\custom_nodes\efficiency-nodes-comfyui\efficiency_nodes.py”，第 158 行，在 efficientloader 模型中，clip = load_lora(lora_params, ckpt_name, my_unique_id, cache=lora_cache, ckpt_cache=ckpt_cache, cache_overwrite=True) 文件“D:\SDAI\ComfyUI-aki-v1.3\custom_nodes\efficiency-nodes-comfyui\tsc_utils.py”，第 372 行，在 load_lora lora_model 中，lora_clip = recursive_load_lora(lora_params, ckpt, clip, id, ckpt_cache, cache_overwrite, folder_paths) 文件“D:\SDAI\ComfyUI-aki-v1.3\custom_nodes\efficiency-nodes-comfyui\tsc_utils.py”, 第 366 行, 在 recursive_load_lora 中返回 recursive_load_lora(lora_params[1:], lora_model, lora_clip, id, ckpt_cache, cache_overwrite, folder_paths) 文件“D:\SDAI\ComfyUI-aki-v1.3\custom_nodes\efficiency-nodes-comfyui\tsc_utils.py”, 第 363 行, 在 recursive_load_lora 中 lora_model, lora_clip = comfy.sd.load_lora_for_models(ckpt, clip, comfy.utils.load_torch_file(lora_path), strength_model, strength_clip) 文件“D:\SDAI\ComfyUI-aki-v1.3\comfy\utils.py”，第 14 行，在 load_torch_file 中，如果 ckpt.lower().endswith(".safetensors")：
@rong028 似乎是Lora文件丢失，你看看你的Lora有没有替换
@downpink 谢谢博主回复，我的lora放在SD里面共享的，和这个有没有关系呢？我注意到博主的模型是使用的1.5，是只能使用1.5还是也可以使用XL模型呢？
@rong028 使用1.5模型
@downpink 谢谢博主回复，现在换1.5模型和lora了，还是显示上面的问题，可能是效率加载器不对，我重新下载试试。
@rong028 如果需要帮助，你可以私信我
what advice to fix this error ?
!!! Exception during processing!!! '_io.StringIO' object has no attribute 'sync_write'
Wow! Great workflow. using the same resources as you it worked on the first try.
Is there a way to adapt this for sdxl? Or are they not really adapted for it?
just for sd1.5
Hi, I gets error "Error occurred when executing ControlNetLoaderAdvanced: join() argument must be str, bytes, or os.PathLike object, not 'NoneType'" at step "AnimateDiff Loader [Legacy]". Could I miss somethings? Pls help me with your advices
have you check your controlnet model? or update your Comfyui?
@downpink I already update ComfyUI, for Controlnet models, I only put 1 model "\sd-webui-controlnet\models\control_openpose-fp16.safetensors" ? Should I need install others?
@trieuviews119 you need install QRCode Controlnet model
@downpink Thank you for your info. Let me try with QRCode Controlnet model.
@downpink Hi, after installing QRCode model, I continue get an error in next step KSampler (Efficient). The log is "[AnimateDiffEvo] - INFO - Using motion module AnimateLCM_sd15_t2v.ckpt:v2.
!!! Exception during processing!!! 'ModuleList' object has no attribute '1'". Could you pls give me your advices?. Thanks.
@trieuviews119 give me dm message
thanks for sharing, very cool
Error occurred when executing Efficient Loader: 'NoneType' object has no attribute 'lower' File "E:\AI\Pinokio\api\comfyui.git\app\execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "E:\AI\Pinokio\api\comfyui.git\app\execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "E:\AI\Pinokio\api\comfyui.git\app\execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) File "E:\AI\Pinokio\api\comfyui.git\app\custom_nodes\efficiency-nodes-comfyui\efficiency_nodes.py", line 158, in efficientloader model, clip = load_lora(lora_params, ckpt_name, my_unique_id, cache=lora_cache, ckpt_cache=ckpt_cache, cache_overwrite=True) File "E:\AI\Pinokio\api\comfyui.git\app\custom_nodes\efficiency-nodes-comfyui\tsc_utils.py", line 372, in load_lora lora_model, lora_clip = recursive_load_lora(lora_params, ckpt, clip, id, ckpt_cache, cache_overwrite, folder_paths) File "E:\AI\Pinokio\api\comfyui.git\app\custom_nodes\efficiency-nodes-comfyui\tsc_utils.py", line 366, in recursive_load_lora return recursive_load_lora(lora_params[1:], lora_model, lora_clip, id, ckpt_cache, cache_overwrite, folder_paths) File "E:\AI\Pinokio\api\comfyui.git\app\custom_nodes\efficiency-nodes-comfyui\tsc_utils.py", line 363, in recursive_load_lora lora_model, lora_clip = comfy.sd.load_lora_for_models(ckpt, clip, comfy.utils.load_torch_file(lora_path), strength_model, strength_clip) File "E:\AI\Pinokio\api\comfyui.git\app\comfy\utils.py", line 14, in load_torch_file if ckpt.lower().endswith(".safetensors"):
check your input model, like Lora. checkpoint, controlnet, aniamtediff, and upscale model, and don't forget to update your comfyUI all
@downpink i see theres a problem. i am on pinokio. on checkpoint it says "ckpt" but i have dreamshape_8lcm.safetensors not ckpt i guess thats the only difference how do i get the ckpt version im unable to find
For some reason, only brown images appear, but I don't know what the problem is...
Is there anything you notice?
check if you use correct controlnet model OR code master
@downpink Thank you for your reply.
controlnet
↓
control_v1p_sd15_qrcode_monster.ckpt
What is code master?
However, VAE cannot use vae-ft-mse-840000-ema-pruned and works with Baked VAE.
Please tell me the code master.m__m
hh, sorry I wrote wrong, it just QRCode Controlnet model, no code master. how's going?
@downpink Why can't I select 840000? I was looking into it ^^
I want to make this wonderful picture you made too...
@__sofia__ if you install correct, fresh your UI, https://civitai.com/models/276082/vae-ft-mse-840000-ema-pruned-or-840000-or-840k-sd15-vae?modelVersionId=311162
@downpink oh! You can do it if you download it!
I thought so too...
I downloaded it and rebooted.
but
「Error occurred when executing Efficient Loader:
VAE.__init__() got an unexpected keyword argument 'ckpt_path'」
Then from earlier
Where do I set the QRCode Controlnet model?
@__sofia__ Check if you vae model name is correct? Your controlnet model should be here: ComfyUI_windows_portable\ComfyUI\models\controlnet
@downpink QRCode Controlnet has been placed! I also checked the screenshot you uploaded earlier, and it seems that there is a problem with VAE. I'll try to find out why it's not being recognized...
@downpink I want to make the picture you made soon too.
@downpink When I restarted my PC, for some reason I was able to generate it only once, but it remained a brown image and was no longer recognized after that...
The mystery of not being able to use VAE...
Is there anything you notice? ><
I want to draw a picture like you soon...
@__sofia__ dm me
I can't read the text in the screenshot because he's too small, could you put the screenshot in the zip together, thanks!
This is extremely satisfying!! so beautiful and eye pleasing, I can't see a glitch even if there was any because the movement is smooth and nice.
Thanks for your affords looks beautiful.I double check every models etc. but i gor this error.
Error occurred when executing KSampler (Efficient): 'NoneType' object has no attribute 'shape'
dm me
Hey) I have a quastion on how to change background to be less dark. It always darken any background I use. Thanks
you can reduce the strength of qr controlnet, and add lineart controlnet
@downpink Thanks, did you mean in reduce the strength in Controlnet Stacker node?
I can't find how to solve this problem:
Error occurred when executing KSampler (Efficient): mat1 and mat2 shapes cannot be multiplied (1232x2048 and 768x320)
Could someone please guide me?
- try converting the images to RGB with a "Image RGB" node? I forget where I got mine.
- Check for SDXL/SD compatibility too in case there's a switch
- And CN load advanced instead of normal CN load (adv apply not necessarily needed)
I've run into this before but forgot the exact issue