This workflow uses Mimic Motion to drive the generation of the pose of head. Background is replaced with a static image, ReActor is used to do a face swap and differential diffusion is used on a AnimateLCM pass to animate the background and clean up the singer. Live Portrait is then used to transfer the lips and/or eyes.
ComfyUI Version
This workflow was tested on ComfyUI v0.21 and should not have package conflicts when installing custom nodes from manager: https://github.com/comfyanonymous/ComfyUI/releases/tag/v0.2.1
Nodes
Red nodes are models that need to be downloaded and selected in the workflow.
Blue nodes are nodes that require user input.
Recommended to use ComfyUI manager to install the following custom nodes. You will need to follow the instructions at the ReActor node github page to install insight face (https://github.com/Gourieff/comfyui-reactor-node?tab=readme-ov-file#troubleshooting)
ComfyUI's ControlNet Auxiliary Preprocessors
ComfyUI Frame Interpolation
AnimateDiff Evolved
ComfyUI-VideoHelperSuite
ReActor Node for ComfyUI
ComfyUI Essentials
KJNodes for ComfyUI
ComfyUI-MimicMotionWrapper
ComfyUI-LivePortraitKJ
Crystools
Models
Manually downloaded and placed in folder
AnimateLCM: https://huggingface.co/wangfuyun/AnimateLCM Put AnimateLCM_sd15_t2v.ckpt in "models/animatediff_models" and put AnimateLCM_sd15_t2v_lora.safetensors in "models/loras"
SD1.5 Checkpoint: you can use your favourite SD1.5 Checkpoint. I am using Realistic Vision.
Download using ComfyUI Manager
control_v11p_sd15_openpose.safetensors
control_v11f1p_sd15_depth_fp16.safetensors
Automatically Downloaded
MimicMotion
ReActor
Description
Updated the workflow to get a more static background.
FAQ
Comments (12)
Getting an error:
Error no file named diffusion_pytorch_model.fp16.bin found in directory C:...\ComfyUI\models\diffusers\stable-video-diffusion-img2vid-xt-1-1
Where would I be able to find that file?
This model is required by Mimic Motion. It is downloaded automatically by the (Down)Load MimicMotionModel node. I just tested with a fresh install of ComfyUI and the model downloaded automatically with no issues.
1) Did you watch your console output as the model was downloading and was there any error reported?
2) Do you normally have issues downloading models from huggingface?
If you are still having difficulties, I guess you can try manually downloading from here: https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt-1-1 and placing the model in the models\diffusers\stable-video-diffusion-img2vid-xt-1-1 folder
Hope this helps you to resolve the error.
thanks for the workflow
Trying to figure out why I'm getting this error:
got prompt
!!! Exception during processing !!! No frames generated
Traceback (most recent call last):
File "/home/test/ComfyUI/execution.py", line 323, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
File "/home/test/ComfyUI/execution.py", line 198, in get_output_data
return_values = mapnode_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
File "/home/test/ComfyUI/execution.py", line 169, in mapnode_over_list
process_inputs(input_dict, i)
File "/home/test/ComfyUI/execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))
File "/home/test/ComfyUI/custom_nodes/ComfyUI-VideoHelperSuite/videohelpersuite/load_video_nodes.py", line 271, in load_video
return load_video_cv(**kwargs)
File "/home/test/ComfyUI/custom_nodes/ComfyUI-VideoHelperSuite/videohelpersuite/load_video_nodes.py", line 207, in load_video_cv
raise RuntimeError("No frames generated")
RuntimeError: No frames generated
Prompt executed in 0.17 seconds
!!! Exception during processing !!! No frames generated
Traceback (most recent call last):
File "D:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\execution.py", line 323, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\execution.py", line 198, in get_output_data
return_values = mapnode_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\execution.py", line 169, in mapnode_over_list
process_inputs(input_dict, i)
File "D:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-VideoHelperSuite\videohelpersuite\load_video_nodes.py", line 271, in load_video
return load_video_cv(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-VideoHelperSuite\videohelpersuite\load_video_nodes.py", line 207, in load_video_cv
raise RuntimeError("No frames generated")
RuntimeError: No frames generated
There seems to be an issue with your driving video. Check your frame load cap and skip first frames settings to ensure that you have some frames loaded.
Hi
Came across your workflow on a Reddit Post's comment
Was looking for better mimic motion workflows out there as their stock workflow was not that good.
Just wanted to ask if you plan to release an SDXL compatible workflow in the future?
AnimateDiff works better on SD1.5 than SDXL so I don't think there will be an update to this workflow until I see something better come up.
Hey There! Trying to get thisd workflow working but ReActor Node is not working. I have it installed but the manager says "Import Failed". Is it possible that it is not working in comfy right now?
you can disable ReActor to check if the rest of the workflow is working.
for ReActor troubleshooting, check the troubleshooting section of the ReActor github page: https://github.com/Gourieff/comfyui-reactor-node
reactor is off
i havent tried the new repo, but you can test: https://github.com/Gourieff/ComfyUI-ReActor