Introduction
This workflow is used to create AI Tiktok dance videos, with the action of the AI avatar driven by an actual dance video.
v7 includes audio reactive background using Depthflow and RyanOnThe Inside nodes.
Models Needed
The models required are the red nodes in the workflow, explanation is in the notes (brown nodes) in the workflow. Recommend to install with ComfyUI Manager.
SD1.5 LCM: https://civarchive.com/models/81458?modelVersionId=256668
SDXL Lightning: https://civarchive.com/models/133005/juggernaut-xl?modelVersionId=920957
AnimateLCM_sd15_t2v.ckpt (https://huggingface.co/wangfuyun/AnimateLCM)
Install Using Manager:
IPAdapter Models. ip-adapter_sd15.safetensors, ip-adapter-plus_sd15.safetensors.
Clip-Vision. CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors and CLIP-ViT-bigG-14-laion2B-39B-b160k.safetensors.
AnimateDiff V3: v3_sd15_mm.ckpt, v3_sd15_adapter.ckpt.
xinsir/ControlNet++: All-in-one ControlNet (ProMax model).
control_v11f1p_sd15_depth.safetensors
depth_anything_v2_vitl.pth
Custom Nodes Needed
Install missing custom nodes using manager.
ComfyUI's ControlNet Auxiliary Preprocessors
ComfyUI Frame Interpolation
ComfyUI_IPAdapter_plus
ComfyUI-Advanced-ControlNet
AnimateDiff Evolved
ComfyUI-VideoHelperSuite
rgthree's ComfyUI Nodes
ComfyUI Essentials
KJNodes for ComfyUI
Crystools
ComfyUI-Inspyrenet-Rembg
Depthflow Nodes
RyanOnTheInside
Description
Updated workflow to use AnimateDiff LCM as it gave more realistic output
Adjusted strength_model and strength_clip in the Load LoRA node
Adjusted beta_schedule in the Use Evolved Sampling node
FAQ
Comments (7)
How to make the face better? It really maintains the custom but the facial details come weird , why?
You can try to use the face ID IP adapters to improve face generation.
If you upload your source video, reference image (background + character) and face and PM me the download link, I can help to work on it to improve the generation.
@PixelMuseAI where can i upload?
@loneillustrator you can upload to sites like Imgur or Google drive
How to keep the face of the original video consistent with the face of the resulting video?
I used a 100% denoise for this workflow, which means nothing from the original video is retained.
Try to incorporate one or more of the following to retain the original face:
IPAdapter methods
1) Pass an image with the original face to the IPAdapter (easiest method, check to see if this works first)
2) Add a IPAdapter face ID node
Retaining Parts of the original video (try 3 and if results are not satisfactory, add 4)
3) VAE Encode the original video and use a lower denoise to retain some of the original features
4) Incorporate an inpainting workflow where you mask out the face using MediaPipe Facemesh. Denoise can be adjusted back to 1.
Post Processing Methods (use either 5 or 6)
5) Use ReActor to swap the original face back into the video
6) Use face detailer with a FaceID IPAdapter, select every 2nd frame in the original video and use RIFE to interpolate to avoid flickering on the face
If you need any help, upload your original dance video and an image of your AI avatar to Google drive, send me the link in a DM and I'll see what I can do to update the workflow.
@PixelMuseAI thank u so much