Introduction

This is my first guide on Civitai. I have been exploring new things in Animatediff and found an excellent guide by Inner Reflection, who is the OG of Animatediff. He recently wrote an article on generating upsampling Animatediff, which is very helpful and interesting. Inspired by Luka Tisler’s post, I experimented with combining live portrait video-to-video with the upsampling Animatediff results. The outcomes were amazing, showcasing the potential of this workflow. In this guide, you'll gain key insights about upsampling Animatediff and live portrait video-to-video

Here is the unsampling Animatediff by inner reflection article link: https://civitai.com/articles/5906/guide-unsampling-for-animatediffhotshot-an-inner-reflections-guide

Quick Overview: What is Unsampling Animatediff?

In the standard Animediff process, we can't control the noise during the denoising phase. However, this technique allows us to predict the initial noise and pass it to a custom sampler, resulting in more consistent animation videos.

Here is the live portrait by kijai: https://github.com/kijai/ComfyUI-LivePortraitKJ/tree/main

Quick Overview: What is Live Portrait?

The live portrait node is developed by kijai. It helps us to transfer facial features from a video to another video.

Prerequisite

1. Attachments: Download the “unsample+live+upscale.json” file from the attachments

2. ComfyUI Setup: Ensure that ComfyUI is running with at least 24 GB of VRAM. If you don't have expensive GPUs, don't worry. You can run this workflow on Jarvislabs with a single click: https://jarvislabs.ai/comfyui_workflows/1027

3. Load the "unsample+live+upscale.json" into ComfyUI.

Let me walk you through the step-by-step process.

Unsampling Group

1. Add Your Video: In the "Load Video" node, load the video you wish to animate.

   

2. Choose width and height: Provide the correct height and width of your video. This step is crucial as it ensures that the video is processed correctly within the specified dimensions.

3. Give a Prompt: Write a detailed prompt that describes the desired characteristics of the video. This should include information about the person's emotion, dress color, hair color, eye color, and the background.

4. Animated Video Generation: Click the "Queue Prompt" button to start generating the animated video based on your input and the provided prompt.

5. Output: Once the animated video is generated, save it. This video will be used in the next steps for the live portrait process.

Live Portrait Group

1. Bypass the Unsampling Group: In the workflow, bypass the unsampling part and enable the live portrait part to proceed with the next steps

2. Add Your Animated Video: Load the video that contains expressions into the "Load Video" node marked as 1 within the "Original Video Clip with Expression" group, and load the video to which you want to add expressions into the node marked as 2

   

3. Give 1:1 aspect size: Once again, provide the correct height and width of your video to ensure proper processing

   

4. Video Generation: Click the "Queue Prompt" button to generate your output video. This video will now have the synced expressions from the original input video applied to it.

Tips

Video Preparation

• Image Size: Ensure that the final unsampling AnimateDiff video is rendered in 1024x1024 or 512x512. These resolutions work best with live portrait nodes and produce optimal results. Avoid using non-standard sizes like 1080x1920.

• Stable Facial Reaction: Ensure the first frame of your movie clip or driving video shows a neutral facial expression. This helps the live portrait detect and animate facial features more accurately. Minimize shoulder and head movements in the driving video for better results.

Prompt Engineering

• Detailed Prompts: For detailed prompts, use screenshots from the video to guide your descriptions. Take screenshots of key frames and use a tool like ChatGPT to generate a stable diffusion prompt. Provide the frame and an example prompt to ChatGPT and ask it to write a similar prompt.

• Example Prompt: "Masterpiece, cinematic, 1man, looking up with intense expression, wearing a black coat and white dress shirt, facial reactions, dramatic scene, high resolution, emotional expressions, scars on face, desolate background, focused, atmospheric, 4k."

Model and Parameter Selection

• Model Selection: Choose the appropriate model for your workflow. Recommended models include meinamix v12 and darksushimix. If using the SDXL model, ensure all the motion and VAE models are compatible.

• CFG Parameter: In the unsample group, setting a higher CFG parameter in the SamplerCustom node makes the video more closely resemble the original.

• Steps Scheduler: In the AlignYourStepsScheduler node, 16 steps are used because it provides a good balance by capturing sufficient detail without overly extending processing time.

• Resample Group: Turn off the addnoise parameter in the SamplerCustom node for static results.

Additional Resources

If you face any problems while rendering the video, please join this Discord community where other users can help you: Discord Link

Conclusion

Thank you for following this guide. I hope you find it helpful in creating amazing videos with realistic expressions using the unsampling Animatediff with live portrait workflow.