CivArchive
    JBOOGX & MACHINE LEARNER ANIMATEDIFF WORKFLOW - Vid2Vid + ControlNet + Latent Upscale + Upscale ControlNet Pass + Multi Image IPAdapter + ReActor Face Swap - v1.0
    NSFW
    Preview 3811732

    WELCOME TO THE JBOOGX & MACHINE LEARNER ANIMATEDIFF WORKFLOW!

    Full YouTube Walkthrough of the Workflow:



    1/8 UPDATE
    Added ReActor Face Swap to Low-Res & Upscaler. Added Bypass / Enable Toggle Switches using rgthree node pack.

    12/7 UPDATE
    About this version

    v2 brings a few quality of life changes and updates.

    I've separated all of the ControlNets into individual groups for you so that you can bypass the one's you don't want to use with ease.

    In the IPAdapter, please download and place this file in your comfyui\clip_vision directory. This is for the 'LOAD Clip Vision' node.

    https://drive.google.com/file/d/13KXx6u9JpHnWdemhqswRQJhVqThEE-7q/view?usp=sharing

    You can find the IPAdapter Plus 1.5 model for the 'LOAD IPAdapter Model' node here.

    https://github.com/cubiq/ComfyUI_IPAdapter_plus

    If you don't want to upscale, then bypass all Upscale groups on the bottom right.

    That should be it :)

    Please tag me in anything you make using the workflow and I will share on my social!

    @jboogx.creative on Instagram
    ---------------------------------------------
    DISCLAIMER: This is NOT beginner friendly. If you are a beginner, start with @Inner_Reflections_Ai vid2vid workflow that is linked here:

    https://civarchive.com/articles/2379/guide-comfyui-animatediff-guideworkflows-including-prompt-scheduling-an-inner-reflections-guide

    After many requests, I have decided to share this workflow that I use on my streams publicly. This is capable of the following....

    -------------------------------------------------

    1. Vid2Vid + Control Nets - Bypass these nodes when you don't want to use them and add any CN and preprocessors you need. The one's included are my go to's.

    2. Latent Upscaling - When not Upscaling during testing, make sure to bypass every upscaling group and the very latent upscale video combine node.

    3. A 2nd ControlNet pass during Latent Upscaling - Best practice is to match the same ControlNets you used in first pass with the same strength & weight

    4. Multiple Image IPAdapter Integration - Do NOT bypass these nodes or things will break. Insert an image in each of the IPAdapter Image nodes on the very bottom and whjen not using the IPAdapter as a style or image reference, simply turn the weight and strength down to zero. This will essentially turn it off.

    5. QR Code Illusion Renders - To do this, use a black and white alpha as your input video and use QR Code Monster as your only ControlNet.

      -------------------------------------------------

    This was built off of the base Vid2Vid workflow that was released by @Inner_Reflections_AI via the Civitai Article. The contributors to helping me with various parts of this workflow and getting it to the point its at are the following talented artists (their Instagram handles)...

    @lightnlense

    @pxl.pshr

    @machine.delusions

    @automatagraphics

    @dotsimulate

    Without their help, this would not have provided the many hours of video making enjoyment it has for me. I am not the most technically gifted person in this, so any input from the community or tweaks to further improve upon this would be greatly appreciated (that's really why I want to share it now)

    There may be some node downloading needed, all of which should be accessible via the Comfy manager (I think). You can bypass any number of the LoRAs, ControlNets, and Upscaling as you need for what you are currently working on. Having intermediate to advanced knowledge of the nodes will help you mitigate with any of the errors you may get as you're turning things on and off, if you're a total beginner, I would recommend starting with @Inner_Reflections_AI base Vid2Vid workflow as it was the only way I was able to understand Comfy in the first place.

    The zip file includes both a workflow .json file as well as a png that you can simply drop into your ComfyUI workspace to load everything. Be prepared to download a lot of Nodes via the ComfyUI manager.

    Any issues or questions, I will be more than happy to attempt to help when I am free to do so 馃檪

    If you make cool things with this, I would love for you to tag me on IG so I can share your creations. Also, a follow is greatly appreciated if you extract any value from this or my Vision Weaver GPT 馃檪 If you use it and like it? Leave me a dope review and throw me some stars!

    @jboogx.creative

    Description

    FAQ

    Comments (45)

    syntaxdiffusionNov 22, 20231 reaction
    CivitAI

    Such a sick workflow!! Thanks for sharing!!

    jboogx_creative
    Author
    Nov 22, 2023

    Thanks for downloading!

    1201754Nov 22, 20231 reaction
    CivitAI

    As you predicted, ComfyUI is throwing errors about missing nodes types. However, through the names it gives, like "Text Box" or "Image Save" I am not able to find the custom node name to download. Could you please share a list, or point me to a resource where I can find out more about it? I already have @Inner_Reflections_AI tutorial workflow set up too.

    jboogx_creative
    Author
    Nov 22, 20231 reaction

    I believe the Text Boxes are from Derfuu_ComfyUI. The Image Save should be a standard node. I'll make an edit to this tomorrow with the complete list of everything I have installed....it's a lot lol. Sometimes if you delete the node and re-add it yourself, that fixes a lot of issues.

    1201754Nov 22, 2023

    @jboogx_creative聽you're awesome!

    albamuthNov 22, 20233 reactions

    Pretty sure "Save Image" is the standard node, but "Image Save" sounds like WAS node suite, which I use to save in different formats and control the filenames.

    1201754Nov 22, 2023

    @albamuth聽It is! You're right! Sadly, I'm still missing:

    Apply ControlNet Stack

    IPAdapterModelLoader

    Control Net Stacker

    ImageDuplicator

    IPAdapterApply

    jboogx_creative
    Author
    Nov 22, 20231 reaction

    @emailaleatorio617111聽Controlnet Stacker - Effciency Nodes.

    IPAdapter Nodes - ComfyUI_IPAdapter_plus

    Image Duplicator - ComfyUI-Image-Selector

    Clockwork_OJNov 24, 2023

    @albamuth聽Installed "WAS node Suite" and the "Images Save" error message went away. Thanks!

    elenaveasNov 22, 20232 reactions
    CivitAI

    Haha, I was printscreening it while watching your twitch :))) Thanks a lot for sharing! No errors so far, only one missing node which can be easily installed with ComfyUI Manager. Let's see if I can do something nice :)

    jboogx_creative
    Author
    Nov 22, 20231 reaction

    Please share whatever you make with it :)

    SabuREALNov 22, 2023
    CivitAI

    Broken Workflow, multiple files missing, after fixing still getting errors! Optional = Required ?

    jboogx_creative
    Author
    Nov 22, 2023

    What is missing and what errors are you getting?

    SabuREALNov 22, 2023

    Error(s) in loading state_dict for Resampler: size mismatch for proj_in.weight: copying a param with shape torch.Size([768, 1280]) from checkpoint, the shape in current model is torch.Size([768, 1024]).

    Ip adapter problems idk

    if i cut them off it worked with some warnings

    hwangbill0645Nov 23, 20231 reaction

    @poweraipower0957聽You will need to have everything up to date. It works for me flawlessly

    hwangbill0645Nov 23, 20231 reaction

    @poweraipower0957 Check in Custom-Nodes There are mostlikely two IP-Adapter folders, copy the models in both folders that resolved this issue for me

    SabuREALNov 25, 2023

    @hwangbill0645聽still dont work idk

    syntaxdiffusionNov 26, 20231 reaction

    this isnt a broken workflow... just download the missing nodes. This is honestly one of the best comfy workflows out there right now. The creator isnt holding anything back

    SabuREALNov 27, 2023

    @syntaxdiffusion聽for me it is idk, i reinstalled the custom nodes, created a ipadapter dict in comfyui, i tryed all ip adapter models that are available... tryed about 7 8 different ckpts or safetensors i still got ipadapter apply error xd

    hwangbill0645Nov 29, 2023

    @poweraipower0957


    I have these:


    cubiq聽
    ComfyUI_IPAdapter_plus

    ComfyUI reference implementation for IPAdapter models. The code is mostly taken from the original IPAdapter repository and laksjdjf's implementation, all credit goes to them. I just made the extension closer to ComfyUI philosophy.

    laksjdjf
    IPAdapter-ComfyUI

    Nodes:Load IPAdapter. This custom nodes provides loader of the IP-Adapter model.

    NOTE: To use this extension node, you need to download the ip-adapter_sd15.bin file and place it in the custom_nodes/IPAdapter-ComfyUI/models directory. Additionally, you need to download the 'Clip vision model' from the 'Install models' menu as well.

    They can be found through the manager typing IPAdapter and you ll see them in the custom node folder

    hwangbill0645Nov 30, 2023

    I just don't know which one worked, but they do not interfere with eachother

    CatzNov 24, 20237 reactions
    CivitAI

    Every custom nodes were found with the "Install Missing Custom Nodes" button but the "Image Save" node. Apparantly it's a YMC node suite node, but my fix was to install the WAS node suite:
    https://github.com/WASasquatch/was-node-suite-comfyui

    Hope it help someone else out there.

    AI_After_DarkNov 25, 20231 reaction

    it did. Thank you!

    Ai_ArtdreamerNov 25, 20231 reaction
    CivitAI

    wow awesome thanks this might be a rookie question but everytime i try to use it i get this error

    When loading the graph, the following node types were not found:

    IPAdapterModelLoader

    IPAdapterApply

    Image Save

    Nodes that have failed to load will show as red on the graph.

    any suggestions. once again thank you so much for sharing this.

    jboogx_creative
    Author
    Nov 25, 20231 reaction

    Here is the Image Save Node:

    https://github.com/WASasquatch/was-node-suite-comfyui

    Search for IPAdapter Plus GitHhub and I believe you can find everything there! Or refer to the IPAdapter stuff in the article I linked in the description!

    pagebrainNov 26, 20234 reactions
    CivitAI

    If anyone else is having trouble running this workflow, I've managed to rerun it in the cloud https://comfy.icu/c/eH4d-UhYlw

    jboogx_creative
    Author
    Nov 26, 2023

    Thanks for sharing this!

    katykakez960Nov 26, 2023
    CivitAI

    The results one can achieve with the workflow is amazing. I was unable to play around with the workflow though. When I try to run the prompt in ComfyUI, I get the following error:

    SyntaxError: Unexpected non-whitespace character after JSON at position 4 (line 1 column 5)

    Would be grateful for anyone's input into the error.

    jboogx_creative
    Author
    Nov 26, 2023

    White Space generally means you have an awkward space or 'return' button press somewhere in the prompt or someplace where you can enter a value. I would try to clear the batch prompt scheduler completely and make sure your syntax being used is spot on. This is not an error i've ever received though :(

    drumicubeNov 26, 20232 reactions
    CivitAI

    Everything is working flawlessly, except for the upscaler, I have OOM issues. It fails in the Ksampler bloc of the 6a.upscaler group. I noticed during the upscaling task, the allocated VRAM is a multiple of the number of frames, so it quickly raise an error, can't go higher than 6 frames. Anyone else having this issue ? Is there a node to split the allocation process frame by frame during upscaling ?
    Thanks.

    jboogx_creative
    Author
    Nov 26, 2023

    Damn, i'm really sorry - this one's above my pay grade. You should ask this in the Banodoco Discord server! A lot of AnimateDiff people who was way smarter than me in there!

    drumicubeNov 26, 20231 reaction

    @jboogx_creative聽Please, don't be sorry. I learn a lot using your workflow, it's really helpful. I'm just switching to comfyui, so spending more time reading docs and readmes is not bad for me ! :)

    GnomeExplorerNov 28, 20231 reaction

    AnimateDiff produces uses / produces batches of latents which is the problem, when you process a batch through ksampler every image is running in parallel so the maximum is going to depend on vram and size (and probably not work at all in directml).

    I haven't tried this but it should probably work.
    The built in node latents->batch->rebatch latents can split them up and process in whatever batch size is optimal or just one at a time. I would do one at a time in your case (the maximum you can do before OOM probably isn't the fastest way to go, but it depends on the card and image size, and there's still VAE decode). Stick one of these before the latent upscaler in that section so there aren't upscaled latents sitting around eating memory before they're being processed.

    If the KSampler in that group relies on an internal random seed that isn't supposed to be changing across the full animation length and was relying on the fact that it was processing a batch to stay constant, it'll need to be set externally so it doesn't change after each image.

    The problem is if you're hitting OOM during the ksampler, doing them one-by-one will probably get you through the VAE decode but you might still need to save an image sequence instead of a video depending on how the video output is implemented.

    Edit: Fixed node -> group in the ksampler / seed info.

    zaitheNov 27, 20231 reaction
    CivitAI

    This is great! Thank you for sharing! I was wondering if there is a way i can run adetailers or something else to fix faces, eyes and hands? I just came from automatic1111 but not sure how you do that in comfyui. Any advice would be great!

    jboogx_creative
    Author
    Nov 27, 20231 reaction

    So I have since added a FaceRestore node w/ a FaceRestore Model in between the first VAE Decoder and the Low-Res Output. Still testing it as the use cases will mostly be for specific videos where the face is prominent. Most of the time, my faces get much better after I do the latent upscale step.

    jonasxkNov 28, 20231 reaction
    CivitAI

    I really like your workflow and I appreciate you sharing this. I run into a little problem though. Any idea how to make the output look more realistic?

    jboogx_creative
    Author
    Nov 28, 2023

    That all comes down to model choice and controlnet settings :)

    zaitheNov 28, 2023
    CivitAI

    Hi, sorry i had another question. For some reason my background keeps showing a pattern instead of my prompt description of the background. I'm using the pose controlnet. Is there some setting I'm missing? Here's a link to an example

    seliko2001Nov 29, 20231 reaction
    CivitAI

    Looks really amazing. Can you share what models are you working with? I usually get really challenging results and suspect the models may be the issue. Thanks!

    lihaoyunlu930Nov 30, 2023
    CivitAI

    thanks for sharing,got an error:

    Error occurred when executing BatchPromptSchedule: list index out of range

    how can i make it work?

    jboogx_creative
    Author
    Nov 30, 2023

    This is either syntax in the prompt scheduler (green box) or your Max Frame number is being funky.....I think

    lihaoyunlu930Dec 1, 2023

    @jboogx_creative聽i dont know never use the prompt scheduler before , change it clip text enc it work

    Eno_oneDec 2, 2023

    @lihaoyunlu930聽I avoided the 'batch prompt scheduler' entirely, used a regular ClipTextEnc as a positive prompt and it fixed the 'index out of range' error.

    zephyrion77Dec 4, 20231 reaction
    CivitAI

    i start comfyUI more than a week ago, i was wandering in the desert until i foud your page. Thanks so much !!

    PaperPlat00nDec 7, 2023
    CivitAI

    great but i have some problems please help me !! would you share the links to the models and etc...

    Workflows
    SD 1.5

    Details

    Downloads
    1,869
    Platform
    CivitAI
    Platform Status
    Available
    Created
    11/21/2023
    Updated
    5/16/2026
    Deleted
    -

    Files

    jboogxMACHINELEARNERANIMATEDIFFWORKFLOW_v10.zip