WELCOME TO THE JBOOGX & MACHINE LEARNER ANIMATEDIFF WORKFLOW!
Full YouTube Walkthrough of the Workflow:
1/8 UPDATE
Added ReActor Face Swap to Low-Res & Upscaler. Added Bypass / Enable Toggle Switches using rgthree node pack.
12/7 UPDATE
About this version
v2 brings a few quality of life changes and updates.
I've separated all of the ControlNets into individual groups for you so that you can bypass the one's you don't want to use with ease.
In the IPAdapter, please download and place this file in your comfyui\clip_vision directory. This is for the 'LOAD Clip Vision' node.
https://drive.google.com/file/d/13KXx6u9JpHnWdemhqswRQJhVqThEE-7q/view?usp=sharing
You can find the IPAdapter Plus 1.5 model for the 'LOAD IPAdapter Model' node here.
https://github.com/cubiq/ComfyUI_IPAdapter_plus
If you don't want to upscale, then bypass all Upscale groups on the bottom right.
That should be it :)
Please tag me in anything you make using the workflow and I will share on my social!
@jboogx.creative on Instagram
---------------------------------------------
DISCLAIMER: This is NOT beginner friendly. If you are a beginner, start with @Inner_Reflections_Ai vid2vid workflow that is linked here:
https://civarchive.com/articles/2379/guide-comfyui-animatediff-guideworkflows-including-prompt-scheduling-an-inner-reflections-guide
After many requests, I have decided to share this workflow that I use on my streams publicly. This is capable of the following....
-------------------------------------------------
Vid2Vid + Control Nets - Bypass these nodes when you don't want to use them and add any CN and preprocessors you need. The one's included are my go to's.
Latent Upscaling - When not Upscaling during testing, make sure to bypass every upscaling group and the very latent upscale video combine node.
A 2nd ControlNet pass during Latent Upscaling - Best practice is to match the same ControlNets you used in first pass with the same strength & weight
Multiple Image IPAdapter Integration - Do NOT bypass these nodes or things will break. Insert an image in each of the IPAdapter Image nodes on the very bottom and whjen not using the IPAdapter as a style or image reference, simply turn the weight and strength down to zero. This will essentially turn it off.
QR Code Illusion Renders - To do this, use a black and white alpha as your input video and use QR Code Monster as your only ControlNet.
-------------------------------------------------
This was built off of the base Vid2Vid workflow that was released by @Inner_Reflections_AI via the Civitai Article. The contributors to helping me with various parts of this workflow and getting it to the point its at are the following talented artists (their Instagram handles)...
@lightnlense
@pxl.pshr
@machine.delusions
@automatagraphics
@dotsimulate
Without their help, this would not have provided the many hours of video making enjoyment it has for me. I am not the most technically gifted person in this, so any input from the community or tweaks to further improve upon this would be greatly appreciated (that's really why I want to share it now)
There may be some node downloading needed, all of which should be accessible via the Comfy manager (I think). You can bypass any number of the LoRAs, ControlNets, and Upscaling as you need for what you are currently working on. Having intermediate to advanced knowledge of the nodes will help you mitigate with any of the errors you may get as you're turning things on and off, if you're a total beginner, I would recommend starting with @Inner_Reflections_AI base Vid2Vid workflow as it was the only way I was able to understand Comfy in the first place.
The zip file includes both a workflow .json file as well as a png that you can simply drop into your ComfyUI workspace to load everything. Be prepared to download a lot of Nodes via the ComfyUI manager.
Any issues or questions, I will be more than happy to attempt to help when I am free to do so 馃檪
If you make cool things with this, I would love for you to tag me on IG so I can share your creations. Also, a follow is greatly appreciated if you extract any value from this or my Vision Weaver GPT 馃檪 If you use it and like it? Leave me a dope review and throw me some stars!
@jboogx.creative
Description
This version has been rearranged with the same functionality, to work with the newly refactored and updated IPAdapter nodes. I also updated the Upscaler to a much easier to use High Res Fix Script that will make your lives much easier.
FAQ
Comments (53)
not working i can't figure out where to put the ip adaptater model i tried clip vision folder but it seems the comfyui say "NULL" so it cannot detect the model, reactor not even working too, i give up.
wtf is wrong with you, you cant just give up xd
ComfyUI\models\IPAdapter
you could easyly find this out by checking the ipadapter github page. same for reactor, there a troubleshooting section on the page. https://github.com/Gourieff/comfyui-reactor-node?tab=readme-ov-file#troubleshooting
Like @Rvage mentioned, the github page has detailed instructions, create a new folder called ipadapter in your models directory "ComfyUI/models/ipadapter" and add the ip-adapter-plus_sd15.bin
https://github.com/cubiq/ComfyUI_IPAdapter_plus
https://huggingface.co/h94/IP-Adapter/tree/main/models
anyone know how to solve
Error occurred when executing CLIPVisionLoader: 'NoneType' object has no attribute 'lower'
there is like 6 errors from execution.py/nodes.py/utils.py/clip_vision.py/etc
looks like you probably have either the wrong clip vision model or it's in the wrong folder.
on the video he posted, he talks about it, minute 22:00 you have to colapse and select the pytorch file
That so insane thanks a lot.
Is it possible to use an external video as QR Code contolnet tho?
I'm using another setup where is possible so i'm going to assume is possible even there
Yup. You just have to select the QR Controlnet in the box I have labeled for it <3
anyone with the same error?
When loading the graph, the following node types were not found:
IPAdapterApplyEncoded
Nodes that have failed to load will show as red on the graph.
they dont use apply anymore , so updated to latest advance
The latest version update of the workflow fixes this w/ the new IPadapters <3
Ksampler crashes on 100%
100%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅| 30/30 [10:04<00:00, 20.16s/it]
[AnimateDiffEvo] - INFO - Sliding context window activated - latents passed in (50) greater than context_length 16.
[AnimateDiffEvo] - INFO - Using motion module v3_sd15_mm.ckpt:v3.
0%| | 0/12 [00:00<?, ?it/s]
Someone with the same problem?
me to :/
Any idea of what to do as im trying to use the new work flow, I got told to maybe rename the ksampler but I did this to no effect. its not going pass this stage, I had to by pass the nodes that make the ipadaptors zoom in and out as I could never get this working on the older workflow but never had a problem with that. Ive been out the game for a while now it seems like lots changed. cant see to find anything on the YouTube channel talking about this. much help would be approbated I have a project to finish but sadly cant use comfy with these issue. Thanks
Error since today:
When loading the graph, the following node types were not found:
ReActorFaceSwap
Text box
The Text Box Error is new and I don't really care about the Face Swap. I ignored that for Months now, but without the Text Box - No prompting.
I tried the uninstall/re-install a couple of times and also just deleting the Folder. And I even tried a fresh Comfy Install... the Error persists.
Do we have any alternative Nodes that are able to replace Derfuu's ?
Hmmm, not sure as I've never encountered an error w/ the text boxes that I use
i have the same issue with text box
I already know this stupid text box problem.
I think I had already forgotten that in the note from ... name... sergej... blah...
Here I deleted the two text boxes and inserted the ones from derfuu, they have a string link for the output.
@Jadewind聽I have the same problem, text box & reactor
@med_boussouni聽deleted the two text boxes and inserted the ones from derfuu
Hi, i am problems with:
When loading the graph, the following node types were not found:
Text box
Nodes that have failed to load will show as red on the graph.
"PRE TEXT and APP TEXT"
This problem is in 2 Prompts input of workflow.
Can you help me?
In addition information, i see in installed nodes in "Manager": 聽
(IMPORT FAILED) ComfyUI Nodes for Inference.Core
I find for internet but i don麓t see solved this problem.
Thank you very much!
this problem is easily solved. If you can't do basic troubleshooting then maybe tech stuff isn't for you. Literally tells you what node you are having a problem with and then you look at the console you'll see whether a module is missing. Yes, i am gonna be an ass about it as i am in a bad mood and reading comments from incompetent people like you make my day even worse.
@FriendlyFriend聽no one is born a learned friend! Thanks
@FriendlyFriend聽When your having a good day I'd really appreciate any kinda guidance. :)
@FriendlyFriend聽please, if you are unable to give a positive and constructive answer, just don't respond at all. thank you - lars
I'd hate to make @FriendlyFriend even more mad, but I'm having this exact issue and can't troubleshoot it at all. Has anyone figured this out yet?
@rayraythedivine聽you need to check your console log and install any missing modules. use the embedded python environment within comfyui, not the global python.
@FriendlyFriend聽I apologize for my greenness, but I don't really know how to do any of that haha. I'm basically just bouncing from issue to issue trying to troubleshoot any time I can find a guide typed out in a step-by-step manner cause otherwise it's another language to me.
@rayraythedivine聽there is not really a guide. you start comfyui by running a bat file right? a command line window opens, this is your log. look for errors and post them here.
I managed to import everything but the "PRE Text and APP Text" do not belong to any custom node.
There is also this Error with the .json
SyntaxError: Unexpected non-whitespace character after JSON at position 4 (line 1 column 5
@dkain76聽that means one of your nodes is still 'red' aka not available. honestly there is plenty of other vid2vid workflows that just work. don't bother with this one.
i got the same issue, i tried hard to figure out how to fix doing researches on google and chatgpt, couldn`t...
what i did was delete those nodes and try to run the workflow, i think i did this on the V2 and worked, not saying worked on the v4
@FriendlyFriend聽I got it to work now. The json was pointing to the Text boxes, i just made new String to Text and replaced them. works great now.
you can switch for primitive or string literal
I've tried with different models and samplers, but on the Sampler stage, it always burns my images into some incomprehensible acid-coloured mess, and than crashes (cuda out of memory). Is it something with my graphic card, or am I doing something wrong? Thank you.
Same problem...
same
For people having issues with Text box. They are just regular text box, you can delete and add new ones. Jboogx mentioned that he doesn't even use them anymore. He just writes in the prompt travel at 0 without the text box.
thanks for the info, what nodes should i put instead?
@pretsbr聽Just the regular text node. Double click somewhere empty and write text in the search. Should just be a field to write text. You can duplicate it and add one to the pre_text input and the other to app_text input of the Batch Prompt Schedule.
Or you can remove the old text node and simply write in the Batch prompt schedule. The text in the Batch Prompt Schedule node should be like so
"0": "Write Here"
You can look at the reference of the workflow in the picture above.
Note: If this is your first workflow in ComfyUI, I strongly suggest to try something way easier. Like in the description, you should follow this article and try the attached workflows for a better understanding of what nodes do:
https://civitai.com/articles/2379/guide-comfyui-animatediff-guideworkflows-including-prompt-scheduling-an-inner-reflections-guide
@pretsbr聽string literals
@DigitalAF聽thanks, after couple of weeks i could finally run the workflow, is amazing the results, i found the Primitive Node works too
the man is speaking
works great for me!! thanks, already post my result! i like to work with low resolution videos, so i downscale my video to 512x384, and then run the workflow, i bypassed IPadapter, i couldn't get a good result with, frame load cap 150 per job, to not run out of memory.
specs of my pc
i7 14gen
4060 16gb
32gb ddr5
Can anyone please assist?
I'm getting this error with clipvision and model loader for ipadapter
Error occurred when executing IPAdapterEmbeds: Error(s) in loading state_dict for ImageProjModel: size mismatch for proj.weight: copying a param with shape torch.Size([3072, 1024]) from checkpoint, the shape in current model is torch.Size([3072, 1280]). File "E:\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-0246\utils.py", line 381, in new_func res_value = old_func(*final_args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(slice_dict(input_data_all, i))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 1088, in apply_ipadapter return ipadapter_execute(model.clone(), ipadapter_model, clip_vision, **ipa_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 386, in ipadapter_execute ipa = IPAdapter( ^^^^^^^^^^ File "E:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 70, in init self.image_proj_model.load_state_dict(ipadapter_model["image_proj"]) File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 2189, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
I am getting error Cannot execute because a node is missing the class_type property.: Node ID '#119
Anyone from the community can help ?
When installing the workflow, ReActorFaceSwap, can't be installed correctly.
I'm running ComfyUI_windows_portable Release v0.0.1
I tried to install it, fix it, uninstall/reInstall and nothing works yet.
Other steps, is trying to install it from the GitHub page : https://github.com/Gourieff/comfyui-reactor-node?tab=readme-ov-file#latestupdate
But I'm blocked at step 5.
5.Go to the SD WebUI root folder, open Console or Terminal and run (Windows users).\venv\Scripts\activate or (Linux/MacOS)venv/bin/activate
I can't find the directory. Available on discord!
Thanks in advance, cheers!
For anyone struggling to install it, ive achieved it with this guide : https://www.youtube.com/watch?v=zCvik-ohKko
Missing node - text box
I have installed all the nodes but still this pop up comes
use primitive
@brunoc_pe聽how?
When loading the graph, the following node types were not found:
ReActorFaceSwap
IPAdapterApplyEncoded
Nodes that have failed to load will show as red on the graph.
Installing missing nodes just gives an error
This is So Awesome! Great!