WELCOME TO THE JBOOGX & MACHINE LEARNER ANIMATEDIFF WORKFLOW!
Full YouTube Walkthrough of the Workflow:
1/8 UPDATE
Added ReActor Face Swap to Low-Res & Upscaler. Added Bypass / Enable Toggle Switches using rgthree node pack.
12/7 UPDATE
About this version
v2 brings a few quality of life changes and updates.
I've separated all of the ControlNets into individual groups for you so that you can bypass the one's you don't want to use with ease.
In the IPAdapter, please download and place this file in your comfyui\clip_vision directory. This is for the 'LOAD Clip Vision' node.
https://drive.google.com/file/d/13KXx6u9JpHnWdemhqswRQJhVqThEE-7q/view?usp=sharing
You can find the IPAdapter Plus 1.5 model for the 'LOAD IPAdapter Model' node here.
https://github.com/cubiq/ComfyUI_IPAdapter_plus
If you don't want to upscale, then bypass all Upscale groups on the bottom right.
That should be it :)
Please tag me in anything you make using the workflow and I will share on my social!
@jboogx.creative on Instagram
---------------------------------------------
DISCLAIMER: This is NOT beginner friendly. If you are a beginner, start with @Inner_Reflections_Ai vid2vid workflow that is linked here:
https://civarchive.com/articles/2379/guide-comfyui-animatediff-guideworkflows-including-prompt-scheduling-an-inner-reflections-guide
After many requests, I have decided to share this workflow that I use on my streams publicly. This is capable of the following....
-------------------------------------------------
Vid2Vid + Control Nets - Bypass these nodes when you don't want to use them and add any CN and preprocessors you need. The one's included are my go to's.
Latent Upscaling - When not Upscaling during testing, make sure to bypass every upscaling group and the very latent upscale video combine node.
A 2nd ControlNet pass during Latent Upscaling - Best practice is to match the same ControlNets you used in first pass with the same strength & weight
Multiple Image IPAdapter Integration - Do NOT bypass these nodes or things will break. Insert an image in each of the IPAdapter Image nodes on the very bottom and whjen not using the IPAdapter as a style or image reference, simply turn the weight and strength down to zero. This will essentially turn it off.
QR Code Illusion Renders - To do this, use a black and white alpha as your input video and use QR Code Monster as your only ControlNet.
-------------------------------------------------
This was built off of the base Vid2Vid workflow that was released by @Inner_Reflections_AI via the Civitai Article. The contributors to helping me with various parts of this workflow and getting it to the point its at are the following talented artists (their Instagram handles)...
@lightnlense
@pxl.pshr
@machine.delusions
@automatagraphics
@dotsimulate
Without their help, this would not have provided the many hours of video making enjoyment it has for me. I am not the most technically gifted person in this, so any input from the community or tweaks to further improve upon this would be greatly appreciated (that's really why I want to share it now)
There may be some node downloading needed, all of which should be accessible via the Comfy manager (I think). You can bypass any number of the LoRAs, ControlNets, and Upscaling as you need for what you are currently working on. Having intermediate to advanced knowledge of the nodes will help you mitigate with any of the errors you may get as you're turning things on and off, if you're a total beginner, I would recommend starting with @Inner_Reflections_AI base Vid2Vid workflow as it was the only way I was able to understand Comfy in the first place.
The zip file includes both a workflow .json file as well as a png that you can simply drop into your ComfyUI workspace to load everything. Be prepared to download a lot of Nodes via the ComfyUI manager.
Any issues or questions, I will be more than happy to attempt to help when I am free to do so 🙂
If you make cool things with this, I would love for you to tag me on IG so I can share your creations. Also, a follow is greatly appreciated if you extract any value from this or my Vision Weaver GPT 🙂 If you use it and like it? Leave me a dope review and throw me some stars!
@jboogx.creative
Description
v2 brings a few quality of life changes and updates.
I've separated all of the ControlNets into individual groups for you so that you can bypass the one's you don't want to use with ease.
In the IPAdapter, please download and place this file in your comfyui\clip_vision directory. This is for the 'LOAD Clip Vision' node.
https://drive.google.com/file/d/13KXx6u9JpHnWdemhqswRQJhVqThEE-7q/view?usp=sharing
You can find the IPAdapter Plus 1.5 model for the 'LOAD IPAdapter Model' node here.
https://github.com/cubiq/ComfyUI_IPAdapter_plus
If you don't want to upscale, then bypass all Upscale groups on the bottom right.
That should be it :)
Please tag me in anything you make using the workflow and I will share on my social!
@jboogx.creative on Instagram
FAQ
Comments (25)
Thank you very Much !!! But where can i find the GOOGLE Drive Link?
It's under the 'About This Version' on the right hand side :)
Thanks Jboogx. The workflow file is missing from the Zip file for V2. There is currently only a ..v2.png file. I download the pytorch file but there is only a .bin file.
Same here guess we are stuck
Hey! The png is the workflow! Just drop the png into the ComfyUI workspace and the workflow will load :)
The .bin file is correct. That is what goes into your Clip_Vision folder :)
@wght545 please see my other reply :)
@jboogx_creative Thanks Bro
@jboogx_creative Thanks So much. Not sure how its possible for ComfyUI to read a PNG but it loads.
Getting this error on queue. "Error occurred when executing BatchPromptSchedule: Unterminated string starting at: line 1 column 7 (char 6) File "C:\ComfyUI_windows_portable\ComfyUI\execution.py", line 153, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "C:\ComfyUI_windows_portable\ComfyUI\execution.py", line 83, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "C:\ComfyUI_windows_portable\ComfyUI\execution.py", line 76, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) File "C:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_FizzNodes\ScheduledNodes.py", line 121, in animate animation_prompts = json.loads(inputText.strip()) File "json\__init__.py", line 346, in loads File "json\decoder.py", line 337, in decode File "json\decoder.py", line 353, in raw_decode" Anyone know the fix?
I'm getting the same thing. Would like to know too
@ceilingfan1234567990 think its your syntax's, triple check its correct
Different fault here, all default settings, same with new or old version. My text boxes seem correct. a comma in 1st line none in 2nd, and the line No's seem correct, ill try a reinstall when i can get round to it. But will check back here jic anyone can help, many thanks, im sure ill get it working some day.
Here is the fault :-
Error occurred when executing IPAdapterEncoder: 'NoneType' object has no attribute 'encode_image' File "C:\ComfyUI_windows_portable\ComfyUI\execution.py", line 153, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI_windows_portable\ComfyUI\execution.py", line 83, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI_windows_portable\ComfyUI\execution.py", line 76, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 642, in preprocess clip_embed = clip_vision.encode_image(image) ^^^^^^^^^^^^^^^^^^^^^^^^
@scraggydog Igot over the same error yesterday, the solution is in the first line : error executing IpadpterEncoder : the pipe (nonetype) doesnt bring the (encode image) data . the error is not on the underlined node (ipadapter node) but from the node before IPadapter, connected to its encode image input.
check your prompt with a json validator , and double check the nodes before ipadapter node. for my part i resolved this by using another checkpoint and changing some loras to make it work
@virusvjvisuals588 Many thanks, I got a bit frustrated, uninstalled Comfy only to get same fault, eventually found the wrong .ckpt (Version3) so thankyou very much. spot on.
I think your workflow is so great!
I have two questions
Where should I start to test how to do 50 frame test
Currently, I am looking at the prompts listed after the video generation is complete
I want to see the images like the sample images on the way.
I can't do anything while the video is generating, so please let me know!
In which situation is the alpha mask used?
The first video upload is inserting a video with the background removed.
I don't know where to put the alpha mask
Im having this error when trying to run it:
"Error occurred when executing IPAdapterModelLoader: invalid IPAdapter model C:\ComfyUI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\models\pytorch_model.bin File "C:\ComfyUI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 153, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "C:\ComfyUI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 83, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "C:\ComfyUI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 76, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) File "C:\ComfyUI\ComfyUI_windows_portable_nvidia_cu118_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus\IPAdapterPlus.py", line 396, in load_ipadapter_model raise Exception("invalid IPAdapter model {}".format(ckpt_path))"
Same problem!I guess we download a wrong IPA model,but i cant comfirm that
same problem, any updates by chance?
Hello, thanks for the workflow. I would like to know how to disable unnecessary CN and Lora. Do you need to delete nodes?
Just right click on the Group Box for the control net and select 'Bypass Group Nodes'. If you would like to reenable it, then do the same thing and select "Set Group Nodes to Always"
@jboogx_creative Thank you very much, I also wanted to know about one problem. I fixed all the errors, the generation is underway, but at the end of the Combine there is no video, only a black screen and even there is no video in the Output folder. What could be the mistake?
@shihgereev Hmmm, not something I've ran into. Maybe delete, and read the Video Combine node and make sure all the settings in it are good.
@jboogx_creative I tested 3 variants of video combine and different formats, but the result is the same. I didn't change the settings much in workflow, except for prompt and Lora. As soon as I'm free, I'll try to figure out this problem myself. If I find a solution, I will unsubscribe here, perhaps someone will also face such a problem
@jboogx_creative Eeeee, I found a solution to the problem. It's all because of the BatchPromptSchedule. As I replaced it with a regular prompt input window, everything began to work
Many thanks, ill check this out, just having a few issues with Ipadapter fault.
Finally re-installed and all good, just having a play round.