If looking for Video to video with modern modelsVACE wan 2.1 is what you are looking for.<a target="_blank" rel="ugc" href="https://civarchive.com/models/1604221/native-vace-video-to-video-with-ref">https://civarchive.com/models/1604221/native-vace-video-to-video-with-ref</a><hr />This wont be updated anymore sorryV5.1 (V6 Beta) - This was a lot of work, please ask questions and give any feedback in comments. A LOT has changed so i really want to make sure everything works.- Redone from nearly scratchWhats new1. Model Loading- Now 1 model default with second model optional if enabled.- Changing of animate diff settings/model for upscale for either model used.- Motion lora loading- Removed all saving of controls inside this flow.- Move resizing to beside prompting.2. masking- 3 main controls loaded from video or image folder or generated in the flow.- 5 optional mask loaders- Sam 2 single mask creation- Mask subtracting/adding- Optional upscale mask.3. Backround Controls- Inpaint Simplified- Simplified 4 background options1. Original Frames2. Video import background3. Image Load4. Image Generation- Image Load and generation both have a crop group to place subjects.- Background Controls are strait forward for default copies of any of the chosen backgrounds.4. Subject Controls- Previews to show what masked controls see when you mask them- Line art, depth, and openpose controls- Outfit to outfit / admotion stabalizers.- Control cutouts5. Audio Replacement loading6. Clearer path for upscale controls.<hr />Controls 5.11 - i dont know what went wrong but it was saving to the wrong spot, I replaced the video text box with the same exact node and it started working again. I dont know if this was a "me" issue or the the node was updated and bugged out.<ul><li>changed openpose to properly bypass instead of disable</li></ul>Sam 5.1 - working on article for this flow now.Simple sam 2 masking - both flows in one now.There are so many damn ways to do this... i am giving you the 2 best i have found.Features- seperation of people or objects in a video/image- tracks people or objects throughout videos- mask faces, hands, clothing, skin, hair, or any object.<ul><li>load from project folder or any location.</li></ul><a target="_blank" rel="ugc" href="https://civarchive.com/articles/8035">https://civarchive.com/articles/8035</a> <a target="_blank" rel="ugc" href="https://civarchive.com/articles/8360/using-sam-2-masking">https://civarchive.com/articles/8360/using-sam-2-masking</a><hr />Controls 5.1 is the start of 6.0, still have work to do on the others so thought i would throw this up for now, its very streamlined for video/pose/depth/lineart saving with optional masking with sam1 or remove background- Removed face mesh- removed depth masking- removed optional controls- removed anything that was not needed anymore.- Changed Interface to be more streamlined- Kept sam1 and remove background masking options if needed- video source, Depth, pose, lineart video saving- optional depth and video source image saving<a target="_blank" rel="ugc" href="https://civarchive.com/articles/8036/video-to-video-project-setups">https://civarchive.com/articles/8036/video-to-video-project-setups</a><hr />5.01- added back downscale options for all controls (large vram savings when using HD controls)- loading in all masks for ip adapter perhaps will add 1 more in next update.- fixed a few minor bugsImportant note : All save will have a 00001 etc attached at the end, you can remove this or just use the names as is to load in the video flow.I feel this should be made clear, V5 comes with many extra workflows. They are all meant to be used in the order of the folders.confyui/output/projects is the main folder, you choose the video name inside this folder to save each videoFollowing flowsmasksandcontrols<ul><li>used to save the video/controlnets/masks to the proper folder to load in for the videos flow.</li><li>These will have 00001 etc attached to the names, you can rename them or use those default names when loading in the videos flow</li></ul>sam 2 masking<ul><li>used to load in the saved video from the masking flow and create sam 2 masks for it saving everything to the same project folder</li></ul>Videos flow<ul><li>This loads in the video and controls generated above to create a video.</li><li>This saves everything to a folder called base and a folder called upscaled.</li></ul>upscale flow<ul><li>This loads in any videos from the upscaled folder inside your video folder, these are the saves from the video flow.</li><li>This saves to a folder called upscale2 inside the same project/videoname folder</li><li>this allows you to load in a mask from the project folder</li></ul>face fix flow<ul><li>this loads from the upscale2 folder to process the faces</li><li>this saves to the folder called faceenhance</li></ul>Live portrait flow<ul><li>this loads from the faceenhance folder the video of your choice</li><li>this saves to the main video folder, i believe.</li></ul>All of these have options to change where they are loading from, so you can skip a step or 2 if you want.V5 has the ability to generate all controls in the flow. This was my attempt to integrate the masking flow with the videos flow. It worked, but it is HIGHLY recommended to run the masksandcontrols.json and generate the controls needed for your video.<ul><li>It takes less than 5 minutes, and is easy to follow. This flow creates ALL folders and controls you need to run the videos flow and places them inside the project/videoname folder for you. You can load any video in this flow from anywhere on your pc to process.</li></ul><ul><li>Then you can load up sam 2 masking and load the video from the project folder the masking flow saved. After this it only loads from this folder.</li></ul><ul><li>If you skip this step things get more confusing when reloading masks and controls.</li></ul>All examples use no upscale or face fixing, the girl with curly hair running had a face fix appliedExplanation of all node groups<a target="_blank" rel="ugc" href="https://civarchive.com/articles/7063">https://civarchive.com/articles/7063</a>Enjoy everyone. :)V5- Reworked loading to work better with loaders- Interface cleaned up- moved prompting to beside base render- changed the loading of models for easy turn off and on of settings- removed pix2pixThe following is set up to run with the videos from the main video flow using project folder.- including SAM 2 masking flow- including masking/controlnet flow- including upscale flow- including face fix flow- including Live Portrait flow- added article with info on video gen workflow- 2 example projects included- looped spin- runningMasking flow can now save images for frames and depth to help with compression artifactingIm tired.Beta 3 - I am separating v2 and v3 beta because there have been many changes to comfy, and bugs introduced that i dont know if i need to fix or will be fixed with comfy updates. V2 may work better?- a lot of minor fixes, hopefully did not break other things.- added sam masking to main the mask save area- added depth mask saving to main depth gen area.- added saving of custom sam masks generated / added mask loader to load in your custom saved masks- Moved all controlnet Model loading to the start, ensure you have loaded your models properly.- tested and fixed the dual mask prompting, i think it works properly.- auto adding commas to all schedules to ensure they work properly- tagger tags integrated into all prompts if enabled, schedules it adds to pre text.- edited the placement background option... not sure if it was broke before...- removed meshgraphormer hand refiner. IMO this is meant for text to video, not video to video. Hand masking can be done with sam masks and you can use depth/lineart with those custom masks.- added the nodes needed to combine any of your masks together.- added skip every override to all control loaders to allow you to load in controls you saved with skip every enabled.- inlcuded project folder with a animation from examples. called "Stop"I am sure there will be errors for the masking in some cases due to the downscale option, so many previews depend on it.If you have an error with any combine masks disable the downscale masks option and it should work. I am not even sure downscale does anything i think it auto downscales when rendering.Included is a project folder with a video to load as default.<hr />The below issue has been fixed with updated comfy UI<a target="_blank" rel="ugc" href="https://github.com/comfyanonymous/ComfyUI/pull/4535">https://github.com/comfyanonymous/ComfyUI/pull/4535</a>**********IMPORTANT**********Bug in video loaders that has been reported and hoping for a fix- when loading a video via string combines as this flow does the loaders needs to load things 2 times to cache things properly... I dont know why, it should take only 1possible solutions1. I could change all loaders to load videos directly but that would need you to set the video for every video control you load and break any simple text boxes that load videos. Its a workaround that i dont want to do TBH, as it makes more work for both me and you.2. Keep it as is with knowledge you need to set up all controls and load them Twice before starting your render. This can be done by just disabling the first and second pass renders, and running the flow a few times until it just starts and stops immediatly waiting for changes. It works on first render, but if you enable upscale after doing base it will restart base if you have not loaded Twice. This is currently what i am doing while waiting for a response to my bug report.In summary, Disable both first and second render and run the flow to load all controls (2 or 3) times until the flow does not reprocess anything when you press q. This prevents restarts when enabling things further down in the flow like the upscale.<hr />Beta 2 -fixed save location for pose and line artfixed batching and re-batching for SAM custom masksadded a default project folder with a default video its 400+ frames original so limit the frames if you have a lower vram card to use the default. Sheesh Dance, example in previewsPut project folder inside comfy ui output folder to load default workflow<hr />V5 beta- reworked interface- added simple masking creation and save- added dual mask loading- added depth, pose, lineart generation and save- Reworked loading controls- Reworked prompting- Simple Prompting, Scheduled Prompting, Regional Prompting- added more custom masking options- using constants to prevent line spam- updated all video combinesFixes- not using clipvison the wrong wayRemoved-Depth lineart-Face Masking-Face Mesh-Face Controls-Background scroll/panLooking to get feedback if this is better, worse or whatever.Leave a comment if you think its worse for some reason, or a thumb up i you like itNot sure if its better than v4 but its a lot cleaner.<hr />Updated workflow to fix the spin1 example, it now has the needed files, sorry.<hr />Starting to go through this to optimize and fix anything comment on anything you would like added or changed.masking 4.1- added Depth Anything v2- added dense pose saving for testing- added proper saving locations, now all control net files are saved to the projects folder inside the chosen video folder for easier sortingnote - all files still need to be renamed to remove the 00001 extensions.- added proper switch to change the save directory name<hr />V4Simplified - this version is cut down to only offer less confusing options with a clear path.- I dont know many more things to make this easier. Some advanced things removed, and UI is now a walkthrough.- Removed RGB masks \ multi prompting (use advanced version for that)- Added Step groups to help guild new users.- Added new notesAll 4 parts included in 1MaskingVideosUpscaleExtras<ul><li>2 example projects</li><li>1 spin as usual</li><li>1 anime dance that was requested.</li><li>a few old projects to show examples, they are problably out of date but should show you settings i used.</li></ul>If You tried this workflow before and thought it was too complicated, this version is for you. This took a while to put together.<hr />some possible nodes needed<a target="_blank" rel="ugc" href="https://github.com/ltdrdata/ComfyUI-Manager">https://github.com/ltdrdata/ComfyUI-Manager</a><a target="_blank" rel="ugc" href="https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite">https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite</a><a target="_blank" rel="ugc" href="https://github.com/crystian/ComfyUI-Crystools">https://github.com/crystian/ComfyUI-Crystools</a><a target="_blank" rel="ugc" href="https://github.com/mav-rik/facerestore_cf">https://github.com/mav-rik/facerestore_cf</a><a target="_blank" rel="ugc" href="https://github.com/ltdrdata/ComfyUI-Impact-Pack">https://github.com/ltdrdata/ComfyUI-Impact-Pack</a><a target="_blank" rel="ugc" href="https://github.com/WASasquatch/was-node-suite-comfyui">https://github.com/WASasquatch/was-node-suite-comfyui</a><a target="_blank" rel="ugc" href="https://github.com/Nourepide/ComfyUI-Allor">https://github.com/Nourepide/ComfyUI-Allor</a><a target="_blank" rel="ugc" href="https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes">https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes</a><a target="_blank" rel="ugc" href="https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet">https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet</a><a target="_blank" rel="ugc" href="https://github.com/cubiq/ComfyUI_essentials">https://github.com/cubiq/ComfyUI_essentials</a><a target="_blank" rel="ugc" href="https://github.com/Fannovel16/ComfyUI-Frame-Interpolation">https://github.com/Fannovel16/ComfyUI-Frame-Interpolation</a><a target="_blank" rel="ugc" href="https://github.com/SLAPaper/ComfyUI-Image-Selector">https://github.com/SLAPaper/ComfyUI-Image-Selector</a><a target="_blank" rel="ugc" href="https://github.com/FizzleDorf/ComfyUI_FizzNodes">https://github.com/FizzleDorf/ComfyUI_FizzNodes</a><a target="_blank" rel="ugc" href="https://github.com/LonicaMewinsky/ComfyUI-MakeFrame">https://github.com/LonicaMewinsky/ComfyUI-MakeFrame</a>Controlnet Models usedThink these2 are the same, or they produce very similar results.(unknown name) adMotion - <a target="_blank" rel="ugc" href="https://huggingface.co/crishhh/animatediff_controlnet/resolve/main/controlnet_checkpoint.ckpt?download=true">https://huggingface.co/crishhh/animatediff_controlnet/resolve/main/controlnet_checkpoint.ckpt?download=true</a>Animatediff Controlnet - <a target="_blank" rel="ugc" href="https://civarchive.com/models/232617/animatediff-controlnet-models">https://civarchive.com/models/232617/animatediff-controlnet-models</a>Media Pipe Face Mesh - <a target="_blank" rel="ugc" href="https://huggingface.co/CrucibleAI/ControlNetMediaPipeFace/tree/main">https://huggingface.co/CrucibleAI/ControlNetMediaPipeFace/tree/main</a>Outfit to Outfit - <a target="_blank" rel="ugc" href="https://civarchive.com/models/191956/outfit-to-outfit-controlnet-outfit2outfit">https://civarchive.com/models/191956/outfit-to-outfit-controlnet-outfit2outfit</a>pix2pix - <a target="_blank" rel="ugc" href="https://civarchive.com/models/38784/controlnet-11-models">https://civarchive.com/models/38784/controlnet-11-models</a>

Video to video - Workflows

3.11 - small fixes - changed cropping padding to have switches for on/off, noticed it filled ram use even when setting to 0. Ensure all nodes in these groups are bypassed (or enabled if using them) AFTER loading in control nets or a new video, saves a lot of ram. - manual HxW setting switch, going with the video rescale. - added option to turn off loading in video/frames if you are setting HxW / frames yourself - can now turn off all control nets and just run text to video with animate diff and the prompts. 3.1 - full overhaul - large optional prompting changes - extra controlnet sources- Full UI rework -- Full backend rework- added optional loading of MP4 based controlnets from masking flow- added toggles to load either mp4/images for all controlnets- changed inputs to only require the foldername to load any of the controls from masking flow- added lora/embedding support to all prompts (Auto1111 style)- this allows loras and embeddings to be used on a per-mask basis (separating subject and background loras)- this allows loras and embeddings to be scheduled and weighted throughout the video (have not found a use for this yet with vid to vid)- changed model loading to be an optional second model. Before it would load 2 models no matter if you used one or both.- added the ability to use the same first model to upscale without the animate diff lora.- (testing) Added color correction prompting to hopefully help color bleed from items. I don't think it works well but could be useful (fully optional)- changed prompts to save all to one file.

Files with SHA256: a306186ad756751eaceef3bcb5324fdf7e1e6ec3916093adf6821398e0f00e6d