Based on 100 experiments, this refined 4-step WAN 2.1 workflow in ComfyUI shows how to use Controlnet type control of your video generation using VACE 14B. I've fine-tuned this workflow to be almost universal. This isn’t guesswork—it’s tested, repeatable, and built to work.
Description
Based on 100 experiments, this refined 4-step WAN 2.1 workflow in ComfyUI shows how to use Controlnet type control of your video generation using VACE 14B. I've fine-tuned this workflow to be almost universal. This isn’t guesswork—it’s tested, repeatable, and built to work. Please comment what video I should make next!
FAQ
Comments (22)
Which video card did you test on? Will the 3090 pull the full model?
RTX 4090, you can see his tutorial here: https://www.youtube.com/watch?v=ZE9kxEut7lo
If somebody can get it to work on a 3090 and let us know, I'd be super grateful! ^_^
@linkerlinkeador I run on RTX3090TI it works great
I get out of memmory in my 3090, i try to use the gguf version but says me that needs the gguf version of module, and i cant find it....
What is workflowFlorence_Video_Caption? your workflow requires it and I have no idea where to get it. ComfyUI can't find it.
ПРИ ОШИБКЕ = workflowFlorence_Video_Caption
СКАЧИВАЕМ = ComfyUI-Florence2
https://github.com/kijai/ComfyUI-Florence2
И СТАВИМ ПРИМЕР В =
E:\ComfyUI 2.1\ComfyUI_windows_portable\ComfyUI\custom_nodes
ИЛИ
E:\ComfyUI-StableDif-t2.7-p312-cu128-Triton\ComfyUI\comfy_api_nodes
ДАЛЕЕ ....ОБЯЗАТЕЛЬНО ЗАХОДИМ В ПАПКУ = ПРИМЕР ГДЕ ( ComfyUI_windows_portable )
E:\ComfyUI 2.1\ComfyUI_windows_portable
ИЛИ
E:\ComfyUI-StableDif-t2.7-p312-cu128-Triton
ЗАПУСКАЕМ CMD В ПАПКЕ И ПРОПИСЫВАЕМ
python_embeded\python.exe -m pip install -r ComfyUI\custom_nodes\ComfyUI-Florence2\requirements.txt
ПОСЛЕ ЭТОГО ОБНОВИТСЯ transformers version 4.38.0 И ВСЕ ЗАРАБОТАЕТ.
УДАЧИ РЕБЯТА
@SERGEY73 I'm also having difficulty getting this running in runpod
For anyone facing this error, try a few things:
1- Make sure you have Florence-2 component installed as indicated by Sergey73 (https://github.com/kijai/ComfyUI-Florence2)
2- Make sure all other required but missing nodes are also installed
3- If you still see this specific error, open the workflow JSON file e.g. (WAN 2.1 VACE 3B Hallett - Anime.json) (don't worry, you don't need too much technical expertise for this - I'll guide you):
- It is a large file, so best way to get where you want to get to is to search for "workflow>Florence_Video_Caption"
- Once you find this, you are in the right 'section' - you will need to change a few things coming after this:
-- Find the following key/value pairs and change them:
--- "link": 378 - change this to "link": null
--- "link": 383 - change this to "link": null
--- "link": [379] - change this to "link": null
Save the file and reload your workflow. Hopefully this will take care of your error.
Let me know if it works, or if it doesn't by commenting please :)
@shadowf yeah hi i have all the components and also tried to edit the JSON and still keep getting the error.. sorry
Я скачал готовую версию у олега -ComfyUI Portable - Настроенная сборка для nvidia 5000-1000 series. выполнил-что написал-и все заработало.я не спец -любитель-писарь-я ))
I downloaded - ComfyUI Portable - Customized assembly for nvidia 5000-1000 series. I did what I wrote there and everything worked. I'm not a specialist.
has anyone found the solution to this ? Its really frustrating and I am stuck at this for 3 days. Helppppppp !!!!
gocool6757131 I was getting a strange error with modules not appearing, but I got it working by installing this comfyUI. https://github.com/YanWenKun/ComfyUI-Windows-Portable?tab=readme-ov-file Try asking Grok; it helped me figure out what to do.
@gocool6757131 I discovered that if you search for that node in the node search on the left using the word Florence, it appears. So I just had to add it and connect it with other nodes and specifications that the one that appears in red already has because comfyui can't find it. It's strange because it says it can't find it, but it does have it ready to be put in.
Updating everything, reinstalling nodes. nothing gets me past Depth Anything V2 error.
Cannot set attribute 'src' directly. Use '_unsafe_update_src()' and manually clear .hash of all callersinstead.
Hi, I've been trying to use the workflow, but it asks for this node "Bjornulf_CombineTexts". I've tried installing it with ComfyUI Manager, but it gives me a permissions error. I've tried installing it by cloning it in the custom nodes folder using git clone https://github.com/justUmen/Bjornulf_custom_nodes, but that error also occurs. It clones it, but that node is still not installed.
Does anyone know how to make it work? Is it essential? I see some text in multiple languages. Will anything bad happen if I remove it?
Ah, okay, I finally got the node, I installed it correctly using try fix.
I'm getting this error, how do I install it? The subtitle is missing from Florence's workflow video.
sadly like most people here, had Florence issues for hours, tried everything ! Even working with Chat and Grok to find some kind of answer, ended up deleting one node that was stuck and fixed that but then the same old DownloadAndLoadFlorence2Model Error even if reloaded, deleted , refreshed. Its one of those that you will spend 4 or 5 days trying to fix.
Its old now, but just delete the Florence, its not required. Just easy prompt aid.
@MattHVisual thanks mate, yeah I figured out in the end :)