Flux-1.dev.fp8 + ControlNet Canny
At the time of writing this guide, it is possible to try out controlnet canny only by switching your ComfyUI from master to xlabs_flux_controlnet (git branch).
Let's jump into the prerequisites:
Prerequisites
Fetch and update all custom nodes
Update ComfyUI from the ComfyUI manager
Install Missing Custom Nodes from ComfyUI manager and Restart it
HOW DOES IT WORK
Requisites
download UNET
flux-dev-fp8https://huggingface.co/Comfy-Org/flux1-dev/blob/main/flux1-dev-fp8.safetensors, save intomodels/unet(only this one work, not xlabs.ai or others)download VAE
aemodel https://huggingface.co/black-forest-labs/FLUX.1-schnell/blob/main/ae.safetensors save it intomodels/vaedownload CLIPS:
t5xxl_fp16.safetensorsclip_l.safetensors
(optional if your machine has less than 32GB of TvT ram) t5xxl_fp8_e4m3fn.safetensors
at: https://huggingface.co/comfyanonymous/flux_text_encoders/tree/main
save them into ComfyUI/models/clip/
1 UNET + CLIP - Group
1 we load flux1-dev-fp8.safetensors, the only one working as of today (Comfy.org) https://huggingface.co/Comfy-Org/flux1-dev/blob/main/flux1-dev-fp8.safetensors and the dual clip loader for Flux-1.
2 select 1 for unet only, select 2 for unet + flux_realism_lora
3 we set some globals to avoid node splines cluttering via SetNode.
2 Base Image - Group
1 Load your base image
3 Prompts - Group
1 Image to text via Florence2
2 concatenate Florence caption Generic prompt via String Function and pass it to conditioning global PROMPT_POS-COND
4 CNET - Group (Controlnet)
select 1 for prompt only OR select 2 for prompt + canny-controlnet
A few Notes
Seed 688106326432651
Cnet 0.76 head facing the camera , still nice image
Cnet 0.77 pulls up her head, still nice image
Cnet 78 the cnet kick in violently and fuck up her arms
Lora:
Keep it low, it's very strong
A Big shoutout to u/comfyanoymus and the amazing Comfy.org project/community
Description
FAQ
Comments (27)
I didn't even know this was possible. Going to study this flow its interesting because it realy keeps things clean. Thank you.
Happy to see people are enjoying it. I am committed to bring you the newest tasty treats from Flux and generally ComfyUI
Thanks for the wf! But I'm experiencing the following issue, even after updating comfyiu after checking out to xlabs branch in ComfyUI folder:
Error occurred when executing ControlNetLoader: MMDiT.__init__() got an unexpected keyword argument 'image_model'
Hi Primaveri, could you please send a screenshot for more context?
Hi @KaceyTrax I can send you the error, tracking. Notice I'm already on xlabs_flux_controlnet branch and I've updated every custom node and Comfyui:
Error occurred when executing ControlNetLoader: 'NoneType' object has no attribute 'keys' File "X:\ComfyPortable\ComfyUI\execution.py", line 152, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "X:\ComfyPortable\ComfyUI\execution.py", line 82, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "X:\ComfyPortable\ComfyUI\execution.py", line 75, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "X:\ComfyPortable\ComfyUI\nodes.py", line 720, in load_controlnet controlnet = comfy.controlnet.load_controlnet(controlnet_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "X:\ComfyPortable\ComfyUI\comfy\controlnet.py", line 492, in load_controlnet return load_controlnet_mmdit(controlnet_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "X:\ComfyPortable\ComfyUI\comfy\controlnet.py", line 406, in load_controlnet_mmdit model_config, operations, load_device, unet_dtype, manual_cast_dtype = controlnet_config(new_sd) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "X:\ComfyPortable\ComfyUI\comfy\controlnet.py", line 379, in controlnet_config model_config = comfy.model_detection.model_config_from_unet(sd, "", True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "X:\ComfyPortable\ComfyUI\comfy\model_detection.py", line 284, in model_config_from_unet unet_config = detect_unet_config(state_dict, unet_key_prefix) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "X:\ComfyPortable\ComfyUI\comfy\model_detection.py", line 37, in detect_unet_config state_dict_keys = list(state_dict.keys()) ^^^^^^^^^^^^^^^
Thank you!
@Primaveri I had the same problem, with XLabs Flux_dev_fp8.
Try this model instead https://huggingface.co/Comfy-Org/flux1-dev/blob/main/flux1-dev-fp8.safetensors
@KaceyTrax Thank you, but I guess thats the one I'm already using. I'll download it again, huggingface models are kind giving some errors to me because they get stuck for a long period when I try to download them sometimes.
@Primaveri sorry to hear that, must be painful. I feel you :(
@KayKayTrax I've managed to make it work. Just had to download the files from xlabs branch manually :/. Thanks!
When i install all missed node for this workflow, my comfy UI was totally out of work!!
you need to restart ComfyUI. If after restarting you get an error send me a screenshot and i can help.
Hey there
what's the vram needed to apply canny?
I got allocation error
fp8 models should work with less than 24GB - 12GB is the minimum requirement, I have a GTX3090 tho
Reducing res from 1024x1024 to 512x512 the gpu oom go away, but had very bad image result, tested with 30 steps, also I have 3060 12GB
@astolfo_hue you need to try and error CFG and steps, there are not absolute values to set in
any image2image workflow. just by changing a few steps can output a magnificent image or a terrible one. Time and patience are key, happy tweaking!
Thank you for your hard work.
the git checkout xlabs_flux_controlnet origin xlabs_flux_controlnet command returns an error error: pathspec 'xlabs_flux_controlnet' did not match any file(s) known to git
error: pathspec 'origin' did not match any file(s) known to git
error: pathspec 'xlabs_flux_controlnet' did not match any file(s) known to git.
What is the reason for this?
Try:
1) git checkout xlabs_flux_controlnet
2) git pull
thanks for the help
\ComfyUI>git checkout xlabs_flux_controlnet
error: pathspec 'xlabs_flux_controlnet' did not match any file(s) known to git
:(
@onoffzvuk502 run git fetch first and then checkout the branch mentioned.
Let me know if that works
@KayKayTrax Yes, it worked.
but now
Error occurred when executing SamplerCustomAdvanced: Allocation on device
File "D:\ARS\ComfyUI FACE 2\ComfyUI\execution.py", line 152, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\ComfyUI\execution.py", line 82, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\ComfyUI\execution.py", line 75, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\ComfyUI\comfy_extras\nodes_custom_sampler.py", line 612, in sample samples = guider.sample(noise.generate_noise(latent), latent_image, sampler, sigmas, denoise_mask=noise_mask, callback=callback, disable_pbar=disable_pbar, seed=noise.seed) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\ComfyUI\comfy\samplers.py", line 716, in sample output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\ComfyUI\comfy\samplers.py", line 695, in inner_sample samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\ComfyUI\comfy\samplers.py", line 600, in sample samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\python_embeded\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\ComfyUI\comfy\k_diffusion\sampling.py", line 143, in sample_euler denoised = model(x, sigma_hat s_in, *extra_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\ComfyUI\comfy\samplers.py", line 299, in call out = self.inner_model(x, sigma, model_options=model_options, seed=seed) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\ComfyUI\comfy\samplers.py", line 682, in call return self.predict_noise(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\ComfyUI\comfy\samplers.py", line 685, in predict_noise return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\ComfyUI\comfy\samplers.py", line 279, in sampling_function out = calc_cond_batch(model, conds, x, timestep, model_options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\ComfyUI\comfy\samplers.py", line 228, in calc_cond_batch output = model.apply_model(input_x, timestep_, c).chunk(batch_chunks) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\ComfyUI\comfy\model_base.py", line 124, in apply_model model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1532, in wrappedcall_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1541, in callimpl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\ComfyUI\comfy\ldm\flux\model.py", line 150, in forward out = self.forward_orig(img, img_ids, context, txt_ids, timestep, y, guidance, control) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\ComfyUI\comfy\ldm\flux\model.py", line 129, in forward_orig img = block(img, vec=vec, pe=pe) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1532, in wrappedcall_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1541, in callimpl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ARS\ComfyUI FACE 2\ComfyUI\comfy\ldm\flux\layers.py", line 233, in forward output = self.linear2(torch.cat((attn, self.mlp_act(mlp)), 2)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Guys, thanks for the help. I have solved the problem
@onoffzvuk502 nice! well done! Happy tweaking! :D
@KayKayTrax great workflow!
When loading the graph, the following node types were not found:
SetNode
GetNode
Florence2ModelLoader
Florence2Run
TextInput_
ShowText|pysssss
ReActorFaceSwap
ReActorFaceBoost
SDXLEmptyLatentSizePicker+
ReActorRestoreFace
Fast Groups Bypasser (rgthree)
CannyEdgePreprocessor
ImpactSwitch
StringFunction|pysssss
ttN hiresfixScale
Nodes that have failed to load will show as red on the graph.
meaning you need to install the missing custom_nodes.
Please navigate to the manager. install missing custom nodes and restart.
Keep getting this error:
ComfyUI Error Report
## Error Details
- Node ID: 23
- Node Type: SamplerCustomAdvanced
- Exception Type: ValueError
- Exception Message: This Controlnet needs a VAE but none was provided, please use a ControlNetApply node with a VAE input and connect it.
it happens because months ago they changed the node structure. As the error suggests, you need to search fro ControlNetApply node, connect your VAE to it, positive and negative conditioning, the reference image and the controlnet loaded