ControlNetFlux (CNFlux) - A collection of Controlnet models for Flux.1 [dev]
This collection strives to create a convenient download location of all currently available Controlnet models for Flux. Please do read the version info for model specific instructions and further resources. If you are the author of one of these models and don't want it to appear here, please contact me to sort this out. Support at the moment is limited to ComfyUI and Forge as far as i know.
NEW: windsingai-openpose
A ComfyUI workflow for InstantX's union model can be found here. The necessary nodes here.
A huge thanks to all the authors, devs and contributors including but not limited to: Automatic1111, Black-Forest-Labs, comfyanonymous, dashtoon, eesahe, Illasviel, InstantX, kadirnar, lucataco, PromeAI, Shakker-Labs, windsingai, XLabs-AI
Description
Union (Alpha) model released by InstantX on huggingface
FAQ
Comments (13)
Running an additional 6gb controlnet on top of everything else does not seem to work with 16gb VRAM. Using the gguf 4.1 version and tried the T5xxl_fp8. Not happening unfortunately. I really hoped to get rid of the xlabs sampler :(
do the other ones work, though? what's the vram usage with them on your system?
@eurotaku Just stumbled upon your message by chance and did not get a notification apparently. Sorry for the delay.
Yes, the other controlnets all work. Just the union is a bit too much. The entire union model concept was based on being convenient for those with some VRAM to spare. And in case of SDXL it made sense and worked nicely for many. But with Flux, being oversized as it is, I think no one is looking to spend more vram for nothing 😄
About VRAM usage, not sure. Especially considering the major update recently and the daily attempts at fixing all the little messes here and there, it changes day to day. All while still having to use the Xlabs sampler, which does not leave the most professional impression either.
Don't work in forge sadly, hope it will be fixed
with the correct settings in forge canny it works.
Sampling method is DPM++ 2M
Denoising strength 0.7-0.8
Can you please specify the correct settings, or point to somewhere that has them?
Denoising? Wouldn't that be img2img? Is there a way to make it work with txt2img?
I thought flux controlnet was not supported in Forge yet ? I've just tried in img2img and it didn't work, and looking at the discussions in Forge github the flux controlnet update isn't finished yet. Are you sure ?
I have it working in Comfyui - however my machine isn't powerful enough to make it work efficiently (RTX4070ti, 64GB, i9, Win11Pro). ie it takes a long time (3-4 mins). Is there a way to make it work more efficiently by tweaking the settings to reduce the 'intensity' or 'query' depth? I'm obviously not a coder - but I hope you get the sentiment?
smaller images is about it sadly. i have the same issue with controlnets, ipadapters and inpainting.
but luckily they all are kind of just.. meh.. anyways so I don't usually need to use them.
Try to enable "use backup memory" in your "manage 3d" of your NVIDIA pannel settings and enable use maximum power in the same menu, i'm french so i didn't remember how it's exactly called. the backup memory does this if you for exemple oom your memory cause not enought VRAM on your gpu it will offload auto the remaining required memory innto your ram. the "use maximum power" turn your GPU on "optimal mode" as the windows option for your processor but here it's for GPU.
Details
Files
controlnetfluxCnflux_instantxUnion.safetensors
Mirrors
flux.1-dev-controlnet-union.safetensors
diffusion_pytorch_model.safetensors
InstantX_FLUX.1-dev-Controlnet-Union.safetensors
diffusion_pytorch_model.safetensors
flux_union_controlnet.safetensors
diffusion_pytorch_model.safetensors
Instant_flux-union.safetensors
Controlnet_diffusion_pytorch_model_FLUX.1.safetensors
FLUX.1-dev-ControlNet-Union-Pro.safetensors
Available On (1 platform)
Same model published on other platforms. May have additional downloads or version variants.