Original ControlNet Difference Models HuggingFace Repository
STOP! THESE MODELS ARE NOT FOR PROMPTING/IMAGE GENERATION AND ARE FOR USE WITH ANY STABLE DIFFUSION 1.5 BASED MODELS (which are most models unless stated to be SD 2.X) AS CONTROL NETWORKS. THE ONES SHARED AT https://civarchive.com/models/9251/controlnet-pre-trained-models ARE ControlNets FOR THE BASE SD 1.5 MODEL ITSELF, NOT DIFFERENCE ONES.
Control Nets allow you to guide the structure of an image based on existing ones and preprocessing them, or by using images that match the expected format for a given control net. More information regarding ControlNet and the different types can be found at https://github.com/lllyasviel/ControlNet.
All examples generated using preprocessors, with the AbyssOrangeMix2 - Hardcore model, with the prompt used being the same as the original input image (but with different seeds). All settings default except where mentioned on a specific control net type. Canvas width set to 512 x 768 to match the input image.
You only need to download the models you want to use, you do not need to download all of them.
These are the models required for the ControlNet extension, converted to Safetensor and "pruned" to extract the ControlNet neural network. I have tested them with AOM2, and they work. Download these models and place them in the \stable-diffusion-webui\extensions\sd-webui-controlnet\models directory.
Diagram was shared by Kohya and attempts to visually explain the difference between the original controlnet models, and the difference ones.
Description
FAQ
Comments (46)
Hey!!! does this mean I don't need to download those 5.71GB files??? please say yes (if the answer is yes 馃槀) also does it work with realistic models too?
@kleind
Correct, you do not need to use the larger files with the ControlNetwork extension, these 700MB pruned ones will work perfectly fine, perhaps better, as they will take up less disk, and system memory, and maybe even vram (as some people reported higher resolutions were possible with the pruned models, if not a coincidence).
Similar to the base 1.5 models, I'd think these could work with other models, BUT I can't speak to how accurate it will be. The reason I created and shared these was I noticed a distinct difference in the quality when using the base 1.5 ones with AOM2 vs testing with converted ones.
Edit: Going forward, everyone should use the diff versions of the control nets, they are compatible with any model. The ones available to download here have been changed to the diff versions.
You don't need the big files at all, these are the extracted standard 700 Meg files.
https://huggingface.co/webui/ControlNet-modules-safetensors/tree/main
https://huggingface.co/kohya-ss/ControlNet-diff-modules/tree/main
@kleind
To be clear, these 700MB ones I uploaded here are different from those shared by iamtherealian001, those are the base SD 1.5 ones, and your results if you use them with AOM2 will be different, and from my testing, not as good.
Edit: Going forward, everyone should use the diff versions of the control nets, they are compatible with any model. The ones available to download here have been changed to the diff versions.
@Havoc聽thank you for the efforts 馃槶
@iamtherealian001 Thanks for the links. I really appreciate!聽
Are these different from the other controlnet models which were uploaded earlier today?
@Polygon
Yes, as per the description, these are ones converted specifically with AOM2_hard. From my testing, the results are more accurate to AOM2_hard and higher quality.
Edit: Going forward, everyone should use the diff versions of the control nets, they are compatible with any model. The ones available to download here have been changed to the diff versions.
@Havoc聽Now I understand, thanks.
I'm confused in the description as it says "I plan to add versions for NSFW and SFW later"... what does this current one do then? NSFW or SFW?? lol
@Someone5544
AOM2 has 3 variants, hardcore, NSFW, and SFW, this one was converted using hardcore. So it can do NSFW of course, but isn't the same as the variant that is NSFW.
@Havoc聽Why's this needed, though? It seems controlnet can work with any model, SFW or not
@Polygon聽If you are using the difference models, this isn't needed, and I plan to update this page to reflect this. At the time I wasn't aware of the difference model variants. The ones shared by theally are not the difference models. The transfer control models I created for AOM2's results align with the difference models, so there is no longer a need for specific ones if using those.
Three files are duplicated. Please check it again.
control_aom2_hard_hed
control_aom2_hard_scribble
control_aom2_hard_openpose
@skkkim will check
@skkkim I think it was an issue with how Civitai truncates and combines the model page name with the version, I have adjusted the naming conventions to make sure all the types have their own proper names.
@Havoc聽thankful, gratefu~!
Can you add comparisons to the usual models, like what's different and how much it improves?
@chaosleges as of now, you can simply use the difference version of the control net models and they will work correctly with any model. Model specific control nets are no longer required.
Umm, the original ones from huggingface also works any custom model except these ones works way better the storage space is overkill. Not sure about this.
@davizca
Edit: Going forward, everyone should use the diff versions of the control nets, they are compatible with any model. The ones available to download here have been changed to the diff versions.
Yes, it was mentioned that SD 1.5 works with any custom model, but the results are not equivalent to using one that has been processed via the approach outlined in discussion. The extension may eventually "transfer control" as they call it, automatically, but right now that is not enabled by default due to some concerns around how that affects the results in some cases (it used to be on by default), as far as I know.
If these work better for AOM2 now, I believe it makes sense to use them if storage space isn't an issue until the extension can handle "transferring control" effectively and by default, as users having to know to turn it on case by case, figuring that out themselves, or in a buried menu won't be very effective.
@Havoc聽Sure, I think the extension at some point could fix that issues but tested yesterday with some custom models and worked right, probably with some models that's not the case. Thanks for the upload.
@davizca聽
Ideally a solution can be found with the extension, as I definitely agree, it won't be sustainable for every model to have up to eight additional control nets uploads, either as part of it's model page (which could lead to confusion), or a separate one, haha. It's a pain enough managing these, but I felt it worth it to make sure people have better introduction to ControlNets with a popular model. A sliver lining is having this may introduce the awareness of ControlNets to more people.
@Havoc聽No worries, Im already doing the tool_transfer_tool.py from some of my custom models so I can test better the difference of them. Thanks for this mate.
@davizca聽as an update, someone created what they call difference models which solve this entirely. I was unaware until this morning, and have since uploaded those here in place of the AOM2 specific ones. They should work with any custom models without transferring control.
@Havoc聽theally ones?
@davizca聽no, theirs are not the difference ones, they are ones specifically for the SD 1.5 base model itself. The ones you can download on this page now, or via the linked HF repo, are the difference ones. The difference ones results align with my AOM2 control transfer ones, the non-difference ones (which include the original unpruned, theally, and the pruned safetensors ones that aren't diff), still produce the different and lower quality results when used with AOM2 (and likely any other custom model too different from the base SD 1.5 model).
@Havoc聽oh.. I mean the second pack. She has two. Pretrained normal and pretrained difference. I thought you meant these last ones
This:
https://civitai.com/models/9868/controlnet-pre-trained-difference-models
@davizca聽Ah, yes, those should be the same as the ones here now and the ones from the HuggingFace repo. Though I will mention, I don't know why theally is applying restrictive permissions to the ones they uploaded, the originals fall under Apache licensing or Openrail and are not restrictive.
Can I only download the openpose model for the openpose, or I have to download them all?
Hello @taijioctopus ,
You only need to download the controlnet models you want to use.
Awesome, so i just put these in the controlnet models folder, and use them in img2img with the AOM2 model loaded, right?
Hey @IntiArt ,
Yep, they go into the control net extension's model folder, and you can use ControlNets with txt2img or img2img.
@Havoc聽So with txt2img, it generates a map from scratch based on your text prompt, right? Then uses that map and references the prompt again to create the details based on that map? Sounds like a weird process.
@IntiArt聽Actually, in text2img, you give it an image to use as the basis, either as is, or having it preprocessed to align with a certain controlnet type. You should see a canvas where it states to select or drag an image.
There's only 1 model here, right?
And I can combine this with any of the other control nets?
Hey @testuser123 ,
There are eight models here, one for each control net type. They are all the pre-trained difference models, pruned, fp16, and safetensors. The difference versions of the control nets seem to work well with any model, so you shouldn't need any other versions than these and the extension can use them without you having to do any manual conversions.
Do these perform the same function as the original controlnet models, or is there something unique about them?
@Polygon these are not the same, the original ones are ones specifically for the SD 1.5 base model, if you use them on other models they will "work" but there will be incoherence and distortions, e.x. faces. These are difference ones as they are the difference only, i.e. removing the parts that were specific to the base SD 1.5 model, meaning it relies on whatever main model you have loaded. There is no reason, that I am aware of, to use anything other than the difference versions of the control nets.
I added a visual explanation from Kohya as well, it's in the description and one of the preview images for control_sd15_normal
@Havoc聽Thank you!
@Havoc聽ControlNet V1.1 is out with the lineart models, will there be difference models for those as well?
鎴戣繕浠ヤ负鏄彧鐢熸垚娉曠嚎鍥惧憿馃槗
I'm assuming, from the blatant absence and non mention of that the following is not required...
but it would be assuring, for anyone who previously followed SD-CN's setup, to know that these don't require .yamls like a nun needs church.
unless i'm mistaken and you're supposed to copy+convert the originals to suite these versions, but I doubt it. :P
I guess i'll know for sure in a few mins ^.^
I'm sure I'm screwing something up here, but the normal map model only spits out abstract modern art for me. I got the depth model working fine though.
Why isnt this a thing for sdxl