This Controlnet model accepts DensePose annotation as input
How to use
Put the .safetensors file under ../stable diffusion/models/ControlNet/
About DensePose
This repo explains WHAT is DensePose and HOW to use it: https://github.com/facebookresearch/detectron2/tree/main/projects/DensePose
To use the annotator, you need to install Detectron2 as instructed. The DensePose annotator relies on apply_net.py to run. Please refer to the last image in the post for necessary modification of this script.
Transforming this project into a preprocessor turns out to be too challenging for me.
Model Feature
Strength
Stable body pose
Good performance on inferring hands
Ability to infer tricky poses
Great potential with Depth Controlnet
Weakness
Like Openpose, depth information relies heavily on inference and Depth Controlnet
Unstable direction of head
Intention to infer multiple person (or more precisely, heads)
Issues that you may encouter
Q: This model tends to infer multiple person
A: Avoid leaving too much empty space on your annotation. Or use it with depth Controlnet. Or write the prompt as what I did (eg. (one:1.2) girl).
Q: This model doesn't perform well
A:
The DensePose annotator doesn't work well. You can mannually modify the annotation. It should be easier than modifying an OpenPose
The depth Annotator doesn't work well. In fact, I recommend to use
glpn-nyuas the depth annotator. Please refer to https://huggingface.co/vinvino02/glpn-nyuThe pose is too tricky. It goes beyonds the model's ability.
Q: This model doesn't perform well with my LoRA
A:
That probably means your LoRA is not trained on enough data. It turns out that LoRA trained on enough amount of data will have fewer conflicts with Controlnet or your prompts
Change your LoRA IN block weights to 0. It's always the IN block that causes all the conflicts. Please refer to https://github.com/hako-mikan/sd-webui-lora-block-weight
*Note: The second pose generation in the post uses depth Controlnet (w=0.25)
Description
FAQ
Comments (29)
This seems cool. But why CKPT not Safetensors? I'll pass until Safetensors.
I'll second that. If you can work out how to create a ControlNet model, you can work out how to convert from .ckpt to .safetensors
This could be a gamechanger, amazing for animations. But I must agree with the previous post, CKPT is a nono for me.
Why are people upvoting the reference photos?
我认为这个东西非常的重要能否详细讲一下如何使用预处理器
good!good!
isnt it too big?
Hi, Where can i find the .yaml file for DensePose? I have downloaded and put it inside controlnet Model but it wont work!
2023-08-07 20:16:34,909 - ControlNet - INFO - Loading model: DensePose [32b069f5]
2023-08-07 20:16:39,476 - ControlNet - INFO - Loaded state_dict from [D:\A\extensions\controlnet\models\DensePose.safetensors]
ERROR: ControlNet cannot find model config [D:\A\extensions\controlnet\models\DensePose.yaml]
ERROR: ControlNet will use a WRONG config [D:\A\extensions\controlnet\models\cldm_v15.yaml] to load your model.
ERROR: The WRONG config may not match your model. The generated results can be bad.
ERROR: You are using a ControlNet model [DensePose] without correct YAML config file.
ERROR: The performance of this model may be worse than your expectation.
ERROR: If this model cannot get good results, the reason is that you do not have a YAML file for the model.
Solution: Please download YAML file, or ask your model provider to provide [D:\A\extensions\controlnet\models\DensePose.yaml] for you to download.
Hint: You can take a look at [D:\A\extensions\controlnet\models] to find many existing YAML files.
This model is supposed to run without .yaml
Just ignore the errors. It seems that you didn't put it under the correct folder. Don't put it under extensions/controlnet. Please double check my instruction
hey, hoping it will work on auto1111, I have paste your safetensors file in auto1111/stable-diffusion-webui/models/ControlNet. And doing so when running auto1111 it shows the model name in models sections of the control net tab. Its not showing good results and since its not showing densepose of the input image when using allow preview. I am not sure whether it is working correctly. Correct me if i am doing something wrong
Sorry for the confusion. I have edited my post to highlight the fact that this model doesn't contain a preprocessor. That's why you should follow Detectron2 repo for DensePose generation.
thanks for the reply, it seems you are using densepose segmentation data or label(I) data
are you using Segmant anything?
Please provide detailed description of input image for DensePose ControlNet
perform well in sd1.5. I write my own densepose preprocessor node for comfyui. I wonder how much dataset is needed
You came up with a workflow for preprocessing in comfy? Would you be willing to share your process? I don't really know comfy, but I bet it would give me the details to figure out creating a preprocessor for automatic1111.
请问在哪可以使用您写的densepose预处理器节点，comfy cn aux插件的节点无法读取这个模型，谢谢
an explanation of how to use would help, what directory do we git pull the detectron2 stuff to?
Without the preprocessor it is almost useless unfortunately. But the idea is great, OpenPose is crap except simple frontal full poses with spreaded limbs...
All of the reference annotations you have are segmentation visualizations. apply_net seems to support a lot more visualization modes, such as UV contours. Does this model support those too? Or does it only support segment annotations?
For what it's worth, this seems to be on it's way into the ControlNet plugin with its own preprocessor.
https://github.com/Mikubill/sd-webui-controlnet/pull/2314
Are there SDXL implementations as well?
i too would love a sdxl densepose cn
@dipstik I trained one myself now: https://huggingface.co/jschoormans/controlnet-densepose-sdxl
@jasper_schoormans396 this is awesome! now i need a preprocessor that allows legs over the head. it keeps changing the leg color to arms when they lift them overt the head
@jasper_schoormans396 , awesome work
@jasper_schoormans396 Winner Winner - Chicken dinner!!!!
For Automatic1111 users the installation method above is deprecated. It is integrated into Controlnet and usable right from the bat.
thanks