This Controlnet model accepts DensePose annotation as input
How to use
Put the .safetensors file under ../stable diffusion/models/ControlNet/
About DensePose
This repo explains WHAT is DensePose and HOW to use it: https://github.com/facebookresearch/detectron2/tree/main/projects/DensePose
To use the annotator, you need to install Detectron2 as instructed. The DensePose annotator relies on apply_net.py to run. Please refer to the last image in the post for necessary modification of this script.
Transforming this project into a preprocessor turns out to be too challenging for me.
Model Feature
Strength
Stable body pose
Good performance on inferring hands
Ability to infer tricky poses
Great potential with Depth Controlnet
Weakness
Like Openpose, depth information relies heavily on inference and Depth Controlnet
Unstable direction of head
Intention to infer multiple person (or more precisely, heads)
Issues that you may encouter
Q: This model tends to infer multiple person
A: Avoid leaving too much empty space on your annotation. Or use it with depth Controlnet. Or write the prompt as what I did (eg. (one:1.2) girl).
Q: This model doesn't perform well
A:
The DensePose annotator doesn't work well. You can mannually modify the annotation. It should be easier than modifying an OpenPose
The depth Annotator doesn't work well. In fact, I recommend to use
glpn-nyuas the depth annotator. Please refer to https://huggingface.co/vinvino02/glpn-nyuThe pose is too tricky. It goes beyonds the model's ability.
Q: This model doesn't perform well with my LoRA
A:
That probably means your LoRA is not trained on enough data. It turns out that LoRA trained on enough amount of data will have fewer conflicts with Controlnet or your prompts
Change your LoRA IN block weights to 0. It's always the IN block that causes all the conflicts. Please refer to https://github.com/hako-mikan/sd-webui-lora-block-weight
*Note: The second pose generation in the post uses depth Controlnet (w=0.25)