This Controlnet model accepts DensePose annotation as input<h2 id="heading-8014">How to use</h2>Put the <code>.safetensors</code> file under <code>../stable diffusion/models/ControlNet/</code><h2 id="heading-8015">About DensePose</h2>This repo explains WHAT is DensePose and HOW to use it: <a target="_blank" rel="ugc" href="https://github.com/facebookresearch/detectron2/tree/main/projects/DensePose">https://github.com/facebookresearch/detectron2/tree/main/projects/DensePose</a>To use the annotator, you need to install <code>Detectron2</code> as instructed. The DensePose annotator relies on <code>apply_net.py</code> to run. Please refer to the last image in the post for necessary modification of this script.Transforming this project into a preprocessor turns out to be too challenging for me.<h2 id="heading-8016">Model Feature</h2><h3 id="heading-8017">Strength</h3><ol><li>Stable body pose</li><li>Good performance on inferring hands</li><li>Ability to infer tricky poses</li><li>Great potential with Depth Controlnet</li></ol><h3 id="heading-8018">Weakness</h3><ol><li>Like Openpose, depth information relies heavily on inference and Depth Controlnet</li><li>Unstable direction of head</li><li>Intention to infer multiple person (or more precisely, heads)</li></ol><h2 id="heading-8019">Issues that you may encouter</h2>Q: This model tends to infer multiple personA: Avoid leaving too much empty space on your annotation. Or use it with depth Controlnet. Or write the prompt as what I did (eg. (one:1.2) girl).Q: This model doesn't perform wellA:<ol><li>The DensePose annotator doesn't work well. You can mannually modify the annotation. It should be easier than modifying an OpenPose</li><li>The depth Annotator doesn't work well. In fact, I recommend to use <code>glpn-nyu</code> as the depth annotator. Please refer to <a target="_blank" rel="ugc" href="https://huggingface.co/vinvino02/glpn-nyu">https://huggingface.co/vinvino02/glpn-nyu</a></li><li>The pose is too tricky. It goes beyonds the model's ability.</li></ol>Q: This model doesn't perform well with my LoRAA:<ol><li>That probably means your LoRA is not trained on enough data. It turns out that LoRA trained on enough amount of data will have fewer conflicts with Controlnet or your prompts</li><li>Change your LoRA IN block weights to 0. It's always the IN block that causes all the conflicts. Please refer to <a target="_blank" rel="ugc" href="https://github.com/hako-mikan/sd-webui-lora-block-weight">https://github.com/hako-mikan/sd-webui-lora-block-weight</a></li></ol>*Note: The second pose generation in the post uses depth Controlnet (w=0.25)

Files with SHA256: e4d3eb5eb6f3cc3ac3bc4976327576ee2d91f87c1007a22a47e2c76cf82f3e8f