ControlNeXt

🤖 Github | 📝 Project Page | 📚 Paper | 🗂️ Demo (SDXL)

ControlNeXt is our official implementation for controllable generation, supporting both images and videos while incorporating diverse forms of control information. In this project, we propose a new method that reduces trainable parameters by up to 90% compared with ControlNet, achieving faster convergence and outstanding efficiency. This method can be directly combined with other LoRA techniques to alter style and ensure more stable generation. Please refer to the examples for more details.

We provide an online demo of ControlNeXt-SDXL. Due to the high resource requirements of SVD, we are unable to offer it online.

🎥 Examples

🛠️ Usage

🚀 Quick Start

Here, we use ControlNeXt-SDXL AnimeCanny as an example.

First of all, you have to make sure that you have Python installed (since you play AI, this should be default for you).

Launch your terminal and run the following command to clone our Github repository:

git clone https://github.com/dvlab-research/ControlNeXt
cd ControlNeXt/ControlNeXt-SDXL

And install the required packages:

pip install -r requirements.txt

(Optional) Download the LoRA weight, such as Amiya (Arknights) Fresh Art Style. And put them under lora/ directory (or other directory you like).

Finally, run our example:

bash examples/anime_canny/run.sh

If you download models by yourself, you may revise the bash script:

# examples/anime_canny/run.sh
python run_controlnext.py --pretrained_model_name_or_path "neta-art/neta-xl-2.0" \
  --unet_model_name_or_path "Eugeoter/controlnext-sdxl-anime-canny" \ # Huggingface repo id or local file path to your ControlNeXt UNet model.
  --controlnet_model_name_or_path "Eugeoter/controlnext-sdxl-anime-canny" \ # Huggingface repo id or local file path to your ControlNeXt ControlNet model.
  --controlnet_scale 1.0 \ # controlnet scale factor used to adjust the strength of the control condition
  --vae_model_name_or_path "madebyollin/sdxl-vae-fp16-fix" \
  --validation_prompt "3d style, photorealistic style, 1girl, arknights, amiya (arknights), solo, white background, upper body, looking at viewer, blush, closed mouth, low ponytail, black jacket, hooded jacket, open jacket, hood down, blue neckwear" \
  --negative_prompt "worst quality, abstract, clumsy pose, deformed hand, fused fingers, extra digits, fewer digits, fewer fingers, extra fingers, extra arm, missing arm, extra leg, missing leg, signature, artist name, multi views, disfigured, ugly" \
  --validation_image "examples/anime_canny/condition_0.png" \ # input canny image
  --output_dir "examples/anime_canny" \
  --load_weight_increasement # load weight increasement

📝 More

For more usages and examples, please refer to our Github project page.