SDXL lora trained on SD1.5. Trained on 'headshot of person' where person is a wildcard.

update much smaller lora + cleaner dataset (2671 filtered images) + trained longer (8025 steps at batch size 16)

Larger images at https://ntcai.xyz/sdxl/headshot

Usage:

https://ntcai.xyz/comfyui/lorasimplesdxl.json comfyui

Quick note: Support us and receive an exclusive NFT airdrop as a token of our appreciation. patreon.com/NTCAI

This process is:

1. Use img2img on SDXL output using SD1.5. I wrote a tutorial on this here: https://civarchive.com/articles/1430/applying-sd-15-models-and-loras-to-sdxl-1024x1024-comfyui ( mirror )
   Create a large dataset with this technique. I ran this overnight on two A6000s.
   

   Hint: choose a subject matter that sd1.5 knows well and perhaps reject any distorted images.

2. Get your files in the correct form. This tutorial helped me:

3. Train SDXL using https://github.com/kohya-ss/sd-scripts on the generated images.

   # Full command
   CUDA_VISIBLE_DEVICES=0 accelerate launch --num_cpu_threads_per_process=2 "sdxl_train_network.py" --enable_bucket --pretrained_model_name_or_path="/ml2/trained/ComfyUI/models/checkpoints/stable-diffusion-xl-base-1.0/sd_xl_base_1.0.safetensors" --train_data_dir="/ml2/trained/sd-scripts/data/headshot" --resolution="1024,1024" --output_dir="/ml2/trained/ComfyUI/models/loras/Lora/sdxl" --logging_dir="./logs" --save_model_as=safetensors --output_name="headshot2" --network_alpha="1" --network_dim=32 --network_module=networks.lora --text_encoder_lr=0.0004 --unet_lr=0.0004 --lr_scheduler="constant" --train_batch_size="16" --max_train_steps="10000" --mixed_precision="bf16" --save_every_n_epochs="1" --save_precision="bf16" --caption_extension=".txt" --optimizer_type="Adafactor" --optimizer_args scale_parameter=False relative_step=False warmup_init=False --max_data_loader_n_workers="0" --bucket_reso_steps=64 --gradient_checkpointing --xformers --bucket_no_upscale

My hope is that this can help creators migrate their work. Happy training!

Resources used:

• eyes lora https://civarchive.com/models/72447/eyes-trained-with-no-data-trick

• SD1.5 aniverse https://civarchive.com/models/107842

• person wildcards https://civarchive.com/models/98622/clonecleaner-wildcards-increase-character-consistency