Training Guide for Illustrious LoRA

Documenting my own process to create captions and train LoRAs, so a post that I can link to exists when needed~

Dataset Captions

"Explain my rather... unconventional, or even controversial, ethos on how to caption the dataset"

The majority of anime models on CivitAI were simply trained on datasets that only got auto-captioned, without any manual pruning at all, which results in the Trigger Words being a giant wall of text. So now, to recreate the look of a character alone, you already have to waste the entire default token count...

Instead, in my opinion, the caption should only consist of "variables" that do not define the subject. In essence, the model should learn to associate the trigger words with the subject. Take a character with blonde hair for example, the blonde hair tag should be removed from the caption, as that "trait" should be linked to the character's name instead. Therefore, I prefer to manually prune the captions before training.

Some people would say this method reduces the flexibility of the model, as you can no longer control each individual detail precisely. However, this goes both ways. I have tried models where if you do not prompt for every single detail, the result ends up not resembling the character at all. So all in all, personally I'd rather use few tags to recreate a character.

Training Parameters

"Explain some of the parameters that worked the best for me after hours of testing"

• mixed_precision: Use bf16 if your GPU supports it for better results

• dynamo_backend: Improves training speed by ~5% if your GPU supports it

• optimizer: Use Prodigy for automatically managed learning rate

  • learning_rate: Set to 1.0

• network_alpha: Set to 0.5x ~ 0.25x of network_dim

  • d_coef: Use more than 1.0 due to the need of higher learning rate caused by network_alpha and train_batch_size

• lr_scheduler: Use cosine to combat Prodigy's tendency of keeping learning rate high

• clip_skip: Use 2 for illustrious

• debiased_estimation_loss: enable to improve the color accuracy