<h2 id="training-guide-for-illustrious-lora-ut6hk0bfz">Training Guide for Illustrious LoRA</h2>Documenting my own process to create captions and train LoRAs, so a post that I can link to exists when needed~<h3 id="dataset-captions-3bt2uuf4f">Dataset Captions</h3>"Explain my rather... unconventional, or even controversial, ethos on how to caption the dataset"The majority of anime models on CivitAI were simply trained on datasets that only got auto-captioned, without any manual pruning at all, which results in the Trigger Words being a giant wall of text. So now, to recreate the look of a character alone, you already have to waste the entire default token count...Instead, in my opinion, the caption should only consist of "variables" that do not define the subject. In essence, the model should learn to associate the trigger words with the subject. Take a character with blonde hair for example, the <code>blonde hair</code> tag should be removed from the caption, as that "trait" should be linked to the character's name instead. Therefore, I prefer to manually prune the captions before training.Some people would say this method reduces the flexibility of the model, as you can no longer control each individual detail precisely. However, this goes both ways. I have tried models where if you do not prompt for every single detail, the result ends up not resembling the character at all. So all in all, personally I'd rather use few tags to recreate a character.<h3 id="training-parameters-7rwqo6io7">Training Parameters</h3>"Explain some of the parameters that worked the best for me after hours of testing"<ul><li>mixed_precision: Use <code>bf16</code> if your GPU supports it for better results</li><li>dynamo_backend: Improves training speed by ~5% if your GPU supports it</li><li>optimizer: Use <code>Prodigy</code> for automatically managed <code>learning rate</code><ul><li>learning_rate: Set to <code>1.0</code></li></ul></li><li>network_alpha: Set to <code>0.5x</code> ~ <code>0.25x</code> of <code>network_dim</code><ul><li>d_coef: Use more than <code>1.0</code> due to the need of higher learning rate caused by <code>network_alpha</code> and <code>train_batch_size</code></li></ul></li><li>lr_scheduler: Use <code>cosine</code> to combat <code>Prodigy</code>'s tendency of keeping learning rate high</li><li>clip_skip: Use <code>2</code> for illustrious </li><li>debiased_estimation_loss: <code>enable</code> to improve the color accuracy</li></ul>

Illustrious LoRA Training Guide

The file is a <code>.json</code> config for Kohya_SS<ul><li>Simply load the config and adjust the paths to your model and dataset</li><li>Also adjust the repeat/epoch count, so that each unique concept (character) gets trained for ~1500 steps in total</li></ul>

Files with SHA256: e62653f4a5a9d6c46f78c2293b7a59c086a8d0a7d94730b82c9fed4de675be5f