Kohaku-XL Zeta

DiT is not all you need

join us: https://discord.gg/tPBsKDyRR5

Highlights

• Resume from Kohaku-XL-Epsilon rev2

• More stable, long/detailed prompt is not a requirement now.

• Better fidelity on style and character, support more style.

  • CCIP metric surpass Sanae XL anime. have over 2200 character with CCIP score > 0.9 in 3700 character set.

• Trained on both danbooru tags and natural language, better ability on nl caption.

• Trained on combined dataset, not only danbooru

  • danbooru (7.6M images, last id 7832883, 2024/07/10)

  • pixiv (filtered from 2.6M special set, will release the url set)

  • pvc figure (around 30k images, internal source)

  • realbooru (around 90k images, for regularization)

  • 8.46M images in total

• Since the model is trained on both kind of caption, the ctx length limit is extended to 300.

Usage (PLEASE READ THIS SECTION)

Recommended Generation Settings

• resolution: 1024x1024 or similar pixel count

• cfg scale: 3.5~6.5

• sampler/scheduler:

  • Euler (A) / any scheduler

  • DPM++ series / exponential scheduler

  • for other sampler, I personally recommend exponential scheduler.

• step: 12~50

Prompt Gen

DTG series prompt gen can still be used on KXL zeta. A brand new prompt gen for cooperating both tag and nl caption is under developing.

Prompt Format

As same as Kohaku XL Epsilon or Delta, but you can replace "general tags" with "natural language caption". You can also put both together.

Special Tags

• Quality tags: masterpiece, best quality, great quality, good quality, normal quality, low quality, worst quality

• Rating tags: safe, sensitive, nsfw, explicit

• Date tags: newest, recent, mid, early, old

Rating tags

General: safe

Sensitive: sensitive

Questionable: nsfw

Explicit: nsfw, explicit

Dataset

For better ability on some certain concepts, I use full danbooru dataset instead of filterd one. Than use crawled Pixiv dataset (from 3~5 tag with popularity sort) as addon dataset. Since Pixiv's search system only allow 5000 page per tag so there is not much meaningful image, and some of them are duplicated with danbooru set(but since I want to reinforce these concept I directly ignore the duplication)

As same as kxl eps rev2, I add realbooru and pvc figure images for more flexibility on concept/style.

Training

• Hardware: Quad RTX 3090s

• Dataset

  • Num Images: 8,468,798

  • Resolution: 1024x1024

  • Min Bucket Resolution: 256

  • Max Bucket Resolution: 4096

  • Caption Tag Dropout: 0.2

  • Caption Group Dropout: 0.2 (for dropping tag/nl caption entirely)

• Training

  • Batch Size: 4

  • Grad Accumulation Step: 32

  • Equivalent Batch Size: 512

  • Total Epoch: 1

  • Total Steps: 16548

  • Training Time: 430 hours (wall time)

  • Mixed Precision: FP16

• Optimizer

  • Optimizer: Lion8bit

  • Learning Rate: 1e-5 for UNet / TE training disabled

  • LR Scheduler: Constant (with warmup)

  • Warmup Steps: 100

  • Weight Decay: 0.1

  • Betas: 0.9, 0.95

• Diffusion

  • Min SNR Gamma: 5

  • Debiased Estimation Loss: Enabled

  • IP Noise Gamma: 0.05

Why do you still use SDXL but not any Brand New DiT-Based Models?

Unless any one give me reasonable compute resources or any team release efficient enough DiT or I will not train any DiT-based anime base model.
But if you give me 8xH100 for an year, I can even train lot of DiT from scratch (If you want)

License

Fair-AI-public-1.0-sd