Yes this is most likely due to the small dataset where only 3 photos are busts, and one is an example of a body without head that the lorra would corect the body

Yes, of course.

It was trained on basic PonyXL, and the customizations were one of the old standard presets in Kohya for SDXL (I can't find them in the latest version).

In short, it's adafactor Scheduler and Optimizer.

Text Encoder and Unet learning rate 0.

Network Rank (Dimension) 128\128 alpha

According to my observations it is much easier to get even with fewer steps with dynamic learning rate than with cosine or linear.

And the most important thing is the dataset. For my personal needs I usually have enough even 1 image in the dataset for 100 steps. The best way to style it is to make a crop on the bust of a character with minimum clues. And it is better that the original image was at least 1.5 times the resolution per one than 1024. So during training much better retains the necessary details.

@LUUNA I honestly haven't delved into this topic. I had a primary task in fast learning with often very limited dataset size, as most of the time I had to manually edit the source from various tattoos, random details, etc. And since my primary goal was to make it adaptive and not destructive to the original style of the control point, these settings for me consistently produce results in fewer iterations.

@LUUNA Anyway since I'm not training any particular concept I think it's best not to train extra lore and let it interfere with too many things when generating. In general, this is just my speculation as I am far from being a neural network training specialist. Mostly I need them for my personal needs as an inpaint model

@LUUNA I didn't even know about the existence of any optimizing scripts for kohya, so I always trained with the original settings that are provided as standard when installing kohya.