HunyuanDit One Image Lora Experiment

As of the writing day, hunyuanDit training tool is not very accessible. I write up my custom training script for this experiment.

code repo: https://github.com/lrzjason/T2ITrainer

train_hunyuan_lora_ui.py

Original image:

prompt: cotton doll, A plush toy character with pink hair, a yellow crown adorned with a red gem, and large blue eyes. She is dressed in a white dress with a blue pendant around her neck. Surrounding her are cotton balls, which appear soft and fluffy. The backdrop is a solid peach color, providing contrast to the character's pastel hues. The overall composition gives off a warm and cute vibe, emphasizing the character's innocence and royalty.

hyperparameter:

adamw constant 1e-4 constant repeats 100 epoch 20

The image overfitted at 1600 steps as following and doesn't follow prompt at this checkpoint

prompt: cotton doll, A plush toy character of a policy officer

The 400 steps lora learned the concept and it has pretty good generalization

prompt: cotton doll, A plush toy character of a policy officer

The trainer repo is still wip. So, if things don't work, you might make a issue and let me know. But I don't sure would it be fixed or not.