This lora produces the likeness of the model/porn star Nancy Ace.
Wan - v1.0: Just realized that I only captioned a few of the images with the keyword. It still works really well, but retraining a v1.1 model with all images captioned.
Training
I trained this lora on a 4090 using diffusion-pipe. I followed this tutorial to set diffusion-pipe up on my machine. Trained with 20 images at 1024x1024 and 800x1024px images for 30 epochs, rank 32, 10 repeats, using fp8_e4m3fn quant (see config.toml, dataset.toml below). Character/likeness loras seem to be best trained with images, while activities seem to best be trained with short videos. Then in order to mix the character with the activity, use both a character lora and an activity lora.
The images I used are a mixture of full body/clothed, full body/not clothed, and close ups of her face. I annotated manually, keeping the annotations simple and focusing on the the things I didn't want baked into the lora (pose, clothing, some surroundings). Training took around 4 hours.
Example annotation: "nancya is squatting on stairs with her legs spread wide. She is wearing a blue long sleeve jumper and white tennis shoes."
I'm really impressed with how Hunyuan reproduces both body likeness and facial likeness! But facial likeness suffers a bit for full body shots. Not sure if this is a common problem with loras. I've seen some people suggest post processing with React to fix faces... my goal is to avoid doing that so I'll keep experimenting to make this better.
I'll also continue to experiment to determine how to generate a quality lora with the fewest images and (related) more quickly. My first pass at this included only non-clothed and close up face images. When using the resulting lora, I found that prompts that included clothing did not reproduce the character likeness. Continuing to experiment.
Generation
Videos can be generated using Kijai Hunyuan video nodes or built in comfy Hunyan nodes. I use Kijai because I feel it gives better results and more control. Download and drag videos to comfyui. Use Comfyui manager to install missing nodes.
Prompting: Use nancya as the keyword. Do not describe any features of her person/body (eg: don't use 'blonde straight hair', 'skinny', etc) since her likeness is already baked into the lora and using these terms often results in a different likeness being generated.
For example:
"nancya standing in front of a pool. She is wearing a red t-shirt and cut off jeans shorts."
Also, prompting for specific articles of clothing or body parts often helps frame the shot. If you want a full body shot, prompting for feet or shoes usually works. You can use 'full body shot' or 'close up shot', but I find that 'full body shot' doesn't always work.
Diffusion-pipe Hunyuan configuration files
config.toml
# Project paths
output_dir = '/mnt/d/Projects/hunyuan-training/nancya/output'
dataset = '/mnt/d/Projects/hunyuan-training/nancya/dataset.toml'
# Training settings
epochs = 40
micro_batch_size_per_gpu = 1
pipeline_stages = 1
gradient_accumulation_steps = 4
gradient_clipping = 1.0
warmup_steps = 100
# eval settings
eval_every_n_epochs = 5
eval_before_first_step = true
eval_micro_batch_size_per_gpu = 1
eval_gradient_accumulation_steps = 1
# misc settings
save_every_n_epochs = 5
checkpoint_every_n_epochs = 5
#checkpoint_every_n_minutes = 30
activation_checkpointing = true
partition_method = 'parameters'
save_dtype = 'bfloat16'
caching_batch_size = 1
steps_per_print = 1
video_clip_mode = 'single_middle'
[model]
type = 'hunyuan-video'
transformer_path = '/mnt/d/Projects/hunyuan-training/diffusion-pipe/models/hunyuan/hunyuan_video_720_cfgdistill_fp8_e4m3fn.safetensors'
#transformer_path = '/mnt/d/Projects/hunyuan-training/diffusion-pipe/models/hunyuan/hunyuan_video_720_cfgdistill_bf16.safetensors'
vae_path = '/mnt/d/Projects/hunyuan-training/diffusion-pipe/models/hunyuan/hunyuan_video_vae_bf16.safetensors'
llm_path = '/mnt/d/Projects/hunyuan-training/diffusion-pipe/models/llm/llava-llama-3-8b-text-encoder-tokenizer'
clip_path = '/mnt/d/Projects/hunyuan-training/diffusion-pipe/models/clip/clip-vit-large-patch14'
dtype = 'bfloat16'
transformer_dtype = 'float8'
timestep_sample_method = 'logit_normal'
[adapter]
type = 'lora'
rank = 32
dtype = 'bfloat16'
[optimizer]
type = 'adamw_optimi'
lr = 2e-5
betas = [0.9, 0.99]
weight_decay = 0.01
eps = 1e-8dataset.toml
# Resolution settings.
# Can adjust this to 1024 for image training, especially on 24gb cards.
resolutions = [[1024,1024],[800,1024]]
#Aspect ratio bucketing settings
enable_ar_bucket = true
min_ar = 0.5
max_ar = 2.0
num_ar_buckets = 7
# Frame buckets (1 is for images)
frame_buckets = [1]
[[directory]]
# Set this to where your dataset is
path = '/mnt/d/Projects/hunyuan-training/nancya/1024px/'
# Reduce as necessary
num_repeats = 10
Diffusion-pipe Wan configuration files
config.wan.toml
# Dataset config file.
output_dir = '/mnt/d/Projects/video-training/nancya/output'
dataset = '/mnt/d/Projects/video-training/nancya/dataset.toml'
# Training settings
epochs = 100
micro_batch_size_per_gpu = 1
pipeline_stages = 1
gradient_accumulation_steps = 4
gradient_clipping = 1.0
warmup_steps = 100
# eval settings
eval_every_n_epochs = 5
eval_before_first_step = true
eval_micro_batch_size_per_gpu = 1
eval_gradient_accumulation_steps = 1
# misc settings
save_every_n_epochs = 10
checkpoint_every_n_epochs = 10
#checkpoint_every_n_minutes = 30
activation_checkpointing = true
partition_method = 'parameters'
save_dtype = 'bfloat16'
caching_batch_size = 1
steps_per_print = 1
video_clip_mode = 'single_middle'
blocks_to_swap = 20
[model]
type = 'wan'
# 1.3B
#ckpt_path = '/mnt/d/software_tools/diffusion-pipe/models/wan/Wan2.1-T2V-1.3B'
# 14B
ckpt_path = '/mnt/d/software_tools/diffusion-pipe/models/wan/Wan2.1-T2V-14B'
transformer_path = '/mnt/d/software_tools/diffusion-pipe/models/wan/Wan2_1-T2V-14B_fp8_e5m2.safetensors' #kijai
vae_path = '/mnt/d/software_tools/diffusion-pipe/models/wan/Wan_2_1_VAE_bf16.safetensors' #kijai
llm_path = '/mnt/d/software_tools/diffusion-pipe/models/wan/umt5-xxl-enc-bf16.safetensors'#kijai
dtype = 'bfloat16'
# You can use fp8 for the transformer when training LoRA.
#transformer_dtype = 'float8'
timestep_sample_method = 'logit_normal'
[adapter]
type = 'lora'
rank = 32
dtype = 'bfloat16'
[optimizer]
type = 'adamw_optimi'
lr = 5e-5
betas = [0.9, 0.99]
weight_decay = 0.01
eps = 1e-8
dataset.toml
# Resolution settings.
# Can adjust this to 1024 for image training, especially on 24gb cards.
resolutions = [1024]
#Aspect ratio bucketing settings
enable_ar_bucket = true
min_ar = 0.5
max_ar = 2.0
num_ar_buckets = 7
# Frame buckets (1 is for images)
frame_buckets = [1]
[[directory]]
# Set this to where your dataset is
path = '/mnt/d/Projects/video-training/nancya/1024px/'
# Reduce as necessary
num_repeats = 5
Description
Updated keyword captioning to make sure it's complete for each pic in dataset. Results seem much more consistent and less plastic.
FAQ
Comments (17)
Hi, nice results with the Wan lora! What is your opinion after training the same char for both HY and Wan, which do you prefer and what differences do you notice training? I've done some tests with Wan likeness and diffusion-pipe and it seems to take a little longer than HY but the results are good. Helps to train with high-res images 720p or higher.
Thanks! I have been super impressed with both hy and wan.
From a training perspective, does wan take longer to train given the same dataset? Maybe a little. Is it egregiously longer? I haven't measured that closely, but I don't think it is. I also think that wan can be trained on lower quality images and still get really good results. Again, I haven't explicitly tested this yet but anecdotally, I'm in the process of training a wan activity lora and have gotten remarkable results with 128x128, 24fps - 48 frame videos. It only took 2-3 hours to train.
The quality of your dataset makes a huge difference in the quality of the lora. The higher resolution images you use (to a limit, i imagine) the better the fidelity of your character or activity. I think that's true for both hy and wan.
I like the results that wan produces a lot more than hy, so I'm planning to focus primarily on wan.
Was just having a conversation with @ComfyTinker and they pointed out that we're not training the CLIP in wan/hunyuan so training with a new token like 'nancya' likely means that the lora becomes a 'concept' lora, where no matter what you prompt, you'll get nancy ace. Basically equivalent to no captioning at all.
My gpu is busy, but I want to experiment with this more to better understand the implications. Will try to get back here with a comment later today or early tomorrow.
@lowcaloriesyrup Thanks for the reply! Yeah I've not used trigger words for the last few of my loras, I came to the same conclusion. I usually caption by describing briefly the character and environment, maybe the camera movement and then a sentence or 2 for the movement.
I've been having great results training with low-res videos and high-res images at the same time, seems to be a good combination when you need the lora to learn detail and movement. Images don't use much VRAM so I use 1280px images.
@definitelynotadog I just verified that this lora is a 'concept' lora. That is, the weights will be applied regardless of the tokens used to in the prompt. For example, if my prompt is 'a plant', I get nancy ace in front of a plant. Ha!
I guess the implication is that it makes the lora harder to use with other loras or other concepts because the weights applied by a made-up-token lora will always be applied and will fight weights associated with other tokens.
I've tried simple manual captioning and using joycaption2 for batch captioning in the past. I would, for example, describe the subject's clothing and something simple about the environment. It didn't work well for that particular lora, so I switched to using made up tokens, which I'm learning was wrong.
It still seems correct to associate the character with a particular token so that captioning remains short/simple, but ensure that the token already exists in the CLIP. For this lora, maybe: 'A woman' or 'a model'.
Thoughts?
@lowcaloriesyrup Yes of course you mostly want to caption things you want people to have control over, so location, clothes/make-up/hairstyle, objects, that kind of thing. Because if there was a plant pot in every dataset image you used and you didn't caption it, then every time you used the lora it's likely a plant pot would show up along with nancy ace even without prompting it.
IMO trying to caption nancy ace with any word is redundant because if you're using the lora you expect her to show in your generated video, so I would think just "a woman/person" would be fine.
@definitelynotadog true, although I wonder about multi-person scenes. Maybe we can get more targeted results if the loras we create a more targeted? Need to play around with this more.
Regardless, every bit of additional understanding about how to refine training is valuable!
I took a look at your config files for the Nancy Ace training and noticed a few things you could potentially tweak for maybe better performance, especially if you're on an RTX 4090 24GB.
I saw you're running with
gradient_accumulation_steps = 4 and a pretty high
blocks_to_swap = 20
to get that 1024px training stable. I was thinking you might need settings like that partly because of the optimizer you listed (adamw_optimi). I've found the AdamW8bitKahan optimizer (you set it with type = 'AdamW8bitKahan') is amazing for saving VRAM with these huge models on the 4090 – it's definitely the best one we've tested for this. It uses way less memory than standard AdamW.
Also, make sure you're actually getting the benefit of 8-bit precision where possible!
Double-check that the line transformer_dtype = 'float8' is uncommented and active in your [model] section – that's crucial to load the DiT weights in FP8 and save massive amounts of VRAM. Using an FP8 version of the T5 LLM file helps a bit too.
And the last tip: for activation_checkpointing, setting it specifically to 'unsloth' seems much more effective at reducing memory peaks during training than just setting it to true.
With those changes combined (especially activating FP8 DiT properly and using the 8-bit Kahan optimizer), you might find you can dramatically lower your blocks_to_swap (maybe even down to 5 or 10?) and potentially use gradient_accumulation_steps = 2, which really speeds up the training steps.
The main benefit is that you can likely reach the same LoRA quality but significantly speed up the whole training process.
Just sharing what worked well on my end! Hope it helps!
Cheers,
Thanks for sharing! This kind of info is invaluable! I'll incorporate your inputs into my next character training and share results with the model when I publish.
I had played with AdamW8bitKahan when I saw it become available in diffusion-pipe. At the time I was experimenting with other training parameters and I decided to go back to adamw_optimi to reduce the set of variables I had changed. Forgot to go back and experiment later, although I did accidentally leave it on AdamW8bitKahan for my latest Nicole Eggert model training. Given your input and the results I got there, I'll definitely stay with AdamW8bitKahan going forward.
@CyberAImania have you run both settings and compared quality, VRAM usage, and training time, so we can see the difference? Interesting info, but curious about the actual numbers and results.
@civitai7_ Hey! Yeah, good question. I have run tests comparing different settings, mainly looking at training T2V LoRAs at 1024px vs 512px on my RTX 4090 24GB using diffusion-pipe.
Honestly, in my personal opinion (IMO), pushing for 1024px using settings that require gradient_accumulation_steps = 2 (or 4) and blocks_to_swap = 5-10 (or more) just doesn't feel worth it for the time investment.
To get enough training steps (like 8k-10k+) for potentially good quality at 1024px with those stabler (but slower) settings, the total training time easily ballooned to around 24 hours for me [calculated estimate]. And while the 1024px can theoretically capture finer details, the final quality difference in the usable LoRA often wasn't big enough to justify that huge time sink, you know?
On the other hand, training at 512px using the much faster settings (gradient_accumulation_steps = 1, blocks_to_swap = 0, with maybe 7-10 epochs (depending on repeats) usually finishes in just ~4 hours [calculation]! And the quality at 512px? It's generally really good!.
Most of the LoRAs I've released here were actually trained that way at 512px.
https://civitai.com/user/CyberAImania/models
It just seems to hit a much better sweet spot between final quality and reasonable training time on this hardware.
So yeah, that's just my two cents based on my runs! For me, 512px with faster settings currently wins for T2V appearance LoRAs.
Cheers!
I made a simple configurator for TOML files. This is an ALPHA version — it only suggests settings, so you need to be cautious and keep that in mind. It's mainly intended for GPUs with 24GB of VRAM.
https://drive.google.com/file/d/1Kr-zRImaRJ0by6wO2sQiGnxBWl2jCM_S/view?usp=sharing
If diffusion pipe is already set up, running Hunyuan Video loras.. what needs to be changed to start training WAN loras? You can use the same setup (obviously making sure WAN models are in the correct places) and then just use a different WAN configuration file? Is it basically all the same as Hunyuan Video and you can easily switch between training for either of them, or what else is different?
Yep! You can easily switch between them to train for both, using the same dataset, just by using separate config.toml files.
Trying to think what/if I changed anything significant between the config files, other than the models. I can't recall anything I did intentionally different going from hunyuan to wan.
@lowcaloriesyrup Nice and easy. Sounds good. Thanks. I'll be trying WAN loras in the next few days most likely.
Just looking at some of the settings, I went back and am now finally running Hunyuan test lora with video clips. I was previously trying too high resolution (same as pics), that's why I could never get it to work. I went with 232 res, based on that divisible by 464p clips I'm doing my first test on and it's running.
@lowcaloriesyrup I noticed "blocks_to_swap" is a new setting for WAN in the config that's not in Hunyuan. I've seen example configs with 15 and 20 for this value so far. Will need some understanding and testing to figure out what's best here, if there's any quality or speed difference.
@lowcaloriesyrup ckpt_path also new.. seems a bit confusing what that's supposed to be, since it's to a folder in your config and not a file. The file path to the checkpoint is then given in the next line 'transformer_path'
learning rate I see is 5e-5 instead of Hunyuan I had 2e-5, just to note another difference, but that's no probablem, just training speed. But the ckpt_path mentioned above, I don't understand.
Details
Files
Available On (1 platform)
Same model published on other platforms. May have additional downloads or version variants.