Got two of these Lora's in beta, one trained on 3 second video clips and one trained on images. I HIGHLY SUGGEST USING THE IMAGE TRAINED VERSION IF YOU ARE ONLY GOING TO USE ONE.
Trained on a H100 in RunPod using Diffusion Pipe UI. Cost me a pretty penny but the results seem to be worth it. Buzz tips are appreciated to help me offset my costs
Still working on training some other iterations.
Image Trained Version. Got some later epochs in the pipeline but still testing.
Seems to work best around 30 steps but you can get acceptable results for some prompts as low as 10.
All examples run through up-scaling a frame interpolation
Lightly trained in the following concepts
lighting: indoor lighting, professional lighting, natural lighting.
Background: bedroom, neon background
concepts: fox ears, cat ears, neko_pose, hand bra, underboob
Clothing: black choker, crop top, long sleeve crop top, tank top, bikini, french maid outfit, thigh high stocking, thigh high socks, mini skirts, plaid skirts
Lora Models in the works:
Yoga Pants / Sporty Girls
Petite / A Cup
Frieren: Beyond Journey's End
Frieren
Fern
Ubel
Amateur Porn
Tan Lines
Let me know if there are any other Loras you may be interested in by leaving a comment and I might just make it. It would be extra likely if you gather a quality training set to share (I'll take care of the captioning)
See my latest article about my video comparison tool I'm developing to test Loras.
https://civarchive.com/posts/11389777
Description
Trained on 29, 3 second videos. I prefer the image version but this one does decent with thinks like tik-tok dances and ahegao.
a little more half baked than the image version.
FAQ
Comments (13)
Awesome work!
Thanks! Just trained a new version at 66 Epoch training, 64 lora rank. ~ 300 steps.
Need to test it out some more tomorrow but should drop soon with a high quality than before.
I do have an idea for a lora I was sure I was gonna train, until I read "pretty penny" X) So maybe I could give the dataset to you once I compile it. How long does the training run actually take? How big the dataset should be? I also don't quite understand if images or videos are needed. I'm surprised the image based one got the motion down so well. What are those images by the way are they sequences of screenshots multiple from same video, or just random screenshots and photos?
So what a dataset should contain? How many images and videos? I understand that they need to be as diverse as possible in order to generalize better, I think some other loras out here are struggling with that.
Also after seeing these two loras I would theorise that training on videos and then fine-tuning on images at the end could yield the optimal results? Have you tried or considered it?
P.S. Personally I found instances for $0.15 per compute hour for 3090 (24GB) +8 CPU cores +16GB RAM. I just need to learn docker now.
The time really depends on how many images/videos in your training set, the number of steps / epochs and the amount of VRAM you have.
For example I made a new lora last night, took about 6-8 hours overnight on my 4090 to get 28 epochs (450 steps).
I have two versions of this lora posted. one trained on 30 images and one on 30 videos (3 second clips). In my opinion the image version is much better but that could be a factor that it takes a lot longer to train on videos and I didn't let it run as long on RunPod since it was getting pricey.
Hello, so for image training you choose like half headshots and the other part is half body photos of your character? How hard is it to train this? And what range of motions can you achieve with this once it's trained, how long video clips can you generate? Thanks
@ArconSeptim I would say I chose 2/4 full body, 1/4 half body and 1/4 close up shots in terms of shot training.
That hardest part of training is honestly just getting the damn environment set up and then experimenting with the training parameters to get the best results to fit in your VRAM.
I didn't caption motion on the images for the dataset so it should be able to achieve any range of motion Hunyuan can already do natively but I haven't experimented too much.
For length of video the general standard seems to be around 3 seconds maximum for the model. Maybe you could get more with additional VRAM but it could be a model limit. I'm not certain.
can you use custom lora with this?
I'm not sure if I understand the question. This is a custom Lora already.
If you are asking if you can use multiple Loras. I believe so, I've seen some workflows for it but have not tested myself.
@azeli In this case you have to, because of the expression. The best you can do here is to have a larger more diverse dataset. 30 IMO is really not enough, and everything should be as diverse as possible to not get stuck in the weights, but any currently unwanted weights can be shaken up by further training on a diversified and/or new data set while also giving the lora more of an idea about the core concept.
@azeli as @illinar said you need to include faces to replicate the eGirl style. But I also did minimal captioning for facial features as not to overfit to singular people. Mostly just describing makeup.
Illinar is also right that the next training set probably needs so more diversity.
How is the LORA, with diffusion pipe or other? formed, what are the necessary PC capacities?
All the info should be in the diffusion-pipe github: https://github.com/alisson-anjos/diffusion-pipe-ui
But I think the lowest VRAM you could get away with is 12-16GB but it will take a long time.
Wan 2.2 version please?
Details
Files
Available On (1 platform)
Same model published on other platforms. May have additional downloads or version variants.