This is my own version of Ellie from The last of us (part 1).
What's special about this LORA is that every training images are generated in Blender with original game model with complete white background, which results in every generations being constant and stable in quality. Thus naming it, CLEAN LORA.
Images with most popular camera angles are prioritized in learning with minor camera angles still being learnt but with less repetition. Ensuring most generations resulting in good camera shots. 35 repetitions are portrait and upper body shots. 25 are full body shots. 15 are extreme angles and sitting, and character looking away.
3 bake versions are uploaded, AnyLora and Stable Diffusion v1.5 base. And then GhostMix v3 bake which is for using with GhostMix mainly.
Prompts:
TLOU_EW is an optional token that can be used in prompts but unnecessary. Use this to get your image closer to trained images.
MUST PLACE (bruise, dirty:1.3) in NEGATIVES to produce clean face without artifacts.
Recommended LORA strength = 0.5 ~ 0.6
Machine Learning Stats:
Dimension = 512 x 768
(25 images x 35 repeats = 1015 steps)
+ (17 images x 25 repeats = 425 steps)
+ (21 images x 15 repeats = 315 steps)
/3 batch process = 585 steps per epoch
585 steps x 22 epoch = 12870 total steps
Description
v1.0 - released June 6th.
FAQ
Comments (12)
While I respect the idea (something I've thought about myself), the issue I had always figured would pop up with this method is what happens when you don't change the lighting scheme.
It's perfectly fine to take a bunch of images of a 3d model from any angle you can think of, but if you don't change the lighting, all that work will amount to nothing. LoRAs are more clever than we give them credit for, but also dumber than you'd really think. When you do something like this without changing the lighting, as I had thought, it gets baked into the LoRA.
As is the case with yours. The biggest smoking gun is the shadow underneath the head that goes at a 45 degree angle down the neck. It's present in all of your example images, and present in your image showing the training data. Looking at the face as well, the shadows are the same in each. That being said, it is a nice experiment.
The best fix I can recommend, is to do a few different rounds of the same set, each with different lighting conditions (maybe an outdoor sunny, overcast, indoors, night). Changing the light direction as well for each of them.
I still think you'll run into issues of the lighting getting baked into the model, but the only other solution I can really think of is doing the different sets, and doing different sets of lighting directions for each set. So as an example: an outdoors sunny set with front lighting, another with back, and another with side, and then repeating with each lighting condition. Depending on how many source images you had in your first set though, it might balloon your dataset to a large amount. 25 images per set, 75 per lighting condition (front/back/side), totaling to 300 images if you just did sunny/overcast/indoors/night conditions.
Whether or not that's worth doing though is up to you, but it's just food for thought.
Thank you for a thoughtful comment. I had that thought too but this is my first experiment and I just went it with only two lighting conditions and see how it turns out. I found the lighting basic and not very annoying (alright for most renders), although it could cause some issues when in dark environment. Thank you for an advice, it is noted.
This seems promising, keeping a look out for updates!
Thank you! Sub on my profile for more future models (:
I'd really like to read a full article of how you managed this using blender step by step. I recently tried using a 3d model in blender for shots, but it came out awful, and haven't tried another lora since. Kinda demoralized me hard.
I didn't read any articles but chance is that, what you did isn't a lot different to what I did. I made plenty of Loras so far but image quality and resolution, and using correct settings are important factors it seems. 512 x 512 and 768 x 768 make staggering differences in quality. But I guess you know this already? If you have any questions I'd be happy to answer.
You need to make sure that each image is properly rendered, you can't just slap a model, view it in viewport in texture mode and just render that, make sure you either render with cycles (path tracing) or eevee (set up good materials so it looks realistic). To avoid same issues as OP make sure to capture face from all angles but change lighting and background every time, also add several shots of the whole body in different poses, each one also with slightly different lighting and backgrounds, if the rig has different clothing you can alter them too in order to train other outfits.
Also if you're doing a video game character it might be good to also add screenshots from the game or cutscenes, assuming the model looks the same or at least very similar to the 3D model you have.
@alicepelle Just generated another lora with various lighting angle. It works but with a downside and chances of scenes with dark or very bad lighting being produced with high frequency. Also talk about the issue of lighting direction not being able to control with prompting with few exception of prompts such as backlighting...
At this point im questioning is it worth going through all the effort to bake in images with various directional lightings considering it inflates the training images and collection time by staggering amount. Only two directional lighting baked into this Lora, and gotta admit that is too basic but honestly it is alright I thought considering outputs are all still images. (AI is clever enough to generate scenes and background with only two directional lightings that is integral - scene that "makes sense".
AND - AI still manages to synthetize directional lighting with correct prompting despite the training images.
The issue is again, there aren't much prompt tokens which controls lighting.
For some reason this lora is not showing up in automatic, even when properly downloaded.
Rename the lora file to something without "(" and ")"
lookin great. any chance of a sarah lora in the future?
Details
Files
Available On (1 platform)
Same model published on other platforms. May have additional downloads or version variants.