Lora Experimentation while adjusting the CFG
Hello reader,
I am a budding AI artist, named Render. Like many of you, I too have poured my heart and soul into crafting intricate prompts for the image generator I use. At times I have dreamed of vibrant images and fantastical reviews. Yet, the vast majority of time the results were often… a LOT different than I wanted!
Colors were muddy, details were blurry, and sometimes, the AI would wander off into bizarre, unintended tangents. It was a frustrating dance, a constant battle because of the limitations of what I knew how to do, as well as my fear of the unknown.
One evening, I stumbled upon discussions about LoRA and CFG adjustments. LoRA, or Low-Rank Adaptation, was described as a technique to fine-tune AI models for specific styles or characteristics without the need to retrain the entire model. CFG (Classifier-Free Guidance) Scale, controlled how closely the AI adhered to the prompt, balancing creativity with adherence to instructions.
I always saw many others here on this site using them, but did not really understand until recently why they would do so. Intrigued by what I learned I decided to experiment. I downloaded a few LoRA models and started tweaking the CFG scale. The initial attempts were very much hit-or-miss.
One thing that helped me was starting at the recommended settings as shown in the downloaded Lora as well as the suggested CFG that the Checkpoint recommended. It took quite a while but with each adjustment I made, the more I tried to learn from others images that they created and I admired, I noticed subtle improvements.
LoRA and CFG are not just technical terms; they are the keys to unlocking the true potential of AI art. Experimenting with these parameters will allow you to:
Improve Image Quality and Detail: LoRA helped the AI generate sharper images, with intricate details and enhanced clarity, especially valuable for high-resolution outputs.
Achieve Style Consistency: By helping you practice your prompts with subtle adjustment based on your chosen Lora.
Explore Artistic Freedom: By understanding and manipulating CFG, help you to find the sweet spot between adhering to the prompt and allowing the AI creative freedom.
Remember while AI provides the tools, it's the artist's willingness to experiment and explore its depths that truly unlocks its magic.
I wanted to share with you a few examples of what I mean.
I first found an image that I wanted to imitate and looked at what prompts and Loras they used. But I did not want to just make the same image they did, rather I wanted to be my own artist, to make something unique. So while borrowing their prompts, I decided to see what other Lora's and CFG settings would get me.
With that in mind I copied a prompt from an image I liked and tried it with some Lora's I already had installed.
Here is the first image I got.
To create this I used the following Lora's:
https://civitai.com/models/1143473/detail-art-enhancement-fluxv10
https://civitai.com/models/210689/fluffy-style-fluxsdxl
https://civitai.com/models/677725/cute-3d-cartoon-flux
I have seen many who say that CFG settings should when using FLUX should not exceed 1-3.5. Others say it should be higher. So I experimented with a CFG setting of 3.5. I did not work the way I wanted. I dropped it back down to 1. And that is where the above image came from.
Now, I also encourage you to experiment when also upscaling.
When I went to upscale the above image, I lowered the the CFG down to 0.4 and ran it through my FLUX upscale workflow expecting to see the same thing just with a bigger size. What I was not expecting was for an entirely new but similar image to appear, because with the upscaler not using Lora's I thought it would still be the same. Instead the lack of Lora's with a low setting just enhanced it a bit. I got this instead.
Notice how there is more detail to the garments, hair, and even face? That all came from simply playing with things. But at the same time, you also may get weird things in your images you were perhaps not expecting. Sometimes that is good, sometimes that is bad. Notice the weird face just to the right of the top of her head. It looks like an alien face or a pair of breasts carved into the rock.
Undaunted, I wanted to keep experimenting. What I was seeing was like rewatching Toy Story 1 immediately after watching Toy Story 4. big difference in the images.
So I switched things up. Still using the same seed, same prompt, I now decided to change the Loras I used. Here they are.
https://civitai.com/models/832858/anime-art
https://civitai.com/models/974790/greg-rutkowski
https://civitai.com/models/1059859/painterly-fantasy-by-chronoknight-flux-and-il
While I changed my Lora's to those three shown above, I still kept this one from the previous experiment.
https://civitai.com/models/1143473/detail-art-enhancement-fluxv10
What were the results? I am glad you asked. Nothing like what I had saw and admired from the person I had borrowed the prompt from, nor like what I had just created. The look and style were different, as they should be. Here is what was created.
I honestly did not expect this style either, but still thought it was interesting. So I decided to continue the experiment by doing the same thing as I had done for the previous images. I ran this one through a upscaler dropped the CFG down to 0.4 and got something that was more cartoonish in nature yet not horrible either, really depending on what you are looking for.
This was the result.
In some cases they bear a similarity to each other, in other cases they are nothing alike. But that is the joy of being your own artist and experimenting. The one thing I have learned and I want to share, do not be afraid to ask questions of others, to make mistakes, to even experiment. The end result, you will be a better Artist in the end.
While I never succeeded in getting anything close to what I saw another artist had done, I also never would have gotten what I did if I had not tried. Never be afraid to try. At times you will fail, at times you will succeed beyond your wildest dreams. But it will only come if you keep working at it.
I hope that this helps.