Have you ever prompted a character to wear something but it just never turns out exactly as you like? Maybe a bikini that doesn't have the fancy strap style you want. Or a t-shirt that is too long in the waist or too high/low cut at the neck. You try to help by drawing a crude shirt shape, only for the generation to warp the shape. And it looks like it is a completely different style than the underlying image. 😡

This LoRA attempts to fix that by only changing EXACTLY what is colored red, blue, or green into the clothes you describe in the style of the image.

NOTE: This model prioritizes accuracy over aesthetics, so further refining will likely be required afterwards.

Prompting:

[type of clothing] : [color] - For example, draw pure red on an image and prompt bikini: red and the drawn red will be replaced by bikini style clothing. Multiple prompts can be used in combination, like bikini: red, jacket:blue though I've noticed lower quality.

colors:red, blue, green were the only ones used and tested, with red being the most commonly used in training. The colors to not need to be exactly perfect red/blue/green.

quality tags like masterpiece do appear to improve the result.

Usage:

1. Install a Cosmos reference node (https://github.com/Anzhc/Anzhc-ComfyUI-Cosmos-Reference)

2. Use the sample workflow

   1. I highly recommend using Krita or an image editor to get full use. I've included a Krita workflow using the AI image Diffusion Plugin.

3. Make further adjustments afterwards because it prioritizes accuracy over aesthetics.

And check out other cosmos reference based LoRAs, like Anima Edit: (https://civarchive.com/models/2650553/anima-edit?modelVersionId=3089149)

If using Forge Neo, you can use this lora in img2img if you enable reference mode for anima, found in settings > Stable Diffusion.

Misc:

After using my model a bit, I realize that it sucks for describing any type of fine detail or baggy clothing. it still works quite well and making general silhouettes for clothes which was my original goal, but now I find it lacking.

It also doesn't produce the best quality for clothes. Its possible that is my limited training data, though i wonder if using synthetic data where the base image is more grokable by the learning algorithm could also help improve quality.

I think I'll make an updated version of this, but instead use scribble or outlines since that could still allow outlines while also allowing for more detailed clothing descriptiveness.

How This Was Trained:

This used the conditioning training script (https://github.com/Mirumo0u0/sd-scripts/blob/main/docs/anima_conditioning_training.md) run on Linux, because it threw errors when i tried on Windows. I really just used the SAM3 model to identify and mask clothing, then made sure to fill in the clothing with appropriate color. In theory this could also be used with a canny filter to keep only the cloth's edges but colored red/blue/green and get an effect similar to the controlnet canny model, except usable on an already drawn image.