V2: Massively overhauled the dataset, adding around 10k unique pairs of various types of edit pairs. Also included a few more experimental datasets like line art reference and restyling. Nearly every image is captioned with natural language now, making the model much easier to use overall. Trained with mixed resolutions of 512, 768, 1024, mitigating the issues experienced with the beta version when trying to edit with higher res images.
Beta: This first version of V2 was exclusively trained at a resolution of 512 and even after a small amount of testing I could tell there are some heavy limitations at higher resolutions, like detail preservation and unwarranted edits. I will probably continue the training run with mixed resolutions and hope that those issues can be mitigated.
Usage: First you will need to install the ComfyUI-Cosmos-Reference custom node and then use the sample workflow. For prompting use mostly natural language and supplement with tags if something doesn't work.
If you have any suggestions for reference control or edit tasks, just leave them in the comments or dm me. As always, data is the main factor to making a better model, so if anyone has any high quality datasets they would like to share, I'd happily include them in training.
V1: First of all, I got to mention the original LoRA that this was based on, here. Similarly to that model, it aims to bring editing concepts natively into Anima via difference tagging. Currently the majority of the data is for removing clothes, but I also reversed the data so the model also understands how to put clothes on as well. The train was done with just over 5k unique pairs of data meaning over 10k pairs after reversed.
Description
FAQ
Comments (31)
changing style works nicely
V2 will be incredibly helpful for creating image pairs for training loras! Take an image with the concept you want to train, use this to remove the concept, and include that image in the training set as well for a clean differential.
This lora model is incredibly well-done—by far the best one available. Here are some ideas I have for future versions: - Subtle changes to the pose, such as arms outstretched, legs crossed, standing, sitting, etc. - Recoloring in either anime or monochrome manga styles. - Changing the character while keeping everything else the same. - Adding concepts from Loras that the model doesn’t support. I've also noticed that the model doesn't recognize many types of clothing, such as bodysuits, many styles of bikinis, and some cosplay outfits that v1 did recognize. I know I’m asking for a lot, and I’m sorry for that—these are just ideas I have. Thank you so much for your work.
great work, i do wish this was actually implemented in comfy itself (https://github.com/Comfy-Org/ComfyUI/pull/13392) but doesn't seem like there's any progress :P
Is this for img2img?
yep, it works! But I failed with other than base 1.0 versions of the model, is it expected behavior?
Pretty much, the model was trained off of base and the task is completely new, so I wouldn't expect it transfer very well to other finetunes.
@mattehe some little number of models that I ususally don't like due to low quality in comparison to other realistic models anima based I use- they manage basic tasks. Even basic model do things with real world photo tasks. Just to mention.
tried it with a bun ch of finetunes, it worked with most, just one didn't do anything.
Is there a way to save only the edited image instead of the before-and-after side-by-side? Also, is there support for realistic models, or is that already supported? Thanks! Oh and can I stack another LoRA (like a detail slider) on top of this?
Instead of using an image stitching node, directly append an image save node after the VAE decoder.
Edit: Ok it just seems very hit and miss, prompts from example images work perfectly, but trying more custom instructions leaves the image identical.
It seems that the Apply Cosmos Reference changed name of the input latent... I was wondering why the sample workflow did nothing. Just had to move the latent to the new input.
Actually maybe not it... it just seems to change nothing with prompts that are too simple like "change the background to be a forest"
Tip:
Text prompts remove <object1> and keep <object2> have an effect. Use them for fine-grained edits. Example
You can avoid subtle changes across unprompted areas of the image by using masks.
Use: Mask -> Set Latent Noise -> Apply Cosmos Reference & KSampler.
Thanks a lot for sharing! It's really helpful. But I'm wondering, how can we make the mask more intelligent in the next step? Right now, we can only draw it manually.
@q148009946 I've had success using the SAM 3.1 model and clip + SAM3 Detect node.
Depends on what you're trying to mask.
Thank you for your work!
Building on your workflow, I tried integrating ControlNet into the image generation pipeline. However, I found that Anima-LLLite and Cosmos Reference do not seem to be compatible out of the box, because both Cosmos Reference and Anima-LLLite modify the MODEL through model_function_wrapper.
Specifically, Cosmos Reference appends the reference latent to the input token sequence, which causes the number of control embedding tokens expected by LLLite to no longer match the model's input token sequence length.
With the help of AI, I created a new node to address this issue. By setting preserve_wrapper=True, both wrappers remain active. Then, a compatibility patch pads LLLite's cond_emb from the original token length to the expanded token length introduced by Cosmos Reference.
With this approach, it's possible to generate images using the ControlNet and prompt from image B while preserving the reference information from image A.
I've been using this new node for character and art style replacement tasks on existing image variant sets. Once the first image is established as the anchor, I only need to provide the ControlNet and prompt for each subsequent image to perform image-to-image generation and create a new variant set. In my experience, this produces significantly better results than applying image-to-image generation to the entire existing variant set directly.
I hope you can shared the node pls :)
11/10 can add cat ears
It works with turbo lora too. Use latent noise mask to focus on specific changes.
Added the ability to use edit loras to my inpaint workflow. It pastes the inpainted area to original image, which helps with the image not degrading with vae encode/decode for surrounding areas that are not edited.
https://civitai.red/models/128556/comfyui-inpaint-workflow-or-updated-for-anima
Forge Neo now supports cosmos reference so you can use this lora in the img2img tab. (Make sure to enable it in Settings->Stable Diffusion) This is an example of me adding cat ears and changing hair color of an image.
Fine cat ears (～￣▽￣)～
absolute king!! thank you for informing!!!
i enabled it. then i cant seem to make it work. can u help please im a newbie
@Ssartvcx Just copy the settings in the image posted by Haoming02 in the Forge Neo link above.
@Fish788 thank youuu
Amazing and weird.
I mean, it works, it can change poses, replace characters, remove background...
But i just can't get a grip on it. Sometimes it does nothing at all. Sometimes it immedeately does what i want. Sometimes you just take your prompt and rewrite it similarly - and it suddenly works. Would love more examples of HOW to prompt it specifically.
But still, an amazing thing.
Works great when I manage to get the prompt right.
But it doesn't seem to understand https://danbooru.donmai.us/wiki_pages/zoom_layer and how to remove them. It doesn't register it as background if I do 'remove background' or 'remove zoom layer'. 'remove background character' or 'remove girl in background' didn't work either.
It would be great that it could be used for controling the style based on artist style tags.
Have you thought about using
https://thetacursed.github.io/Anima-Style-Explorer/
as a dataset for teaching this lora to change style of the image?