A Canny control LoRA trained for Anima (link). Currently requires my fork of ComfyUI.
See the version notes for training details.
The example workflow is in the image(s).
I may either make a custom node or try to get this merged upstream...
Description
first proof of concept version
FAQ
Comments (29)
Absolutely essential tooling, good job.
Glad to hear that, any control types I should prioritize for training?
@levzzz I could list off quite a few. NoobAI had an Inpaint (infill) model, Reference, Lineart (softer canny), Depth, etc. I could recommend a bunch that are helpful but I think getting support in across WebUIs would be equally as important. Right now this is specific to your fork and that might limit adoption. Anima is compatible with other WebUIs like Forge Neo where Inpainting is 99% less painful vs. Comfy.
Looking forward to anything you might cook next.
Is this training pipeline an adaptation over a public repo (sd-scripts, diffusion pipe, whatever else) or entirely custom? Could this idea of reference based generation be exploited for more transformational effects? I'm thinking like QwenImageEdit or similar. Or something more rudimentary like training Diff/Copier/etc. LoRAs that learn specific deltas.
@inpaint the reference latent implementation is nearly exactly the same as flux.2 so anything it can do is possible to train with anima now. Any UI that supports f.2 or f.2k would have no trouble implementing Anima edit. First step would be getting it merged into comfy at least. The trainer I used is a fork of diffusion-pipe, will release it later. Right now I don't really have the disk space nor compute for a proper booru-based editing dataset but it is possible.
@levzzz Good to know, in that case the future of this model is very strong. Thanks for your work.
Hello there. Weirdly enough, I was just learning more about Canny yesterday. You're saying it requires your own forked Comfy, but don't you just add the OpenCV lib to your python? Thank you 🙌
This LoRA essentially turns the model into an editing one, so requires ComfyUI to implement editing for a specific model.
What training tools do you use?
A fork of diffusion-pipe
@levzzz Is there a project URL?
@Recursion https://github.com/levzzz5154/diffusion-pipe there it is
@levzzz thank you😊
I don't quite understand, is this a ControlNet mode? How do I use it in ComfyUI?
sd-webui-forge-neo:
For those who are lazy to go through the lines, just replace existing file.
https://gist.github.com/benfy95/70f1f50bcabeea9c5a63d4d99a9895c0
this causing control net to stop working for me when I replace this anima.py file
@turkey910
Theres a new version of ControlNet available. Works with Forge Neo out of the box (update Forge to latest ver)
https://huggingface.co/kohya-ss/Anima-LLLite
@soralz lmao you're kidding, I wish I saw this two hours ago, I saw the huggingtree but it didn't work on my forge so I made a patch and an article on how to apply it lmao. Guess I just needed to update forge yikes
@turkey910 I tried, it works but not stable enough though. Not sure because the model is not "large enough" to contain the weight? Because noobai ver almost 5GB per control model and it works really well, but the new anima controlnet is <100MB in size.
Regarding the ControlNet on this model page, the author has no plans to support it, so I guess it doesnt work by default on Forge Neo, and will break in future update if anything changes.
@soralz I asked codex to take a look too and it said my patch was better anyways. I've got it working pretty decently for depth on neo
@soralz kohya's new controlnet is pretty great, tho i think limited to simpler control methods. someone trained a more or less full editing lora with my method
how to use this lora in comfyui?can you give a rough workflow to use this?
it's in the images in the gallery
why create a fork broooooooo just create a patcher with encoded image and model
Ref_latents broooooo
sorry mate not very familiar with comfy internals
I don't understand how to use it in Forge Neo. I already replaced the two files, but when I select anima-preview-canny-v0.2 in ControlNet, it doesn't work.
Any hero without a cape who can help me, please?
Why my image completely different with the reference? What setting should i change for fix this?
is this for the comfy ui