A Canny control LoRA trained for Anima (link). Currently requires my fork of ComfyUI.
See the version notes for training details.
The example workflow is in the image(s).
I may either make a custom node or try to get this merged upstream...
Description
Same dataset, 15 epochs
FAQ
Comments (20)
Absolutely essential tooling, good job.
Glad to hear that, any control types I should prioritize for training?
@levzzz I could list off quite a few. NoobAI had an Inpaint (infill) model, Reference, Lineart (softer canny), Depth, etc. I could recommend a bunch that are helpful but I think getting support in across WebUIs would be equally as important. Right now this is specific to your fork and that might limit adoption. Anima is compatible with other WebUIs like Forge Neo where Inpainting is 99% less painful vs. Comfy.
Looking forward to anything you might cook next.
Is this training pipeline an adaptation over a public repo (sd-scripts, diffusion pipe, whatever else) or entirely custom? Could this idea of reference based generation be exploited for more transformational effects? I'm thinking like QwenImageEdit or similar. Or something more rudimentary like training Diff/Copier/etc. LoRAs that learn specific deltas.
@inpaint the reference latent implementation is nearly exactly the same as flux.2 so anything it can do is possible to train with anima now. Any UI that supports f.2 or f.2k would have no trouble implementing Anima edit. First step would be getting it merged into comfy at least. The trainer I used is a fork of diffusion-pipe, will release it later. Right now I don't really have the disk space nor compute for a proper booru-based editing dataset but it is possible.
@levzzz Good to know, in that case the future of this model is very strong. Thanks for your work.
Hello there. Weirdly enough, I was just learning more about Canny yesterday. You're saying it requires your own forked Comfy, but don't you just add the OpenCV lib to your python? Thank you 🙌
This LoRA essentially turns the model into an editing one, so requires ComfyUI to implement editing for a specific model.
What training tools do you use?
A fork of diffusion-pipe
@levzzz Is there a project URL?
@Recursion https://github.com/levzzz5154/diffusion-pipe there it is
@levzzz thank you😊
I don't quite understand, is this a ControlNet mode? How do I use it in ComfyUI?
sd-webui-forge-neo:
For those who are lazy to go through the lines, just replace existing file.
https://gist.github.com/benfy95/70f1f50bcabeea9c5a63d4d99a9895c0
how to use this lora in comfyui?can you give a rough workflow to use this?
it's in the images in the gallery
why create a fork broooooooo just create a patcher with encoded image and model
Ref_latents broooooo
sorry mate not very familiar with comfy internals
I don't understand how to use it in Forge Neo. I already replaced the two files, but when I select anima-preview-canny-v0.2 in ControlNet, it doesn't work.
Any hero without a cape who can help me, please?
I re-tested by download from my github page and try the patch still works on Forge Neo (commit f8524e93c7db6aa4750375696feb6e7dc523f886) 4/18. Are you sure you select the correct control method and the correct model?