https://github.com/TkskKurumi/DiffusersFastAPI/blob/main/model_as_lora.py
I've attempted to develop an algorithm that can calculate LoRA from subtracting two trained full model. The core is to compress the full Δw (weight difference) matrix into LoRA.
I think my matrix compression algorithm is not good enough. I'm not good at linear algebra (or I don't even know what fields of knowledge is needed for the task).
Compress matrix W of shape(m, n) into matul of A and B of shapes (m, rank) and (rank, n). The compress ratio according to parameter amount is rank*(m+n)/(m*n). I've found that ratio = 25% may nearly represent Fantexi subtract my anime model. And ratio=5% with lora alpha doubled has adequate result.
I've also tried the algorithm to take image as input and use 2 matrixs of shapes (width, rank) (rank, height). Following image has original 1080 pixels width and height and compressed into low rank 32.
Description
The LoRA ≈ A - B. Compress ratio = 25%
FAQ
Comments (4)
Very interesting. I adapted the code to my own code and ran a basic test, it worked pretty well. I'll continue doing some more tests with regular extraction and resizing to see how it compares, will write again if I find anything relevant
Would like to know teh findings regarding the dimensions of the output LoRA or LyCORIS, since there are too many niche Checkpoints that are better off treated as an "extra".
@TomLucidor I'll have to say after a lot of tests that it works really well. I've done exactly that, I have a few models and have used them to extract from other models (that were probably better as a style LoRA anyways) and resized the result using a factor of 0.9. The trick is to find the most similar ones to use for the extraction and then the LoRA can be smaller (the distance between the two models in the "model space" would be shorter)
Nice to see this being done, and would like to check against other Checkpoints like this https://civitai.com/models/5386/zack3d-kinky https://civitai.com/models/14503/latex-generator-15-or-by-latexid https://civitai.com/models/19117/chillout-latex-mix https://civitai.com/models/6593/latex-reality-v30-final