This loha converts the base text and image to video model to reference to video. It is not perfect, there is difference between result of this loha and what native version gives. The results are not bad however. Just different. (what can you expect with mean cossim of 0.6774)
loha should be reversable if you set weigth to -1. DO NOT USE JOINT MODEL+CLIP NODE IN COMFYUI. USE THE MODEL ONLY LORA LOADER.
I used combination of SVD and training.
Description
FAQ
Comments (19)
How does it works ?
from my understanding its takes the ti2va model, add this lora, and it act as ref 2 vid model. question is what workflow because ti2va is only support first last frame, and if you use ref 2 v you can use up to 9 images + video. so my guess is use a ref 2 vid workflow with loading the ti2va model. i'll test it, seems interesting.
edit: so i tried now an ti2va lora on ref2v and got very bad results, maybe this way you can use an ti2va lora + this lora and a ref2v workflow.. making some generations and will update, maybe even upload some examples if i think about this correctly and get lucky.
Yes, It converts ti2va to ref2vid. Using this lora you can use ti2va as if it was ref2va. Just a heads up, I did not test the fp8 version
Also in theory it should be able to go in reverse by putting -1 to weigth. Did not test that.
@mkDaniel from ref2v model to ti2va behavior? cool
Because both models are the same core model, just fine-tuned for each type of work. If you put the R2V model into an I2V workflow, it will work normally. Most importantly, just write the prompt in the correct H3 format.
its changes water to h2o
This seems cool.
What advantage do we gain using the base model with this lora vs just the ref model alone? I suppose you wouldn't need to load two models if you needed both for a workflow.
Size, mostly. This is made to free up space on SSD by not needing 2 20GB models.
From my limited testing (to make sure it works) it can a bit better follow prompt.
If you only need reference model or only the base text/image to video then this is worthless (might turn reference model to the base one with weigth -1).
I think extracting the LoRA from R2V and using it with FL2VA would probably be better. Honestly, FL2VA can actually do R2V as well, but you have to write the prompt correctly according to the R2V guide.
Hold on, why did you use <image2>
@g1263495582 that Is how ref2va points to references
Tried that. Got a mess of pixelů.
So I Tool ref And the other version, extracted delta Matrix (just ref-fl. Split that using SVD and then again. I continued with training these Split matricies So when combined inside sampler back into delta Matrix And added to fl version it Is as close as possible to the ref version.
Whole thing took And hour on a 5090. The result Is honestly not that good.
@mkDaniel Hold on, what are you mainly using? ComfyUI or something else? And did you write the script you're using yourself?
@mkDaniel If it's ComfyUI, use <Picture 1> not <image1>. Also, I suggest checking https://huggingface.co/MiniMaxAI/MiniMax-H3/tree/main/docs first, then rewrite the r2v prompt and re-gen using fl2va.
@g1263495582 Did not notice the comment about the script. Yes, I did. With a help of AI to get better result. This is the first time I am extracting a trained thing when having a base and trained model. I usually train from data, not extract it.
Hey is there any link online for a more detailed explanation on this? I was wondering about how I can use extra references like using a video as an extra reference for the FL2VA model