MOCHI | Test bench with fast

@LatentDream ah thx

also also - i am testing the force CPU option to load CLIP in FP16 - same speed!
you could give that a try. In V4 i'll like to include a version with your "fast settings", so i'll link your workflow in mine.

@LatentDream oh i see the very low frame count now ! well done, it's a nice catch. i'll give it a mention in tomorrows release and the eventual video. I'm holding back until we know more about the setup.

@LatentDream it can be done, there is that team credit option when you release a model version. it will even split the buzz from ppl actions etc.

I wanted to peek on your setup, but due to my comfyui not having impact pack, it did not load correctly.
I have started "fast" branch in my pack, i'll link to you in the workflow notes etc.

My approach is to start simple with just 25 frames ~5 minutes, then build on that for complexity or speed in two forks. I have my own approach for i2v, but it's really txt2img with OneVision.

i'll see if impact pack will install again today, i dropped it recently because too many people had it fail to install, so maybe they fixed the problem.

@driftjohnson @LatentDream Looks like a builders' meeting here 😄
I am only running a 4080 with 16gb VRAM so I did not even try running Mochi yet, but I just learnt of a memory use enhancing modification to start comfy up with, which could be helpful for Mochi (if it even applies)
It is from a video tutorial @Grockster made and posted here and comes down to modifying the run_nvidia_gpu.bat file to:

.\python_embeded\python.exe -s ComfyUI\main.py --windows-standalone-build --use-quad-cross-attention --fp8_e4m3fn-text-enc --normalvram --dont-upcast-attention

I only did a very small test with it using base Flux (3 images, 3 seeds, same settings) and had no noticeable impact on memory use, but I could use Flux before just fine (with fp8_e4mf3n_fast), so this modification might apply better for other memory hungry cases like CogVideoX, SD3.5, AuraFlow and maybe Mochi I thought.

@redpinkretro interesting - thanks for the tip, i had not seen this.
I think you can squeeze Mochi into 12GB VRAM with using the T5XXL-FP16 (on CPU as in my V5/V6 workflows) and then using the Q4_0 quantized mochi model, however there would be some loss in quality with that, I'll check this out and if it works, it will be in a new update.

Certainly with the Q8_0 model (V6 best quality) this can be on the edge for even a 4090 (if you don't use VAE tiling), Even i got some OOM's with some latents i needed to decode, which is what led me to create the Runpod template.

@LatentDream Yes ! When i tested with the T5XXL-FP8 Clip models, they made worse video quality with the Q8_0 so initially i did not use it either, but when i switched to offloading the T5-FP16 to CPU, the Q8_0 worked better than the FP8 versions, so i reckong that using my V6 setup, it might make the Q4_0 version viable. I had not used the BF16 yet, it can only run on Ampere based RTX GPU, so less people were able to run them, i try to use the most compatible version early and then try those later.

Did you find the BF16 was faster or easier to load?

@driftjohnson Good to know that it is even possible with less than 24gb VRAM. Not sure if time spent and quality reduction make that option preferable to CogVideoX though 😅

@LatentDream I just tried a single text2video run with CogVideoX 5b (50 steps, 49 frames, interpolated to 97 frames) and from load to finish it actually took around 560s with modified comfy startup arguments, while taking only 530s without. So yeah there could be a speed increase noticeable in especially video generation cases. I turned the arguments off for now as well.
Did you notice a reduction in VRAM use though? Speed for VRAM is a common sacrificial trade-off in that kind of voodoo 😄

@LatentDream Damn, that is like the worst possible outcome, sorry about that 😂
I just went through the odyssee of trying to get sageattention working and ran into errors so cryptic I just dropped the attempt after an hour of headaches.
Now I am at Cuda 12.4 and Torch at 2.5.0 with triton running, so that is good/fast enough I suppose. 😅

@redpinkretro  hi , ive also got 4080 16gb and can run mochi with good results , resolution is a bit lower at 609 x 353 and max 49 frames same as cog but results are better than cog but a bit slower at about 5 minutes per gen using qwen2 to prompt from video .

thanks to @driftjohnson and latentdream for all the work they are doing on this

@citydailyai That sounds great, did you get sageattention to run or does it take less than 5min without? Man what I would give for just a couple dozen hours more a day...

@redpinkretro yes got sageattn working , , first i installed triton via a prebuilt wheel ,then sageattn installed with no problems , theres a thread on reditt stable diffusion with good instruction.

edit link to triton info, https://www.reddit.com/r/StableDiffusion/comments/1g45n6n/triton_3_wheels_published_for_windows_and_working/

@citydailyai Yeah, got it working now too, no idea what the issue was before, but now I had no complications whatsoever. Just installed it and all is well. Thanks again! Hopefully get to check out Mochi on the weekend 🤞

@LatentDream thanks for inspiring my fast version, yours is still faster i think !!
Maybe my LoadLatent node can be useful for you too :) removing the tiling gave a big increase in quality :)

@LatentDream this had no impact on speed for me. however i was comparing to the T5XXL_FP8_Scaled (from the SD3.5 release) which was a bit larger and increased detail. When i use the T5-FP16 on CPU, i'm not using the Tiling VAE anymore, so that was the biggest quality bump, but you gotta do the save latents to disk and then load the latent for the decoding stage. I also ran out of memory - it's a tight squeeze!